Re: [I] Ensure `to_timestamp` behaves consistently with PostgreSQL [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #13351: URL: https://github.com/apache/datafusion/issues/13351#issuecomment-246828 The difference in `to_timestamp(1)` is almost certainly tz related as DF udf's don't have access to the context's timezone. As for why the DF version has multiple argumen

Re: [PR] Convert nth_value builtIn function to UDWF [datafusion]

2024-11-11 Thread via GitHub
buraksenn commented on code in PR #13201: URL: https://github.com/apache/datafusion/pull/13201#discussion_r1836726411 ## datafusion/sqllogictest/test_files/errors.slt: ## @@ -119,13 +119,36 @@ regr_slope(c11, '2') over () as min1 from aggregate_test_100 order by c9 -# Window

Re: [PR] [WIP][DO-NOT-MERGE] feat: support array_insert [datafusion-comet]

2024-11-11 Thread via GitHub
SemyonSinchenko commented on PR #1073: URL: https://github.com/apache/datafusion-comet/pull/1073#issuecomment-2468325238 @andygrove Sorry for tagging but I have questions about the ticket (`array_insert`). 1. `array_insert` was added in spark 3.4, so all the 3.3.x tests are obviousl

Re: [I] Add support for string view to a few functions [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #13330: URL: https://github.com/apache/datafusion/issues/13330#issuecomment-2468324874 regexp_match filed as https://github.com/apache/datafusion/issues/13357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

[I] regexp_match does not support Utf8View [datafusion]

2024-11-11 Thread via GitHub
Omega359 opened a new issue, #13357: URL: https://github.com/apache/datafusion/issues/13357 ### Describe the bug regexp match's signature: ``` // TODO: Native support Utf8View for regexp_match. TypeSignature::Exact(vec![Utf8, Utf8]),

[PR] Add support for MYSQL's `CREATE TABLE SELECT` expr [datafusion-sqlparser-rs]

2024-11-11 Thread via GitHub
wugeer opened a new pull request, #1515: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1515 This PR supports `CREATE TABLE SELECT` clause for mysql dialect,. For more information, please refer to: https://dev.mysql.com/doc/refman/8.4/en/create-table-select.html This r

Re: [PR] Add stacker and recursive [datafusion]

2024-11-11 Thread via GitHub
peter-toth commented on code in PR #13310: URL: https://github.com/apache/datafusion/pull/13310#discussion_r1836819378 ## datafusion/sql/src/expr/mod.rs: ## @@ -168,6 +169,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { /// Internal implementation. Use /// [`Self

Re: [PR] Add stacker and recursive [datafusion]

2024-11-11 Thread via GitHub
peter-toth commented on code in PR #13310: URL: https://github.com/apache/datafusion/pull/13310#discussion_r1836813902 ## datafusion/sql/src/query.rs: ## @@ -59,7 +59,12 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { self.select_into(plan, select_into)

Re: [PR] Add stacker and recursive [datafusion]

2024-11-11 Thread via GitHub
peter-toth commented on code in PR #13310: URL: https://github.com/apache/datafusion/pull/13310#discussion_r1836813902 ## datafusion/sql/src/query.rs: ## @@ -59,7 +59,12 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { self.select_into(plan, select_into)

Re: [PR] chore: Refactor binary and math expression serde code [datafusion-comet]

2024-11-11 Thread via GitHub
viirya merged PR #1069: URL: https://github.com/apache/datafusion-comet/pull/1069 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

[I] Make `CreateTable.location` / `AlterTable.location` make more sense [datafusion-sqlparser-rs]

2024-11-11 Thread via GitHub
tv42 opened a new issue, #1518: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1518 `CreateTable.location` has no documentation and is a tricky edge case, not always set even when SQL statement did set location. https://docs.rs/sqlparser/latest/sqlparser/ast/struct.Cre

Re: [I] Make `CreateTable.location` / `AlterTable.location` make more sense [datafusion-sqlparser-rs]

2024-11-11 Thread via GitHub
tv42 commented on issue #1518: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1518#issuecomment-246886 Somewhat similar issue about `CREATE DATABASE .. LOCATION`: https://github.com/apache/datafusion-sqlparser-rs/issues/453 -- This is an automated message from the Apac

Re: [PR] chore: Refactor binary and math expression serde code [datafusion-comet]

2024-11-11 Thread via GitHub
viirya commented on PR #1069: URL: https://github.com/apache/datafusion-comet/pull/1069#issuecomment-2468997678 Thanks @andygrove @kazuyukitanimura -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] fix(minor): Cast `TypeCategory::from` to `TypeCategory::Unknown` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n opened a new pull request, #13365: URL: https://github.com/apache/datafusion/pull/13365 ## Which issue does this PR close? Closes #13358 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] chore: Simplify CometShuffleMemoryAllocator to use Spark unified memory allocator [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove commented on PR #1063: URL: https://github.com/apache/datafusion-comet/pull/1063#issuecomment-2469015292 I'm running into SIGSEGV issues again. ``` │ # A fatal error has been detected by the Java Runtime Environment:

[PR] [WIP] [comet-parquet-exec] Add unit test for reading a struct field from Parquet [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove opened a new pull request, #1075: URL: https://github.com/apache/datafusion-comet/pull/1075 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [I] Overflow in `repeat_arrs_from_indices` [datafusion]

2024-11-11 Thread via GitHub
demetribu commented on issue #13237: URL: https://github.com/apache/datafusion/issues/13237#issuecomment-2469065968 ``` #[test] fn test_array_take_sequential() { use arrow_array::{Int64Array, ArrayRef}; use std::sync::Arc; let int_val

Re: [PR] chore: Simplify CometShuffleMemoryAllocator to use Spark unified memory allocator [datafusion-comet]

2024-11-11 Thread via GitHub
viirya commented on PR #1063: URL: https://github.com/apache/datafusion-comet/pull/1063#issuecomment-2469086104 Let me see if I can reproduce it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] TypeCategory::from does not handle Utf8View [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #13358: URL: https://github.com/apache/datafusion/issues/13358#issuecomment-2469091589 I think this is taken care of by https://github.com/apache/datafusion/pull/13350 -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[PR] fix: Fix failures caused by `array_prepend` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n opened a new pull request, #13367: URL: https://github.com/apache/datafusion/pull/13367 ## Which issue does this PR close? Closes #. ## Rationale for this change Added changes to array_prepend to fix the tests that are failing due to the changes in #13362

[PR] support window functions [datafusion-ballista]

2024-11-11 Thread via GitHub
onursatici opened a new pull request, #1112: URL: https://github.com/apache/datafusion-ballista/pull/1112 # Which issue does this PR close? Closes #542 . # Rationale for this change As stated in #542, datafusion adds `SortExec` and `RepartitionExec` based on the

Re: [I] Empty strings in CSV files aren't being interpreted as null when using a `Dictionary(_, Utf8)` [datafusion]

2024-11-11 Thread via GitHub
edmondop commented on issue #12041: URL: https://github.com/apache/datafusion/issues/12041#issuecomment-2469227106 @alamb before I file an issue to arrow-csv, why is "bob" a valid value for DictionaryArray? don't you need a key and a value for a dictionary? -- This is an automated message

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-11 Thread via GitHub
tbar4 commented on code in PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#discussion_r1837293315 ## python/src/lib.rs: ## @@ -15,18 +15,66 @@ // specific language governing permissions and limitations // under the License. +use ballista::prelude::*;

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-11 Thread via GitHub
tbar4 commented on code in PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#discussion_r1837293454 ## python/src/lib.rs: ## @@ -15,18 +15,66 @@ // specific language governing permissions and limitations // under the License. +use ballista::prelude::*;

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-11 Thread via GitHub
tbar4 commented on code in PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#discussion_r1837293612 ## python/src/lib.rs: ## @@ -15,18 +15,66 @@ // specific language governing permissions and limitations // under the License. +use ballista::prelude::*;

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-11 Thread via GitHub
tbar4 commented on code in PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#discussion_r1837293691 ## python/src/lib.rs: ## @@ -15,18 +15,66 @@ // specific language governing permissions and limitations // under the License. +use ballista::prelude::*;

[PR] wip - CometNativeScan [datafusion-comet]

2024-11-11 Thread via GitHub
parthchandra opened a new pull request, #1076: URL: https://github.com/apache/datafusion-comet/pull/1076 Replaces hacked `CometScan` with `CometNativeScan` so we can switch between original and experimental without rebuilding. I tried removing a lot of the functions in CometNativeScan but

Re: [PR] Implement `Spanned` to retrieve source locations on AST nodes [datafusion-sqlparser-rs]

2024-11-11 Thread via GitHub
alamb commented on PR #1435: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1435#issuecomment-2468945628 Sorry for my silence / inability to find time to review this PR in depth. I am struggling with too many thing TLDR I I think we should: 1. Get the tests passing on

Re: [I] `cargo build --no-default-features` does not build cleanly [datafusion]

2024-11-11 Thread via GitHub
alamb commented on issue #8844: URL: https://github.com/apache/datafusion/issues/8844#issuecomment-2468946506 ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

[PR] improve performance of regexp_count [datafusion]

2024-11-11 Thread via GitHub
Dimchikkk opened a new pull request, #13364: URL: https://github.com/apache/datafusion/pull/13364 ## Which issue does this PR close? Closes #13011 ## Rationale for this change regexp_count becomes as performant as other regexp functions ## What changes are included

Re: [I] Release sqlparser-rs version `0.52.0` [datafusion-sqlparser-rs]

2024-11-11 Thread via GitHub
alamb closed issue #1423: Release sqlparser-rs version `0.52.0` URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1423 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Release sqlparser-rs version `0.52.0` [datafusion-sqlparser-rs]

2024-11-11 Thread via GitHub
alamb commented on issue #1423: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1423#issuecomment-2468926648 The release has been approved and officially released: https://lists.apache.org/thread/b4n5ycp9z922tzg29w2z3vlfg7cwcrc8 Thank you everyone. Next release is track

Re: [I] [DISCUSSION] 2024 Q4 / 2025 Q1 Roadmap [datafusion]

2024-11-11 Thread via GitHub
matthewmturner commented on issue #13274: URL: https://github.com/apache/datafusion/issues/13274#issuecomment-2468934127 @alamb i will work on that next. will ping you when ready for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] Access children `DataType` or return-type in `ScalarUDFImpl::invoke` [datafusion]

2024-11-11 Thread via GitHub
gatesn commented on issue #12819: URL: https://github.com/apache/datafusion/issues/12819#issuecomment-2468831055 I think the objection is that by all understanding, `simplification` is an optional optimization step. The result of an expression shouldn't change based on whether it was run.

Re: [PR] feat: support array_append [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove commented on code in PR #1072: URL: https://github.com/apache/datafusion-comet/pull/1072#discussion_r1837147575 ## native/core/src/execution/datafusion/planner.rs: ## @@ -691,6 +693,34 @@ impl PhysicalPlanner { expr.ordinal as usize,

[PR] minor(fix): Support `Utf8View` in `string_numeric_coercion` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n opened a new pull request, #13366: URL: https://github.com/apache/datafusion/pull/13366 ## Which issue does this PR close? Closes #13359 . ## Rationale for this change ## What changes are included in this PR? Added `utf8view` to `string_nume

Re: [PR] improve performance of regexp_count [datafusion]

2024-11-11 Thread via GitHub
Dimchikkk commented on PR #13364: URL: https://github.com/apache/datafusion/pull/13364#issuecomment-2469020556 > Thanks @Dimchikkk for your contribution. Do you mean the Entry API misbehaved returning Vacant all the time and forced the regexp pattern to be recompiled? Hi @comphead ,

Re: [I] Adjust `Statistics::total_byte_size` in `Filter` with a projection [datafusion]

2024-11-11 Thread via GitHub
jiashenC commented on issue #13224: URL: https://github.com/apache/datafusion/issues/13224#issuecomment-2469023831 Take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] chore: Simplify CometShuffleMemoryAllocator to use Spark unified memory allocator [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove commented on PR #1063: URL: https://github.com/apache/datafusion-comet/pull/1063#issuecomment-2469042196 I increased the off-heap pool size, and now I can run TPC-H q5 @ sf=1TB on the `main` branch, but get SIGSEGV with this PR. -- This is an automated message from the Apache G

Re: [PR] feat: Implement CAST between struct types [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove merged PR #1074: URL: https://github.com/apache/datafusion-comet/pull/1074 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Implement cast between struct types [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove closed issue #815: Implement cast between struct types URL: https://github.com/apache/datafusion-comet/issues/815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] fix: Support `Utf8View` in `string_numeric_coercion` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n closed pull request #13366: fix: Support `Utf8View` in `string_numeric_coercion` URL: https://github.com/apache/datafusion/pull/13366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] fix: Support `Utf8View` in `string_numeric_coercion` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n commented on PR #13366: URL: https://github.com/apache/datafusion/pull/13366#issuecomment-2469141056 close reopen to fix tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

[I] Add how new docs are dealt with to HOWTOs [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n opened a new issue, #13368: URL: https://github.com/apache/datafusion/issues/13368 ### Is your feature request related to a problem or challenge? Newer committers will probably miss how the docs are done dynamically as it isn't very obvious. I suggest adding a new section

Re: [I] regexp_match does not support Utf8View [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #13357: URL: https://github.com/apache/datafusion/issues/13357#issuecomment-2469170847 This ticket is a duplicate of https://github.com/apache/datafusion/issues/11911 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] regexp_match does not support Utf8View [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #13357: URL: https://github.com/apache/datafusion/issues/13357#issuecomment-2469167377 A good part of the reason why this hasn't been done in DF is that the corresponding functions in arrow-rs do not support StringView - https://github.com/apache/arrow-rs/blob/28

Re: [PR] fix(minor): Cast `TypeCategory::from` to `TypeCategory::Unknown` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n closed pull request #13365: fix(minor): Cast `TypeCategory::from` to `TypeCategory::Unknown` URL: https://github.com/apache/datafusion/pull/13365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] fix(minor): Cast `TypeCategory::from` to `TypeCategory::Unknown` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n commented on PR #13365: URL: https://github.com/apache/datafusion/pull/13365#issuecomment-2469143336 Oops, closed and reopened before the fix even merged. 😆 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] fix(minor): Cast `TypeCategory::from` to `TypeCategory::Unknown` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n commented on PR #13365: URL: https://github.com/apache/datafusion/pull/13365#issuecomment-2469140439 close reopen to fix ci tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] fix: Fix failures caused by `array_prepend` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n commented on PR #13367: URL: https://github.com/apache/datafusion/pull/13367#issuecomment-2469138306 @alamb @Omega359 I believe this will probably keep happening from time to time (newer committers being unaware of the dynamic docs for UDFs). I'll open another PR to put this in

Re: [I] Add how new docs are dealt with to HOWTOs [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n commented on issue #13368: URL: https://github.com/apache/datafusion/issues/13368#issuecomment-2469164985 This can be given a good first issue label. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] feat: Optimize `SortPreservingMergeExec` to avoid merging non-overlapping partitions [datafusion]

2024-11-11 Thread via GitHub
suremarc commented on code in PR #13296: URL: https://github.com/apache/datafusion/pull/13296#discussion_r1837262772 ## datafusion/sqllogictest/test_files/optimize_sort_preserving_merge.slt: ## @@ -0,0 +1,89 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or

Re: [I] Update `REGEXP_MATCH` scalar function to support Utf8View [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #11911: URL: https://github.com/apache/datafusion/issues/11911#issuecomment-2469181899 Filed upstream ticket https://github.com/apache/arrow-rs/issues/6717 to have regexp_match updated to support StringViewArray's -- This is an automated message from the Apache

Re: [I] regexp_match does not support Utf8View [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #13357: URL: https://github.com/apache/datafusion/issues/13357#issuecomment-2469187378 Closing as a duplicate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] regexp_match does not support Utf8View [datafusion]

2024-11-11 Thread via GitHub
Omega359 closed issue #13357: regexp_match does not support Utf8View URL: https://github.com/apache/datafusion/issues/13357 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Add support for string view to a few functions [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #13330: URL: https://github.com/apache/datafusion/issues/13330#issuecomment-2469191937 [nvl](https://github.com/apache/datafusion/blob/main/datafusion/functions/src/core/nvl.rs): ```Rust static SUPPORTED_NVL_TYPES: &[DataType] = &[ DataType::Boolean,

Re: [PR] wip - CometNativeScan [datafusion-comet]

2024-11-11 Thread via GitHub
viirya commented on code in PR #1076: URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1837300980 ## common/src/main/scala/org/apache/comet/CometConf.scala: ## @@ -77,6 +77,15 @@ object CometConf extends ShimCometConf { .booleanConf .createWithDefa

Re: [PR] fix: Fix failures caused by `array_prepend` [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on PR #13367: URL: https://github.com/apache/datafusion/pull/13367#issuecomment-2469275115 There is a big warning at the top of those files but adding notes to howto wouldn't hurt at all. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] wip - CometNativeScan [datafusion-comet]

2024-11-11 Thread via GitHub
parthchandra commented on code in PR #1076: URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1837318445 ## common/src/main/scala/org/apache/comet/CometConf.scala: ## @@ -77,6 +77,15 @@ object CometConf extends ShimCometConf { .booleanConf .createWi

Re: [PR] wip - CometNativeScan [datafusion-comet]

2024-11-11 Thread via GitHub
viirya commented on code in PR #1076: URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1837328909 ## common/src/main/scala/org/apache/comet/CometConf.scala: ## @@ -77,6 +77,15 @@ object CometConf extends ShimCometConf { .booleanConf .createWithDefa

Re: [PR] wip - CometNativeScan [datafusion-comet]

2024-11-11 Thread via GitHub
viirya commented on code in PR #1076: URL: https://github.com/apache/datafusion-comet/pull/1076#discussion_r1837328909 ## common/src/main/scala/org/apache/comet/CometConf.scala: ## @@ -77,6 +77,15 @@ object CometConf extends ShimCometConf { .booleanConf .createWithDefa

Re: [PR] Support TypeSignature::ZeroArg [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on PR #13354: URL: https://github.com/apache/datafusion/pull/13354#issuecomment-2469349776 > I think it would be nice to support a signature defined by something like `Vec` then ZeroArg / NoArgs would be just represented by an empty vec. What do you think? `Signa

Re: [PR] Support TypeSignature::ZeroArg [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on code in PR #13354: URL: https://github.com/apache/datafusion/pull/13354#discussion_r1837342675 ## datafusion/expr-common/src/signature.rs: ## @@ -135,6 +134,8 @@ pub enum TypeSignature { /// Null is considerd as `Utf8` by default /// Dictionary

Re: [PR] fix: Support `Utf8View` in `string_numeric_coercion` [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on code in PR #13366: URL: https://github.com/apache/datafusion/pull/13366#discussion_r1837342252 ## datafusion/expr-common/src/type_coercion/binary.rs: ## @@ -1490,6 +1491,92 @@ mod tests { ); } +#[test] +fn test_numeric_string_coerc

[PR] Move filtered SMJ Full filtered join out of `join_partial` phase [datafusion]

2024-11-11 Thread via GitHub
comphead opened a new pull request, #13369: URL: https://github.com/apache/datafusion/pull/13369 ## Which issue does this PR close? Closes #12359 ## Rationale for this change Move the Full Outer filtered SMJ join out of `join_partial` phase to evaluate filter expressions

Re: [PR] Move filtered SMJ Full filtered join out of `join_partial` phase [datafusion]

2024-11-11 Thread via GitHub
comphead commented on code in PR #13369: URL: https://github.com/apache/datafusion/pull/13369#discussion_r1837356945 ## datafusion/core/tests/fuzz_cases/join_fuzz.rs: ## @@ -168,21 +169,19 @@ async fn test_full_join_1k() { JoinType::Full, None, ) -.run

Re: [PR] improve performance of regexp_count [datafusion]

2024-11-11 Thread via GitHub
comphead commented on PR #13364: URL: https://github.com/apache/datafusion/pull/13364#issuecomment-2469372948 Please rebase from latest main to avoid the CI failure and personally I like the numbers -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] coerced_from in functions.rs seems to be missing Utf8View in some cases [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on issue #13363: URL: https://github.com/apache/datafusion/issues/13363#issuecomment-2469396169 Instead of maintain this function, we should change function that rely on `coerced_from` to TypeSignature::String or similar -- This is an automated message from the Apache

Re: [PR] fix(minor): Cast `TypeCategory::from` to `TypeCategory::Unknown` [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on PR #13365: URL: https://github.com/apache/datafusion/pull/13365#issuecomment-2469378960 Duplicated PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Move filtered SMJ Full filtered join out of `join_partial` phase [datafusion]

2024-11-11 Thread via GitHub
comphead commented on PR #13369: URL: https://github.com/apache/datafusion/pull/13369#issuecomment-2469397133 During this work there are some opportunities found to clean up/document/improve testing for existing SMJ code. Planning to file a separate PR for it -- This is an automated mess

Re: [I] Epic: Simplify functions signature with LogicalType [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on issue #13301: URL: https://github.com/apache/datafusion/issues/13301#issuecomment-2469398229 @jonathanc-n You can take a look on this if you are interested, there are tons of functions require the change, not able to resolved in single PR -- This is an automated me

Re: [PR] improve performance of regexp_count [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on code in PR #13364: URL: https://github.com/apache/datafusion/pull/13364#discussion_r1837355468 ## datafusion/functions/src/regex/regexpcount.rs: ## @@ -541,42 +541,45 @@ where let pattern = compile_and_cache

Re: [PR] Move filtered SMJ Full filtered join out of `join_partial` phase [datafusion]

2024-11-11 Thread via GitHub
comphead commented on code in PR #13369: URL: https://github.com/apache/datafusion/pull/13369#discussion_r1837359254 ## datafusion/physical-plan/src/joins/sort_merge_join.rs: ## @@ -852,6 +852,54 @@ fn get_corrected_filter_mask( corrected_mask.extend(vec![Some(true)

Re: [PR] Move filtered SMJ Full filtered join out of `join_partial` phase [datafusion]

2024-11-11 Thread via GitHub
comphead commented on code in PR #13369: URL: https://github.com/apache/datafusion/pull/13369#discussion_r1837357297 ## datafusion/sqllogictest/test_files/sort_merge_join.slt: ## @@ -126,24 +126,21 @@ Alice 50 Alice 1 Alice 50 Alice 2 Bob 1 NULL NULL -# Uncomment when filter

Re: [PR] fix: Fix failures caused by `array_prepend` [datafusion]

2024-11-11 Thread via GitHub
comphead merged PR #13367: URL: https://github.com/apache/datafusion/pull/13367 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] fix(minor): Cast `TypeCategory::from` to `TypeCategory::Unknown` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n closed pull request #13365: fix(minor): Cast `TypeCategory::from` to `TypeCategory::Unknown` URL: https://github.com/apache/datafusion/pull/13365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] Ensure `to_timestamp` behaves consistently with PostgreSQL [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on issue #13351: URL: https://github.com/apache/datafusion/issues/13351#issuecomment-2469407593 Great, but what is each argument represented for is unclear to me? I can only find this query in test. `to_timestamp('2020-09-08T12:00:00+00:00', '2020-09-08 12/00/0

Re: [PR] WIP: Generate docs from macros. [datafusion]

2024-11-11 Thread via GitHub
comphead commented on PR #12822: URL: https://github.com/apache/datafusion/pull/12822#issuecomment-2469408870 I'm closer to this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] WIP: Generate docs from macros. [datafusion]

2024-11-11 Thread via GitHub
comphead commented on PR #12822: URL: https://github.com/apache/datafusion/pull/12822#issuecomment-2469410388 We should also take care on handing things like https://github.com/apache/datafusion/pull/13367 -- This is an automated message from the Apache Git Service. To respond to the mess

Re: [PR] Introduce `information_schema.parameters` table [datafusion]

2024-11-11 Thread via GitHub
goldmedal commented on PR #13341: URL: https://github.com/apache/datafusion/pull/13341#issuecomment-2469413599 Thanks @jayzhan211 for reviewing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Introduce `information_schema.parameters` table [datafusion]

2024-11-11 Thread via GitHub
goldmedal merged PR #13341: URL: https://github.com/apache/datafusion/pull/13341 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] fix: Support `Utf8View` in `string_numeric_coercion` [datafusion]

2024-11-11 Thread via GitHub
jonathanc-n commented on code in PR #13366: URL: https://github.com/apache/datafusion/pull/13366#discussion_r1837368172 ## datafusion/expr-common/src/type_coercion/binary.rs: ## @@ -1490,6 +1491,92 @@ mod tests { ); } +#[test] +fn test_numeric_string_coer

Re: [PR] fix: Support `Utf8View` in `string_numeric_coercion` [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on code in PR #13366: URL: https://github.com/apache/datafusion/pull/13366#discussion_r1837370169 ## datafusion/expr-common/src/type_coercion/binary.rs: ## @@ -1490,6 +1491,92 @@ mod tests { ); } +#[test] +fn test_numeric_string_coerc

Re: [PR] fix: Support `Utf8View` in `string_numeric_coercion` [datafusion]

2024-11-11 Thread via GitHub
jayzhan211 commented on code in PR #13366: URL: https://github.com/apache/datafusion/pull/13366#discussion_r1837370169 ## datafusion/expr-common/src/type_coercion/binary.rs: ## @@ -1490,6 +1491,92 @@ mod tests { ); } +#[test] +fn test_numeric_string_coerc

Re: [PR] Convert nth_value builtIn function to UDWF [datafusion]

2024-11-11 Thread via GitHub
jcsherin commented on code in PR #13201: URL: https://github.com/apache/datafusion/pull/13201#discussion_r1836725370 ## datafusion/sqllogictest/test_files/errors.slt: ## @@ -119,13 +119,36 @@ regr_slope(c11, '2') over () as min1 from aggregate_test_100 order by c9 -# WindowF

Re: [PR] Convert nth_value builtIn function to UDWF [datafusion]

2024-11-11 Thread via GitHub
jcsherin commented on code in PR #13201: URL: https://github.com/apache/datafusion/pull/13201#discussion_r1836725370 ## datafusion/sqllogictest/test_files/errors.slt: ## @@ -119,13 +119,36 @@ regr_slope(c11, '2') over () as min1 from aggregate_test_100 order by c9 -# WindowF

Re: [I] Implement cast between struct types [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove commented on issue #815: URL: https://github.com/apache/datafusion-comet/issues/815#issuecomment-2468374526 We need this feature so that we can support reading structs from Parquet, so I am going to pick this up. @dharanad Let me know if you are interested in collaborating

Re: [PR] support to_timestamp with optional chrono formats [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on PR #8886: URL: https://github.com/apache/datafusion/pull/8886#issuecomment-2468258706 https://datafusion.apache.org/user-guide/sql/scalar_functions.html#to-timestamp I would be very much opposed to changing this to be exactly like pg. -- This is an automated m

Re: [PR] RFC: Add `Precision:AtLeast` and `Precision::AtMost` for more `Statistics`… precision [datafusion]

2024-11-11 Thread via GitHub
crepererum commented on PR #13293: URL: https://github.com/apache/datafusion/pull/13293#issuecomment-2468259496 Why does `Estimage::Range` has both `bounds` and a value? What's the `value` meant to be, e.g. if a parquet data source tells you that the interval is `42..=1337`? -- This is a

Re: [PR] Add stacker and recursive [datafusion]

2024-11-11 Thread via GitHub
peter-toth commented on code in PR #13310: URL: https://github.com/apache/datafusion/pull/13310#discussion_r1836813902 ## datafusion/sql/src/query.rs: ## @@ -59,7 +59,12 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { self.select_into(plan, select_into)

Re: [I] [EPIC] DataFusion Ray Testing [datafusion-ray]

2024-11-11 Thread via GitHub
edmondop commented on issue #18: URL: https://github.com/apache/datafusion-ray/issues/18#issuecomment-2468424828 I wonder whether we should only run tpch queries on Kubernetes and skip completely running them on single ray node locally @andygrove -- This is an automated message from the

Re: [PR] Convert nth_value builtIn function to UDWF [datafusion]

2024-11-11 Thread via GitHub
jcsherin commented on PR #13201: URL: https://github.com/apache/datafusion/pull/13201#issuecomment-2468432128 @buraksenn and @berkaysynnada Thanks! @alamb This PR is ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [I] Add support for string view to a few functions [datafusion]

2024-11-11 Thread via GitHub
Omega359 commented on issue #13330: URL: https://github.com/apache/datafusion/issues/13330#issuecomment-2468481874 Related: - https://github.com/apache/arrow-rs/issues/6713 - https://github.com/apache/arrow-rs/issues/6714 -- This is an automated message from the Apache Git Servi

Re: [PR] feat: support array_append [datafusion-comet]

2024-11-11 Thread via GitHub
NoeB commented on code in PR #1072: URL: https://github.com/apache/datafusion-comet/pull/1072#discussion_r1836921855 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -2313,4 +2313,22 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSpar

[PR] [WIP] feat: Implement CAST between struct types [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove opened a new pull request, #1074: URL: https://github.com/apache/datafusion-comet/pull/1074 ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/815 ## Rationale for this change We need support for casting b

Re: [PR] chore: Simplify CometShuffleMemoryAllocator to use Spark unified memory allocator [datafusion-comet]

2024-11-11 Thread via GitHub
andygrove commented on PR #1063: URL: https://github.com/apache/datafusion-comet/pull/1063#issuecomment-2468580519 > As now the allocator uses all available memory on the executor (we don't specify memory size on the allocator), it should not be an issue for #886 now. @andygrove Do you wan

[I] binary_to_string_coercion in binary.rs does not support utf8view [datafusion]

2024-11-11 Thread via GitHub
Omega359 opened a new issue, #13361: URL: https://github.com/apache/datafusion/issues/13361 ### Describe the bug The coercion rules defined in binary_to_string_coercion do not account for utf8view ### To Reproduce _No response_ ### Expected behavior binary

[PR] minor(docs): Correct array_prepend docs [datafusion]

2024-11-11 Thread via GitHub
NoeB opened a new pull request, #13362: URL: https://github.com/apache/datafusion/pull/13362 ## Which issue does this PR close? ## Rationale for this change While working on adding array_append support for datafusion-comet I noticed the docs had a copy paste error relat

Re: [PR] refactor: replace `instant` with `web-time` [datafusion]

2024-11-11 Thread via GitHub
jonahgao merged PR #13355: URL: https://github.com/apache/datafusion/pull/13355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] build(deps): bump tokio from 1.40.0 to 1.41.1 [datafusion-python]

2024-11-11 Thread via GitHub
Michael-J-Ward merged PR #946: URL: https://github.com/apache/datafusion-python/pull/946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

  1   2   3   >