Re: [I] Potential performance regression for TPCH q18 [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 commented on issue #13188: URL: https://github.com/apache/datafusion/issues/13188#issuecomment-2461547990 Other than https://github.com/apache/arrow-rs/issues/6692. If we create filter version of `GroupColumn` that accumulate the filtered array into array builder **for each col

Re: [PR] feat: Added Timestamp/Binary/Float to fuzz [datafusion]

2024-11-06 Thread via GitHub
LeslieKid commented on code in PR #13280: URL: https://github.com/apache/datafusion/pull/13280#discussion_r1832163669 ## datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs: ## @@ -171,6 +172,25 @@ fn baseline_config() -> DatasetGeneratorConfig { ColumnDescr::new("time32

Re: [PR] feat: Added Timestamp/Binary/Float to fuzz [datafusion]

2024-11-06 Thread via GitHub
LeslieKid commented on code in PR #13280: URL: https://github.com/apache/datafusion/pull/13280#discussion_r1832163669 ## datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs: ## @@ -171,6 +172,25 @@ fn baseline_config() -> DatasetGeneratorConfig { ColumnDescr::new("time32

Re: [PR] Add support for MSSQL's `XQuery` methods [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
gaoqiangz commented on code in PR #1500: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1500#discussion_r1832131298 ## src/parser/mod.rs: ## @@ -1269,6 +1269,50 @@ impl<'a> Parser<'a> { _ => self.expected("an expression", next_token), }?; +

Re: [PR] Add support for MSSQL's `XQuery` methods [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
gaoqiangz commented on code in PR #1500: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1500#discussion_r1832131298 ## src/parser/mod.rs: ## @@ -1269,6 +1269,50 @@ impl<'a> Parser<'a> { _ => self.expected("an expression", next_token), }?; +

[I] CometShuffleMemoryAllocator should use Spark unified memory manager [datafusion-comet]

2024-11-06 Thread via GitHub
viirya opened a new issue, #1064: URL: https://github.com/apache/datafusion-comet/issues/1064 ### What is the problem the feature request solves? Currently Comet JVM shuffle's CometShuffleMemoryAllocator uses a separate unsafe memory allocator and track off-heap memory allocation inte

Re: [PR] Add support for MSSQL's `XQuery` methods [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
gaoqiangz commented on code in PR #1500: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1500#discussion_r1832134100 ## src/ast/mod.rs: ## @@ -560,6 +560,13 @@ pub enum Expr { expr: Box, key: Ident, }, +/// CompositeFunction (mssql) e.g. S

[PR] chore: Simplify CometShuffleMemoryAllocator to use Spark memory allocator [datafusion-comet]

2024-11-06 Thread via GitHub
viirya opened a new pull request, #1063: URL: https://github.com/apache/datafusion-comet/pull/1063 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes te

Re: [PR] Add support for MSSQL's `XQuery` methods [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
gaoqiangz commented on code in PR #1500: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1500#discussion_r1832131298 ## src/parser/mod.rs: ## @@ -1269,6 +1269,50 @@ impl<'a> Parser<'a> { _ => self.expected("an expression", next_token), }?; +

Re: [PR] chore: Move `CoalesceBatches` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 commented on code in PR #13283: URL: https://github.com/apache/datafusion/pull/13283#discussion_r1832121911 ## datafusion/physical-optimizer/src/coalesce_batches.rs: ## @@ -18,19 +18,20 @@ //! CoalesceBatches optimizer that groups batches together rows //! in bigger

Re: [PR] chore: Move `OptimizeAggregateOrder` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 merged PR #13284: URL: https://github.com/apache/datafusion/pull/13284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] chore: Move `OptimizeAggregateOrder` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 commented on PR #13284: URL: https://github.com/apache/datafusion/pull/13284#issuecomment-2461418637 > Sorry, grammar mistake, does it matter how many dependencies are added to the optimizer crate during this transition? The number of dependencies isn’t the issue; what matt

Re: [I] Support numeric string argument in functions [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 commented on issue #13285: URL: https://github.com/apache/datafusion/issues/13285#issuecomment-2461366322 Does this mean we need LogicalType::Unknown πŸ€” Then we can infer type in the later stage -- This is an automated message from the Apache Git Service. To respond to the messa

[I] Support numeric string argument in functions [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 opened a new issue, #13285: URL: https://github.com/apache/datafusion/issues/13285 # Issue abs() is one of the example that support numeric string In Postgres, the type is determined lately so `-1.2` is not string type but inferred into float type for abs() [doc](

Re: [I] Make benchmark binaries (like tpch) compile faster [datafusion]

2024-11-06 Thread via GitHub
2010YOUY01 commented on issue #13277: URL: https://github.com/apache/datafusion/issues/13277#issuecomment-2461288805 > For the quick test during dev, maybe can try following build mode, it will be much faster: > > ``` > cargo build --profile release-nonlto > ``` Thank you

Re: [I] Inferring the possible types from the TypeSignature [datafusion]

2024-11-06 Thread via GitHub
goldmedal commented on issue #13271: URL: https://github.com/apache/datafusion/issues/13271#issuecomment-2461275275 Thanks @delamarch3 This issue follows up on #13255. If you want to start working, you can use #13255 to work and rebase after it is merged. -- This is an automated messa

Re: [PR] Support unparsing plans after applying `optimize_projections` rule [datafusion]

2024-11-06 Thread via GitHub
goldmedal commented on PR #13267: URL: https://github.com/apache/datafusion/pull/13267#issuecomment-2461268897 I'll review this PR tonight. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Add support for MSSQL's `OPENJSON WITH` clause [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
gaoqiangz commented on code in PR #1498: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1498#discussion_r1831991242 ## tests/sqlparser_mssql.rs: ## @@ -192,6 +192,35 @@ fn parse_mssql_apply_join() { ); } +#[test] +fn parse_mssql_cross_apply_json() { +le

Re: [PR] chore: Move `CoalesceBatches` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jonathanc-n commented on PR #13283: URL: https://github.com/apache/datafusion/pull/13283#issuecomment-2461219566 Not too sure whats happening with the hash collisions test here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] chore: Move `OptimizeAggregateOrder` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jonathanc-n commented on PR #13284: URL: https://github.com/apache/datafusion/pull/13284#issuecomment-2461218546 Sorry, grammar mistake, does it matter how many dependencies are added to the optimizer crate during this transition? -- This is an automated message from the Apache Git Servic

Re: [PR] Add support for MSSQL's `OPENJSON WITH` clause [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
gaoqiangz commented on code in PR #1498: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1498#discussion_r1831977121 ## src/parser/mod.rs: ## @@ -1,7 +1,7 @@ impl<'a> Parser<'a> { table_with_joins: Box::new(table_and_joins),

Re: [PR] feat: basic support for executing prepared statements [datafusion]

2024-11-06 Thread via GitHub
jonahgao commented on PR #13242: URL: https://github.com/apache/datafusion/pull/13242#issuecomment-2461172729 Thanks @alamb for the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Store Prepare Logical Plan [datafusion]

2024-11-06 Thread via GitHub
jonahgao closed issue #4549: Store Prepare Logical Plan URL: https://github.com/apache/datafusion/issues/4549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] feat: basic support for executing prepared statements [datafusion]

2024-11-06 Thread via GitHub
jonahgao merged PR #13242: URL: https://github.com/apache/datafusion/pull/13242 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] Use `filtered_null_mask` in `CountGroupsAccumulator ` and `PrimitiveGroupsAccumulator` [datafusion]

2024-11-06 Thread via GitHub
github-actions[bot] commented on PR #11825: URL: https://github.com/apache/datafusion/pull/11825#issuecomment-2461159167 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] JoinOptimization: Add build side pushdown to probe side [datafusion]

2024-11-06 Thread via GitHub
Lordworms commented on PR #13054: URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2461159231 Another weird thing I found after rebasing main is that the ParquetExec time seems to be wrong without optimization https://github.com/user-attachments/assets/6207c16e-69e2-45

Re: [PR] Minor: Clarify use of infallable APIs [datafusion]

2024-11-06 Thread via GitHub
jonahgao merged PR #13217: URL: https://github.com/apache/datafusion/pull/13217 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] chore: Move `OptimizeAggregateOrder` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 commented on PR #13284: URL: https://github.com/apache/datafusion/pull/13284#issuecomment-2461143729 > @jayzhan211 @lewiszlw Are there preferences/limits for how much the dependencies should be moved between the crates? What do you mean by "how much the dependencies"? --

Re: [PR] feat: Require offHeap memory to be enabled (always use unified memory) [datafusion-comet]

2024-11-06 Thread via GitHub
parthchandra commented on PR #1062: URL: https://github.com/apache/datafusion-comet/pull/1062#issuecomment-2461120532 Is https://github.com/apache/datafusion-comet/issues/875 a blocker for this PR -- This is an automated message from the Apache Git Service. To respond to the message, ple

Re: [PR] chore: Move `OptimizeAggregateOrder` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jonathanc-n commented on PR #13284: URL: https://github.com/apache/datafusion/pull/13284#issuecomment-2461101735 @jayzhan211 @lewiszlw Are there preferences/limits for how much the dependencies should be moved between the crates? -- This is an automated message from the Apache Git Service

[PR] chore: Move `OptimizeAggregateOrder` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jonathanc-n opened a new pull request, #13284: URL: https://github.com/apache/datafusion/pull/13284 ## Which issue does this PR close? Part of #11502 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [PR] Use LogicalType for TypeSignature `Numeric` and `String`, `Coercible` [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 commented on PR #13240: URL: https://github.com/apache/datafusion/pull/13240#issuecomment-2461034288 Thanks @goldmedal @findepi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] Use LogicalType for TypeSignature `Numeric` and `String`, `Coercible` [datafusion]

2024-11-06 Thread via GitHub
jayzhan211 merged PR #13240: URL: https://github.com/apache/datafusion/pull/13240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] added a BallistaContext to ballista to allow for Remote or standalone [datafusion-ballista]

2024-11-06 Thread via GitHub
tbar4 commented on PR #1100: URL: https://github.com/apache/datafusion-ballista/pull/1100#issuecomment-2461011649 @andygrove @milenkovicm once these most recent checks pass I think we should merge. I think we can start to focus on follow up tasks related to the python implementation, but w

[PR] chore: Move `CoalesceBatches` from core to optimizer crate [datafusion]

2024-11-06 Thread via GitHub
jonathanc-n opened a new pull request, #13283: URL: https://github.com/apache/datafusion/pull/13283 ## Which issue does this PR close? Part of #11502 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

[PR] feat: Require offHeap memory to be enabled [datafusion-comet]

2024-11-06 Thread via GitHub
andygrove opened a new pull request, #1062: URL: https://github.com/apache/datafusion-comet/pull/1062 ## Which issue does this PR close? Closes #1017 ## Rationale for this change ## What changes are included in this PR? ## How are these chan

Re: [PR] Expand LIKE simplification: cover `NULL` pattern/expression and constant [datafusion]

2024-11-06 Thread via GitHub
findepi commented on code in PR #13260: URL: https://github.com/apache/datafusion/pull/13260#discussion_r1831797615 ## datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs: ## @@ -3633,32 +3697,123 @@ mod tests { #[test] fn test_like_and_ilke() { -

Re: [I] Return the "position" of rows in parquet files after performing a query. [datafusion]

2024-11-06 Thread via GitHub
alamb commented on issue #13261: URL: https://github.com/apache/datafusion/issues/13261#issuecomment-2460932847 I think that API makes a lot of sense. What does `MERGE` based mean? Does it mean that the row_ids come in order? -- This is an automated message from the Apache Git Servi

Re: [I] [DISCUSS] Document criteria for adding new features / what belongs in core DataFusion (e.g. sql syntax, functions, etc) [datafusion]

2024-11-06 Thread via GitHub
alamb commented on issue #12357: URL: https://github.com/apache/datafusion/issues/12357#issuecomment-2460926837 Hi @mkarbo -- DataFusion actually has its own SQL dialect that was implemented as a small extension to the sqlparser https://docs.rs/datafusion/latest/datafusion/sql/parser

Re: [I] Return the "position" of rows in parquet files after performing a query. [datafusion]

2024-11-06 Thread via GitHub
findepi commented on issue #13261: URL: https://github.com/apache/datafusion/issues/13261#issuecomment-2460923423 The positions (numbers) themselves are not enough. We need to return file paths as well. The deletions will go back into TableProvider, right? so the TableProvider itself cou

Re: [PR] WIP: Update to sqlparser-rs 0.52.0 pre-release [datafusion]

2024-11-06 Thread via GitHub
alamb commented on code in PR #13282: URL: https://github.com/apache/datafusion/pull/13282#discussion_r1831787428 ## datafusion/sqllogictest/test_files/distinct_on.slt: ## @@ -153,7 +153,7 @@ b 1 29 -18218 994303988 5983957848665088916 204 9489 3275293996 1485709125918647 c 2

Re: [PR] JoinOptimization: Add build side pushdown to probe side [datafusion]

2024-11-06 Thread via GitHub
Lordworms commented on code in PR #13054: URL: https://github.com/apache/datafusion/pull/13054#discussion_r1831785310 ## datafusion/physical-plan/src/joins/dynamic_filters.rs: ## @@ -0,0 +1,267 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contrib

[PR] WIP: Create separate instance of CometShuffleMemoryAllocator per plan [datafusion-comet]

2024-11-06 Thread via GitHub
andygrove opened a new pull request, #1061: URL: https://github.com/apache/datafusion-comet/pull/1061 ## Which issue does this PR close? Closes: https://github.com/apache/datafusion-comet/issues/886 ## Rationale for this change ## What changes are included

Re: [PR] JoinOptimization: Add build side pushdown to probe side [datafusion]

2024-11-06 Thread via GitHub
Dandandan commented on code in PR #13054: URL: https://github.com/apache/datafusion/pull/13054#discussion_r1831783035 ## datafusion/physical-plan/src/joins/dynamic_filters.rs: ## @@ -0,0 +1,267 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contrib

[PR] chore: Update to sqlparser-rs 0.52.0 pre-release [datafusion]

2024-11-06 Thread via GitHub
alamb opened a new pull request, #13282: URL: https://github.com/apache/datafusion/pull/13282 ## Which issue does this PR close? N/A ## Rationale for this change Related to https://github.com/apache/datafusion-sqlparser-rs/issues/1423 I am testing out the upgrade t

Re: [I] Release sqlparser-rs version `0.52.0` [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on issue #1423: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1423#issuecomment-2460860450 Thanks to @andygrove https://github.com/apache/datafusion-sqlparser-rs/pull/1479 is now ready for review πŸ™ -- This is an automated message from the Apache Git Ser

Re: [PR] Update CHANGELOG.md for `0.52.0` release, add scripts/ instructions for ASF releases [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on code in PR #1479: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1479#discussion_r1809404655 ## dev/release/README.md: ## @@ -0,0 +1,179 @@ + + + +## Process Overview Review Comment: This is liberally based off (aka copy/paste) from https://

Re: [PR] minor: use defaults instead of hard-coding values [datafusion-comet]

2024-11-06 Thread via GitHub
andygrove commented on code in PR #1060: URL: https://github.com/apache/datafusion-comet/pull/1060#discussion_r1831756658 ## spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala: ## @@ -1275,8 +1275,14 @@ object CometSparkSessionExtensions extends Logging {

[PR] minor: use defaults instead of hard-coding values [datafusion-comet]

2024-11-06 Thread via GitHub
andygrove opened a new pull request, #1060: URL: https://github.com/apache/datafusion-comet/pull/1060 ## Which issue does this PR close? N/A ## Rationale for this change In `CometSparkSessionExtension` we have some hard-coded default values that match the

Re: [PR] Move CHANGELOG content [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb merged PR #1503: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1503 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] Add Apache License to additional files [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on PR #1502: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1502#issuecomment-2460851046 Thank you @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Add Apache License to additional files [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb merged PR #1502: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1502 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] Move CHANGELOG content [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on PR #1503: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1503#issuecomment-2460850857 Thank you @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] Move CHANGELOG content [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on PR #1503: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1503#issuecomment-2460845723 FYI @iffyio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Add Apache License to additional files [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on PR #1502: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1502#issuecomment-2460845852 fyi @iffyio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

[PR] minor: doc fixes in fuzz testing [datafusion]

2024-11-06 Thread via GitHub
jonathanc-n opened a new pull request, #13281: URL: https://github.com/apache/datafusion/pull/13281 ## Which issue does this PR close? Closes #. ## Rationale for this change Was working on the fuzz testing and noticed some of the comments were incorrect. #

Re: [I] Release sqlparser-rs version `0.52.0` [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on issue #1423: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1423#issuecomment-2460841252 I made a few prerequisite PRs: 1. https://github.com/apache/datafusion-sqlparser-rs/pull/1502 2. https://github.com/apache/datafusion-sqlparser-rs/pull/1503

Re: [PR] add support for `FOR ORDINALITY` and `NESTED` in JSON_TABLE [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb merged PR #1493: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1493 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] Move CHANGELOG content [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
coveralls commented on PR #1503: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1503#issuecomment-2460793508 ## Pull Request Test Coverage Report for [Build 11711914569](https://coveralls.io/builds/70741840) ### Details * **0** of **0** changed or added rele

Re: [PR] Move CHANGELOG content [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on code in PR #1503: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1503#discussion_r1831711754 ## changelog/0.51.0-pre.md: ## @@ -0,0 +1,1188 @@ + + + +## [0.51.0] 2024-09-11 Review Comment: This is simply the content from `CHANGELOG.md` moved

[PR] Move CHANGELOG content [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb opened a new pull request, #1503: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1503 Part of - https://github.com/apache/datafusion-sqlparser-rs/issues/1423 While working on release scripts in https://github.com/apache/datafusion-sqlparser-rs/pull/1479 I want to

Re: [I] Inferring the possible types from the TypeSignature [datafusion]

2024-11-06 Thread via GitHub
delamarch3 commented on issue #13271: URL: https://github.com/apache/datafusion/issues/13271#issuecomment-2460782684 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] add support for `FOR ORDINALITY` and `NESTED` in JSON_TABLE [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
lovasoa commented on PR #1493: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1493#issuecomment-2460782299 Thank you for merging, @alamb :blush: Could you maybe also take a look at https://github.com/apache/datafusion-sqlparser-rs/pull/1490 ? -- This is an automated

Re: [PR] Add Apache License to additional files [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
coveralls commented on PR #1502: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1502#issuecomment-2460781187 ## Pull Request Test Coverage Report for [Build 11711824168](https://coveralls.io/builds/70741744) ### Warning: This coverage report may be inaccurate. Thi

Re: [PR] JoinOptimization: Add build side pushdown to probe side [datafusion]

2024-11-06 Thread via GitHub
Lordworms commented on code in PR #13054: URL: https://github.com/apache/datafusion/pull/13054#discussion_r1831707315 ## datafusion/physical-plan/src/joins/dynamic_filters.rs: ## @@ -0,0 +1,267 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contrib

Re: [PR] add support for `FOR ORDINALITY` and `NESTED` in JSON_TABLE [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on PR #1493: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1493#issuecomment-2460776614 Thanks @lovasoa and @iffyio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] add support for `FOR ORDINALITY` in `JSON_TABLE` [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb closed issue #1492: add support for `FOR ORDINALITY` in `JSON_TABLE` URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1492 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

[PR] Add Apache License to more files [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb opened a new pull request, #1502: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1502 Part of - https://github.com/apache/datafusion-sqlparser-rs/issues/1423 While working on release scripts in https://github.com/apache/datafusion-sqlparser-rs/pull/1479 the RAT c

[PR] feat: Added Timestamp/Binary/Float to fuzz [datafusion]

2024-11-06 Thread via GitHub
jonathanc-n opened a new pull request, #13280: URL: https://github.com/apache/datafusion/pull/13280 ## Which issue does this PR close? Closes #13279. ## What changes are included in this PR? Added timestamp, binary, and float for the fuzz testing --

Re: [PR] add support for `FOR ORDINALITY` and `NESTED` in JSON_TABLE [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
coveralls commented on PR #1493: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1493#issuecomment-2460735202 ## Pull Request Test Coverage Report for [Build 11621463747](https://coveralls.io/builds/70741294) ### Warning: This coverage report may be inaccurate. Thi

Re: [I] Release sqlparser-rs version `0.52.0` [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on issue #1423: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1423#issuecomment-2460705082 Update here is that now that we have @iffyio setup as a committer, I think we will be able to commit PRs and make the first release of this crate under the ASF proces

Re: [PR] Add support for MSSQL's `OPENJSON WITH` clause [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
lovasoa commented on code in PR #1498: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1498#discussion_r1831039009 ## src/ast/query.rs: ## @@ -2346,6 +2386,40 @@ impl fmt::Display for JsonTableColumnErrorHandling { } } +/// A single column definition in MSSQ

Re: [PR] WIP: Create separate instance of CometShuffleMemoryAllocator per plan [datafusion-comet]

2024-11-06 Thread via GitHub
andygrove closed pull request #1054: WIP: Create separate instance of CometShuffleMemoryAllocator per plan URL: https://github.com/apache/datafusion-comet/pull/1054 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Filters on `RANDOM()` are applied incorrectly when pushdown_filters is enabled. [datafusion]

2024-11-06 Thread via GitHub
alamb commented on issue #13268: URL: https://github.com/apache/datafusion/issues/13268#issuecomment-2460591532 I agree something is wrong with volatile expression pushdown -- thank you for the report @adamfaulkner-at -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Support unparsing plans after applying `optimize_projections` rule [datafusion]

2024-11-06 Thread via GitHub
sgrebnov commented on PR #13267: URL: https://github.com/apache/datafusion/pull/13267#issuecomment-2460507776 > Would this be equivalent to disabling `optimize_projections`? @findepi - the logic of `optimize_projections` is still working in this case pushing down only required colum

Re: [I] Access children `DataType` or return-type in `ScalarUDFImpl::invoke` [datafusion]

2024-11-06 Thread via GitHub
alamb commented on issue #12819: URL: https://github.com/apache/datafusion/issues/12819#issuecomment-2460426232 > Cool, https://github.com/spiraldb/datafusion/pull/1/files Thanks! Is there any way you can add an example of the `pack` UDF that shows how this API would be used?

Re: [PR] feat: Implement native version of ColumnarToRow [datafusion-comet]

2024-11-06 Thread via GitHub
parthchandra commented on PR #1034: URL: https://github.com/apache/datafusion-comet/pull/1034#issuecomment-2460421886 @andygrove I tried with your change and I get - ``` Running benchmark: ColumnarToRowExec Running case: Spark Columnar To Row - integer Stopped after 35 itera

Re: [PR] Deprecate `PhysicalSortRequirement::from_sort_exprs` and `PhysicalSortRequirement::to_sort_exprs` [datafusion]

2024-11-06 Thread via GitHub
alamb commented on PR #13222: URL: https://github.com/apache/datafusion/pull/13222#issuecomment-2459490123 Thank you for the review @berkaysynnada -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Update sqlparser requirement from 0.50 to 0.51 [datafusion-ballista]

2024-11-06 Thread via GitHub
dependabot[bot] commented on PR #1083: URL: https://github.com/apache/datafusion-ballista/pull/1083#issuecomment-2460398677 Looks like sqlparser is no longer a dependency, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Expand LIKE simplification: cover `NULL` pattern/expression and constant [datafusion]

2024-11-06 Thread via GitHub
alamb commented on code in PR #13260: URL: https://github.com/apache/datafusion/pull/13260#discussion_r1831473963 ## datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs: ## @@ -1470,34 +1471,67 @@ impl<'a, S: SimplifyInfo> TreeNodeRewriter for Simplifier<'a, S> {

Re: [PR] Expand LIKE simplification: cover `NULL` pattern/expression and constant [datafusion]

2024-11-06 Thread via GitHub
alamb commented on code in PR #13260: URL: https://github.com/apache/datafusion/pull/13260#discussion_r1831471559 ## datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs: ## @@ -3633,32 +3697,123 @@ mod tests { #[test] fn test_like_and_ilke() { -/

Re: [I] Add fuzz support for `Timestamp`, `Binary` and `Float` [datafusion]

2024-11-06 Thread via GitHub
alamb commented on issue #13279: URL: https://github.com/apache/datafusion/issues/13279#issuecomment-2460405536 > @alamb Is there a certain range for which float point numbers are compared equal to each other? any best practices? For determining groups, the comparison is exact equalit

Re: [PR] Update sqlparser requirement from 0.50 to 0.51 [datafusion-ballista]

2024-11-06 Thread via GitHub
dependabot[bot] closed pull request #1083: Update sqlparser requirement from 0.50 to 0.51 URL: https://github.com/apache/datafusion-ballista/pull/1083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Update dashmap requirement from 5.4.0 to 6.1.0 [datafusion-ballista]

2024-11-06 Thread via GitHub
dependabot[bot] closed pull request #1082: Update dashmap requirement from 5.4.0 to 6.1.0 URL: https://github.com/apache/datafusion-ballista/pull/1082 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Update dashmap requirement from 5.4.0 to 6.1.0 [datafusion-ballista]

2024-11-06 Thread via GitHub
dependabot[bot] commented on PR #1082: URL: https://github.com/apache/datafusion-ballista/pull/1082#issuecomment-2460398769 Looks like dashmap is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Update and move deps to workspace [datafusion-ballista]

2024-11-06 Thread via GitHub
andygrove merged PR #1109: URL: https://github.com/apache/datafusion-ballista/pull/1109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] Ballista proto cleanup [datafusion-ballista]

2024-11-06 Thread via GitHub
andygrove merged PR #1110: URL: https://github.com/apache/datafusion-ballista/pull/1110 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [I] Add fuzz support for `Timestamp`, `Binary` and `Float` [datafusion]

2024-11-06 Thread via GitHub
jonathanc-n commented on issue #13279: URL: https://github.com/apache/datafusion/issues/13279#issuecomment-2460353794 @alamb Is there a certain range for which float point numbers are compared equal to each other. -- This is an automated message from the Apache Git Service. To respond to

[PR] Minor: Exclude all DDL statements from Ray scheduling [datafusion-ray]

2024-11-06 Thread via GitHub
ccciudatu opened a new pull request, #42: URL: https://github.com/apache/datafusion-ray/pull/42 Do not execute `EmptyExec` physical plans on the cluster, as this is the outcome of DDL statements (and it's a no-op anyway). -- This is an automated message from the Apache Git Service. To

Re: [PR] Add support for MSSQL's `XQuery` methods [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
iffyio commented on code in PR #1500: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1500#discussion_r183149 ## src/parser/mod.rs: ## @@ -1269,6 +1269,50 @@ impl<'a> Parser<'a> { _ => self.expected("an expression", next_token), }?; +

Re: [I] Return the "position" of rows in parquet files after performing a query. [datafusion]

2024-11-06 Thread via GitHub
alamb commented on issue #13261: URL: https://github.com/apache/datafusion/issues/13261#issuecomment-2460335893 > I agree with the assessment that the information must be coning from the file reader itself. I also agree with this assessment In general I am not sure a SQL level

Re: [PR] Add support for TOP before ALL/DISTINCT [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb merged PR #1495: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1495 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] Add support for TOP before ALL/DISTINCT [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on PR #1495: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1495#issuecomment-2460337347 Thanks @yoavcloud and @iffyio -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Add support for MSSQL's `OPENJSON WITH` clause [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
iffyio commented on code in PR #1498: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1498#discussion_r1831399100 ## tests/sqlparser_mssql.rs: ## @@ -192,6 +192,35 @@ fn parse_mssql_apply_join() { ); } +#[test] +fn parse_mssql_cross_apply_json() { Review Co

Re: [I] Nov 5. 2024: This week in DataFusion [datafusion]

2024-11-06 Thread via GitHub
alamb commented on issue #13265: URL: https://github.com/apache/datafusion/issues/13265#issuecomment-2459506362 Also @jcsherin and @buraksenn are super close to converting the last of the built in window functions: https://github.com/apache/datafusion/pull/13201 -- This is an automated

Re: [PR] TOP before ALL/DISTINCT [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
iffyio commented on code in PR #1495: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1495#discussion_r1830855141 ## src/parser/mod.rs: ## @@ -3534,7 +3534,9 @@ impl<'a> Parser<'a> { pub fn parse_all_or_distinct(&mut self) -> Result, ParserError> { le

Re: [I] Bug: `SELECT * FROM ( SELECT * FROM APPLICATIONS.SALESFORCE.ACCOUNT) outer` fails to parse in snowflake [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
yoavcloud commented on issue #1457: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1457#issuecomment-2460265208 In our fork we encountered the same issue and started separating the RESERVED_FOR_TABLE_ALIAS by dialect. I was planning to introduce that here soon, as far as I t

Re: [I] Remove uses of `#[allow(dead_code)` in favor of names starting with `_` [datafusion]

2024-11-06 Thread via GitHub
ding-young commented on issue #13278: URL: https://github.com/apache/datafusion/issues/13278#issuecomment-2460260279 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add support for TOP before ALL/DISTINCT [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
coveralls commented on PR #1495: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1495#issuecomment-2460255856 ## Pull Request Test Coverage Report for [Build 11707166854](https://coveralls.io/builds/70736418) ### Details * **23** of **25** **(92.0%)** change

Re: [I] [DISCUSSION]: move sqlparser to Apache (DataFusion) governance [datafusion-sqlparser-rs]

2024-11-06 Thread via GitHub
alamb commented on issue #1294: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1294#issuecomment-2460251446 @mobuchowski -- thank you for the kind words The last few months @iffyio has really been the key maintainer. πŸ™ Thank you! -- This is an automated message f

  1   2   >