Re: [PR] NestedLoopJoin Projection Pushdown [datafusion]

2025-01-14 Thread via GitHub
berkaysynnada commented on code in PR #14120: URL: https://github.com/apache/datafusion/pull/14120#discussion_r1916044324 ## datafusion/core/src/physical_optimizer/projection_pushdown.rs: ## @@ -144,7 +144,10 @@ pub fn remove_unnecessary_projections( } else if let Some(

[PR] Add support for `IS [NOT] [form] NORMALIZED` [datafusion-sqlparser-rs]

2025-01-14 Thread via GitHub
alexander-beedie opened a new pull request, #1655: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1655 Supports the `IS [NOT] [form] NORMALIZED -> bool` syntax: Details from the PostgreSQL string function docs: https://www.postgresql.org/docs/current/functions-string.ht

Re: [PR] fix: add support for Decimal128 and Decimal256 types in interval arithmetic [datafusion]

2025-01-14 Thread via GitHub
waynexia commented on PR #14126: URL: https://github.com/apache/datafusion/pull/14126#issuecomment-2591632023 > thanks @waynexia I feed we can also add the initial problematic query into `.slt` file? > > ``` > SELECT ('0.54321543215432154321543215432154321'::DECIMAL(35,35) + 1

[PR] Feat: Support array_join [datafusion-comet]

2025-01-14 Thread via GitHub
erenavsarogullari opened a new pull request, #1290: URL: https://github.com/apache/datafusion-comet/pull/1290 ## Which issue does this PR close? Related to Epic: https://github.com/apache/datafusion-comet/issues/1042 `array_join`: `select array_join(array('hello', '-', 'world'), ' ')` =

Re: [PR] Making the data_imdb and clickbench_1 functions atomic. [datafusion]

2025-01-14 Thread via GitHub
2010YOUY01 commented on PR #14129: URL: https://github.com/apache/datafusion/pull/14129#issuecomment-2591584028 > Even I was thinking the same! But I wanted to ensure first that the same directory isn't being used to store other files. I think the convention followed now is that the

Re: [PR] Making the data_imdb and clickbench_1 functions atomic. [datafusion]

2025-01-14 Thread via GitHub
Spaarsh commented on PR #14129: URL: https://github.com/apache/datafusion/pull/14129#issuecomment-2591568792 Even I was thinking the same! But I wanted to ensure first that the same directory isn't being used to store other files. Since I would be using the ```rm -rf``` command here, that

Re: [PR] feat: add `alias()` method for DataFrame [datafusion]

2025-01-14 Thread via GitHub
jonahgao commented on code in PR #14127: URL: https://github.com/apache/datafusion/pull/14127#discussion_r1915865676 ## datafusion/core/tests/dataframe/mod.rs: ## @@ -2646,3 +2646,135 @@ async fn boolean_dictionary_as_filter() { assert_batches_eq!(expected, &result_df.col

Re: [PR] feat: add `alias()` method for DataFrame [datafusion]

2025-01-14 Thread via GitHub
jonahgao commented on code in PR #14127: URL: https://github.com/apache/datafusion/pull/14127#discussion_r1915865966 ## datafusion/core/tests/dataframe/mod.rs: ## @@ -2646,3 +2646,58 @@ async fn boolean_dictionary_as_filter() { assert_batches_eq!(expected, &result_df.coll

Re: [PR] feat: add `alias()` method for DataFrame [datafusion]

2025-01-14 Thread via GitHub
jonahgao commented on code in PR #14127: URL: https://github.com/apache/datafusion/pull/14127#discussion_r1915864922 ## datafusion/core/tests/dataframe/mod.rs: ## @@ -2646,3 +2646,135 @@ async fn boolean_dictionary_as_filter() { assert_batches_eq!(expected, &result_df.col

[PR] Reuse build side experssions values in HashJoinExec [datafusion]

2025-01-14 Thread via GitHub
lewiszlw opened a new pull request, #14131: URL: https://github.com/apache/datafusion/pull/14131 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested?

Re: [PR] JoinOptimization: Add build side pushdown to probe side [datafusion]

2025-01-14 Thread via GitHub
github-actions[bot] commented on PR #13054: URL: https://github.com/apache/datafusion/pull/13054#issuecomment-2591473073 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] fix: incorrect NATURAL/USING JOIN schema [datafusion]

2025-01-14 Thread via GitHub
jonahgao commented on PR #14102: URL: https://github.com/apache/datafusion/pull/14102#issuecomment-2591453341 Thanks @DDtKey @alamb for the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] Return err if wildcard is not expanded before type coercion [datafusion]

2025-01-14 Thread via GitHub
xudong963 opened a new pull request, #14130: URL: https://github.com/apache/datafusion/pull/14130 ## Rationale for this change If users use the whole analyzer, they won't encounter the issue. But if only use the [TypeCoercionRewriter](https://docs.rs/datafusion/latest/datafus

[I] Unsupported Arrow Vector for export: class org.apache.arrow.vector.complex.ListVector [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove opened a new issue, #1289: URL: https://github.com/apache/datafusion-comet/issues/1289 ### Describe the bug I am trying to add support for distinct aggregates in https://github.com/apache/datafusion-comet/pull/1261. The test "multiple group-by columns + single aggrega

Re: [PR] Feat: Support array_intersect [datafusion-comet]

2025-01-14 Thread via GitHub
erenavsarogullari commented on PR #1271: URL: https://github.com/apache/datafusion-comet/pull/1271#issuecomment-2591389516 Sure @andygrove. I can also work on https://github.com/apache/datafusion-comet/issues/1269 by assigning myself and cover `array_intersect` as part of it. -- This is

Re: [PR] Feat/use uv python management [datafusion-python]

2025-01-14 Thread via GitHub
timsaucer commented on code in PR #994: URL: https://github.com/apache/datafusion-python/pull/994#discussion_r1915787482 ## .github/workflows/build.yml: ## @@ -35,8 +35,6 @@ jobs: - name: Install dependencies uses: astral-sh/setup-uv@v5 -with: -

Re: [PR] feat: Add HasRowIdMapping interface [datafusion-comet]

2025-01-14 Thread via GitHub
viirya commented on code in PR #1288: URL: https://github.com/apache/datafusion-comet/pull/1288#discussion_r1915728249 ## common/src/main/java/org/apache/comet/vector/HasRowIdMapping.java: ## @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + *

[PR] feat: Add HasRowIdMapping interface [datafusion-comet]

2025-01-14 Thread via GitHub
viirya opened a new pull request, #1288: URL: https://github.com/apache/datafusion-comet/pull/1288 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes te

Re: [PR] docs: Fix links and provide complete benchmarking scripts [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove merged PR #1284: URL: https://github.com/apache/datafusion-comet/pull/1284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] chore: Start 0.6.0 development [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove merged PR #1286: URL: https://github.com/apache/datafusion-comet/pull/1286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] Round floats but not decimals in SqlLogicTests [datafusion]

2025-01-14 Thread via GitHub
Omega359 commented on PR #13743: URL: https://github.com/apache/datafusion/pull/13743#issuecomment-2591129891 > float rounding could help with some of these eg > https://private-user-images.githubusercontent.com/144328/395788351-df0d41ff-0a70-4577-b72a-2bb8025b9f97.png?jwt=eyJhbGciOiJIUzI1

Re: [PR] fix: incorrect NATURAL/USING JOIN schema [datafusion]

2025-01-14 Thread via GitHub
alamb merged PR #14102: URL: https://github.com/apache/datafusion/pull/14102 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] fix: incorrect NATURAL/USING JOIN schema [datafusion]

2025-01-14 Thread via GitHub
alamb commented on PR #14102: URL: https://github.com/apache/datafusion/pull/14102#issuecomment-2591103278 Thanks @jonahgao and @DDtKey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Regression: `DataFrame::schema` returns incorrect schema for NATURAL JOIN [datafusion]

2025-01-14 Thread via GitHub
alamb closed issue #14058: Regression: `DataFrame::schema` returns incorrect schema for NATURAL JOIN URL: https://github.com/apache/datafusion/issues/14058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Feat: Support array_intersect [datafusion-comet]

2025-01-14 Thread via GitHub
erenavsarogullari commented on code in PR #1271: URL: https://github.com/apache/datafusion-comet/pull/1271#discussion_r1915588010 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -2545,4 +2545,20 @@ class CometExpressionSuite extends CometTestBase with

Re: [PR] feat: add `alias()` method for DataFrame [datafusion]

2025-01-14 Thread via GitHub
comphead commented on code in PR #14127: URL: https://github.com/apache/datafusion/pull/14127#discussion_r1915584242 ## datafusion/core/tests/dataframe/mod.rs: ## @@ -2646,3 +2646,58 @@ async fn boolean_dictionary_as_filter() { assert_batches_eq!(expected, &result_df.coll

Re: [PR] Feat/use uv python management [datafusion-python]

2025-01-14 Thread via GitHub
timsaucer commented on PR #994: URL: https://github.com/apache/datafusion-python/pull/994#issuecomment-2591078566 @kylebarron ready for review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Chore: refactor DataSink traits to avoid duplication [datafusion]

2025-01-14 Thread via GitHub
ozankabak commented on code in PR #14121: URL: https://github.com/apache/datafusion/pull/14121#discussion_r1915581273 ## datafusion/core/src/datasource/physical_plan/mod.rs: ## @@ -63,14 +64,53 @@ use crate::{ }; use arrow::datatypes::{DataType, SchemaRef}; +use datafusion_c

Re: [PR] Feat: Support array_intersect [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove commented on code in PR #1271: URL: https://github.com/apache/datafusion-comet/pull/1271#discussion_r1915556986 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -2545,4 +2545,20 @@ class CometExpressionSuite extends CometTestBase with Adaptiv

Re: [PR] test: show a mismatch for initcap between Spark and DataFusion [datafusion-comet]

2025-01-14 Thread via GitHub
kazuyukitanimura commented on PR #1051: URL: https://github.com/apache/datafusion-comet/pull/1051#issuecomment-2591039449 @Blizzara please feel free to reopen once the proper fix is ready -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Chore: refactor DataSink traits to avoid duplication [datafusion]

2025-01-14 Thread via GitHub
alamb commented on PR #14121: URL: https://github.com/apache/datafusion/pull/14121#issuecomment-2591013987 Thank you @mertak-synnada -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Chore: refactor DataSink traits to avoid duplication [datafusion]

2025-01-14 Thread via GitHub
alamb commented on code in PR #14121: URL: https://github.com/apache/datafusion/pull/14121#discussion_r1915535246 ## datafusion/core/src/datasource/physical_plan/mod.rs: ## @@ -63,14 +64,53 @@ use crate::{ }; use arrow::datatypes::{DataType, SchemaRef}; +use datafusion_commo

[PR] chore: Start 0.6.0 development [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove opened a new pull request, #1286: URL: https://github.com/apache/datafusion-comet/pull/1286 ## Which issue does this PR close? N/A ## Rationale for this change Update version numbers now that we have created the `branch-0.5` release branch.

[PR] Making the data_imdb and clickbench_1 functions atomic. [datafusion]

2025-01-14 Thread via GitHub
Spaarsh opened a new pull request, #14129: URL: https://github.com/apache/datafusion/pull/14129 ## Which issue does this PR close? Closes #14128 ## Rationale for this change Due to non-atomic downloads, the user would need to manually remove files/folders created by

[I] Making Downloads Atomic in bench.sh [datafusion]

2025-01-14 Thread via GitHub
Spaarsh opened a new issue, #14128: URL: https://github.com/apache/datafusion/issues/14128 ### Is your feature request related to a problem or challenge? As discussed under [issue#13896](https://github.com/apache/datafusion/issues/13896#issuecomment-2561745899) the downloading operat

[PR] [comet-parquet-exec] wip: merge main into comet-parquet-exec [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove opened a new pull request, #1285: URL: https://github.com/apache/datafusion-comet/pull/1285 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## How are these changes

Re: [PR] [comet-parquet-exec] wip: merge main into comet-parquet-exec [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove closed pull request #1285: [comet-parquet-exec] wip: merge main into comet-parquet-exec URL: https://github.com/apache/datafusion-comet/pull/1285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] Comet web site is no longer updated to reflect recent documentation changes [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove closed issue #1282: Comet web site is no longer updated to reflect recent documentation changes URL: https://github.com/apache/datafusion-comet/issues/1282 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [I] Comet web site is no longer updated to reflect recent documentation changes [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove commented on issue #1282: URL: https://github.com/apache/datafusion-comet/issues/1282#issuecomment-2590610317 I manually pushed a whitespace change to the asf-site branch and that fixed it for now -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Minor: Document the rationale for the lack of Cargo.lock [datafusion]

2025-01-14 Thread via GitHub
comphead commented on code in PR #14071: URL: https://github.com/apache/datafusion/pull/14071#discussion_r1915282401 ## README.md: ## @@ -146,3 +146,27 @@ stable API, we also improve the API over time. As a result, we typically deprecate methods before removing them, according

Re: [PR] Add sqlite sqllogictest run to extended.yml [datafusion]

2025-01-14 Thread via GitHub
Omega359 commented on code in PR #14101: URL: https://github.com/apache/datafusion/pull/14101#discussion_r1915280445 ## .github/workflows/extended.yml: ## @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -name: Rust Hash Collis

Re: [I] Add `union_extract` function [datafusion]

2025-01-14 Thread via GitHub
gstvg commented on issue #11081: URL: https://github.com/apache/datafusion/issues/11081#issuecomment-2590587357 Hi @tobixdev My bad, the arrow kernel ended up being enough for me and I got busy with other work, leaving this for later. I'm already started updating #12116, I should pu

Re: [PR] Add sqlite sqllogictest run to extended.yml [datafusion]

2025-01-14 Thread via GitHub
comphead commented on code in PR #14101: URL: https://github.com/apache/datafusion/pull/14101#discussion_r1915276899 ## .github/workflows/extended.yml: ## @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -name: Rust Hash Collis

Re: [PR] chore: extract math_funcs expressions to folders based on spark grouping [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove commented on PR #1219: URL: https://github.com/apache/datafusion-comet/pull/1219#issuecomment-2590569143 @rluvaton I managed to get most of these PRs merged. Would you have time to help fix merge conflicts on this one? -- This is an automated message from the Apache Git Service

Re: [PR] fix: cast timestamp to decimal is unsupported [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove merged PR #1281: URL: https://github.com/apache/datafusion-comet/pull/1281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Cast timestamp to decimal is unsupported [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove closed issue #1280: Cast timestamp to decimal is unsupported URL: https://github.com/apache/datafusion-comet/issues/1280 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[PR] docs: Fix links and provide complete be benchmarking scripts [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove opened a new pull request, #1284: URL: https://github.com/apache/datafusion-comet/pull/1284 ## Which issue does this PR close? N/A ## Rationale for this change The links were incorrect for the TPC-H results. I also thought it would be best to pr

Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

2025-01-14 Thread via GitHub
alamb commented on issue #14115: URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2590528988 Maybe once we have done one or two PRs we can use them as examples and file tickets for the other uses (to do the work in parallel). It is going to be awesoem -- This is an aut

Re: [PR] chore: whitespace change to force docs to be published [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove merged PR #1283: URL: https://github.com/apache/datafusion-comet/pull/1283 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

[PR] chore: whitespace change to force docs to be published [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove opened a new pull request, #1283: URL: https://github.com/apache/datafusion-comet/pull/1283 ## Which issue does this PR close? Related to https://github.com/apache/datafusion-comet/issues/1282 ## Rationale for this change Recent changes in `asf-s

Re: [I] Comet web site is no longer updated to reflect recent documentation changes [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove commented on issue #1282: URL: https://github.com/apache/datafusion-comet/issues/1282#issuecomment-2590518319 The `asf-site` branch has been updated but has not been published -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

[PR] feat: add `alias()` method for DataFrame [datafusion]

2025-01-14 Thread via GitHub
jonahgao opened a new pull request, #14127: URL: https://github.com/apache/datafusion/pull/14127 ## Which issue does this PR close? Closes #14112. ## Rationale for this change The new `alias()` method applies an alias to an existing DataFrame by wrapping a `SubqueryA

Re: [I] Add a hint about normalization in error message [datafusion]

2025-01-14 Thread via GitHub
comphead closed issue #14089: Add a hint about normalization in error message URL: https://github.com/apache/datafusion/issues/14089 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Add a hint about normalization in error message (#14089) [datafusion]

2025-01-14 Thread via GitHub
comphead merged PR #14113: URL: https://github.com/apache/datafusion/pull/14113 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

[I] Comet web site is no longer updated to reflect recent documentation changes [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove opened a new issue, #1282: URL: https://github.com/apache/datafusion-comet/issues/1282 ### Describe the bug Comet web site is no longer updated to reflect recent documentation changes ### Steps to reproduce _No response_ ### Expected behavior _No r

Re: [PR] Distinguish None and empty projection in unparser [datafusion]

2025-01-14 Thread via GitHub
comphead merged PR #14116: URL: https://github.com/apache/datafusion/pull/14116 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] fix: cast timestamp to decimal is unsupported [datafusion-comet]

2025-01-14 Thread via GitHub
codecov-commenter commented on PR #1281: URL: https://github.com/apache/datafusion-comet/pull/1281#issuecomment-2590444208 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/1281?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [PR] fix: handle loading of complex types into CometVector correctly in iceberg_compat scans [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove merged PR #1279: URL: https://github.com/apache/datafusion-comet/pull/1279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] WIP Upgrade to arrow-rs/parquet `54.0.0` [datafusion]

2025-01-14 Thread via GitHub
Owen-CH-Leung commented on PR #13663: URL: https://github.com/apache/datafusion/pull/13663#issuecomment-2590366307 > > I would appreciate this PR getting in before the next datafusion release 🙏 > > Yes I hope to -- if anyone has time to help get it in shape / make a new PR it would b

Re: [PR] fix: encode should work with non-UTF-8 binaries [datafusion]

2025-01-14 Thread via GitHub
Omega359 commented on code in PR #14087: URL: https://github.com/apache/datafusion/pull/14087#discussion_r1915125736 ## datafusion/functions/src/encoding/inner.rs: ## @@ -87,7 +87,20 @@ impl ScalarUDFImpl for EncodeFunc { } fn return_type(&self, arg_types: &[DataType

Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

2025-01-14 Thread via GitHub
Chen-Yuan-Lai commented on issue #14115: URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2590270808 @alamb Sure, thanks for the suggestion -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] Feat/use uv python management [datafusion-python]

2025-01-14 Thread via GitHub
timsaucer commented on PR #994: URL: https://github.com/apache/datafusion-python/pull/994#issuecomment-2590252479 Remaining things to do: - [ ] Try enabling the cache - [ ] Update instructions for local docs building - [ ] Update readme - [ ] Update online docs -- This is

Re: [PR] Feat/use uv python management [datafusion-python]

2025-01-14 Thread via GitHub
timsaucer commented on code in PR #994: URL: https://github.com/apache/datafusion-python/pull/994#discussion_r1915049658 ## .github/workflows/build.yml: ## @@ -46,7 +46,13 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: dtolnay/

Re: [I] Support `MemoryExec` in proto `try_from_physical_plan` [datafusion]

2025-01-14 Thread via GitHub
jatin510 commented on issue #14082: URL: https://github.com/apache/datafusion/issues/14082#issuecomment-2590201610 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] fix: cast timestamp to decimal is unsupported [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove commented on PR #1281: URL: https://github.com/apache/datafusion-comet/pull/1281#issuecomment-2590126905 There is a test failure: ``` 2025-01-14T14:27:25.3861150Z CometCastSuite: 2025-01-14T14:27:25.4393419Z - all valid cast combinations covered *** FAILED *** (13 mil

Re: [PR] chore: Update TPC-DS results for 0.5.0 release [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove merged PR #1277: URL: https://github.com/apache/datafusion-comet/pull/1277 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] chore: Add changelog for 0.5.0 [datafusion-comet]

2025-01-14 Thread via GitHub
andygrove merged PR #1278: URL: https://github.com/apache/datafusion-comet/pull/1278 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [PR] Support trailing commas in `FROM` clause [datafusion-sqlparser-rs]

2025-01-14 Thread via GitHub
alamb merged PR #1645: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr.

Re: [PR] Support trailing commas in `FROM` clause [datafusion-sqlparser-rs]

2025-01-14 Thread via GitHub
alamb commented on PR #1645: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1645#issuecomment-2590099156 Thanks @barsela1 and @iffyio ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Move JoinSelection into datafusion-physical-optimizer crate (#14073) [datafusion]

2025-01-14 Thread via GitHub
alamb commented on PR #14085: URL: https://github.com/apache/datafusion/pull/14085#issuecomment-2589929141 Thanks agian @cj-zhukov -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] Move `JoinSelection` into `datafusion-physical-optimizer` crate [datafusion]

2025-01-14 Thread via GitHub
alamb closed issue #14073: Move `JoinSelection` into `datafusion-physical-optimizer` crate URL: https://github.com/apache/datafusion/issues/14073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Move JoinSelection into datafusion-physical-optimizer crate (#14073) [datafusion]

2025-01-14 Thread via GitHub
alamb merged PR #14085: URL: https://github.com/apache/datafusion/pull/14085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] fix: cast timestamp to decimal is unsupported [datafusion-comet]

2025-01-14 Thread via GitHub
wForget commented on code in PR #1281: URL: https://github.com/apache/datafusion-comet/pull/1281#discussion_r1914817644 ## spark/src/test/scala/org/apache/comet/CometCastSuite.scala: ## @@ -838,8 +838,9 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper

Re: [I] [Epic] Remove Sort Merge Join Experimental status [datafusion]

2025-01-14 Thread via GitHub
16pierre commented on issue #9846: URL: https://github.com/apache/datafusion/issues/9846#issuecomment-2589918623 > Adjusting heuristics will likely just make some people's plans better but others worse Yes, what I had in mind was to stay behind the `!prefer_hash_join` flag to use SMJ

[PR] fix: cast timestamp to decimal is unsupported [datafusion-comet]

2025-01-14 Thread via GitHub
wForget opened a new pull request, #1281: URL: https://github.com/apache/datafusion-comet/pull/1281 ## Which issue does this PR close? Closes #1280. ## Rationale for this change ## What changes are included in this PR? Fix test case and mark cas

[I] Cast timestamp to decimal is unsupported [datafusion-comet]

2025-01-14 Thread via GitHub
wForget opened a new issue, #1280: URL: https://github.com/apache/datafusion-comet/issues/1280 ### Describe the bug Found error by fuzz testing: Schema: ``` Created table test0 with schema: ... c57: TimestampType ``` SQL: ``` SELECT c57, cast(c57

[PR] fix: add support for Decimal128 and Decimal256 types in interval arit… [datafusion]

2025-01-14 Thread via GitHub
waynexia opened a new pull request, #14126: URL: https://github.com/apache/datafusion/pull/14126 …hmetic ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/14124 ## Rationale for this change Fixes the panic when de

[PR] Fix: regularize order bys when consuming from substrait [datafusion]

2025-01-14 Thread via GitHub
gabotechs opened a new pull request, #14125: URL: https://github.com/apache/datafusion/pull/14125 ## Which issue does this PR close? - [ ] TODO: add an issue with replication steps? ## Rationale for this change Fixing an issue with the current substrait consumer ##

Re: [PR] Updated slt files with latest changes from datafusion and including switch from skipif Datafusion -> query error [datafusion-testing]

2025-01-14 Thread via GitHub
alamb merged PR #4: URL: https://github.com/apache/datafusion-testing/pull/4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

[I] [Regression] Panic when handling Decimal128 overflow [datafusion]

2025-01-14 Thread via GitHub
discord9 opened a new issue, #14124: URL: https://github.com/apache/datafusion/issues/14124 ### Describe the bug Panic when handling Decimal128 overflow: ```sql SELECT ('0.54321543215432154321543215432154321'::DECIMAL(35,35) + 1)::VARCHAR; ``` ### To Reproduce

Re: [PR] Interface for physical plan invariant checking. [datafusion]

2025-01-14 Thread via GitHub
alamb commented on code in PR #13986: URL: https://github.com/apache/datafusion/pull/13986#discussion_r1914720254 ## datafusion/physical-optimizer/src/optimizer.rs: ## @@ -46,4 +46,23 @@ pub trait PhysicalOptimizerRule: Debug { /// Some of the optimization rules might chang

Re: [I] [EPIC] Run full sqllogic / sqlite test suite against DataFusion [datafusion]

2025-01-14 Thread via GitHub
alamb commented on issue #13811: URL: https://github.com/apache/datafusion/issues/13811#issuecomment-2589743668 Thanks @Omega359 -- I added https://github.com/apache/datafusion/issues/13784 https://github.com/apache/datafusion/issues/2408 and https://github.com/apache/datafusion/issues/13

Re: [PR] Updated slt files with latest changes from datafusion and including switch from skipif Datafusion -> query error [datafusion-testing]

2025-01-14 Thread via GitHub
alamb commented on PR #4: URL: https://github.com/apache/datafusion-testing/pull/4#issuecomment-2589743124 Thanks @Omega359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] `lz0` should be `lzo` [datafusion-python]

2025-01-14 Thread via GitHub
timsaucer closed issue #979: `lz0` should be `lzo` URL: https://github.com/apache/datafusion-python/issues/979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [PR] fix: correct LZ0 to LZO in compression options [datafusion-python]

2025-01-14 Thread via GitHub
timsaucer merged PR #995: URL: https://github.com/apache/datafusion-python/pull/995 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@d

Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

2025-01-14 Thread via GitHub
alamb commented on issue #14115: URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2589715949 BTW @Chen-Yuan-Lai I very much suggest doing this task as multiple smaller PRs if possible (e.g. make a PR to replace the use in Correlation, make another PR to replace a

Re: [PR] Support trailing commas in `FROM` clause [datafusion-sqlparser-rs]

2025-01-14 Thread via GitHub
alamb commented on code in PR #1645: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1645#discussion_r1914691270 ## src/dialect/mod.rs: ## @@ -404,6 +404,12 @@ pub trait Dialect: Debug + Any { self.supports_trailing_commas() } +/// Returns true i

Re: [PR] Support trailing commas in `FROM` clause [datafusion-sqlparser-rs]

2025-01-14 Thread via GitHub
alamb commented on PR #1645: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1645#issuecomment-2589710909 I also restarted the checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Release DataFusion `45.0.0` [datafusion]

2025-01-14 Thread via GitHub
alamb commented on issue #14008: URL: https://github.com/apache/datafusion/issues/14008#issuecomment-2589707968 > > > > I am happy to do it again for 45 if no one else would like the opportunity (see what I did there 😆 ) > > Thanks, alamb, I booked 46 in advance! Awesom

Re: [I] Release DataFusion `46.0.0` [datafusion]

2025-01-14 Thread via GitHub
alamb commented on issue #14123: URL: https://github.com/apache/datafusion/issues/14123#issuecomment-2589707349 I think @xudong963 said they might be interested in running this one: - https://github.com/apache/datafusion/issues/14008#issuecomment-2589148761 > Thanks, alamb, I book

[I] Release DataFusion `46.0.0` [datafusion]

2025-01-14 Thread via GitHub
alamb opened a new issue, #14123: URL: https://github.com/apache/datafusion/issues/14123 ### Is your feature request related to a problem or challenge? ### Is your feature request related to a problem or challenge? Tracking ticket for next release, also a place to track desired

Re: [I] Clarify logic in `nth_value` function [datafusion]

2025-01-14 Thread via GitHub
alamb closed issue #14063: Clarify logic in `nth_value` function URL: https://github.com/apache/datafusion/issues/14063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [PR] clarify logic in nth_value window function [datafusion]

2025-01-14 Thread via GitHub
alamb merged PR #14104: URL: https://github.com/apache/datafusion/pull/14104 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Chore/single sink exec [datafusion]

2025-01-14 Thread via GitHub
ozankabak commented on PR #14121: URL: https://github.com/apache/datafusion/pull/14121#issuecomment-2589595470 Due to Rust's incomplete specialization implementation, we have some trivial duplication regarding `write_all`, but they will go away when Rust catches up -- This is an automated

Re: [PR] Support trailing commas in `FROM` clause [datafusion-sqlparser-rs]

2025-01-14 Thread via GitHub
iffyio commented on PR #1645: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1645#issuecomment-2589568188 @barsela1 could you take a look to resolve the conflicts on the branch? -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] Propagate table constraints through physical plans to optimize sort operations [datafusion]

2025-01-14 Thread via GitHub
ozankabak commented on PR #14111: URL: https://github.com/apache/datafusion/pull/14111#issuecomment-2589365147 I left my reviews here: https://github.com/synnada-ai/datafusion-upstream/pull/53#pullrequestreview-2549129230 -- This is an automated message from the Apache Git Service. To res

[PR] Minor: fix duplicated SharedBitmapBuilder definitions [datafusion]

2025-01-14 Thread via GitHub
lewiszlw opened a new pull request, #14122: URL: https://github.com/apache/datafusion/pull/14122 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested?

Re: [PR] Chore/single sink exec [datafusion]

2025-01-14 Thread via GitHub
mertak-synnada commented on code in PR #14121: URL: https://github.com/apache/datafusion/pull/14121#discussion_r1914428755 ## datafusion/core/src/datasource/physical_plan/mod.rs: ## @@ -90,13 +109,23 @@ pub struct FileSinkConfig { pub insert_op: InsertOp, /// Controls

[PR] Chore/single sink exec [datafusion]

2025-01-14 Thread via GitHub
mertak-synnada opened a new pull request, #14121: URL: https://github.com/apache/datafusion/pull/14121 ## Which issue does this PR close? Closes #. This PR creates an abstraction for FileSinkers' generic behaviors and centralizes some of the functionalities. The main in