Re: [PR] fix: Disable tests that require SubqueryBroadcastExec [datafusion-comet]

2024-07-10 Thread via GitHub
viirya commented on code in PR #647: URL: https://github.com/apache/datafusion-comet/pull/647#discussion_r1671722907 ## dev/diffs/4.0.0-preview1.diff: ## @@ -442,7 +442,7 @@ index 2c24cc7d570..50a2ce86117 100644 - test("partition pruning in broadcast hash joins with aliases

Re: [I] Remove additional unnecessary qualified names in the code [datafusion]

2024-07-10 Thread via GitHub
Weijun-H commented on issue #11317: URL: https://github.com/apache/datafusion/issues/11317#issuecomment-2219788432 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

[PR] Expose the correct namespaces for Python modules [datafusion-python]

2024-07-10 Thread via GitHub
datapythonista opened a new pull request, #751: URL: https://github.com/apache/datafusion-python/pull/751 Currently, in the `datafusion` module there is no match between what `dir()` returns, and what's actually inside the module. For example: ```python >>> import datafusion

Re: [PR] Python wrapper classes for all user interfaces [datafusion-python]

2024-07-10 Thread via GitHub
datapythonista commented on PR #750: URL: https://github.com/apache/datafusion-python/pull/750#issuecomment-2219816929 This looks great in terms of functionality, but I personally don't think it's worth to have wrappers for every DataFusion functionality just for the types and the docs. Af

Re: [PR] Enable `clone_on_ref_ptr` clippy lint on optimizer [datafusion]

2024-07-10 Thread via GitHub
Dandandan merged PR #11346: URL: https://github.com/apache/datafusion/pull/11346 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] Enable `clone_on_ref_ptr` clippy lint on optimizer [datafusion]

2024-07-10 Thread via GitHub
Dandandan commented on PR #11346: URL: https://github.com/apache/datafusion/pull/11346#issuecomment-2219819628 Thank you @lewiszlw -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Introduce `resources_err!` error macro [datafusion]

2024-07-10 Thread via GitHub
Weijun-H commented on PR #11374: URL: https://github.com/apache/datafusion/pull/11374#issuecomment-2219892946 It would be better to replace the following but also be done on a follow-on. https://github.com/apache/arrow-datafusion/blob/37428bb2034abc1aba5d4cd52cf8f2885623bada/datafusio

[PR] Update sqlparser requirement from 0.47 to 0.48 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] opened a new pull request, #11377: URL: https://github.com/apache/datafusion/pull/11377 Updates the requirements on [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) to permit the latest version. Changelog Sourced from https://github.com/sqlparser-rs/sqlpar

[PR] Update pbjson requirement from 0.6.0 to 0.7.0 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] opened a new pull request, #11378: URL: https://github.com/apache/datafusion/pull/11378 Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]:

[PR] Update prost-build requirement from =0.12.6 to =0.13.1 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] opened a new pull request, #11379: URL: https://github.com/apache/datafusion/pull/11379 Updates the requirements on [prost-build](https://github.com/tokio-rs/prost) to permit the latest version. Changelog Sourced from https://github.com/tokio-rs/prost/blob/master/CH

Re: [PR] Update prost-build requirement from =0.12.6 to =0.13.0 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] commented on PR #11352: URL: https://github.com/apache/datafusion/pull/11352#issuecomment-2219903622 Superseded by #11379. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Update prost-build requirement from =0.12.6 to =0.13.0 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] closed pull request #11352: Update prost-build requirement from =0.12.6 to =0.13.0 URL: https://github.com/apache/datafusion/pull/11352 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[PR] Enable `clone_on_ref_ptr` clippy lint on sql [datafusion]

2024-07-10 Thread via GitHub
lewiszlw opened a new pull request, #11380: URL: https://github.com/apache/datafusion/pull/11380 ## Which issue does this PR close? Part of https://github.com/apache/datafusion/issues/11143. Code splited from https://github.com/apache/datafusion/pull/11157. ## Rationale

[PR] Update substrait requirement from 0.36.0 to 0.37.3 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] opened a new pull request, #11381: URL: https://github.com/apache/datafusion/pull/11381 Updates the requirements on [substrait](https://github.com/substrait-io/substrait-rs) to permit the latest version. Release notes Sourced from https://github.com/substrait-io/su

[PR] Update pbjson-build requirement from =0.6.2 to =0.7.0 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] opened a new pull request, #11382: URL: https://github.com/apache/datafusion/pull/11382 Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]:

Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]

2024-07-10 Thread via GitHub
jayzhan211 commented on PR #11361: URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2219914336 For `make_map(k1, v1, k2, v2...)`, ideally we can arrange the order of args and call `map([k1, k2..], [v1, v2])`. I think we can arrange it in ExprPlanner, so we can have only one

[PR] Update termtree requirement from 0.4.1 to 0.5.0 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] opened a new pull request, #11383: URL: https://github.com/apache/datafusion/pull/11383 Updates the requirements on [termtree](https://github.com/rust-cli/termtree) to permit the latest version. Changelog Sourced from https://github.com/rust-cli/termtree/blob/main/C

[PR] Enable `clone_on_ref_ptr` clippy lint on common [datafusion]

2024-07-10 Thread via GitHub
lewiszlw opened a new pull request, #11384: URL: https://github.com/apache/datafusion/pull/11384 ## Which issue does this PR close? Part of https://github.com/apache/datafusion/issues/11143. Code splited from https://github.com/apache/datafusion/pull/11157. ## Rationale

[I] Regression in TPC-DS Q75 [datafusion]

2024-07-10 Thread via GitHub
gruuya opened a new issue, #11385: URL: https://github.com/apache/datafusion/issues/11385 ### Describe the bug While logical and physical planning work fine for TPC-DS Q75, as of v39.0.0 the execution throws the following error ``` Error: External(External(ArrowError(InvalidArg

[PR] Coerce types for all union children plans when eliminating nesting [datafusion]

2024-07-10 Thread via GitHub
gruuya opened a new pull request, #11386: URL: https://github.com/apache/datafusion/pull/11386 ## Which issue does this PR close? Closes #11385. ## Rationale for this change Investigating the above issue led me to identify a couple of aspects that need to align in order

Re: [I] Remove additional unnecessary qualified names in the code [datafusion]

2024-07-10 Thread via GitHub
Weijun-H commented on issue #11317: URL: https://github.com/apache/datafusion/issues/11317#issuecomment-2219948733 Consider if there is a better way to automatically detect this issue in Clippy or other crates. 🤔 -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] fix: Disable tests that require SubqueryBroadcastExec [datafusion-comet]

2024-07-10 Thread via GitHub
kazuyukitanimura commented on code in PR #647: URL: https://github.com/apache/datafusion-comet/pull/647#discussion_r1671906662 ## dev/diffs/4.0.0-preview1.diff: ## @@ -442,7 +442,7 @@ index 2c24cc7d570..50a2ce86117 100644 - test("partition pruning in broadcast hash joins wi

Re: [PR] Improve `CommonSubexprEliminate` rule with surely and conditionally evaluated stats [datafusion]

2024-07-10 Thread via GitHub
peter-toth commented on PR #11357: URL: https://github.com/apache/datafusion/pull/11357#issuecomment-2219957257 > Thank you @peter-toth the CI clippy error was fixed in #11368 so if you merge up from main the tests should now pass > > I will review this PR tomorrow Thank you @a

Re: [I] StateBackend in DataFusion's RuntimeEnv [datafusion]

2024-07-10 Thread via GitHub
ozankabak commented on issue #11365: URL: https://github.com/apache/datafusion/issues/11365#issuecomment-2220076686 We actually do something like this in our use of DF for stream processing. Since it could remain unused/irrelevant for cases other than stream processing, it may be an overfit

Re: [I] Remove additional unnecessary qualified names in the code [datafusion]

2024-07-10 Thread via GitHub
alamb commented on issue #11317: URL: https://github.com/apache/datafusion/issues/11317#issuecomment-2220094166 > Consider if there is a better way to automatically detect this issue in Clippy or other crates. 🤔 I think @findepi said there was some feature in RustRover that did it. A

[PR] Support serialization/deserialization for custom physical exprs in proto [datafusion]

2024-07-10 Thread via GitHub
lewiszlw opened a new pull request, #11387: URL: https://github.com/apache/datafusion/pull/11387 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/11350. ## Rationale for this change ## What changes are included in this

Re: [PR] Remove redundant `unalias_nested` calls for creating Filter's [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11340: URL: https://github.com/apache/datafusion/pull/11340#issuecomment-2220097367 Thanks again @jonahgao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] Deterministic IDs for ExecutionPlan [datafusion]

2024-07-10 Thread via GitHub
mustafasrepo commented on issue #11364: URL: https://github.com/apache/datafusion/issues/11364#issuecomment-2220122107 During planning there are couple of physical optimization rules which modify the `ExecutionPlan` (rules can be found in [the link](https://github.com/apache/datafusion/blob

Re: [PR] Impl a general get results from stats [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11261: URL: https://github.com/apache/datafusion/pull/11261#issuecomment-2220163407 > Ok, been a bit busy the past couple of days, continue to read the related codes and think a relatively good way to solve this today... One simple way I could think to solve it tempor

Re: [PR] Update `prost`, `prost-derive`, `pbjson`, `tonic` ecosystem [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11372: URL: https://github.com/apache/datafusion/pull/11372#issuecomment-2220221076 Thank you @tisonkun It appears to me that compilation is failing now as arrow-flight is using an older version of prost/tonic, but since datafusion itself doesn't depend on any

Re: [PR] Blog post for release 40.0.0 [datafusion-site]

2024-07-10 Thread via GitHub
alamb commented on PR #6: URL: https://github.com/apache/datafusion-site/pull/6#issuecomment-2220231640 Thanks @phillipleblanc -- I added your suggestions. I hope to clean this up more over the next few days -- This is an automated message from the Apache Git Service. To respond to the m

Re: [I] Implement SQLancer (a end-to-end SQL fuzz testing library) [datafusion]

2024-07-10 Thread via GitHub
alamb commented on issue #11030: URL: https://github.com/apache/datafusion/issues/11030#issuecomment-2220233006 This is really nice work @2010YOUY01 -- thank you so much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [I] Add StreamingWindowExec to DataFusion physical plan to support aggregations over unbounded data [datafusion]

2024-07-10 Thread via GitHub
ozankabak commented on issue #11366: URL: https://github.com/apache/datafusion/issues/11366#issuecomment-2220241361 To add to the comment above -- `BoundedWindowAggExec` also supports streaming (when possible). -- This is an automated message from the Apache Git Service. To respond to the

[I] Improvements to user guide for DataFrames [datafusion]

2024-07-10 Thread via GitHub
alamb opened a new issue, #11388: URL: https://github.com/apache/datafusion/issues/11388 First of all, I'm not sure we need the distinction between "user guide" and "library user guide" when it comes to data frames. The only way you can use a data frame is if you are using it as library? I'

Re: [PR] Improve `DataFrame` Users Guide [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11324: URL: https://github.com/apache/datafusion/pull/11324#issuecomment-2220255395 Thanks @efredine and @comphead I have not forgotten about @efredine 's feedback in https://github.com/apache/datafusion/pull/11324#issuecomment-2214357564. I filed https://git

Re: [PR] feat(11344): track parquet encoding memory usage [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11345: URL: https://github.com/apache/datafusion/pull/11345#issuecomment-2220257359 FYI @devinjdangelo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] feat(11344): track parquet encoding memory usage [datafusion]

2024-07-10 Thread via GitHub
alamb commented on code in PR #11345: URL: https://github.com/apache/datafusion/pull/11345#discussion_r1672096842 ## datafusion/core/src/datasource/file_format/parquet.rs: ## @@ -895,16 +912,23 @@ async fn send_arrays_to_col_writers( fn spawn_rg_join_and_finalize_task( col

Re: [PR] feat(11344): track parquet encoding memory usage [datafusion]

2024-07-10 Thread via GitHub
alamb commented on code in PR #11345: URL: https://github.com/apache/datafusion/pull/11345#discussion_r1672097639 ## datafusion/core/src/datasource/file_format/parquet.rs: ## @@ -1015,15 +1050,21 @@ async fn concatenate_parallel_row_groups( while let Some(task) = serialize_

Re: [PR] feat: add UDF to_local_time() [datafusion]

2024-07-10 Thread via GitHub
alamb commented on code in PR #11347: URL: https://github.com/apache/datafusion/pull/11347#discussion_r1672115527 ## datafusion/functions/src/datetime/to_local_time.rs: ## @@ -0,0 +1,564 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor li

Re: [I] Review use of logical expressions in physical AggregateFunctionExpr [datafusion]

2024-07-10 Thread via GitHub
jayzhan211 commented on issue #11359: URL: https://github.com/apache/datafusion/issues/11359#issuecomment-2220320006 @ozankabak had asked for whether there is anyway to entirely getting rid of logical expressions in discord, so I think we can review about the challenge I had before.

Re: [I] Review use of logical expressions in physical AggregateFunctionExpr [datafusion]

2024-07-10 Thread via GitHub
ozankabak commented on issue #11359: URL: https://github.com/apache/datafusion/issues/11359#issuecomment-2220396909 > With this approach, function like limited_convert_logical_expr_to_physical_expr is no longer needed after this change. We have at least two such functions and it woul

Re: [PR] feat: continue visit children expr of short-circuit expr [datafusion]

2024-07-10 Thread via GitHub
zhuliquan closed pull request #11033: feat: continue visit children expr of short-circuit expr URL: https://github.com/apache/datafusion/pull/11033 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] Python wrapper classes for all user interfaces [datafusion-python]

2024-07-10 Thread via GitHub
timsaucer commented on code in PR #750: URL: https://github.com/apache/datafusion-python/pull/750#discussion_r1672230398 ## python/datafusion/expr.py: ## @@ -15,9 +15,256 @@ # specific language governing permissions and limitations # under the License. +from __future__ impor

Re: [PR] fix: Fix eq properties regression from #10434 [datafusion]

2024-07-10 Thread via GitHub
alamb merged PR #11363: URL: https://github.com/apache/datafusion/pull/11363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Regression in eliminating monotonic sorts [datafusion]

2024-07-10 Thread via GitHub
alamb closed issue #11362: Regression in eliminating monotonic sorts URL: https://github.com/apache/datafusion/issues/11362 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] fix: Fix eq properties regression from #10434 [datafusion]

2024-07-10 Thread via GitHub
alamb commented on code in PR #11363: URL: https://github.com/apache/datafusion/pull/11363#discussion_r1672247911 ## datafusion/physical-expr/src/equivalence/properties.rs: ## @@ -2454,30 +2478,49 @@ mod tests { ]; for case in cases { -let mut pro

[I] Remove the uses of `termtree` dependency [datafusion]

2024-07-10 Thread via GitHub
alamb opened a new issue, #11389: URL: https://github.com/apache/datafusion/issues/11389 ### Is your feature request related to a problem or challenge? Whatever we can do to make it faster/easier to test and compile datafusion is good, and this includes keeping the dependency chain sm

Re: [PR] Update termtree requirement from 0.4.1 to 0.5.0 [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11383: URL: https://github.com/apache/datafusion/pull/11383#issuecomment-2220504303 filed https://github.com/apache/datafusion/issues/11389 to remove this dependency eventually -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Add parser option enable_options_value_normalization [datafusion]

2024-07-10 Thread via GitHub
xinlifoobar commented on code in PR #11330: URL: https://github.com/apache/datafusion/pull/11330#discussion_r1672258635 ## datafusion/sql/tests/sql_integration.rs: ## @@ -149,6 +153,68 @@ fn parse_ident_normalization() { } } +#[test] +fn test_parse_options_value_normaliz

Re: [PR] Update termtree requirement from 0.4.1 to 0.5.0 [datafusion]

2024-07-10 Thread via GitHub
alamb merged PR #11383: URL: https://github.com/apache/datafusion/pull/11383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] feat: add UDF to_local_time() [datafusion]

2024-07-10 Thread via GitHub
appletreeisyellow commented on code in PR #11347: URL: https://github.com/apache/datafusion/pull/11347#discussion_r1672265164 ## datafusion/functions/src/datetime/to_local_time.rs: ## @@ -0,0 +1,564 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more co

Re: [PR] Python wrapper classes for all user interfaces [datafusion-python]

2024-07-10 Thread via GitHub
timsaucer commented on PR #750: URL: https://github.com/apache/datafusion-python/pull/750#issuecomment-2220507454 It's a well made point @datapythonista and maintainability is definitely a concern. I would argue that going with a `.pyi` file approach is **worse** for maintainability

Re: [PR] Update prost requirement from 0.12.0 to 0.13.1 [datafusion]

2024-07-10 Thread via GitHub
alamb closed pull request #11370: Update prost requirement from 0.12.0 to 0.13.1 URL: https://github.com/apache/datafusion/pull/11370 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Update prost requirement from 0.12.0 to 0.13.1 [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11370: URL: https://github.com/apache/datafusion/pull/11370#issuecomment-2220508943 Contained in https://github.com/apache/datafusion/pull/11372. Closing this PR to get it out of the review queue -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Update prost requirement from 0.12.0 to 0.13.1 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] commented on PR #11370: URL: https://github.com/apache/datafusion/pull/11370#issuecomment-2220509049 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version

Re: [PR] Update pbjson-build requirement from =0.6.2 to =0.7.0 [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11382: URL: https://github.com/apache/datafusion/pull/11382#issuecomment-2220509304 Contained in https://github.com/apache/datafusion/pull/11372. Closing this PR to get it out of the review queue -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Update pbjson-build requirement from =0.6.2 to =0.7.0 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] commented on PR #11382: URL: https://github.com/apache/datafusion/pull/11382#issuecomment-2220509409 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version

Re: [PR] Update prost-build requirement from =0.12.6 to =0.13.1 [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11379: URL: https://github.com/apache/datafusion/pull/11379#issuecomment-2220509664 Contained in https://github.com/apache/datafusion/pull/11372. Closing this PR to get it out of the review queue -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Update pbjson requirement from 0.6.0 to 0.7.0 [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11378: URL: https://github.com/apache/datafusion/pull/11378#issuecomment-2220511457 Contained in https://github.com/apache/datafusion/pull/11372. Closing this PR to get it out of the review queue -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Update pbjson requirement from 0.6.0 to 0.7.0 [datafusion]

2024-07-10 Thread via GitHub
alamb closed pull request #11378: Update pbjson requirement from 0.6.0 to 0.7.0 URL: https://github.com/apache/datafusion/pull/11378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Update substrait requirement from 0.36.0 to 0.37.3 [datafusion]

2024-07-10 Thread via GitHub
alamb closed pull request #11381: Update substrait requirement from 0.36.0 to 0.37.3 URL: https://github.com/apache/datafusion/pull/11381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Update pbjson-build requirement from =0.6.2 to =0.7.0 [datafusion]

2024-07-10 Thread via GitHub
alamb closed pull request #11382: Update pbjson-build requirement from =0.6.2 to =0.7.0 URL: https://github.com/apache/datafusion/pull/11382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Update substrait requirement from 0.36.0 to 0.37.3 [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11381: URL: https://github.com/apache/datafusion/pull/11381#issuecomment-2220509482 Contained in https://github.com/apache/datafusion/pull/11372. Closing this PR to get it out of the review queue -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Update prost-build requirement from =0.12.6 to =0.13.1 [datafusion]

2024-07-10 Thread via GitHub
alamb closed pull request #11379: Update prost-build requirement from =0.12.6 to =0.13.1 URL: https://github.com/apache/datafusion/pull/11379 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] Update substrait requirement from 0.36.0 to 0.37.3 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] commented on PR #11381: URL: https://github.com/apache/datafusion/pull/11381#issuecomment-2220509576 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version

Re: [PR] Update prost-build requirement from =0.12.6 to =0.13.1 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] commented on PR #11379: URL: https://github.com/apache/datafusion/pull/11379#issuecomment-2220509747 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version

Re: [PR] Support `NULL` literals in where clause [datafusion]

2024-07-10 Thread via GitHub
xinlifoobar commented on PR #11266: URL: https://github.com/apache/datafusion/pull/11266#issuecomment-2220510013 > Thanks @xinlifoobar -- this makes sense to me > > Another potential way to fix this issue would be to cast / coerce NULL literals in the WHERE clause to boolean >

Re: [PR] Update pbjson requirement from 0.6.0 to 0.7.0 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] commented on PR #11378: URL: https://github.com/apache/datafusion/pull/11378#issuecomment-2220511550 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version

Re: [PR] Update tonic requirement from 0.11 to 0.12 [datafusion]

2024-07-10 Thread via GitHub
dependabot[bot] commented on PR #11353: URL: https://github.com/apache/datafusion/pull/11353#issuecomment-2220513428 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version

Re: [I] Remove the uses of `termtree` dependency [datafusion]

2024-07-10 Thread via GitHub
avantgardnerio commented on issue #11389: URL: https://github.com/apache/datafusion/issues/11389#issuecomment-2220512705 I added it because it was a quick and easy way to debug & document tree-related tests. I think seeing it visually is helpful, but I have no strong opinion about how that

Re: [PR] Update tonic requirement from 0.11 to 0.12 [datafusion]

2024-07-10 Thread via GitHub
alamb closed pull request #11353: Update tonic requirement from 0.11 to 0.12 URL: https://github.com/apache/datafusion/pull/11353 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Update tonic requirement from 0.11 to 0.12 [datafusion]

2024-07-10 Thread via GitHub
alamb commented on PR #11353: URL: https://github.com/apache/datafusion/pull/11353#issuecomment-2220513324 Contained in https://github.com/apache/datafusion/pull/11372. Closing this PR to get it out of the review queue -- This is an automated message from the Apache Git Service. To respo

Re: [PR] feat: add UDF to_local_time() [datafusion]

2024-07-10 Thread via GitHub
appletreeisyellow commented on code in PR #11347: URL: https://github.com/apache/datafusion/pull/11347#discussion_r1672271610 ## datafusion/functions/src/datetime/to_local_time.rs: ## @@ -0,0 +1,564 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more co

Re: [PR] feat: add UDF to_local_time() [datafusion]

2024-07-10 Thread via GitHub
appletreeisyellow commented on code in PR #11347: URL: https://github.com/apache/datafusion/pull/11347#discussion_r1672273877 ## datafusion/functions/src/datetime/to_local_time.rs: ## @@ -0,0 +1,564 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more co

Re: [PR] feat: add UDF to_local_time() [datafusion]

2024-07-10 Thread via GitHub
appletreeisyellow commented on code in PR #11347: URL: https://github.com/apache/datafusion/pull/11347#discussion_r1672273877 ## datafusion/functions/src/datetime/to_local_time.rs: ## @@ -0,0 +1,564 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more co

Re: [PR] feat: add UDF to_local_time() [datafusion]

2024-07-10 Thread via GitHub
appletreeisyellow commented on PR #11347: URL: https://github.com/apache/datafusion/pull/11347#issuecomment-2220519822 @alamb Thank you for the careful review! > The other thing we should do is document this function in the function reference: [datafusion.apache.org/user-guide/sql/sc

Re: [PR] Python wrapper classes for all user interfaces [datafusion-python]

2024-07-10 Thread via GitHub
Michael-J-Ward commented on code in PR #750: URL: https://github.com/apache/datafusion-python/pull/750#discussion_r1671267473 ## src/common/data_type.rs: ## @@ -764,7 +764,7 @@ pub enum SqlType { #[allow(non_camel_case_types)] #[allow(clippy::upper_case_acronyms)] #[derive(De

Re: [PR] Python wrapper classes for all user interfaces [datafusion-python]

2024-07-10 Thread via GitHub
datapythonista commented on PR #750: URL: https://github.com/apache/datafusion-python/pull/750#issuecomment-2220543813 Thanks for the detailed information @timsaucer, and for all the work with this. I'm not an expert on type hints but it's not immediately clear to me what would ever

Re: [PR] fix: Disable tests that require SubqueryBroadcastExec [datafusion-comet]

2024-07-10 Thread via GitHub
viirya commented on code in PR #647: URL: https://github.com/apache/datafusion-comet/pull/647#discussion_r1672302253 ## dev/diffs/4.0.0-preview1.diff: ## @@ -442,7 +442,7 @@ index 2c24cc7d570..50a2ce86117 100644 - test("partition pruning in broadcast hash joins with aliases

Re: [PR] Python wrapper classes for all user interfaces [datafusion-python]

2024-07-10 Thread via GitHub
Michael-J-Ward commented on PR #750: URL: https://github.com/apache/datafusion-python/pull/750#issuecomment-2220558214 The wrappers also allow us to offer more pythonic interfaces when the `rust` API is "clunky", at least by `python` standards. An example of this is how @timsaucer cl

Re: [PR] feat: Use unified allocator for execution iterators [datafusion-comet]

2024-07-10 Thread via GitHub
viirya commented on code in PR #613: URL: https://github.com/apache/datafusion-comet/pull/613#discussion_r1672310479 ## spark/src/test/scala/org/apache/spark/sql/CometTPCDSQuerySuite.scala: ## @@ -141,9 +139,45 @@ class CometTPCDSQuerySuite "q98", "q99") -

Re: [PR] feat: Use unified allocator for execution iterators [datafusion-comet]

2024-07-10 Thread via GitHub
viirya commented on code in PR #613: URL: https://github.com/apache/datafusion-comet/pull/613#discussion_r1672311644 ## spark/src/test/scala/org/apache/spark/sql/CometTPCDSQuerySuite.scala: ## @@ -141,9 +139,45 @@ class CometTPCDSQuerySuite "q98", "q99") -

Re: [PR] Impl a general get results from stats [datafusion]

2024-07-10 Thread via GitHub
Rachelint commented on PR #11261: URL: https://github.com/apache/datafusion/pull/11261#issuecomment-2220576084 > > Ok, been a bit busy the past couple of days, continue to read the related codes and think a relatively good way to solve this today... One simple way I could think to solve it

Re: [PR] Python wrapper classes for all user interfaces [datafusion-python]

2024-07-10 Thread via GitHub
timsaucer commented on PR #750: URL: https://github.com/apache/datafusion-python/pull/750#issuecomment-2220606862 @datapythonista I think your PR is great! It definitely cleans up a problem, but it's not the same one as what I was trying to say. I know there are many users who look a

[I] Some memory reservations of GroupedHashAggregateStream seem to be mis-tagged as spillable while they do not allow spilling [datafusion]

2024-07-10 Thread via GitHub
Ablu opened a new issue, #11390: URL: https://github.com/apache/datafusion/issues/11390 ### Describe the bug When using a grouping (I tested with `distinct_on`) in combination with a `FairSpillPool` not all memory seems to be tagged correctly. While the grouping itself does gracefull

[PR] Minor: return "not supported" for `COUNT DISTINCT` with multiple arguments [datafusion]

2024-07-10 Thread via GitHub
jonahgao opened a new pull request, #11391: URL: https://github.com/apache/datafusion/pull/11391 ## Which issue does this PR close? Closes #11303. ## Rationale for this change `COUNT` with multiple arguments is a feature of [Spark](https://spark.apache.org/docs/lates

Re: [PR] Implement ScalarFunction `MAKE_MAP` and `MAP` [datafusion]

2024-07-10 Thread via GitHub
goldmedal commented on PR #11361: URL: https://github.com/apache/datafusion/pull/11361#issuecomment-2220709681 > For `make_map(k1, v1, k2, v2...)`, ideally we can arrange the order of args and call `map([k1, k2..], [v1, v2])`. I think we can arrange it in ExprPlanner, so we can have only on

Re: [I] Extract registering default features from `SessionState` and into its own function [datafusion]

2024-07-10 Thread via GitHub
Omega359 commented on issue #11320: URL: https://github.com/apache/datafusion/issues/11320#issuecomment-2220715348 Would we want to then deprecate the SessionState::new_with_config_rt and SessionState::new_with_config_rt_and_catalog_list as well (basically all the SessionState::new_* functi

Re: [I] Investigate TPC-H queries that are slower when Comet is enabled [datafusion-comet]

2024-07-10 Thread via GitHub
andygrove closed issue #530: Investigate TPC-H queries that are slower when Comet is enabled URL: https://github.com/apache/datafusion-comet/issues/530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [I] Investigate TPC-H queries that are slower when Comet is enabled [datafusion-comet]

2024-07-10 Thread via GitHub
andygrove commented on issue #530: URL: https://github.com/apache/datafusion-comet/issues/530#issuecomment-2220728731 I think we can close this issue now that we have specific issues filed for q14, q16, and q19 -- This is an automated message from the Apache Git Service. To respond to th

[PR] Add customizable equality and hash functions to UDFs [datafusion]

2024-07-10 Thread via GitHub
joroKr21 opened a new pull request, #11392: URL: https://github.com/apache/datafusion/pull/11392 ## Which issue does this PR close? Closes #127. ## Rationale for this change After #9436 it's possible to write and use all kinds of custom UDF functions e.g. paramet

Re: [PR] Add customizable equality and hash functions to UDFs [datafusion]

2024-07-10 Thread via GitHub
joroKr21 commented on code in PR #11392: URL: https://github.com/apache/datafusion/pull/11392#discussion_r1672449433 ## datafusion/expr/src/udaf.rs: ## @@ -72,20 +76,19 @@ pub struct AggregateUDF { impl PartialEq for AggregateUDF { fn eq(&self, other: &Self) -> bool { -

Re: [I] Plan Comet 0.1.0 Release [datafusion-comet]

2024-07-10 Thread via GitHub
andygrove commented on issue #369: URL: https://github.com/apache/datafusion-comet/issues/369#issuecomment-2220742737 I think we are getting close to being able to release 0.1.0 now that we are using an official DataFusion release again (or will be in a few days when DF 40 is released to c

Re: [PR] feat: Use unified allocator for execution iterators [datafusion-comet]

2024-07-10 Thread via GitHub
andygrove commented on code in PR #613: URL: https://github.com/apache/datafusion-comet/pull/613#discussion_r1672471830 ## spark/src/test/scala/org/apache/spark/sql/CometTPCDSQuerySuite.scala: ## @@ -112,7 +108,9 @@ class CometTPCDSQuerySuite "q69", "q70",

Re: [PR] feat: Use unified allocator for execution iterators [datafusion-comet]

2024-07-10 Thread via GitHub
andygrove commented on code in PR #613: URL: https://github.com/apache/datafusion-comet/pull/613#discussion_r1672476021 ## spark/src/test/scala/org/apache/spark/sql/CometTPCDSQuerySuite.scala: ## @@ -157,9 +191,11 @@ class CometTPCDSQuerySuite conf.set(CometConf.COMET_EXEC_

Re: [PR] feat: Use unified allocator for execution iterators [datafusion-comet]

2024-07-10 Thread via GitHub
andygrove commented on code in PR #613: URL: https://github.com/apache/datafusion-comet/pull/613#discussion_r1672479688 ## spark/src/test/scala/org/apache/spark/sql/CometTPCDSQuerySuite.scala: ## @@ -112,7 +108,9 @@ class CometTPCDSQuerySuite "q69", "q70",

Re: [PR] Introduce `resources_err!` error macro [datafusion]

2024-07-10 Thread via GitHub
comphead commented on PR #11374: URL: https://github.com/apache/datafusion/pull/11374#issuecomment-2220768713 Thanks folks for the review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [PR] Introduce `resources_err!` error macro [datafusion]

2024-07-10 Thread via GitHub
comphead merged PR #11374: URL: https://github.com/apache/datafusion/pull/11374 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] feat: Use unified allocator for execution iterators [datafusion-comet]

2024-07-10 Thread via GitHub
andygrove commented on code in PR #613: URL: https://github.com/apache/datafusion-comet/pull/613#discussion_r1672490835 ## spark/src/test/scala/org/apache/spark/sql/CometTPCDSQuerySuite.scala: ## @@ -112,7 +108,9 @@ class CometTPCDSQuerySuite "q69", "q70",

Re: [PR] Minor: remove clones and unnecessary Arcs in `from_substrait_rex` [datafusion]

2024-07-10 Thread via GitHub
comphead commented on code in PR #11337: URL: https://github.com/apache/datafusion/pull/11337#discussion_r1672490967 ## datafusion/substrait/src/logical_plan/consumer.rs: ## @@ -411,11 +411,11 @@ pub async fn from_substrait_rel( from_substrait_rex(ctx, e

  1   2   3   >