Re: [PR] Int64 as default type for make_array function empty or null case [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 commented on code in PR #10790: URL: https://github.com/apache/datafusion/pull/10790#discussion_r1988207529 ## datafusion/functions-array/src/make_array.rs: ## @@ -131,6 +131,11 @@ impl ScalarUDFImpl for MakeArray { } } +// Empty array is a special case that i

Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]

2025-03-10 Thread via GitHub
comphead commented on issue #13814: URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2711668760 might be compiler related? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] feat: add `register_metadata` function for `GroupsAccumulator` [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 commented on code in PR #15022: URL: https://github.com/apache/datafusion/pull/15022#discussion_r1988229738 ## datafusion/expr-common/src/groups_accumulator.rs: ## @@ -251,3 +261,18 @@ pub trait GroupsAccumulator: Send { /// compute, not `O(num_groups)` fn s

Re: [PR] Find a way to communicate the ordering of a file back with the existi… [datafusion]

2025-03-10 Thread via GitHub
github-actions[bot] closed pull request #13933: Find a way to communicate the ordering of a file back with the existi… URL: https://github.com/apache/datafusion/pull/13933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Add tests for simplification and coercion of `SessionContext::create_physical_expr` [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 commented on PR #15034: URL: https://github.com/apache/datafusion/pull/15034#issuecomment-2712168720 Thanks @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] preserve sql formatting through a parse + display roundtrip (partial implementation) [datafusion-sqlparser-rs]

2025-03-10 Thread via GitHub
github-actions[bot] closed pull request #1636: preserve sql formatting through a parse + display roundtrip (partial implementation) URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1636 -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] Add tests for simplification and coercion of `SessionContext::create_physical_expr` [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 merged PR #15034: URL: https://github.com/apache/datafusion/pull/15034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] shell script to collect Benchmarks [datafusion]

2025-03-10 Thread via GitHub
logan-keede commented on PR #15144: URL: https://github.com/apache/datafusion/pull/15144#issuecomment-2712783895 One problem that I sometimes encounter is that cargo decides to use `arrow-arith v53.4.0` for particular releases which ends up giving compilation error. I’m not sure why this

Re: [I] Change mapping of SQL `VARCHAR` from `Utf8` to `Utf8View` [datafusion]

2025-03-10 Thread via GitHub
zhuqi-lucas commented on issue #15096: URL: https://github.com/apache/datafusion/issues/15096#issuecomment-2712795395 Create the ticket for avro: - [ ] Support Utf8View for avro [#7262](https://github.com/apache/arrow-rs/issues/7262) -- This is an automated message from the

Re: [PR] Renaming Internal Structs [datafusion-python]

2025-03-10 Thread via GitHub
Spaarsh commented on code in PR #1059: URL: https://github.com/apache/datafusion-python/pull/1059#discussion_r1988468214 ## src/expr.rs: ## @@ -100,22 +100,37 @@ pub mod window; use sort_expr::{to_sort_expressions, PySortExpr}; +// Define the new RawExpr struct and impleme

Re: [PR] add support for `with` clauses (CTEs) in `delete` statements [datafusion-sqlparser-rs]

2025-03-10 Thread via GitHub
lovasoa commented on code in PR #1764: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1764#discussion_r1988500551 ## src/parser/mod.rs: ## @@ -10202,19 +10209,25 @@ impl<'a> Parser<'a> { } } +/// Parse a `WITH` clause, i.e. a `WITH` keyword foll

Re: [PR] add support for `with` clauses (CTEs) in `delete` statements [datafusion-sqlparser-rs]

2025-03-10 Thread via GitHub
lovasoa commented on code in PR #1764: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1764#discussion_r1988500551 ## src/parser/mod.rs: ## @@ -10202,19 +10209,25 @@ impl<'a> Parser<'a> { } } +/// Parse a `WITH` clause, i.e. a `WITH` keyword foll

Re: [I] `flatten` should be single-step, not recursive [datafusion]

2025-03-10 Thread via GitHub
delamarch3 commented on issue #13757: URL: https://github.com/apache/datafusion/issues/13757#issuecomment-2711632124 Hi @logan-keede, is it ok if I pick this up? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-10 Thread via GitHub
alan910127 commented on code in PR #15110: URL: https://github.com/apache/datafusion/pull/15110#discussion_r1988192389 ## datafusion/optimizer/src/simplify_expressions/unwrap_cast.rs: ## @@ -177,6 +192,45 @@ pub(super) fn is_cast_expr_and_support_unwrap_cast_in_comparison_for_i

Re: [I] Implement tree explain for `PlaceholderRowExec` [datafusion]

2025-03-10 Thread via GitHub
Standing-Man commented on issue #15138: URL: https://github.com/apache/datafusion/issues/15138#issuecomment-2712286093 @irenjj -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]

2025-03-10 Thread via GitHub
tustvold commented on issue #13814: URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2711684269 IIRC that relates to type checking expressions, and therefore this would suggest the compiler is spending a lot of time resolving generics. At least historically non-boxed asyn

Re: [I] Change in behavior for deep structure columns with the latest sql parser upgrade [datafusion]

2025-03-10 Thread via GitHub
chenkovsky commented on issue #15118: URL: https://github.com/apache/datafusion/issues/15118#issuecomment-2710431152 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] chore: Stop disabling readside padding in TPC stability suite [datafusion-comet]

2025-03-10 Thread via GitHub
andygrove opened a new pull request, #1491: URL: https://github.com/apache/datafusion-comet/pull/1491 ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/1472 ## Rationale for this change In the TPC-DS stability suit

Re: [I] March 2025 ASF Board Report (March 12) [datafusion]

2025-03-10 Thread via GitHub
alamb commented on issue #13713: URL: https://github.com/apache/datafusion/issues/13713#issuecomment-2710480146 I have incorporated @robtandy and @kevinjqliu 's comments. Here is the current draft ``` ## Description: The mission of Apache DataFusion is the creation and maintenan

Re: [PR] chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) [datafusion]

2025-03-10 Thread via GitHub
Blizzara commented on code in PR #15123: URL: https://github.com/apache/datafusion/pull/15123#discussion_r1987288818 ## datafusion/core/tests/physical_optimizer/projection_pushdown.rs: ## @@ -89,6 +92,10 @@ impl ScalarUDFImpl for DummyUDF { fn return_type(&self, _arg_types:

Re: [I] TPC-H benchmark does not run q15 [datafusion-ray]

2025-03-10 Thread via GitHub
zhangx commented on issue #81: URL: https://github.com/apache/datafusion-ray/issues/81#issuecomment-2710625487 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Per file filter evaluation [datafusion]

2025-03-10 Thread via GitHub
adriangb commented on PR #15057: URL: https://github.com/apache/datafusion/pull/15057#issuecomment-2712042344 The example is now working and even does stats pruning of shredded columns 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] fix: enable full decimal to decimal support [datafusion-comet]

2025-03-10 Thread via GitHub
andygrove commented on code in PR #1385: URL: https://github.com/apache/datafusion-comet/pull/1385#discussion_r1987807688 ## spark/src/test/scala/org/apache/comet/CometCastSuite.scala: ## @@ -1210,27 +1213,36 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanH

Re: [I] Change naming of rust exposed structs to ease debugging [datafusion-python]

2025-03-10 Thread via GitHub
kylebarron commented on issue #853: URL: https://github.com/apache/datafusion-python/issues/853#issuecomment-2711787803 fwiw I always define my classes exported to Python with a Py prefix on the rust side, and then rename the actual export from within the pyclass macro -- This is an auto

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-10 Thread via GitHub
alan910127 commented on code in PR #15110: URL: https://github.com/apache/datafusion/pull/15110#discussion_r1988192389 ## datafusion/optimizer/src/simplify_expressions/unwrap_cast.rs: ## @@ -177,6 +192,45 @@ pub(super) fn is_cast_expr_and_support_unwrap_cast_in_comparison_for_i

Re: [PR] fix: unparse for subqueryalias [datafusion]

2025-03-10 Thread via GitHub
alamb commented on code in PR #15068: URL: https://github.com/apache/datafusion/pull/15068#discussion_r1987794389 ## datafusion/core/tests/sql/select.rs: ## @@ -350,3 +351,48 @@ async fn test_version_function() { assert_eq!(version.value(0), expected_version); } + +#[tok

[PR] Implement tree rendering for `SortPreservingMergeExec` [datafusion]

2025-03-10 Thread via GitHub
Standing-Man opened a new pull request, #15140: URL: https://github.com/apache/datafusion/pull/15140 ## Which issue does this PR close? - Closes #15139 and Part of #14914. ## Rationale for this change ## What changes are included in this PR? Imp

Re: [I] Implement tree explain for `CoalesceBatchesExec` [datafusion]

2025-03-10 Thread via GitHub
Standing-Man commented on issue #15141: URL: https://github.com/apache/datafusion/issues/15141#issuecomment-2712465988 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] Make it easier to run TPCH queries with datafusion-cli [datafusion]

2025-03-10 Thread via GitHub
clflushopt commented on issue #14608: URL: https://github.com/apache/datafusion/issues/14608#issuecomment-2712464986 For anyone following this issue I have a full port here https://github.com/clflushopt/tpchgen-rs and I am working on completing a first release (I have issues to track that m

Re: [I] Implement tree explain for `CoalesceBatchesExec` [datafusion]

2025-03-10 Thread via GitHub
Standing-Man commented on issue #15141: URL: https://github.com/apache/datafusion/issues/15141#issuecomment-2712465888 @irenjj -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[I] Implement tree explain for `CoalesceBatchesExec` [datafusion]

2025-03-10 Thread via GitHub
Standing-Man opened a new issue, #15141: URL: https://github.com/apache/datafusion/issues/15141 ### Is your feature request related to a problem or challenge? Part of #14914 ### Describe the solution you'd like _No response_ ### Describe alternatives you've conside

Re: [PR] Implement tree explain for `RepartitionExec` and `WorkTableExec` [datafusion]

2025-03-10 Thread via GitHub
2010YOUY01 commented on code in PR #15137: URL: https://github.com/apache/datafusion/pull/15137#discussion_r1988371514 ## datafusion/sqllogictest/test_files/explain_tree.slt: ## @@ -179,19 +185,31 @@ physical_plan 06)└─┬─┘ 07)┌─┴

Re: [I] Rewrite `datafusion-sqlancer` in Rust [datafusion]

2025-03-10 Thread via GitHub
2010YOUY01 commented on issue #14535: URL: https://github.com/apache/datafusion/issues/14535#issuecomment-2712646798 > Hello, I am interested in applying to work on this project for GSoC. After reading through [#11030](https://github.com/apache/datafusion/issues/11030) , it looks like the t

Re: [I] beautify default column names [datafusion]

2025-03-10 Thread via GitHub
NevroHelios commented on issue #2027: URL: https://github.com/apache/datafusion/issues/2027#issuecomment-2712675671 Since it is still open can I work on it and submit a pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Minor: Fix invalid query in test [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 commented on PR #15131: URL: https://github.com/apache/datafusion/pull/15131#issuecomment-2712172487 Thanks @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] Make it easier to run TPCH queries with datafusion-cli [datafusion]

2025-03-10 Thread via GitHub
matthewmturner commented on issue #14608: URL: https://github.com/apache/datafusion/issues/14608#issuecomment-2712539973 @clflushopt this is _awesome_. Once you release I will likely add this to [dft](https://github.com/datafusion-contrib/datafusion-dft). -- This is an automated message

Re: [PR] Renaming Internal Structs [datafusion-python]

2025-03-10 Thread via GitHub
Spaarsh commented on code in PR #1059: URL: https://github.com/apache/datafusion-python/pull/1059#discussion_r1988360570 ## src/expr.rs: ## @@ -100,22 +100,37 @@ pub mod window; use sort_expr::{to_sort_expressions, PySortExpr}; +// Define the new RawExpr struct and impleme

Re: [PR] Implement tree explain for AggregateExec [datafusion]

2025-03-10 Thread via GitHub
Weijun-H commented on code in PR #15103: URL: https://github.com/apache/datafusion/pull/15103#discussion_r1988354902 ## datafusion/physical-plan/src/aggregates/mod.rs: ## @@ -809,8 +809,60 @@ impl DisplayAs for AggregateExec { } } Displ

[PR] Minor: fix extend sqllogical consistent with main test [datafusion]

2025-03-10 Thread via GitHub
zhuqi-lucas opened a new pull request, #15145: URL: https://github.com/apache/datafusion/pull/15145 ## Which issue does this PR close? fix extend sqllogical consistent with main test ## Rationale for this change Similar to: https://github.com/apache/datafusion/issu

Re: [PR] Implement tree rendering for `SortPreservingMergeExec` [datafusion]

2025-03-10 Thread via GitHub
2010YOUY01 commented on PR #15140: URL: https://github.com/apache/datafusion/pull/15140#issuecomment-2712575632 Thank you for making this happen. I have a suggestion: I think the only field needed inside `SPM` is the sort keys, how about making it consistent with those in `SortExec`?

Re: [PR] feat: Add `datafusion-spark` crate [datafusion]

2025-03-10 Thread via GitHub
andygrove closed pull request #14392: feat: Add `datafusion-spark` crate URL: https://github.com/apache/datafusion/pull/14392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] chore: add an "expr_planners" method to SessionState [datafusion]

2025-03-10 Thread via GitHub
niebayes commented on PR #15119: URL: https://github.com/apache/datafusion/pull/15119#issuecomment-2712347513 @alamb I wonder if we can remove the `register_expr_planners` and `expr_planners` from the `FunctionRegistry` trait. I have checked the codebase and they're only used by a test. And

Re: [I] Implement tree explain for `PlaceholderRowExec` [datafusion]

2025-03-10 Thread via GitHub
pranavJibhakate commented on issue #15138: URL: https://github.com/apache/datafusion/issues/15138#issuecomment-2712342026 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-10 Thread via GitHub
alan910127 commented on code in PR #15110: URL: https://github.com/apache/datafusion/pull/15110#discussion_r1988197176 ## datafusion/optimizer/src/simplify_expressions/unwrap_cast.rs: ## @@ -177,6 +192,45 @@ pub(super) fn is_cast_expr_and_support_unwrap_cast_in_comparison_for_i

[PR] fixed PushDownFilter bug [15047] [datafusion]

2025-03-10 Thread via GitHub
Jiashu-Hu opened a new pull request, #15142: URL: https://github.com/apache/datafusion/pull/15142 …revent this specific situation ## Which issue does this PR close? - Closes #[15047](https://github.com/apache/datafusion/issues/15047). ## Rationale for this change

Re: [PR] Minor: Fix invalid query in test [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 merged PR #15131: URL: https://github.com/apache/datafusion/pull/15131 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

[I] Invalid schema for unions in ViewTable [datafusion]

2025-03-10 Thread via GitHub
Friede80 opened a new issue, #15134: URL: https://github.com/apache/datafusion/issues/15134 ### Describe the bug When a ViewTable is created, the plan is run through the `Analyzer` with the `ExpandWildcardRule` and `TypeCoercion` rules. When this ViewTable is later inlined, it is run

Re: [PR] Add all missing table options to be handled in any order [datafusion-sqlparser-rs]

2025-03-10 Thread via GitHub
iffyio commented on code in PR #1747: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1747#discussion_r1988442245 ## src/ast/dml.rs: ## @@ -138,6 +143,30 @@ pub struct CreateTable { pub engine: Option, pub comment: Option, pub auto_increment_offset: O

Re: [I] `ScalarValue::to_array` panics when getting statistics for List column [datafusion]

2025-03-10 Thread via GitHub
trueleo closed issue #5706: `ScalarValue::to_array` panics when getting statistics for List column URL: https://github.com/apache/datafusion/issues/5706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] add support for `with` clauses (CTEs) in `delete` statements [datafusion-sqlparser-rs]

2025-03-10 Thread via GitHub
iffyio commented on code in PR #1764: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1764#discussion_r1988453837 ## src/parser/mod.rs: ## @@ -10202,19 +10209,25 @@ impl<'a> Parser<'a> { } } +/// Parse a `WITH` clause, i.e. a `WITH` keyword follo

[PR] shell script to collect Benchmarks [datafusion]

2025-03-10 Thread via GitHub
logan-keede opened a new pull request, #15144: URL: https://github.com/apache/datafusion/pull/15144 ## Which issue does this PR close? - Part of #5504 ## Rationale for this change > Here is a suggestion on how to proceed with this project: > 1. Create the converte

Re: [PR] Fix broken `serde` feature [datafusion]

2025-03-10 Thread via GitHub
vadimpiven commented on code in PR #15124: URL: https://github.com/apache/datafusion/pull/15124#discussion_r1988170629 ## datafusion/core/Cargo.toml: ## @@ -79,7 +79,7 @@ recursive_protection = [ "datafusion-physical-optimizer/recursive_protection", "datafusion-sql/rec

Re: [PR] fix: unparse for subqueryalias [datafusion]

2025-03-10 Thread via GitHub
goldmedal commented on code in PR #15068: URL: https://github.com/apache/datafusion/pull/15068#discussion_r1987731975 ## datafusion/core/tests/sql/select.rs: ## @@ -350,3 +351,48 @@ async fn test_version_function() { assert_eq!(version.value(0), expected_version); } + +#

Re: [PR] feat: add `register_metadata` function for `GroupsAccumulator` [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 commented on code in PR #15022: URL: https://github.com/apache/datafusion/pull/15022#discussion_r1988229738 ## datafusion/expr-common/src/groups_accumulator.rs: ## @@ -251,3 +261,18 @@ pub trait GroupsAccumulator: Send { /// compute, not `O(num_groups)` fn s

Re: [PR] Expand wildcard to actual expressions in `prepare_select_exprs` [datafusion]

2025-03-10 Thread via GitHub
alamb commented on code in PR #15090: URL: https://github.com/apache/datafusion/pull/15090#discussion_r1987814170 ## datafusion/sqllogictest/test_files/order.slt: ## @@ -985,13 +985,20 @@ drop table ambiguity_test; statement ok create table t(a0 int, a int, b int, c int) as va

Re: [PR] feat: add `register_metadata` function for `GroupsAccumulator` [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 commented on code in PR #15022: URL: https://github.com/apache/datafusion/pull/15022#discussion_r1988229738 ## datafusion/expr-common/src/groups_accumulator.rs: ## @@ -251,3 +261,18 @@ pub trait GroupsAccumulator: Send { /// compute, not `O(num_groups)` fn s

[I] Implement tree explain for `SortPreservingMergeExec` [datafusion]

2025-03-10 Thread via GitHub
Standing-Man opened a new issue, #15139: URL: https://github.com/apache/datafusion/issues/15139 ### Is your feature request related to a problem or challenge? Part of #14914 ### Describe the solution you'd like _No response_ ### Describe alternatives you've conside

[PR] Implement tree explain for `RepartitionExec` and `WorkTableExec` [datafusion]

2025-03-10 Thread via GitHub
Standing-Man opened a new pull request, #15137: URL: https://github.com/apache/datafusion/pull/15137 ## Which issue does this PR close? - Closes #15097 and part of #14914. ## Rationale for this change ## What changes are included in this PR?

Re: [PR] support run mutiple queries in TPC-H benchmark [datafusion-ray]

2025-03-10 Thread via GitHub
zhangx commented on PR #82: URL: https://github.com/apache/datafusion-ray/pull/82#issuecomment-2712446883 > @zhangx thank you for submitting this! > > I submitted one at the same time with a similar fix, combined with a few other small changes that came up during benchmarking.

[PR] WIP: test parquet modular encryption support [datafusion]

2025-03-10 Thread via GitHub
alamb opened a new pull request, #15133: URL: https://github.com/apache/datafusion/pull/15133 ## Which issue does this PR close? - related to https://github.com/apache/arrow-rs/pull/6637 ## Rationale for this change I am using this PR to help verify that the chang

Re: [I] Implement tree explain for `SortPreservingMergeExec` [datafusion]

2025-03-10 Thread via GitHub
Standing-Man commented on issue #15139: URL: https://github.com/apache/datafusion/issues/15139#issuecomment-2712433934 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat/improve ruff test coverage [datafusion-python]

2025-03-10 Thread via GitHub
CrystalZhou0529 commented on code in PR #1055: URL: https://github.com/apache/datafusion-python/pull/1055#discussion_r1988370240 ## python/datafusion/udf.py: ## @@ -111,7 +111,27 @@ def __call__(self, *args: Expr) -> Expr: args_raw = [arg.expr for arg in args]

[PR] feat: Attach `Diagnostic` to more than one column errors in scalar_subquery and in_subquery [datafusion]

2025-03-10 Thread via GitHub
changsun20 opened a new pull request, #15143: URL: https://github.com/apache/datafusion/pull/15143 ## Which issue does this PR close? - Closes #14438. ## Rationale for this change This pull request enhances diagnostic information by attaching the `Diagnos

Re: [PR] Add all missing table options to be handled in any order [datafusion-sqlparser-rs]

2025-03-10 Thread via GitHub
tomershaniii commented on code in PR #1747: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1747#discussion_r1986746865 ## src/ast/dml.rs: ## @@ -138,6 +143,30 @@ pub struct CreateTable { pub engine: Option, pub comment: Option, pub auto_increment_off

Re: [I] Regression: count(*) does not work unless using default expression planners [datafusion]

2025-03-10 Thread via GitHub
jayzhan211 commented on issue #15114: URL: https://github.com/apache/datafusion/issues/15114#issuecomment-2709816452 `expr_planners` can be a method of `trait Session` similar to `scalar_functions`, `aggregate_functions`. -- This is an automated message from the Apache Git Service. To res

[PR] chore(deps): bump clap from 4.5.30 to 4.5.31 [datafusion]

2025-03-10 Thread via GitHub
dependabot[bot] opened a new pull request, #15115: URL: https://github.com/apache/datafusion/pull/15115 Bumps [clap](https://github.com/clap-rs/clap) from 4.5.30 to 4.5.31. Release notes Sourced from https://github.com/clap-rs/clap/releases";>clap's releases. v4.5.31 [4.5.

Re: [PR] Add DataFrame fill_nan/fill_null [datafusion-python]

2025-03-10 Thread via GitHub
kosiew commented on PR #1019: URL: https://github.com/apache/datafusion-python/pull/1019#issuecomment-2709822987 thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Make it easier to run TPCH queries with datafusion-cli [datafusion]

2025-03-10 Thread via GitHub
alamb commented on issue #14608: URL: https://github.com/apache/datafusion/issues/14608#issuecomment-2710281339 > Hey [@alamb](https://github.com/alamb) as of today I have a fully working implementation that matches Apache Trino and OLTPBenchmark's, I found the issue I mentionned in the mes

Re: [PR] chore: Add `native_iceberg_compat` CI checks [datafusion-comet]

2025-03-10 Thread via GitHub
andygrove merged PR #1487: URL: https://github.com/apache/datafusion-comet/pull/1487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

Re: [I] Attach `Diagnostic` to "more than one column in subquery" error [datafusion]

2025-03-10 Thread via GitHub
eliaperantoni commented on issue #14438: URL: https://github.com/apache/datafusion/issues/14438#issuecomment-2710537490 Thank you so much @changsun20 for the incredible work, that's a very nice and comprehensive plan! I think it looks awesome and I definitely like it. Just one minor t

[PR] chore: remove ScalarUDFImpl::return_type_from_exprs [datafusion]

2025-03-10 Thread via GitHub
Blizzara opened a new pull request, #15130: URL: https://github.com/apache/datafusion/pull/15130 use `return_type_from_args` instead ## Which issue does this PR close? - Closes #14729 ## Rationale for this change Implementing `return_type_from_exprs` is almost

Re: [PR] Per file filter evaluation [datafusion]

2025-03-10 Thread via GitHub
adriangb commented on code in PR #15057: URL: https://github.com/apache/datafusion/pull/15057#discussion_r1988140038 ## datafusion/datasource-parquet/src/opener.rs: ## @@ -111,18 +109,18 @@ impl FileOpener for ParquetOpener { .schema_adapter_factory .cr

Re: [PR] perf: unwrap cast for comparing ints =/!= strings [datafusion]

2025-03-10 Thread via GitHub
alamb commented on code in PR #15110: URL: https://github.com/apache/datafusion/pull/15110#discussion_r1987055078 ## datafusion/optimizer/src/simplify_expressions/unwrap_cast.rs: ## @@ -468,6 +510,10 @@ mod tests { // the 999 is not within the range of MAX(int32

Re: [PR] Int64 as default type for make_array function empty or null case [datafusion]

2025-03-10 Thread via GitHub
joroKr21 commented on code in PR #10790: URL: https://github.com/apache/datafusion/pull/10790#discussion_r1987078389 ## datafusion/functions-array/src/make_array.rs: ## @@ -131,6 +131,11 @@ impl ScalarUDFImpl for MakeArray { } } +// Empty array is a special case that is

Re: [I] Regression in supported plans for Spark 3.5.2+ [datafusion-comet]

2025-03-10 Thread via GitHub
andygrove closed issue #1458: Regression in supported plans for Spark 3.5.2+ URL: https://github.com/apache/datafusion-comet/issues/1458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [I] Regression in supported plans for Spark 3.5.2+ [datafusion-comet]

2025-03-10 Thread via GitHub
andygrove commented on issue #1458: URL: https://github.com/apache/datafusion-comet/issues/1458#issuecomment-2710378853 Fixed by https://github.com/apache/datafusion-comet/pull/1470 and https://github.com/apache/datafusion-comet/pull/1471 -- This is an automated message from the Apache G

Re: [PR] chore(deps): bump substrait from 0.53.2 to 0.54.0 [datafusion]

2025-03-10 Thread via GitHub
alamb commented on PR #15043: URL: https://github.com/apache/datafusion/pull/15043#issuecomment-2710162871 FYI @Blizzara and @vbarua -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] fix: Support datatype cast for insert api same as insert into sql [datafusion]

2025-03-10 Thread via GitHub
alamb merged PR #15091: URL: https://github.com/apache/datafusion/pull/15091 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] implement tree rendering for StreamingTableExec [datafusion]

2025-03-10 Thread via GitHub
alamb merged PR #15085: URL: https://github.com/apache/datafusion/pull/15085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] implement tree explain for GlobalLimitExec [datafusion]

2025-03-10 Thread via GitHub
alamb commented on PR #15100: URL: https://github.com/apache/datafusion/pull/15100#issuecomment-2710170906 I also ran the tests locally and made sure they still passed after a merge from main -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] chore(deps): bump semver from 1.0.25 to 1.0.26 [datafusion]

2025-03-10 Thread via GitHub
alamb merged PR #15116: URL: https://github.com/apache/datafusion/pull/15116 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] implement tree rendering for StreamingTableExec [datafusion]

2025-03-10 Thread via GitHub
alamb commented on PR #15085: URL: https://github.com/apache/datafusion/pull/15085#issuecomment-2710152588 I also verified the tests pass after merging to main locally. Thanks again @Standing-Man -- This is an automated message from the Apache Git Service. To respond to the message, plea

Re: [I] Weekly Plan (Andrew Lamb) March 10, 2025 [datafusion]

2025-03-10 Thread via GitHub
alamb commented on issue #15121: URL: https://github.com/apache/datafusion/issues/15121#issuecomment-2710258428 Review Queue DataFusion: Bugs/UX/Performance - [ ] https://github.com/apache/datafusion/pull/15033 - [ ] https://github.com/apache/datafusion/pull/14331 - [ ] https:

Re: [PR] Renaming Internal Structs [datafusion-python]

2025-03-10 Thread via GitHub
timsaucer commented on code in PR #1059: URL: https://github.com/apache/datafusion-python/pull/1059#discussion_r1987943713 ## src/expr.rs: ## @@ -100,22 +100,37 @@ pub mod window; use sort_expr::{to_sort_expressions, PySortExpr}; +// Define the new RawExpr struct and imple

Re: [PR] fix: enable full decimal to decimal support [datafusion-comet]

2025-03-10 Thread via GitHub
himadripal commented on code in PR #1385: URL: https://github.com/apache/datafusion-comet/pull/1385#discussion_r1987983268 ## spark/src/test/scala/org/apache/comet/CometCastSuite.scala: ## @@ -1210,27 +1213,36 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlan

Re: [PR] Minor: Fix invalid query in test [datafusion]

2025-03-10 Thread via GitHub
alamb commented on code in PR #15131: URL: https://github.com/apache/datafusion/pull/15131#discussion_r1987868432 ## datafusion/sqllogictest/test_files/order.slt: ## @@ -986,17 +986,26 @@ statement ok create table t(a0 int, a int, b int, c int) as values (1, 2, 3, 4), (5, 6, 7,

Re: [I] Implement `tree` explain for `NestedLoopJoinExec`, `CrossJoinExec`, `SortMergeJoinExec` and `SymmetricHashJoinExec` [datafusion]

2025-03-10 Thread via GitHub
2010YOUY01 closed issue #15080: Implement `tree` explain for `NestedLoopJoinExec`, `CrossJoinExec`, `SortMergeJoinExec` and `SymmetricHashJoinExec` URL: https://github.com/apache/datafusion/issues/15080 -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [I] Make it easier to run TPCH queries with datafusion-cli [datafusion]

2025-03-10 Thread via GitHub
clflushopt commented on issue #14608: URL: https://github.com/apache/datafusion/issues/14608#issuecomment-2710564179 @alamb Hey yeah sorry it just by habit I like to complete things before "releasing" them, but just made it open ! -- This is an automated message from the Apache Git Servic

Re: [PR] Fix wasm32 build on version 46 [datafusion]

2025-03-10 Thread via GitHub
alamb commented on PR #15102: URL: https://github.com/apache/datafusion/pull/15102#issuecomment-2710565149 I have been thinking about this issue I plan to: 1. Make a ticket proposing a new patch release for datafusion 46 2. Make a ticket to cover doing something with parquet via

Re: [PR] #15108 issue: "Non Panic Task error" is not an internal error [datafusion]

2025-03-10 Thread via GitHub
alamb commented on PR #15109: URL: https://github.com/apache/datafusion/pull/15109#issuecomment-2710561497 Thank you @Satyam018 It looks like there are some problems with the tests on this PR. Any chance you can fix them ? -- This is an automated message from the Apache Git Servi

Re: [PR] chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) [datafusion]

2025-03-10 Thread via GitHub
Weijun-H commented on code in PR #15123: URL: https://github.com/apache/datafusion/pull/15123#discussion_r1987269201 ## datafusion/core/tests/physical_optimizer/projection_pushdown.rs: ## @@ -89,6 +92,10 @@ impl ScalarUDFImpl for DummyUDF { fn return_type(&self, _arg_types:

Re: [PR] Order Requirement Analysis [datafusion-site]

2025-03-10 Thread via GitHub
alamb commented on PR #58: URL: https://github.com/apache/datafusion-site/pull/58#issuecomment-2710581497 Giving it another read now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Blog: Using Ordering for Better Plans in Apache DataFusion [datafusion-site]

2025-03-10 Thread via GitHub
ozankabak commented on code in PR #58: URL: https://github.com/apache/datafusion-site/pull/58#discussion_r1987861882 ## content/blog/2025-03-05-ordering-analysis.md: ## @@ -0,0 +1,353 @@ +--- +layout: post +title: Analysis of Ordering for Better Plans +date: 2025-03-05 +author:

Re: [PR] Update python min version to 3.9 [datafusion-python]

2025-03-10 Thread via GitHub
kevinjqliu commented on PR #1043: URL: https://github.com/apache/datafusion-python/pull/1043#issuecomment-2711411064 @timsaucer nope this LGTM. I double check the changes in `examples/ffi-table-provider/Cargo.lock` -- This is an automated message from the Apache Git Service. To respond t

Re: [I] Update python min version to 3.9 [datafusion-python]

2025-03-10 Thread via GitHub
timsaucer closed issue #1042: Update python min version to 3.9 URL: https://github.com/apache/datafusion-python/issues/1042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] Fix invalid schema for unions in ViewTables [datafusion]

2025-03-10 Thread via GitHub
Friede80 commented on PR #15135: URL: https://github.com/apache/datafusion/pull/15135#issuecomment-271158 I'm not sure if there are still valid uses of `coerce_union_schema` given only the set of logical plans, but if we can't change the api of a public function, it would be easy enough

Re: [I] Improve Parsing for KV Format in `tree` explain. [datafusion]

2025-03-10 Thread via GitHub
alamb closed issue #15098: Improve Parsing for KV Format in `tree` explain. URL: https://github.com/apache/datafusion/issues/15098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] Enable take comments to assign issues to users [datafusion-python]

2025-03-10 Thread via GitHub
timsaucer merged PR #1058: URL: https://github.com/apache/datafusion-python/pull/1058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...

Re: [PR] Refactor EnforceDistribution test cases to demonstrate dependencies across optimizer runs. [datafusion]

2025-03-10 Thread via GitHub
alamb commented on PR #15074: URL: https://github.com/apache/datafusion/pull/15074#issuecomment-2710816307 I'll plan to merge this later today unless anyone else would like more time to review -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] fix: enable full decimal to decimal support [datafusion-comet]

2025-03-10 Thread via GitHub
kazuyukitanimura commented on code in PR #1385: URL: https://github.com/apache/datafusion-comet/pull/1385#discussion_r1987953695 ## spark/src/test/scala/org/apache/comet/CometCastSuite.scala: ## @@ -1210,27 +1213,36 @@ class CometCastSuite extends CometTestBase with AdaptiveSpa

  1   2   >