Re: [PR] Move average unit tests to slt [datafusion]

2024-05-07 Thread via GitHub
jayzhan211 merged PR #10401: URL: https://github.com/apache/datafusion/pull/10401 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

[PR] Move array_agg unit tests to slt [datafusion]

2024-05-07 Thread via GitHub
lewiszlw opened a new pull request, #10402: URL: https://github.com/apache/datafusion/pull/10402 ## Which issue does this PR close? part of https://github.com/apache/datafusion/issues/10384. ## Rationale for this change ## What changes are included in this

Re: [PR] Move array_agg unit tests to slt [datafusion]

2024-05-07 Thread via GitHub
jayzhan211 merged PR #10402: URL: https://github.com/apache/datafusion/pull/10402 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dat

Re: [PR] Add simplify method to aggregate function [datafusion]

2024-05-07 Thread via GitHub
milenkovicm commented on code in PR #10354: URL: https://github.com/apache/datafusion/pull/10354#discussion_r1592018104 ## datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs: ## @@ -1307,6 +1309,39 @@ impl<'a, S: SimplifyInfo> TreeNodeRewriter for Simplifier<'a, S

Re: [PR] improve monotonicity api [datafusion]

2024-05-07 Thread via GitHub
tinfoil-knight commented on code in PR #10117: URL: https://github.com/apache/datafusion/pull/10117#discussion_r1592020936 ## datafusion/expr/src/signature.rs: ## @@ -346,13 +346,81 @@ impl Signature { } } -/// Monotonicity of the `ScalarFunctionExpr` with respect to its

Re: [PR] Fix and improve `CommonSubexprEliminate` rule [datafusion]

2024-05-07 Thread via GitHub
peter-toth commented on code in PR #10396: URL: https://github.com/apache/datafusion/pull/10396#discussion_r1592032033 ## datafusion/optimizer/src/common_subexpr_eliminate.rs: ## @@ -720,43 +782,53 @@ impl TreeNodeRewriter for CommonSubexprRewriter<'_> { return Ok(T

Re: [I] binary_op should be handled by sql_expr_to_logical_expr.. This was likely caused by a bug in DataFusion's code [datafusion]

2024-05-07 Thread via GitHub
jayzhan211 commented on issue #10310: URL: https://github.com/apache/datafusion/issues/10310#issuecomment-2097763697 > @jayzhan211 > > I understand that the issue runs a bit deeper as we employ sqlparser for converting SQL queries into statements. > > `"SELECT c FROM t WHERE c

Re: [PR] Add simplify method to aggregate function [datafusion]

2024-05-07 Thread via GitHub
milenkovicm commented on code in PR #10354: URL: https://github.com/apache/datafusion/pull/10354#discussion_r1592050865 ## datafusion/expr/src/udaf.rs: ## @@ -195,6 +197,21 @@ impl AggregateUDF { pub fn create_groups_accumulator(&self) -> Result> { self.inner.creat

Re: [PR] Fix and improve `CommonSubexprEliminate` rule [datafusion]

2024-05-07 Thread via GitHub
peter-toth commented on code in PR #10396: URL: https://github.com/apache/datafusion/pull/10396#discussion_r1592032033 ## datafusion/optimizer/src/common_subexpr_eliminate.rs: ## @@ -720,43 +782,53 @@ impl TreeNodeRewriter for CommonSubexprRewriter<'_> { return Ok(T

Re: [PR] feat: short-circuiting expression simplifier (second version) [datafusion]

2024-05-07 Thread via GitHub
jayzhan211 commented on code in PR #10386: URL: https://github.com/apache/datafusion/pull/10386#discussion_r1592084830 ## datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs: ## @@ -1670,17 +1734,106 @@ fn inlist_except(mut l1: InList, l2: InList) -> Result {

Re: [PR] feat: short-circuiting expression simplifier (second version) [datafusion]

2024-05-07 Thread via GitHub
jayzhan211 commented on code in PR #10386: URL: https://github.com/apache/datafusion/pull/10386#discussion_r1592084830 ## datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs: ## @@ -1670,17 +1734,106 @@ fn inlist_except(mut l1: InList, l2: InList) -> Result {

Re: [PR] feat: short-circuiting expression simplifier (second version) [datafusion]

2024-05-07 Thread via GitHub
jayzhan211 commented on code in PR #10386: URL: https://github.com/apache/datafusion/pull/10386#discussion_r1592109596 ## datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs: ## @@ -1670,17 +1734,106 @@ fn inlist_except(mut l1: InList, l2: InList) -> Result {

Re: [PR] feat: short-circuiting expression simplifier (second version) [datafusion]

2024-05-07 Thread via GitHub
jayzhan211 commented on code in PR #10386: URL: https://github.com/apache/datafusion/pull/10386#discussion_r1592109596 ## datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs: ## @@ -1670,17 +1734,106 @@ fn inlist_except(mut l1: InList, l2: InList) -> Result {

Re: [I] May 2024 ASF Board Report [datafusion]

2024-05-07 Thread via GitHub
alamb commented on issue #10281: URL: https://github.com/apache/datafusion/issues/10281#issuecomment-2097874752 Apparently it needed to be done today Here is the text of the report that was submitted: ``` ## Description: The mission of Apache DataFusion is the creation and

Re: [I] May 2024 ASF Board Report [datafusion]

2024-05-07 Thread via GitHub
alamb closed issue #10281: May 2024 ASF Board Report URL: https://github.com/apache/datafusion/issues/10281 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [I] June 2024 ASF Board Report [datafusion]

2024-05-07 Thread via GitHub
alamb commented on issue #10155: URL: https://github.com/apache/datafusion/issues/10155#issuecomment-2097879998 See #10281 for example -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Add `SessionContext`/`SessionState::create_physical_expr()` to create `PhysicalExpressions` from `Expr`s [datafusion]

2024-05-07 Thread via GitHub
alamb commented on PR #10330: URL: https://github.com/apache/datafusion/pull/10330#issuecomment-2097919890 Thanks for your reviews @phillipleblanc and @westonpace @andygrove I can't merge this PR until it is approved by a committer, based on the rules in this repo, but from my pers

Re: [I] Blog Post about graduating to a new top level project [datafusion]

2024-05-07 Thread via GitHub
alamb closed issue #10135: Blog Post about graduating to a new top level project URL: https://github.com/apache/datafusion/issues/10135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-05-07 Thread via GitHub
alamb closed issue #9691: [EPIC] Tasks for a new Top Level Apache Project URL: https://github.com/apache/datafusion/issues/9691 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-05-07 Thread via GitHub
alamb closed issue #9691: [EPIC] Tasks for a new Top Level Apache Project URL: https://github.com/apache/datafusion/issues/9691 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[I] Make ASF public press release [datafusion]

2024-05-07 Thread via GitHub
alamb opened a new issue, #10403: URL: https://github.com/apache/datafusion/issues/10403 ### Is your feature request related to a problem or challenge? DataFusion PMC, I'm writing on behalf of ASF M&P about Apache DataFusion's graduation to TLP. We typically announce project gr

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-05-07 Thread via GitHub
alamb commented on issue #9691: URL: https://github.com/apache/datafusion/issues/9691#issuecomment-2097964759 DataFusion Top Level Project announcement is live: https://arrow.apache.org/blog/2024/05/07/datafusion-tlp/ -- This is an automated message from the Apache Git Service. To respond

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-05-07 Thread via GitHub
alamb commented on issue #9691: URL: https://github.com/apache/datafusion/issues/9691#issuecomment-2097967163 Also, we got a suggestion to make an official ASF press releas https://github.com/apache/datafusion/issues/10403 -- This is an automated message from the Apache Git Service. To re

Re: [I] Make ASF public press release [datafusion]

2024-05-07 Thread via GitHub
alamb commented on issue #10403: URL: https://github.com/apache/datafusion/issues/10403#issuecomment-2097972778 > Kindly note that all ASF press releases are kept private (= limited to private@) during the drafting process to ensure no inadvertent leaks happen on email lists or directly to

Re: [I] Support simplification that requires multiple applications of constant folding / simplification [datafusion]

2024-05-07 Thread via GitHub
alamb closed issue #1160: Support simplification that requires multiple applications of constant folding / simplification URL: https://github.com/apache/datafusion/issues/1160 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] feat: run expression simplifier in a loop until a fixedpoint or 3 cycles [datafusion]

2024-05-07 Thread via GitHub
alamb merged PR #10358: URL: https://github.com/apache/datafusion/pull/10358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] feat: run expression simplifier in a loop until a fixedpoint or 3 cycles [datafusion]

2024-05-07 Thread via GitHub
alamb commented on PR #10358: URL: https://github.com/apache/datafusion/pull/10358#issuecomment-2098051955 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

Re: [PR] feat: short-circuiting expression simplifier (second version) [datafusion]

2024-05-07 Thread via GitHub
alamb commented on PR #10386: URL: https://github.com/apache/datafusion/pull/10386#issuecomment-2098054270 Other PR has been merged in: https://github.com/apache/datafusion/pull/10358 I think we can merge / rebase this PR now and mark it ready for review -- This is an automated mess

[PR] Simplify Format Options [datafusion]

2024-05-07 Thread via GitHub
berkaysynnada opened a new pull request, #10404: URL: https://github.com/apache/datafusion/pull/10404 ## Which issue does this PR close? Closes #9945. ## Rationale for this change ## What changes are included in this PR? ## Are these

[PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]

2024-05-07 Thread via GitHub
alamb opened a new pull request, #10405: URL: https://github.com/apache/datafusion/pull/10405 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/10209 ## Rationale for this change Make planning faster by not copying as much ## Wha

Re: [I] Create fixed size list table with syntax [] [datafusion]

2024-05-07 Thread via GitHub
vaibhawvipul commented on issue #10303: URL: https://github.com/apache/datafusion/issues/10303#issuecomment-2098176829 @jayzhan211 Do we need this for external tables too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10405: URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592317393 ## datafusion/optimizer/src/optimize_projections/mod.rs: ## @@ -1184,13 +1206,32 @@ mod tests { assert_optimized_plan_equal(plan, expected) } +//

Re: [PR] improve monotonicity api [datafusion]

2024-05-07 Thread via GitHub
ozankabak commented on code in PR #10117: URL: https://github.com/apache/datafusion/pull/10117#discussion_r1592322084 ## datafusion/expr/src/signature.rs: ## @@ -346,13 +346,61 @@ impl Signature { } } -/// Monotonicity of the `ScalarFunctionExpr` with respect to its argu

[PR] Minor: Add more docs and examples for `Expr::unalias` [datafusion]

2024-05-07 Thread via GitHub
alamb opened a new pull request, #10406: URL: https://github.com/apache/datafusion/pull/10406 ## Which issue does this PR close? Part of #7013 ## Rationale for this change I have had to look up the behavior of this function several times when working on optimizer rules,

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-05-07 Thread via GitHub
andygrove closed issue #10181: Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 URL: https://github.com/apache/datafusion/issues/10181 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Add `SessionContext`/`SessionState::create_physical_expr()` to create `PhysicalExpressions` from `Expr`s [datafusion]

2024-05-07 Thread via GitHub
andygrove merged PR #10330: URL: https://github.com/apache/datafusion/pull/10330 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

Re: [PR] chore: Store EXTENSION_INFO as Set[String] instead of newline-delimited String [datafusion-comet]

2024-05-07 Thread via GitHub
andygrove commented on PR #386: URL: https://github.com/apache/datafusion-comet/pull/386#issuecomment-2098326148 Thanks for the reviews @parthchandra and @advancedxy @viirya I need a committer approval to be able to merge. Could you review? -- This is an automated message from the A

Re: [I] DataFusion `38.0.0` Release [datafusion]

2024-05-07 Thread via GitHub
andygrove commented on issue #10217: URL: https://github.com/apache/datafusion/issues/10217#issuecomment-2098375093 I will start the process of creating the first release candidate today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Create fixed size list table with syntax [] [datafusion]

2024-05-07 Thread via GitHub
jayzhan211 commented on issue #10303: URL: https://github.com/apache/datafusion/issues/10303#issuecomment-2098392913 I think the change here does not matter the table is external or not -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Support custom SchemaAdapter on ParquetExec [datafusion]

2024-05-07 Thread via GitHub
tustvold commented on issue #10398: URL: https://github.com/apache/datafusion/issues/10398#issuecomment-2098455325 I wonder if the way to achieve this might be something like https://github.com/apache/datafusion/issues/2293, this would allow making schema adaption a standard execution node,

Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10405: URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592520545 ## datafusion/sqllogictest/test_files/cte.slt: ## @@ -31,10 +31,9 @@ query TT EXPLAIN WITH "NUMBERS" AS (SELECT 1 as a, 2 as b, 3 as c) SELECT "NUMBERS".* FROM "NU

[I] chore: Prepare release process and scripts for official source releases [datafusion-comet]

2024-05-07 Thread via GitHub
andygrove opened a new issue, #394: URL: https://github.com/apache/datafusion-comet/issues/394 ### What is the problem the feature request solves? At some point we will want to start creating official releases of Comet. ASF projects produce source releases, not binary releases,

[PR] Prepare 38.0.0 release candidate 1 [datafusion]

2024-05-07 Thread via GitHub
andygrove opened a new pull request, #10407: URL: https://github.com/apache/datafusion/pull/10407 ## Which issue does this PR close? N/A ## Rationale for this change It is time for a new release. ## What changes are included in this PR?

[PR] Remove unnecessary `exprlist_to_fields_aggregate` [datafusion]

2024-05-07 Thread via GitHub
jonahgao opened a new pull request, #10408: URL: https://github.com/apache/datafusion/pull/10408 ## Which issue does this PR close? Closes #10346. ## Rationale for this change [exprlist_to_fields_aggregate](https://github.com/apache/datafusion/blob/a0fccbf886346fde5d

Re: [PR] Prepare 38.0.0 release candidate 1 [datafusion]

2024-05-07 Thread via GitHub
comphead commented on code in PR #10407: URL: https://github.com/apache/datafusion/pull/10407#discussion_r1592562326 ## Cargo.toml: ## @@ -44,14 +44,14 @@ members = [ resolver = "2" [workspace.package] -authors = ["Apache Arrow "] +authors = ["Apache DataFusion "] Review Co

Re: [PR] Add simplify method to aggregate function [datafusion]

2024-05-07 Thread via GitHub
milenkovicm commented on PR #10354: URL: https://github.com/apache/datafusion/pull/10354#issuecomment-2098527849 Another option could be to make a simplify return optional closure ``` fn simplify( &self ) -> Option datafusion_common::tree_node::Transformed>> {

Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10405: URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592578860 ## datafusion/expr/src/expr.rs: ## @@ -1087,13 +1087,60 @@ impl Expr { } /// Remove an alias from an expression if one exists. +/// +/// If the e

Re: [PR] Fix: Sort Merge Join LeftSemi issues [datafusion]

2024-05-07 Thread via GitHub
comphead commented on PR #10304: URL: https://github.com/apache/datafusion/pull/10304#issuecomment-2098539325 fuzztests failing... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10405: URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592580823 ## datafusion/optimizer/src/optimize_projections/mod.rs: ## @@ -102,101 +118,19 @@ impl OptimizerRule for OptimizeProjections { /// - `Ok(Some(LogicalPlan))`: An op

Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10405: URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592582483 ## datafusion/optimizer/src/optimize_projections/mod.rs: ## @@ -102,101 +118,19 @@ impl OptimizerRule for OptimizeProjections { /// - `Ok(Some(LogicalPlan))`: An op

Re: [PR] Prepare 38.0.0 release candidate 1 [datafusion]

2024-05-07 Thread via GitHub
andygrove commented on PR #10407: URL: https://github.com/apache/datafusion/pull/10407#issuecomment-2098560873 Thanks for the review @comphead -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Prepare 38.0.0 release candidate 1 [datafusion]

2024-05-07 Thread via GitHub
andygrove merged PR #10407: URL: https://github.com/apache/datafusion/pull/10407 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@data

[PR] [MINOR]: Move transpose code to under common [datafusion]

2024-05-07 Thread via GitHub
mustafasrepo opened a new pull request, #10409: URL: https://github.com/apache/datafusion/pull/10409 ## Which issue does this PR close? Closes #. ## Rationale for this change `fn transpose` code is under `` ## What changes are included in this PR?

[PR] Add `LogicalPlan::recompute_schema` for handling rewrite passes [datafusion]

2024-05-07 Thread via GitHub
alamb opened a new pull request, #10410: URL: https://github.com/apache/datafusion/pull/10410 ## Which issue does this PR close? Part of https://github.com/apache/datafusion/issues/9637 ## Rationale for this change `LogicalPlan::recompute_schema` is needed for se

Re: [PR] Stop copying LogicalPlan and Exprs in `TypeCoercion` (10% faster planning) [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10356: URL: https://github.com/apache/datafusion/pull/10356#discussion_r1592666938 ## datafusion/expr/src/logical_plan/plan.rs: ## @@ -467,6 +468,200 @@ impl LogicalPlan { self.with_new_exprs(self.expressions(), inputs.to_vec()) } +

Re: [PR] Add `LogicalPlan::recompute_schema` for handling rewrite passes [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10410: URL: https://github.com/apache/datafusion/pull/10410#discussion_r1592667315 ## datafusion/expr/src/logical_plan/plan.rs: ## @@ -467,6 +468,200 @@ impl LogicalPlan { self.with_new_exprs(self.expressions(), inputs.to_vec()) } +

Re: [PR] build(deps): upgrade sqlparser to 0.46.0 [datafusion]

2024-05-07 Thread via GitHub
tisonkun commented on PR #10392: URL: https://github.com/apache/datafusion/pull/10392#issuecomment-2098683236 @jmhain Thanks for your patch. I've applied it to this branch. Let us continue fixing the breaking changes here when we have time :D -- This is an automated message from the Apach

Re: [PR] Minor: Add more docs and examples for `Expr::unalias` [datafusion]

2024-05-07 Thread via GitHub
alamb commented on PR #10406: URL: https://github.com/apache/datafusion/pull/10406#issuecomment-2098685377 Thanks for the speedy review @comphead -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Minor: Add more docs and examples for `Expr::unalias` [datafusion]

2024-05-07 Thread via GitHub
alamb merged PR #10406: URL: https://github.com/apache/datafusion/pull/10406 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Avoid inlining non deterministic CTE [datafusion]

2024-05-07 Thread via GitHub
tgujar commented on issue #10337: URL: https://github.com/apache/datafusion/issues/10337#issuecomment-2098697931 I think this would push the responsibility to the user to figure out what may be non-deterministic. I am not sure if this would be a good approach -- This is an automated messa

[PR] feat: Supports UUID column [datafusion-comet]

2024-05-07 Thread via GitHub
huaxingao opened a new pull request, #395: URL: https://github.com/apache/datafusion-comet/pull/395 ## Which issue does this PR close? Closes #. ## Rationale for this change Supports UUID column. This is for Iceberg/Comet integration ## What changes ar

Re: [PR] chore: Store EXTENSION_INFO as Set[String] instead of newline-delimited String [datafusion-comet]

2024-05-07 Thread via GitHub
andygrove merged PR #386: URL: https://github.com/apache/datafusion-comet/pull/386 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@da

Re: [PR] feat: Supports UUID column [datafusion-comet]

2024-05-07 Thread via GitHub
huaxingao commented on PR #395: URL: https://github.com/apache/datafusion-comet/pull/395#issuecomment-2098794942 > should we get this covered by tests This is for iceberg/comet integration. I don't think there is an easy way to test this now. I have tested this on my local, though.

Re: [PR] feat: Implement Spark-compatible CAST from floating-point to decimal [datafusion-comet]

2024-05-07 Thread via GitHub
andygrove commented on code in PR #384: URL: https://github.com/apache/datafusion-comet/pull/384#discussion_r1592732303 ## spark/src/test/scala/org/apache/comet/CometCastSuite.scala: ## @@ -402,7 +402,7 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper

Re: [PR] feat: Implement Spark-compatible CAST from floating-point to decimal [datafusion-comet]

2024-05-07 Thread via GitHub
andygrove commented on PR #384: URL: https://github.com/apache/datafusion-comet/pull/384#issuecomment-2098808791 @vaibhawvipul Do you plan on supporting `FloatType -> DecimalType` as well as `DecimalType -> DecimalType` in this PR? -- This is an automated message from the Apache Git Servi

[PR] minor: Remove [RUST][DataFusion] from release vote email subject line [datafusion]

2024-05-07 Thread via GitHub
andygrove opened a new pull request, #10411: URL: https://github.com/apache/datafusion/pull/10411 ## Which issue does this PR close? N/A ## Rationale for this change This information is not relevant now that DataFusion is a TLP. ## What changes are

Re: [PR] RFC: Make it easier to call window functions via expression API (and add example) [datafusion]

2024-05-07 Thread via GitHub
timsaucer commented on PR #6746: URL: https://github.com/apache/datafusion/pull/6746#issuecomment-2098821089 I'm willing to work on this, but I'd like to wrap up the examples I have going into datafusion-python first. I think those will have more impact. I only have an hour or so a day to w

Re: [PR] feat: Implement Spark-compatible CAST from floating-point to decimal [datafusion-comet]

2024-05-07 Thread via GitHub
vaibhawvipul commented on PR #384: URL: https://github.com/apache/datafusion-comet/pull/384#issuecomment-2098823494 > @vaibhawvipul Do you plan on supporting `FloatType -> DecimalType` as well as `DecimalType -> DecimalType` in this PR? no, but do you want me to? I thought that the is

Re: [PR] feat: Supports UUID column [datafusion-comet]

2024-05-07 Thread via GitHub
comphead commented on PR #395: URL: https://github.com/apache/datafusion-comet/pull/395#issuecomment-2098828743 > > should we get this covered by tests > > This is for iceberg/comet integration. I don't think there is an easy way to test this now. I have tested this on my local, thoug

Re: [PR] minor: Remove [RUST][DataFusion] from release vote email subject line [datafusion]

2024-05-07 Thread via GitHub
alamb merged PR #10411: URL: https://github.com/apache/datafusion/pull/10411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] minor: Remove [RUST][DataFusion] from release vote email subject line [datafusion]

2024-05-07 Thread via GitHub
alamb commented on PR #10411: URL: https://github.com/apache/datafusion/pull/10411#issuecomment-2098838949 Thanks @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: Supports UUID column [datafusion-comet]

2024-05-07 Thread via GitHub
huaxingao commented on PR #395: URL: https://github.com/apache/datafusion-comet/pull/395#issuecomment-2098856203 > Maybe we can return uuid value and assert it somehow, although its non-determenistic? I'm thinking if we can be protected from regression if anyone else changes this code later

Re: [PR] Fix: Sort Merge Join LeftSemi issues [datafusion]

2024-05-07 Thread via GitHub
comphead commented on code in PR #10304: URL: https://github.com/apache/datafusion/pull/10304#discussion_r1592785692 ## datafusion/core/tests/fuzz_cases/join_fuzz.rs: ## @@ -79,20 +79,20 @@ async fn test_full_join_1k() { } #[tokio::test] -async fn test_semi_join_10k() { +asy

Re: [PR] Remove ScalarFunctionDefinition [datafusion]

2024-05-07 Thread via GitHub
alamb commented on PR #10325: URL: https://github.com/apache/datafusion/pull/10325#issuecomment-2098892166 Now that https://github.com/apache/datafusion/pull/10407 is merged, I believe main is open for 38.0.0 business ! 🚀 -- tanks again @lewiszlw -- This is an automated message f

Re: [PR] Remove ScalarFunctionDefinition [datafusion]

2024-05-07 Thread via GitHub
alamb merged PR #10325: URL: https://github.com/apache/datafusion/pull/10325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] Remove ScalarFunctionDefinition [datafusion]

2024-05-07 Thread via GitHub
alamb closed issue #10175: Remove ScalarFunctionDefinition URL: https://github.com/apache/datafusion/issues/10175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[I] Substrait integration doesn't recognize typed functions [datafusion]

2024-05-07 Thread via GitHub
Blizzara opened a new issue, #10412: URL: https://github.com/apache/datafusion/issues/10412 ### Describe the bug Since https://github.com/substrait-io/substrait/pull/537, the standard has declared that functions must be named with compound names consisting of both the function and it

Re: [I] Create fixed size list table with syntax [] [datafusion]

2024-05-07 Thread via GitHub
alamb commented on issue #10303: URL: https://github.com/apache/datafusion/issues/10303#issuecomment-2098927570 I think we may be able to use the syntax https://docs.rs/sqlparser/latest/sqlparser/ast/enum.DataType.html#variant.Array So something like this to create a 256 integer array

Re: [I] DataFusion weekly project plan (Andrew Lamb) - May 6, 2024 [datafusion]

2024-05-07 Thread via GitHub
alamb commented on issue #10395: URL: https://github.com/apache/datafusion/issues/10395#issuecomment-2098931190 Review Queue: - [ ] https://github.com/apache/datafusion/pull/10386 - [ ] https://github.com/apache/datafusion/pull/10404 - [ ] https://github.com/apache/datafusion/pull/10

Re: [PR] chore(docs): update subquery documentation with more information [datafusion]

2024-05-07 Thread via GitHub
alamb merged PR #10361: URL: https://github.com/apache/datafusion/pull/10361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Fix incorrect Schema over aggregate function, Remove unnecessary `exprlist_to_fields_aggregate` [datafusion]

2024-05-07 Thread via GitHub
alamb commented on PR #10408: URL: https://github.com/apache/datafusion/pull/10408#issuecomment-2098945511 FYI @timsaucer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] fix dml logical plan output schema [datafusion]

2024-05-07 Thread via GitHub
comphead commented on code in PR #10394: URL: https://github.com/apache/datafusion/pull/10394#discussion_r1592831759 ## datafusion/sqllogictest/test_files/insert.slt: ## @@ -259,7 +259,7 @@ statement error Error during planning: Column count doesn't match insert query! insert

Re: [PR] fix dml logical plan output schema [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10394: URL: https://github.com/apache/datafusion/pull/10394#discussion_r1592832784 ## datafusion/sqllogictest/test_files/aggregate.slt: ## @@ -3260,7 +3260,7 @@ SELECT STRING_AGG(column1, '|') FROM (values (''), (null), ('')); statement ok CREAT

Re: [PR] fix dml logical plan output schema [datafusion]

2024-05-07 Thread via GitHub
alamb merged PR #10394: URL: https://github.com/apache/datafusion/pull/10394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [I] For DML plans, `LogicalPlan::schema` returns the input schema instead of output schema [datafusion]

2024-05-07 Thread via GitHub
alamb closed issue #10393: For DML plans, `LogicalPlan::schema` returns the input schema instead of output schema URL: https://github.com/apache/datafusion/issues/10393 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] feat: Implement Spark-compatible CAST from floating-point to decimal [datafusion-comet]

2024-05-07 Thread via GitHub
parthchandra commented on code in PR #384: URL: https://github.com/apache/datafusion-comet/pull/384#discussion_r1592834377 ## core/src/execution/datafusion/expressions/cast.rs: ## @@ -395,6 +399,58 @@ impl Cast { Ok(cast_array) } +fn cast_float64_to_decimal12

Re: [I] DataFusion `38.0.0` Release [datafusion]

2024-05-07 Thread via GitHub
alamb commented on issue #10217: URL: https://github.com/apache/datafusion/issues/10217#issuecomment-2098957547 Vote thread: https://lists.apache.org/thread/nnhtnxt8rs65k5x59scfdc0qkmqghy3n -- This is an automated message from the Apache Git Service. To respond to the message, please log o

[PR] build: Add scala-version to matrix [datafusion-comet]

2024-05-07 Thread via GitHub
snmvaughan opened a new pull request, #396: URL: https://github.com/apache/datafusion-comet/pull/396 ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/388 ## Rationale for this change scalafix check will fail when running Scala 2.

[I] Incorrect results with common subexpression elimination [datafusion]

2024-05-07 Thread via GitHub
alamb opened a new issue, #10413: URL: https://github.com/apache/datafusion/issues/10413 ### Describe the bug DataFusion will sometimes resolve expressions incorrectly when the alias shadows an expression ### To Reproduce ```sql DataFusion CLI v37.1.0 > select

Re: [PR] feat: Implement Spark-compatible CAST from floating-point to decimal [datafusion-comet]

2024-05-07 Thread via GitHub
vaibhawvipul commented on code in PR #384: URL: https://github.com/apache/datafusion-comet/pull/384#discussion_r1592854344 ## core/src/execution/datafusion/expressions/cast.rs: ## @@ -395,6 +399,58 @@ impl Cast { Ok(cast_array) } +fn cast_float64_to_decimal12

Re: [PR] feat: Implement Spark unhex [datafusion-comet]

2024-05-07 Thread via GitHub
andygrove commented on PR #342: URL: https://github.com/apache/datafusion-comet/pull/342#issuecomment-2098985033 Thanks for the updates @tshauck. I plan on reviewing later today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [PR] feat: Implement Spark-compatible CAST from floating-point to decimal [datafusion-comet]

2024-05-07 Thread via GitHub
parthchandra commented on code in PR #384: URL: https://github.com/apache/datafusion-comet/pull/384#discussion_r1592861816 ## core/src/execution/datafusion/expressions/cast.rs: ## @@ -395,6 +399,58 @@ impl Cast { Ok(cast_array) } +fn cast_float64_to_decimal12

Re: [PR] build: Add scala-version to matrix [datafusion-comet]

2024-05-07 Thread via GitHub
viirya commented on code in PR #396: URL: https://github.com/apache/datafusion-comet/pull/396#discussion_r1592862382 ## .github/workflows/pr_build.yml: ## @@ -82,11 +83,14 @@ jobs: java_version: [8, 11, 17] test-target: [java] spark-version: ['3.2', '3

Re: [I] Tracking Upgrade to Datafusion 37 [datafusion-python]

2024-05-07 Thread via GitHub
Michael-J-Ward commented on issue #663: URL: https://github.com/apache/datafusion-python/issues/663#issuecomment-2099002670 # Current Status The initial steps are completed and `python -m pytest` now passing in #662 (modulo 1, discussed below). ## TODO Besides the fail

Re: [PR] Draft: upgrading to datafusion 37.1.0 [datafusion-python]

2024-05-07 Thread via GitHub
Michael-J-Ward commented on PR #662: URL: https://github.com/apache/datafusion-python/pull/662#issuecomment-2099008338 Refer to this status update on tracking issue https://github.com/apache/datafusion-python/issues/663#issuecomment-2099002670 ## NOTE I did not rebase / clean-up

Re: [PR] Fix and improve `CommonSubexprEliminate` rule [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10396: URL: https://github.com/apache/datafusion/pull/10396#discussion_r1592864219 ## datafusion/optimizer/src/common_subexpr_eliminate.rs: ## @@ -34,90 +33,63 @@ use datafusion_common::{ use datafusion_expr::expr::Alias; use datafusion_expr::log

Re: [PR] Fix and improve `CommonSubexprEliminate` rule [datafusion]

2024-05-07 Thread via GitHub
alamb commented on code in PR #10396: URL: https://github.com/apache/datafusion/pull/10396#discussion_r1592878495 ## datafusion/sqllogictest/test_files/select.slt: ## @@ -1613,6 +1613,14 @@ select count(1) from v; 1 +# Ensure CSE resolves columns correctly Review Comme

Re: [PR] [MINOR]: Move transpose code to under common [datafusion]

2024-05-07 Thread via GitHub
alamb merged PR #10409: URL: https://github.com/apache/datafusion/pull/10409 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] [MINOR]: Move transpose code to under common [datafusion]

2024-05-07 Thread via GitHub
alamb commented on PR #10409: URL: https://github.com/apache/datafusion/pull/10409#issuecomment-2099020603 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[PR] feat: Add logging to explain reasons for Comet not being able to run a query stage natively [datafusion-comet]

2024-05-07 Thread via GitHub
andygrove opened a new pull request, #397: URL: https://github.com/apache/datafusion-comet/pull/397 ## Which issue does this PR close? Closes https://github.com/apache/datafusion-comet/issues/385 ## Rationale for this change As a user, I would like to unde

  1   2   3   >