Re: [PR] feat: Support murmur3_hash and sha2 family hash functions [datafusion-comet]

2024-04-23 Thread via GitHub
advancedxy commented on code in PR #226: URL: https://github.com/apache/datafusion-comet/pull/226#discussion_r1575746005 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -983,8 +983,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSpa

Re: [PR] feat: Support murmur3_hash and sha2 family hash functions [datafusion-comet]

2024-04-23 Thread via GitHub
advancedxy commented on code in PR #226: URL: https://github.com/apache/datafusion-comet/pull/226#discussion_r1575752513 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -983,8 +983,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSpa

[PR] Introduce `Expr::is_volatile()`, adjust `TreeNode::exists()` [datafusion]

2024-04-23 Thread via GitHub
peter-toth opened a new pull request, #10191: URL: https://github.com/apache/datafusion/pull/10191 This PR introduces `Expr::is_volatile()` instead of `is_volatile_expression(e: &Expr)`. Also, a small change is needed in `TreeNode::exists()`, but it is a fairly new API and the change mak

Re: [PR] feat: Support murmur3_hash and sha2 family hash functions [datafusion-comet]

2024-04-23 Thread via GitHub
advancedxy commented on code in PR #226: URL: https://github.com/apache/datafusion-comet/pull/226#discussion_r1575759131 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -1646,6 +1647,37 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde

Re: [PR] feat: Support murmur3_hash and sha2 family hash functions [datafusion-comet]

2024-04-23 Thread via GitHub
advancedxy commented on code in PR #226: URL: https://github.com/apache/datafusion-comet/pull/226#discussion_r1575761858 ## spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala: ## @@ -1646,6 +1647,37 @@ object QueryPlanSerde extends Logging with ShimQueryPlanSerde

[PR] chore(deps): update object_store requirement from 0.9.1 to 0.10.0 [datafusion]

2024-04-23 Thread via GitHub
dependabot[bot] opened a new pull request, #10192: URL: https://github.com/apache/datafusion/pull/10192 Updates the requirements on [object_store](https://github.com/apache/arrow-rs) to permit the latest version. Changelog Sourced from https://github.com/apache/arrow-rs/blob/master

Re: [PR] Move coalesce to datafusion-functions and remove BuiltInScalarFunction [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10098: URL: https://github.com/apache/datafusion/pull/10098#discussion_r1575897841 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -144,24 +144,11 @@ impl PhysicalExpr for ScalarFunctionExpr { fn evaluate(&self, batch: &RecordBa

Re: [PR] Move coalesce to datafusion-functions and remove BuiltInScalarFunction [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10098: URL: https://github.com/apache/datafusion/pull/10098#discussion_r1575897841 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -144,24 +144,11 @@ impl PhysicalExpr for ScalarFunctionExpr { fn evaluate(&self, batch: &RecordBa

[PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 opened a new pull request, #10193: URL: https://github.com/apache/datafusion/pull/10193 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested

Re: [PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1575925764 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -142,21 +142,11 @@ impl PhysicalExpr for ScalarFunctionExpr { } fn evaluate(&self, batch:

Re: [PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1575925764 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -142,21 +142,11 @@ impl PhysicalExpr for ScalarFunctionExpr { } fn evaluate(&self, batch:

Re: [PR] Move coalesce to datafusion-functions and remove BuiltInScalarFunction [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10098: URL: https://github.com/apache/datafusion/pull/10098#discussion_r1575944617 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -144,24 +144,11 @@ impl PhysicalExpr for ScalarFunctionExpr { fn evaluate(&self, batch: &RecordBa

Re: [PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
alamb commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1575980985 ## datafusion/functions/src/math/random.rs: ## @@ -69,12 +69,13 @@ impl ScalarUDFImpl for RandomFunc { } } -/// Random SQL function fn random(args: &[Column

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2071950006 > Another way to tackle it could be to leave the concept of a GetIndexedField node at the parsing layer and pull it out of Expr (or deprecate). This would force the conversion to

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2071951668 I am thinking I'll try and make a PR with such an API over the next day or two to see how it might look -- This is an automated message from the Apache Git Service. To respond t

Re: [PR] Update .asf.yaml to publish docs to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10190: URL: https://github.com/apache/datafusion/pull/10190#issuecomment-2071962390 > Also I wasn't able to figure out how to add the redirect from arrow.apache.org/datafusion to datafusion.apache.org. @alamb did you have an idea on how to do that? The only thi

Re: [PR] Update .asf.yaml to publish docs to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10190: URL: https://github.com/apache/datafusion/pull/10190#issuecomment-2071962701 Thank you for your work / help and reviews @phillipleblanc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] Create Project Website (datafusion.apache.org) [datafusion]

2024-04-23 Thread via GitHub
alamb closed issue #10151: Create Project Website (datafusion.apache.org) URL: https://github.com/apache/datafusion/issues/10151 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Update .asf.yaml to publish docs to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
alamb merged PR #10190: URL: https://github.com/apache/datafusion/pull/10190 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

[I] Redirect existing docs from https://arrow.apache.org/ to https://datafusion.apache.org/ [datafusion]

2024-04-23 Thread via GitHub
alamb opened a new issue, #10194: URL: https://github.com/apache/datafusion/issues/10194 ### Is your feature request related to a problem or challenge? in https://github.com/apache/datafusion/pull/10190 the datafusion documentation is now published to https://datafusion.apache.org/

Re: [I] Redirect existing docs from https://arrow.apache.org/ to https://datafusion.apache.org/ [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #10194: URL: https://github.com/apache/datafusion/issues/10194#issuecomment-2071979606 Here is the existing list of html files, found using ```shell cd ~/Software/datafusion/docs ./build.sh find build/html -name '*.html' ``` Here is the outpu

Re: [PR] Update .asf.yaml to publish docs to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10190: URL: https://github.com/apache/datafusion/pull/10190#issuecomment-2071980082 Filed https://github.com/apache/datafusion/issues/10194 to track redirects -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] Update .asf.yaml to publish docs to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10190: URL: https://github.com/apache/datafusion/pull/10190#issuecomment-2071981403 It appears to have worked! https://datafusion.apache.org/ now shows: ![Screenshot 2024-04-23 at 6 43 58  AM](https://github.com/apache/datafusion/assets/490673/ec802ae4-b67

Re: [PR] Update .asf.yaml to publish docs to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10190: URL: https://github.com/apache/datafusion/pull/10190#issuecomment-2071983447 It appears that https://arrow.apache.org/datafusion/ has an older snapshot of the site (as in it was not updated as well): ![Screenshot 2024-04-23 at 6 45 03  AM](https://github.

Re: [I] Redirect existing docs from https://arrow.apache.org/ to https://datafusion.apache.org/ [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #10194: URL: https://github.com/apache/datafusion/issues/10194#issuecomment-2071986708 After https://github.com/apache/datafusion/pull/10190 was merged (see https://github.com/apache/datafusion/pull/10190#issuecomment-2071981403) It seems like https://datafusi

Re: [PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576097589 ## datafusion/functions/src/math/random.rs: ## @@ -69,12 +69,13 @@ impl ScalarUDFImpl for RandomFunc { } } -/// Random SQL function fn random(args: &[C

Re: [PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576108924 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -142,21 +142,11 @@ impl PhysicalExpr for ScalarFunctionExpr { } fn evaluate(&self, batch:

Re: [PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576108924 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -142,21 +142,11 @@ impl PhysicalExpr for ScalarFunctionExpr { } fn evaluate(&self, batch:

Re: [I] Redirect existing docs from https://arrow.apache.org/ to https://datafusion.apache.org/ [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #10194: URL: https://github.com/apache/datafusion/issues/10194#issuecomment-2072080961 l played around with this a little with https://github.com/apache/arrow-site/pull/502 It mostly works, but there are still a few corner cases I need to figure out -- This

Re: [PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576121743 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -142,21 +142,11 @@ impl PhysicalExpr for ScalarFunctionExpr { } fn evaluate(&self, batch:

Re: [PR] Avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576139980 ## datafusion/functions/src/math/random.rs: ## @@ -69,12 +69,13 @@ impl ScalarUDFImpl for RandomFunc { } } -/// Random SQL function fn random(args: &[C

[PR] Update links to point to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
phillipleblanc opened a new pull request, #10195: URL: https://github.com/apache/datafusion/pull/10195 ## Which issue does this PR close? Part of #9691 ## Rationale for this change Updates the links in the repo to point to https://datafusion.apache.org instead of https:/

Re: [PR] Update links to point to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
phillipleblanc commented on code in PR #10195: URL: https://github.com/apache/datafusion/pull/10195#discussion_r1576152869 ## dev/release/README.md: ## @@ -463,7 +463,7 @@ svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist - Checkout the `asf-site

Re: [PR] Update links to point to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
phillipleblanc commented on code in PR #10195: URL: https://github.com/apache/datafusion/pull/10195#discussion_r1576150883 ## README.md: ## @@ -46,10 +46,10 @@ in-memory format. [Python Bindings](https://github.com/apache/datafusion-python) Here are links to some important i

Re: [PR] Update links to point to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
phillipleblanc commented on code in PR #10195: URL: https://github.com/apache/datafusion/pull/10195#discussion_r1576154457 ## README.md: ## @@ -46,10 +46,10 @@ in-memory format. [Python Bindings](https://github.com/apache/datafusion-python) Here are links to some important i

Re: [PR] ScalarUDF: Remove `supports_zero_argument` and avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576155672 ## datafusion/functions/src/math/random.rs: ## @@ -83,19 +84,13 @@ fn random(args: &[ColumnarValue]) -> Result { #[cfg(test)] mod test { -use std::sync:

Re: [PR] ScalarUDF: Remove `supports_zero_argument` and avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r157619 ## datafusion/core/tests/user_defined/user_defined_scalar_functions.rs: ## @@ -403,123 +398,6 @@ async fn test_user_defined_functions_with_alias() -> Result<()

Re: [PR] ScalarUDF: Remove `supports_zero_argument` and avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576121743 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -142,21 +142,11 @@ impl PhysicalExpr for ScalarFunctionExpr { } fn evaluate(&self, batch:

Re: [PR] ScalarUDF: Remove `supports_zero_argument` and avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576121743 ## datafusion/physical-expr/src/scalar_function.rs: ## @@ -142,21 +142,11 @@ impl PhysicalExpr for ScalarFunctionExpr { } fn evaluate(&self, batch:

Re: [PR] Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` [datafusion]

2024-04-23 Thread via GitHub
alamb closed pull request #10187: Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` URL: https://github.com/apache/datafusion/pull/10187 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10187: URL: https://github.com/apache/datafusion/pull/10187#issuecomment-2072232791 Closing in favor of https://github.com/apache/arrow-site/pull/502 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] Redirect existing docs from https://arrow.apache.org/ to https://datafusion.apache.org/ [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #10194: URL: https://github.com/apache/datafusion/issues/10194#issuecomment-2072266855 Ok, I think https://github.com/apache/arrow-site/pull/502 is now ready to go -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] Update .asf.yaml to publish docs to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10190: URL: https://github.com/apache/datafusion/pull/10190#issuecomment-2072279922 I think I have figured out the redirects -- see https://github.com/apache/arrow-site/pull/502 -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] Support `FixedSizeList` in array methods [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on issue #6560: URL: https://github.com/apache/datafusion/issues/6560#issuecomment-2072281672 I think we have support FixedSizeList already. Thanks for @Weijun-H -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

Re: [I] Support `FixedSizeList` in array methods [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 closed issue #6560: Support `FixedSizeList` in array methods URL: https://github.com/apache/datafusion/issues/6560 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Minor: Add `Column::from(Tableref, &FieldRef)`, `Expr::from(Column)` and `Expr::from(Tableref, &FieldRef)` [datafusion]

2024-04-23 Thread via GitHub
alamb merged PR #10178: URL: https://github.com/apache/datafusion/pull/10178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Minor: Add `Column::from(Tableref, &FieldRef)`, `Expr::from(Column)` and `Expr::from(Tableref, &FieldRef)` [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10178: URL: https://github.com/apache/datafusion/pull/10178#issuecomment-2072298700 Thanks everyone for the reviews ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #9691: URL: https://github.com/apache/datafusion/issues/9691#issuecomment-2072302680 https://datafusion.apache.org/ now has the website live 🚀 -- thanks @phillipleblanc -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [I] Complete support for `Expr --> String ` [datafusion]

2024-04-23 Thread via GitHub
devanbenz commented on issue #9726: URL: https://github.com/apache/datafusion/issues/9726#issuecomment-2072328675 > Hi @devanbenz , I think `ScalarVariable` has not been done yet and you could have a try, though it may not be possible.🤔 I think maybe it should be converted to `ScalarValue`

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-04-23 Thread via GitHub
lostmygithubaccount commented on issue #9691: URL: https://github.com/apache/datafusion/issues/9691#issuecomment-2072330322 this link currently still works and doesn't redirect -- will it redirect to the new one at some point? https://arrow.apache.org/datafusion/ (I know this is very

Re: [I] Complete support for `Expr --> String ` [datafusion]

2024-04-23 Thread via GitHub
devanbenz commented on issue #9726: URL: https://github.com/apache/datafusion/issues/9726#issuecomment-2072342237 > Here are some thoughts to help people figure out what SQL matches to what expression: > > * `GetIndexedField` --> `SELECT col['field_name'] > * ScalarVariable --> Not

Re: [PR] ScalarUDF: Remove `supports_zero_argument` and avoid creating null array for empty args [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on code in PR #10193: URL: https://github.com/apache/datafusion/pull/10193#discussion_r1576291793 ## datafusion/functions/src/math/pi.rs: ## @@ -62,12 +60,10 @@ impl ScalarUDFImpl for PiFunc { Ok(Float64) } -fn invoke(&self, args: &[Colum

Re: [I] Move coalesce function from math to core [datafusion]

2024-04-23 Thread via GitHub
xxxuuu commented on issue #10174: URL: https://github.com/apache/datafusion/issues/10174#issuecomment-2072373131 I would like to give it a try :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] feat: Improve CometSortMergeJoin statistics [datafusion-comet]

2024-04-23 Thread via GitHub
andygrove merged PR #304: URL: https://github.com/apache/datafusion-comet/pull/304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@da

Re: [I] Improve CometSortMergeJoin statistics [datafusion-comet]

2024-04-23 Thread via GitHub
andygrove closed issue #303: Improve CometSortMergeJoin statistics URL: https://github.com/apache/datafusion-comet/issues/303 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Move coalesce function from math to core [datafusion]

2024-04-23 Thread via GitHub
xxxuuu commented on issue #10174: URL: https://github.com/apache/datafusion/issues/10174#issuecomment-2072400424 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] feat: Support Variance [datafusion-comet]

2024-04-23 Thread via GitHub
andygrove commented on code in PR #297: URL: https://github.com/apache/datafusion-comet/pull/297#discussion_r1576321475 ## core/src/execution/proto/expr.proto: ## @@ -165,6 +167,18 @@ message CovPopulation { DataType datatype = 4; } +message VarianceSample { + Expr child

Re: [I] "Cannot infer common argument type for comparison operation Union..." [datafusion]

2024-04-23 Thread via GitHub
jayzhan211 commented on issue #10180: URL: https://github.com/apache/datafusion/issues/10180#issuecomment-2072405561 I take a look a bit, I found that your `return_type` is `Union`, but If I understand correctly, you should compute the return type based on args. For example, your test `json

Re: [I] CLI unable to execute queries with range(..) function [datafusion]

2024-04-23 Thread via GitHub
Lordworms commented on issue #10177: URL: https://github.com/apache/datafusion/issues/10177#issuecomment-2072419019 I think it is a related issue with https://github.com/apache/datafusion/issues/10069#issue-2241483049 and if it is necessary to register range as a table function, I would

[PR] fix: cargo warnings of import item [datafusion]

2024-04-23 Thread via GitHub
waynexia opened a new pull request, #10196: URL: https://github.com/apache/datafusion/pull/10196 ## Which issue does this PR close? Closes #. ## Rationale for this change I found there are many warnings in the project. It looks like we don't run "cargo check"

[I] Allow expr_to_sql unparsing with no quotes [datafusion]

2024-04-23 Thread via GitHub
phillipleblanc opened a new issue, #10197: URL: https://github.com/apache/datafusion/issues/10197 ### Describe the bug I'm using the Unparser.expr_to_sql function to translate a DataFusion `Expr` into an expression for Spark Connect. Unlike most SQL engines, Spark doesn't like quotin

Re: [PR] fix: cargo warnings of import item [datafusion]

2024-04-23 Thread via GitHub
waynexia commented on code in PR #10196: URL: https://github.com/apache/datafusion/pull/10196#discussion_r1576371100 ## .github/workflows/rust.yml: ## @@ -77,7 +77,7 @@ jobs: run: cargo check --all-targets --no-default-features -p datafusion-functions - name:

[PR] Allow expr_to_sql unparsing with no quotes [datafusion]

2024-04-23 Thread via GitHub
phillipleblanc opened a new pull request, #10198: URL: https://github.com/apache/datafusion/pull/10198 ## Which issue does this PR close? Closes #10197 ## Rationale for this change I'm using the Unparser.expr_to_sql function to translate a DataFusion Expr into an express

Re: [PR] chore: Remove unused functions [datafusion-comet]

2024-04-23 Thread via GitHub
andygrove merged PR #301: URL: https://github.com/apache/datafusion-comet/pull/301 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@da

Re: [I] try to add local limit after the optimizer rule `join_selection` [datafusion]

2024-04-23 Thread via GitHub
Lordworms commented on issue #10170: URL: https://github.com/apache/datafusion/issues/10170#issuecomment-2072517570 maybe related https://github.com/apache/datafusion/issues/9792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] "Cannot infer common argument type for comparison operation Union..." [datafusion]

2024-04-23 Thread via GitHub
samuelcolvin commented on issue #10180: URL: https://github.com/apache/datafusion/issues/10180#issuecomment-2072538449 @jayzhan211 that doesn't work since the argument types don't tell you want type will be returned. e.g.: * if the value in column `foo` is `{"x": "abc"}`, then `jso

Re: [PR] fix: CometExec's outputPartitioning might not be same as Spark expects after AQE interferes [datafusion-comet]

2024-04-23 Thread via GitHub
andygrove commented on code in PR #299: URL: https://github.com/apache/datafusion-comet/pull/299#discussion_r1576395895 ## spark/src/main/scala/org/apache/spark/sql/comet/operators.scala: ## @@ -471,6 +488,9 @@ case class CometLocalLimitExec( child: SparkPlan, override

Re: [PR] fix: Iceberg scan transition should be in front of other data source v2 [datafusion-comet]

2024-04-23 Thread via GitHub
viirya merged PR #302: URL: https://github.com/apache/datafusion-comet/pull/302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] fix: Iceberg scan transition should be in front of other data source v2 [datafusion-comet]

2024-04-23 Thread via GitHub
viirya commented on PR #302: URL: https://github.com/apache/datafusion-comet/pull/302#issuecomment-2072546044 Merged. Thanks @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] fix: CometExec's outputPartitioning might not be same as Spark expects after AQE interferes [datafusion-comet]

2024-04-23 Thread via GitHub
viirya commented on code in PR #299: URL: https://github.com/apache/datafusion-comet/pull/299#discussion_r1576413764 ## spark/src/main/scala/org/apache/spark/sql/comet/operators.scala: ## @@ -471,6 +488,9 @@ case class CometLocalLimitExec( child: SparkPlan, override va

Re: [PR] feat: Support murmur3_hash and sha2 family hash functions [datafusion-comet]

2024-04-23 Thread via GitHub
viirya commented on code in PR #226: URL: https://github.com/apache/datafusion-comet/pull/226#discussion_r1576421464 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -983,8 +983,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPl

Re: [PR] feat: Support murmur3_hash and sha2 family hash functions [datafusion-comet]

2024-04-23 Thread via GitHub
viirya commented on code in PR #226: URL: https://github.com/apache/datafusion-comet/pull/226#discussion_r1576425396 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -983,8 +983,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPl

[PR] chore: Ignore unused variables [datafusion-comet]

2024-04-23 Thread via GitHub
snmvaughan opened a new pull request, #306: URL: https://github.com/apache/datafusion-comet/pull/306 ## Which issue does this PR close? Address scalafix identified errors ## Rationale for this change This ignore unused variables ## What changes are included in this

Re: [PR] feat: Implement Spark-compatible CAST from string to integral types [datafusion-comet]

2024-04-23 Thread via GitHub
andygrove commented on PR #307: URL: https://github.com/apache/datafusion-comet/pull/307#issuecomment-207292 I am now working on refactoring to reduce code duplication by leveraging macros/generics. -- This is an automated message from the Apache Git Service. To respond to the message

[PR] Fix docs on datafusion names [datafusion]

2024-04-23 Thread via GitHub
comphead opened a new pull request, #10199: URL: https://github.com/apache/datafusion/pull/10199 ## Which issue does this PR close? Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested?

Re: [PR] Fix docs on datafusion names [datafusion]

2024-04-23 Thread via GitHub
comphead commented on PR #10199: URL: https://github.com/apache/datafusion/pull/10199#issuecomment-2072673087 Related to https://github.com/apache/datafusion/issues/9691 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[I] `select array_concat([])` panicked [datafusion]

2024-04-23 Thread via GitHub
jonahgao opened a new issue, #10200: URL: https://github.com/apache/datafusion/issues/10200 ### Describe the bug thread 'main' panicked at datafusion/datafusion/functions-array/src/concat.rs:275:32: index out of bounds: the len is 0 but the index is 0 ### To Reproduce

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-04-23 Thread via GitHub
comphead commented on issue #9691: URL: https://github.com/apache/datafusion/issues/9691#issuecomment-2072699184 We should rename slack and discord channels? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #8373: URL: https://github.com/apache/datafusion/issues/8373#issuecomment-2072704562 And apparently I still didn't get it entirely correct: ``` -- Forwarded message - From: Tim Pollitt Date: Tue, Apr 23, 2024 at 8:46 AM Subject: ACM

[PR] Move coalesce function from math to core [datafusion]

2024-04-23 Thread via GitHub
xxxuuu opened a new pull request, #10201: URL: https://github.com/apache/datafusion/pull/10201 ## Which issue does this PR close? closes https://github.com/apache/datafusion/issues/10174 ## Rationale for this change The current location of the coalesce function isn't real

Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #8373: URL: https://github.com/apache/datafusion/issues/8373#issuecomment-2072745195 "third time's the charm" https://github.com/apache/datafusion/assets/490673/76350044-eabf-4407-83a5-c8cedaa6c69a";> [DataFusion Query Engine - SIGMOD 2024-FINAL-mk3.zip

Re: [PR] Fix docs on datafusion names [datafusion]

2024-04-23 Thread via GitHub
andygrove commented on code in PR #10199: URL: https://github.com/apache/datafusion/pull/10199#discussion_r1576483028 ## dev/release/README.md: ## @@ -249,7 +249,7 @@ NOTE: steps in this section can only be done by PMC members. ### After the release is approved Move artifact

Re: [PR] fix: CometExec's outputPartitioning might not be same as Spark expects after AQE interferes [datafusion-comet]

2024-04-23 Thread via GitHub
viirya commented on PR #299: URL: https://github.com/apache/datafusion-comet/pull/299#issuecomment-2072758871 Thank you @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Fix docs on datafusion names [datafusion]

2024-04-23 Thread via GitHub
andygrove commented on code in PR #10199: URL: https://github.com/apache/datafusion/pull/10199#discussion_r1576483296 ## dev/release/README.md: ## @@ -437,7 +437,7 @@ svn ls https://dist.apache.org/repos/dist/dev/arrow | grep datafusion Delete a release candidate: ```bash -

Re: [PR] Fix docs on datafusion names [datafusion]

2024-04-23 Thread via GitHub
andygrove commented on code in PR #10199: URL: https://github.com/apache/datafusion/pull/10199#discussion_r1576483681 ## dev/release/README.md: ## @@ -453,7 +453,7 @@ svn ls https://dist.apache.org/repos/dist/release/arrow | grep datafusion Delete a release: ```bash -svn de

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-04-23 Thread via GitHub
alamb commented on issue #9691: URL: https://github.com/apache/datafusion/issues/9691#issuecomment-2072761667 > this link currently still works and doesn't redirect -- will it redirect to the new one at some point? https://arrow.apache.org/datafusion/ @lostmygithubaccount (😆 ) Yes ab

Re: [PR] Fix docs on datafusion names [datafusion]

2024-04-23 Thread via GitHub
andygrove commented on code in PR #10199: URL: https://github.com/apache/datafusion/pull/10199#discussion_r1576486451 ## docs/source/contributor-guide/communication.md: ## @@ -37,7 +37,7 @@ We use the Slack and Discord platforms for informal discussions and coordination meet o

Re: [PR] Update links to point to datafusion.apache.org [datafusion]

2024-04-23 Thread via GitHub
andygrove commented on code in PR #10195: URL: https://github.com/apache/datafusion/pull/10195#discussion_r1576492917 ## README.md: ## @@ -46,10 +46,10 @@ in-memory format. [Python Bindings](https://github.com/apache/datafusion-python) Here are links to some important inform

[PR] [MINOR] Remove ScalarFunction from datafusion.proto #10173 [datafusion]

2024-04-23 Thread via GitHub
dmitrybugakov opened a new pull request, #10202: URL: https://github.com/apache/datafusion/pull/10202 ## Which issue does this PR close? #10173 ## Rationale for this change ## What changes are included in this PR? ## Are these changes tested

Re: [PR] Fix docs on datafusion names [datafusion]

2024-04-23 Thread via GitHub
comphead commented on PR #10199: URL: https://github.com/apache/datafusion/pull/10199#issuecomment-2072800414 Thanks for the review @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] CometExec's outputPartitioning might not be same as Spark expects after AQE interferes [datafusion-comet]

2024-04-23 Thread via GitHub
viirya closed issue #298: CometExec's outputPartitioning might not be same as Spark expects after AQE interferes URL: https://github.com/apache/datafusion-comet/issues/298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] fix: CometExec's outputPartitioning might not be same as Spark expects after AQE interferes [datafusion-comet]

2024-04-23 Thread via GitHub
viirya merged PR #299: URL: https://github.com/apache/datafusion-comet/pull/299 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

Re: [PR] fix: CometExec's outputPartitioning might not be same as Spark expects after AQE interferes [datafusion-comet]

2024-04-23 Thread via GitHub
viirya commented on PR #299: URL: https://github.com/apache/datafusion-comet/pull/299#issuecomment-2072824296 Merged. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] feat: Support murmur3_hash and sha2 family hash functions [datafusion-comet]

2024-04-23 Thread via GitHub
advancedxy commented on code in PR #226: URL: https://github.com/apache/datafusion-comet/pull/226#discussion_r1576520191 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -983,8 +983,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSpa

Re: [PR] feat: Support murmur3_hash and sha2 family hash functions [datafusion-comet]

2024-04-23 Thread via GitHub
advancedxy commented on code in PR #226: URL: https://github.com/apache/datafusion-comet/pull/226#discussion_r1576529455 ## spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala: ## @@ -983,8 +983,7 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSpa

Re: [PR] chore: Remove unused functions [datafusion-comet]

2024-04-23 Thread via GitHub
kazuyukitanimura commented on PR #301: URL: https://github.com/apache/datafusion-comet/pull/301#issuecomment-2072961155 Thank you all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] implement rewrite for FilterNullJoinKeys [datafusion]

2024-04-23 Thread via GitHub
alamb merged PR #10166: URL: https://github.com/apache/datafusion/pull/10166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] implement rewrite for FilterNullJoinKeys [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10166: URL: https://github.com/apache/datafusion/pull/10166#issuecomment-2073002072 Thanks again @Lordworms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] Implement rewrite for EliminateOneUnion and EliminateJoin [datafusion]

2024-04-23 Thread via GitHub
alamb merged PR #10184: URL: https://github.com/apache/datafusion/pull/10184 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Move coalesce function from math to core [datafusion]

2024-04-23 Thread via GitHub
alamb commented on PR #10201: URL: https://github.com/apache/datafusion/pull/10201#issuecomment-2073010872 Thank you for the contribution @xxxuuu ❤️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

  1   2   >