(datafusion) branch main updated: Update NOTICE.txt to be relevant to DataFusion (#10185)

2024-04-22 Thread agrove
This is an automated email from the ASF dual-hosted git repository. agrove pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 711239fb13 Update NOTICE.txt to be relevant to

Re: [PR] Update .asf.yaml to point to new mailing list [datafusion]

2024-04-22 Thread via GitHub
phillipleblanc commented on PR #10189: URL: https://github.com/apache/datafusion/pull/10189#issuecomment-2071303961 > We should also update: > > ``` > homepage: https://arrow.apache.org/datafusion > ``` I'm working on a separate PR to update those - we need to add the

(datafusion) branch asf-site updated: Publish built docs triggered by 2eb8f12db4096f4ad633987ce4cd0a5bd6db6757

2024-04-22 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/asf-site by this push: new 94b4e0e816 Publish built docs

Re: [PR] Update .asf.yaml to point to new mailing list [datafusion]

2024-04-22 Thread via GitHub
andygrove merged PR #10189: URL: https://github.com/apache/datafusion/pull/10189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(datafusion) branch main updated: Update .asf.yaml to configure new mailing list (#10189)

2024-04-22 Thread agrove
This is an automated email from the ASF dual-hosted git repository. agrove pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 2eb8f12db4 Update .asf.yaml to configure new

Re: [PR] Update .asf.yaml to point to new mailing list [datafusion]

2024-04-22 Thread via GitHub
viirya commented on PR #10189: URL: https://github.com/apache/datafusion/pull/10189#issuecomment-2071296255 And ``` # publishes the content of the `asf-site` branch to # https://arrow.apache.org/datafusion/ ``` -- This is an automated message from the Apache Git Service.

Re: [I] [EPIC] Tasks for a new Top Level Apache Project [datafusion]

2024-04-22 Thread via GitHub
phillipleblanc commented on issue #9691: URL: https://github.com/apache/datafusion/issues/9691#issuecomment-2071292961 Update `.asf.yaml` to point to to the new mailing list: https://github.com/apache/datafusion/pull/10189 I'm also taking a look at

[PR] Update .asf.yaml to point to new mailing list [datafusion]

2024-04-22 Thread via GitHub
phillipleblanc opened a new pull request, #10189: URL: https://github.com/apache/datafusion/pull/10189 ## Which issue does this PR close? Part of #9691 ## Rationale for this change See #9691 ## What changes are included in this PR? Updates `.asf.yaml` to

Re: [I] [EPIC] Add Decimal support [datafusion]

2024-04-22 Thread via GitHub
liukun4515 commented on issue #3523: URL: https://github.com/apache/datafusion/issues/3523#issuecomment-2071292078 > I wonder if we should claim this ticket is complete? I wonder if there is anything else this ticket is tracking I will check above unclosed issue in this week. --

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on code in PR #10125: URL: https://github.com/apache/datafusion/pull/10125#discussion_r1575552547 ## datafusion/execution/src/cache/cache_unit.rs: ## @@ -232,4 +337,64 @@ mod tests { meta.clone() ); } +#[test] Review

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
westonpace commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2071274206 > I don't think there is any particular motivation (or any reason that the conversion needs to be done at either spot) 樂 I think, for me, it's just a general unease

Re: [PR] Add benchmark for DFSchema [datafusion]

2024-04-22 Thread via GitHub
github-actions[bot] commented on PR #7948: URL: https://github.com/apache/datafusion/pull/7948#issuecomment-2071259258 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] Support AND operator as alias of array intersect function [datafusion]

2024-04-22 Thread via GitHub
github-actions[bot] closed pull request #8496: Support AND operator as alias of array intersect function URL: https://github.com/apache/datafusion/pull/8496 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Add mechanism for verifying that source code in documentation is valid [datafusion]

2024-04-22 Thread via GitHub
github-actions[bot] commented on PR #7956: URL: https://github.com/apache/datafusion/pull/7956#issuecomment-2071259218 Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or

Re: [PR] feat: support scientific notation in `parse_float_as_decimal ` [datafusion]

2024-04-22 Thread via GitHub
github-actions[bot] closed pull request #8494: feat: support scientific notation in `parse_float_as_decimal ` URL: https://github.com/apache/datafusion/pull/8494 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Move `create_physical_expr` to `phy-expr-common` #1 [datafusion]

2024-04-22 Thread via GitHub
jayzhan211 commented on PR #10144: URL: https://github.com/apache/datafusion/pull/10144#issuecomment-2071213734 I created an ALL-in-one PR in #10188 to know what it is like after moving all the create-expr function -- This is an automated message from the Apache Git Service. To respond

[PR] Move create_physical_expr to phy-expr-common #3 [datafusion]

2024-04-22 Thread via GitHub
jayzhan211 opened a new pull request, #10188: URL: https://github.com/apache/datafusion/pull/10188 ## Which issue does this PR close? Closes #10074 All in one in this PR. ## Rationale for this change ## What changes are included in this PR?

Re: [PR] Minor: Possibility to strip datafusion error name [datafusion]

2024-04-22 Thread via GitHub
comphead commented on code in PR #10186: URL: https://github.com/apache/datafusion/pull/10186#discussion_r1575504885 ## datafusion/common/src/error.rs: ## @@ -778,6 +817,20 @@ mod test { ); } +#[test] +fn test_strip_error_name() { +let res:

Re: [PR] Minor: Possibility to strip datafusion error name [datafusion]

2024-04-22 Thread via GitHub
edmondop commented on code in PR #10186: URL: https://github.com/apache/datafusion/pull/10186#discussion_r1575501494 ## datafusion/common/src/error.rs: ## @@ -778,6 +817,20 @@ mod test { ); } +#[test] +fn test_strip_error_name() { +let res:

Re: [I] Old URL for CLI docs page is showing 404 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10124: URL: https://github.com/apache/datafusion/issues/10124#issuecomment-2071176791 (I hit this issue when google searching today so figured I would make a PR) This is what I came up with : https://github.com/apache/datafusion/pull/10187 -- This is

Re: [PR] Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10187: URL: https://github.com/apache/datafusion/pull/10187#discussion_r1575500022 ## docs/source/index.rst: ## @@ -103,3 +103,9 @@ Please see the `developer’s guide`_ for contributing and `communication`_ for contributor-guide/roadmap

Re: [PR] Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10187: URL: https://github.com/apache/datafusion/pull/10187#discussion_r1575500022 ## docs/source/index.rst: ## @@ -103,3 +103,9 @@ Please see the `developer’s guide`_ for contributing and `communication`_ for contributor-guide/roadmap

[PR] Add redirect to old cli location `user-guide/cli.html` --> `user-guide/cli/index.html` [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new pull request, #10187: URL: https://github.com/apache/datafusion/pull/10187 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/10124 ## Rationale for this change Old links (and google) direct to cli.html, which I

Re: [PR] Minor: Possibility to strip datafusion error name [datafusion]

2024-04-22 Thread via GitHub
comphead commented on PR #10186: URL: https://github.com/apache/datafusion/pull/10186#issuecomment-2071162551 The implementation might not be super elegant though, please share your thoughts and opininions -- This is an automated message from the Apache Git Service. To respond to the

[PR] Minor: Possibility to strip datafusion error name [datafusion]

2024-04-22 Thread via GitHub
comphead opened a new pull request, #10186: URL: https://github.com/apache/datafusion/pull/10186 ## Which issue does this PR close? Closes #. ## Rationale for this change Currently Datafusion wraps the original error message and includes Datafusion name on top of

Re: [PR] Minor: Possibility to strip datafusion error name [datafusion]

2024-04-22 Thread via GitHub
comphead commented on PR #10186: URL: https://github.com/apache/datafusion/pull/10186#issuecomment-2071162770 @andygrove cc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Update NOTICE.txt to be relevant to DataFusion [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10185: URL: https://github.com/apache/datafusion/pull/10185#discussion_r1575486264 ## NOTICE.txt: ## @@ -1,84 +1,5 @@ -Apache Arrow -Copyright 2016-2019 The Apache Software Foundation +Apache DataFusion +Copyright 2019-2024 The Apache Software

[PR] Update NOTICE.txt to be relevant to DataFusion [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new pull request, #10185: URL: https://github.com/apache/datafusion/pull/10185 ## Which issue does this PR close? Closes https://github.com/apache/datafusion/issues/10131 ## Rationale for this change It seems like `NOTICE.txt` was brought from the arrow

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2071134346 > What's the motivation for doing this at the logical level instead of doing this as part of the conversion from logical to physical? I don't think there is any particular

(datafusion) branch main updated: implement rewrite for ExtractEquijoinPredicate and avoid clone in filter (#10165)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new d9ebd2b83b implement rewrite for

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] implement rewrite for ExtractEquijoinPredicate and avoid clone in filter [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10165: URL: https://github.com/apache/datafusion/pull/10165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10182: URL: https://github.com/apache/datafusion/issues/10182#issuecomment-2071125951 樂 I tested it locally and likewise it seems to work just fine for me in a debug build I rebuilt a release build and it works fine. Not sure what was going on on my

Re: [I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
alamb closed issue #10182: `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo URL: https://github.com/apache/datafusion/issues/10182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on code in PR #10125: URL: https://github.com/apache/datafusion/pull/10125#discussion_r1575451507 ## datafusion/execution/src/cache/cache_unit.rs: ## @@ -232,4 +337,64 @@ mod tests { meta.clone() ); } +#[test] Review Comment:

[PR] Implement rewrite for EliminateOneUnion and EliminateJoin [datafusion]

2024-04-22 Thread via GitHub
Lordworms opened a new pull request, #10184: URL: https://github.com/apache/datafusion/pull/10184 ## Which issue does this PR close? part of #9637 Closes #. ## Rationale for this change ## What changes are included in this PR? ## Are these

Re: [I] [EPIC] Improve the performance of ListingTable [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on issue #9964: URL: https://github.com/apache/datafusion/issues/9964#issuecomment-2071090146 > @Lordworms did you get the chance to compare querying with a filter / pruning involved (ideally with a range) between dashmap and sequence trie? Not sure if the dataset is

Re: [I] [EPIC] Improve the performance of ListingTable [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on issue #9964: URL: https://github.com/apache/datafusion/issues/9964#issuecomment-2071088353 @Lordworms did you get the chance to compare querying with a filter / pruning involved (ideally with a range) between dashmap and sequence trie? Not sure if the dataset is

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on code in PR #10125: URL: https://github.com/apache/datafusion/pull/10125#discussion_r1575440108 ## datafusion/execution/src/cache/cache_unit.rs: ## @@ -232,4 +337,64 @@ mod tests { meta.clone() ); } +#[test] Review

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on PR #10125: URL: https://github.com/apache/datafusion/pull/10125#issuecomment-2071082691 @Lordworms i added one comment, i plan to review more later / tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [PR] LRU DashMap to cache objectMeta [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on code in PR #10125: URL: https://github.com/apache/datafusion/pull/10125#discussion_r1575437882 ## datafusion/execution/src/cache/cache_unit.rs: ## @@ -232,4 +337,64 @@ mod tests { meta.clone() ); } +#[test] Review

Re: [I] [EPIC] Improve the performance of ListingTable [datafusion]

2024-04-22 Thread via GitHub
matthewmturner commented on issue #9964: URL: https://github.com/apache/datafusion/issues/9964#issuecomment-2071068360 @Lordworms apologies it took me longer than expected to get some free time. I plan to review between tonight and tomorrow. -- This is an automated message from the

Re: [I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on issue #10182: URL: https://github.com/apache/datafusion/issues/10182#issuecomment-2071058523 I used debug mode to build this and the query worked fine https://github.com/apache/datafusion/assets/48054792/aa6c1b59-c28a-4041-8897-0f1d257b72cb;>

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
westonpace commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2071008009 > I think this can be controlled by the consumer -- for example if you are walking Exprs in lancedb, you can control when you transform Expr::GetStructField into ScalarUDF

Re: [I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on issue #10182: URL: https://github.com/apache/datafusion/issues/10182#issuecomment-2071007541 Take this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Any plan to support JSON or JSONB? [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #7845: URL: https://github.com/apache/datafusion/issues/7845#issuecomment-2071004118 Thanks @WenyXu -- sounds very neat. FWI I think @samuelcolvin is also thinking about the representation in

Re: [PR] Add example of using Expr::field in `37.1.0` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10183: URL: https://github.com/apache/datafusion/pull/10183#discussion_r1575378141 ## datafusion-examples/examples/expr_api.rs: ## @@ -248,18 +296,35 @@ fn make_ts_field(name: ) -> Field { make_field(name,

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070993030 > The "Expr walking" code now has to be aware of both the GetStructField and the ScalarUDF variants of field access. I think this can be controlled by the consumer -- for

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070989408 Here is an example of how to make Expr::struct work in 37.1.0: https://github.com/apache/datafusion/pull/10183 I think we need a better API to do this for real (in 38.0.0

Re: [PR] Add example of using Expr::field in `37.1.0` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10183: URL: https://github.com/apache/datafusion/pull/10183#discussion_r1575378141 ## datafusion-examples/examples/expr_api.rs: ## @@ -248,18 +296,35 @@ fn make_ts_field(name: ) -> Field { make_field(name,

[PR] Add example of using Expr::field in `37.1.0` [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new pull request, #10183: URL: https://github.com/apache/datafusion/pull/10183 I don't intend to merge this, but I want to put it up to illustrate how to use the 37.1.0 API to do this reasonably ## Which issue does this PR close? Part of

Re: [PR] implement rewrite for ExtractEquijoinPredicate and avoid clone in filter [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on code in PR #10165: URL: https://github.com/apache/datafusion/pull/10165#discussion_r1575372225 ## datafusion/optimizer/src/extract_equijoin_predicate.rs: ## @@ -67,69 +88,100 @@ impl OptimizerRule for ExtractEquijoinPredicate { }) => {

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
westonpace commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070947738 This also leads to a sort of "variant problem" for any code we write that handles `Expr`. For example, we have code that walks through an `Expr` and calculates which

Re: [PR] implement rewrite for ExtractEquijoinPredicate and avoid clone in filter [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10165: URL: https://github.com/apache/datafusion/pull/10165#discussion_r1575341213 ## datafusion/optimizer/src/extract_equijoin_predicate.rs: ## @@ -67,66 +88,97 @@ impl OptimizerRule for ExtractEquijoinPredicate { }) => { Review

Re: [PR] Array agg groups accumulator [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10149: URL: https://github.com/apache/datafusion/pull/10149#issuecomment-2070918994 Thank you for this PR @lkt -- I will take a look at this one carefully shortly -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] DataFusion weekly project plan (Andrew Lamb) - April 22, 2024 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10172: URL: https://github.com/apache/datafusion/issues/10172#issuecomment-2070919255 Review queue: - [ ] https://github.com/apache/datafusion/pull/10149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070917587 I'll work on creating an example shortly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[I] `create table` fails in datafusion-cli with External error: Failed to convert path to URL: foo [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new issue, #10182: URL: https://github.com/apache/datafusion/issues/10182 ### Describe the bug Something is wrong with datafusion-cli and creating external tables ### To Reproduce ``` andrewlamb@Andrews-MacBook-Pro:~/Software/arrow-datafusion$

Re: [PR] Improve documentation on `TreeNode` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10035: URL: https://github.com/apache/datafusion/pull/10035#issuecomment-2070900137   -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

(datafusion) branch main updated: Improve documentation on `TreeNode` (#10035)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 16369d8612 Improve documentation on `TreeNode`

Re: [PR] Improve documentation on `TreeNode` [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10035: URL: https://github.com/apache/datafusion/pull/10035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
ion-elgreco commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070893136 @alamb this is how we create expressions: ```rust /// Parse a string predicate into an `Expr` pub(crate) fn parse_predicate_expression( schema: , expr:

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
westonpace commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070887996 I think I'd be happy with 2. The example you linked is how we are using datafusion. Here is an updated example that fails with the error: ``` // For example,

Re: [I] Consider introducing unique expression IDs in Logical/Physical plan [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #8379: URL: https://github.com/apache/datafusion/issues/8379#issuecomment-2070886046 @tv42 I agree it is certainly related -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] implement short_circuits function for ScalarUDFImpl trait [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10168: URL: https://github.com/apache/datafusion/pull/10168 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(datafusion) branch main updated: implement short_circuits function for ScalarUDFImpl trait (#10168)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new f5ab312165 implement short_circuits function for

Re: [I] Adding the new API `is_short_circuits()`(default to false, when need set to true) to `ScalarUDF` and `ScalarUDFImpl` might be a good way to do this because users may want to define their own s

2024-04-22 Thread via GitHub
alamb closed issue #10162: Adding the new API `is_short_circuits()`(default to false, when need set to true) to `ScalarUDF` and `ScalarUDFImpl` might be a good way to do this because users may want to define their own short-circuit functions. URL:

Re: [PR] Move coalesce to datafusion-functions and remove BuiltInScalarFunction [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10098: URL: https://github.com/apache/datafusion/pull/10098#issuecomment-2070877181 Thanks @Omega359 for filing follow on tickets https://github.com/apache/datafusion/issues/10173 https://github.com/apache/datafusion/issues/10174

Re: [PR] Move coalesce to datafusion-functions and remove BuiltInScalarFunction [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10098: URL: https://github.com/apache/datafusion/pull/10098#discussion_r1575310332 ## datafusion/expr/src/expr.rs: ## @@ -362,10 +362,6 @@ impl Between { #[derive(Debug, Clone, PartialEq, Eq, Hash)] /// Defines which implementation of a function

Re: [PR] implement rewrite for FilterNullJoinKeys [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10166: URL: https://github.com/apache/datafusion/pull/10166#discussion_r1575308618 ## datafusion/optimizer/src/filter_null_join_keys.rs: ## @@ -100,11 +105,9 @@ fn create_not_null_predicate(filters: Vec) -> Expr { .into_iter()

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070872284 I see a few options: 1. We can put `get_field` back into the core crate (and move it out of datafusion-functions) - might be the least surprising but would not allow people to

Re: [I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #10181: URL: https://github.com/apache/datafusion/issues/10181#issuecomment-2070864010 Example from @ion-elgreco @alamb this is the code: ``` let (table, _metrics) = DeltaOps(table) .delete()

Re: [I] Release DataFusion `37.1.0` (non breaking API release) [datafusion]

2024-04-22 Thread via GitHub
alamb commented on issue #9904: URL: https://github.com/apache/datafusion/issues/9904#issuecomment-2070863362 Filed https://github.com/apache/datafusion/issues/10181 to track the issue with `internal error: entered unreachable code: NamedStructField should be rewritten in

[I] Error "entered unreachable code: NamedStructField should be rewritten in OperatorToFunction" after upgrade to 37 [datafusion]

2024-04-22 Thread via GitHub
alamb opened a new issue, #10181: URL: https://github.com/apache/datafusion/issues/10181 ### Is your feature request related to a problem or challenge? In 37.0.0 many of the built in functions have been migrated to `UDF`s as described on #8045 . The migration is completed in 38.0.0

(datafusion-comet) branch main updated: doc: Update DataFusion project name and url (#300)

2024-04-22 Thread viirya
This is an automated email from the ASF dual-hosted git repository. viirya pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git The following commit(s) were added to refs/heads/main by this push: new de42975 doc: Update DataFusion project

(datafusion-comet) branch main updated: feat: Add extended explain info to Comet plan (#255)

2024-04-22 Thread viirya
This is an automated email from the ASF dual-hosted git repository. viirya pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git The following commit(s) were added to refs/heads/main by this push: new 6d01f6a feat: Add extended explain info

Re: [PR] implement rewrite for FilterNullJoinKeys [datafusion]

2024-04-22 Thread via GitHub
Lordworms commented on code in PR #10166: URL: https://github.com/apache/datafusion/pull/10166#discussion_r1575279632 ## datafusion/optimizer/src/filter_null_join_keys.rs: ## @@ -100,11 +105,18 @@ fn create_not_null_predicate(filters: Vec) -> Expr { .into_iter()

Re: [PR] Minor: Add `Column::from(Tableref, )`, `Expr::from(Column)` and `Expr::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10178: URL: https://github.com/apache/datafusion/pull/10178#issuecomment-2070804781 > lgtm thanks @alamb CIL testdoc keeps failing Thanks -- fixed in 4db07ee31 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [PR] implement rewrite for FilterNullJoinKeys [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10166: URL: https://github.com/apache/datafusion/pull/10166#discussion_r1575275630 ## datafusion/optimizer/src/filter_null_join_keys.rs: ## @@ -100,11 +105,18 @@ fn create_not_null_predicate(filters: Vec) -> Expr { .into_iter()

(datafusion) branch main updated: Projection Expression - Input Field Inconsistencies during Projection (#10088)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 07804384cb Projection Expression - Input Field

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10088: URL: https://github.com/apache/datafusion/pull/10088#discussion_r1575269612 ## datafusion/functions-aggregate/src/first_last.rs: ## @@ -895,6 +891,31 @@ fn convert_to_sort_cols( .collect::>() } +fn

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10088: URL: https://github.com/apache/datafusion/pull/10088 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10088: URL: https://github.com/apache/datafusion/pull/10088#issuecomment-2070782001 Thanks @berkaysynnada -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10088: URL: https://github.com/apache/datafusion/pull/10088#discussion_r1575265148 ## datafusion/functions-aggregate/src/first_last.rs: ## @@ -895,6 +891,31 @@ fn convert_to_sort_cols( .collect::>() } +fn

Re: [PR] Projection Expression - Input Field Inconsistencies during Projection [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10088: URL: https://github.com/apache/datafusion/pull/10088#issuecomment-2070754771 Is this PR waiting on anything prior to merge? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Improve `TreeNode` and `LogicalPlan` APIs to accept owned closures, deprecate `transform_down_mut()` and `transform_up_mut()` [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10126: URL: https://github.com/apache/datafusion/pull/10126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Improve `TreeNode` and `LogicalPlan` APIs to accept owned closures, deprecate `transform_down_mut()` and `transform_up_mut()` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on PR #10126: URL: https://github.com/apache/datafusion/pull/10126#issuecomment-2070731173 Thanks again @peter-toth -- epic work -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [I] Deprecate TreeNode `transform_xx_mut` methods [datafusion]

2024-04-22 Thread via GitHub
alamb closed issue #10097: Deprecate TreeNode `transform_xx_mut` methods URL: https://github.com/apache/datafusion/issues/10097 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

(datafusion) branch main updated: Improve `TreeNode` and `LogicalPlan` APIs to accept owned closures, deprecate `transform_down_mut()` and `transform_up_mut()` (#10126)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 3c3cb87198 Improve `TreeNode` and `LogicalPlan`

(datafusion) branch main updated: fix installation section link (#10179)

2024-04-22 Thread alamb
This is an automated email from the ASF dual-hosted git repository. alamb pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/datafusion.git The following commit(s) were added to refs/heads/main by this push: new 44bc16337f fix installation section link

Error while running notifications feature from .asf.yaml in datafusion!

2024-04-22 Thread Apache Infrastructure
An error occurred while running notifications feature in .asf.yaml!: Invalid notification target 'comm...@arrow.apache.org'. Must be a valid @datafusion.apache.org list! - To unsubscribe, e-mail:

Re: [PR] minor: fix installation section link [datafusion]

2024-04-22 Thread via GitHub
alamb merged PR #10179: URL: https://github.com/apache/datafusion/pull/10179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Minor: Add `Column::from(Tableref, )`, `Expr::from(Column)` and `Expr::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10178: URL: https://github.com/apache/datafusion/pull/10178#discussion_r1575252042 ## benchmarks/src/tpch/convert.rs: ## @@ -88,9 +88,7 @@ impl ConvertOpt { .schema() .iter()

Re: [PR] Minor: Add `Column::from(Tableref, )` [datafusion]

2024-04-22 Thread via GitHub
alamb commented on code in PR #10178: URL: https://github.com/apache/datafusion/pull/10178#discussion_r1575249802 ## benchmarks/src/tpch/convert.rs: ## @@ -88,9 +88,8 @@ impl ConvertOpt { .schema() .iter()

Re: [PR] feat: support input reordering for `NestedLoopJoinExec` [datafusion]

2024-04-22 Thread via GitHub
Dandandan commented on PR #9676: URL: https://github.com/apache/datafusion/pull/9676#issuecomment-2070626124 Thank you @korowa  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Range/inequality joins are slow [datafusion]

2024-04-22 Thread via GitHub
Dandandan commented on issue #8393: URL: https://github.com/apache/datafusion/issues/8393#issuecomment-2070623786 I don't think this issue should be closed. #9676 seems to take care of ordering but I think it doesn't improve range/inequality joins much? -- This is an automated

  1   2   >