Re: [PR] test: Fix flaky join tests [datafusion]

findepi commented on code in PR #16860: URL: https://github.com/apache/datafusion/pull/16860#discussion_r2225377710 ## datafusion/sqllogictest/test_files/joins.slt: ## @@ -4164,23 +4164,40 @@ AS VALUES (3, 3, true), (3, 3, false); -query B -SELECT * FROM t0 FULL JOIN t1

Re: [PR] MINOR: add unit tests for chr function [datafusion]

waynexia commented on code in PR #16856: URL: https://github.com/apache/datafusion/pull/16856#discussion_r2225842872 ## datafusion/functions/src/string/chr.rs: ## @@ -132,3 +134,79 @@ impl ScalarUDFImpl for ChrFunc { self.doc() } } + +#[cfg(test)] +mod tests { +

Re: [PR] Chore: refactor Comparison out of QueryPlanSerde [datafusion-comet]

mbutrovich commented on code in PR #2028: URL: https://github.com/apache/datafusion-comet/pull/2028#discussion_r2225843321 ## spark/src/main/scala/org/apache/comet/serde/comparisons.scala: ## @@ -0,0 +1,265 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one +

[PR] fix(datafusion-proto): support serializing/deserilizing ArrowFormat tables [datafusion]

colinmarc opened a new pull request, #16875: URL: https://github.com/apache/datafusion/pull/16875 ## Which issue does this PR close? - Closes #16874 ## What changes are included in this PR? Added protobuf support for ArrowFormat. ## Are these changes teste

Re: [PR] Fix `next_up` and `next_down` behavior for zero float values [datafusion]

findepi commented on PR #16745: URL: https://github.com/apache/datafusion/pull/16745#issuecomment-3109033105 > I mean SQL spec and what DataFusion should be implementing. I wish DataFusion followed SQL spec in everything, but that's not the project design philosophy AFAICT. That wa

Re: [PR] MINOR: add unit tests for chr function [datafusion]

findepi commented on code in PR #16856: URL: https://github.com/apache/datafusion/pull/16856#discussion_r2225398089 ## datafusion/functions/src/string/chr.rs: ## @@ -132,3 +134,79 @@ impl ScalarUDFImpl for ChrFunc { self.doc() } } + +#[cfg(test)] +mod tests { +

Re: [I] Upgrade to sqlparser 0.56.0 [datafusion]

barakalon commented on issue #16405: URL: https://github.com/apache/datafusion/issues/16405#issuecomment-3108756695 Can we jump straight to [0.57.0](https://crates.io/crates/sqlparser/0.57.0)? 😬 -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[PR] Perf: Optimize vectorized append function [datafusion]

zhuqi-lucas opened a new pull request, #16876: URL: https://github.com/apache/datafusion/pull/16876 ## Which issue does this PR close? - Closes part of https://github.com/apache/datafusion/issues/16330 ## Rationale for this change Optimize vectorized append functi

Re: [PR] Update extending-operators.md [datafusion]

Adez017 closed pull request #15832: Update extending-operators.md URL: https://github.com/apache/datafusion/pull/15832 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Perf: Optimize vectorized append function [datafusion]

zhuqi-lucas commented on PR #16876: URL: https://github.com/apache/datafusion/pull/16876#issuecomment-3109152002 Hi @alamb can you trigger the benchmark here, i can't believe the benchmark result from my local, thanks! ```rust critcmp --filter "/vectorized_append" optimize_vectori

[PR] feat: support datetime_field as expr for bigquery [datafusion-sqlparser-rs]

chenkovsky opened a new pull request, #1971: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1971 bigquery supports ``` SELECT DATETIME_TRUNC(CURRENT_DATETIME, WEEK(MONDAY)) ``` `WEEK(MONDAY)` should be parsed to datetime_field -- This is an automated messa

Re: [PR] feat: change Expr OuterReferenceColumn to Box type for reducing expr struct size [datafusion]

zhuqi-lucas commented on PR #16771: URL: https://github.com/apache/datafusion/pull/16771#issuecomment-3109177350 I will try only to make Alias changing to Box soon, this should be the best choice from performance aspect. -- This is an automated message from the Apache Git Service. To resp

Re: [I] Document the SQL dialect DataFusion attempts to follow [datafusion]

findepi commented on issue #13704: URL: https://github.com/apache/datafusion/issues/13704#issuecomment-3109180176 For floating point numbers semantics I propose that we follow Trino, which in turn follows IEEE 754 -- translating floating point standard primitive operations into as-consisten

Re: [PR] Feat: Impl array flatten func [datafusion-comet]

kazantsev-maksim commented on code in PR #2039: URL: https://github.com/apache/datafusion-comet/pull/2039#discussion_r2226264550 ## docs/spark_expressions_support.md: ## @@ -98,7 +98,7 @@ - [x] arrays_overlap - [ ] arrays_zip - [x] element_at - - [ ] flatten + - [x] flatte

Re: [PR] Feat: Impl array flatten func [datafusion-comet]

kazantsev-maksim commented on code in PR #2039: URL: https://github.com/apache/datafusion-comet/pull/2039#discussion_r2226267723 ## spark/src/main/scala/org/apache/comet/serde/arrays.scala: ## @@ -378,3 +378,38 @@ object CometCreateArray extends CometExpressionSerde { }

Re: [PR] speedup `date_trunc` (~7x faster) in some cases [datafusion]

waynexia commented on code in PR #16859: URL: https://github.com/apache/datafusion/pull/16859#discussion_r2226279914 ## datafusion/functions/src/datetime/date_trunc.rs: ## @@ -185,6 +187,21 @@ impl ScalarUDFImpl for DateTruncFunc { ) -> Result { let parsed_

Re: [PR] Fix `next_up` and `next_down` behavior for zero float values [datafusion]

findepi commented on PR #16745: URL: https://github.com/apache/datafusion/pull/16745#issuecomment-3109041262 Again, if we want to follow the SQL spec, you have my support, but it won't bring answers to what actual float equality, comparison and ordering for float values should be. F

[PR] chore: migrate to DF 49.0.0 [datafusion-comet]

comphead opened a new pull request, #2040: URL: https://github.com/apache/datafusion-comet/pull/2040 ## Which issue does this PR close? Closes #1993 . ## Rationale for this change ## What changes are included in this PR? ## How are these cha

[PR] feat: support multiple value for pivot [datafusion-sqlparser-rs]

chenkovsky opened a new pull request, #1970: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1970 https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-pivot.html currently this sql is not supported. ``` SELECT * FROM person PIVOT ( SUM(age)

Re: [PR] Fixes 3 bugs during serialization and deserialization of physical plans [datafusion]

NGA-TRAN commented on code in PR #16858: URL: https://github.com/apache/datafusion/pull/16858#discussion_r2225941942 ## datafusion/proto/src/physical_plan/from_proto.rs: ## @@ -368,7 +368,12 @@ pub fn parse_physical_expr( e.name.as_str(),

Re: [PR] MINOR: add unit tests for chr function [datafusion]

waynexia commented on code in PR #16856: URL: https://github.com/apache/datafusion/pull/16856#discussion_r2226354689 ## datafusion/functions/src/string/chr.rs: ## @@ -132,3 +134,79 @@ impl ScalarUDFImpl for ChrFunc { self.doc() } } + +#[cfg(test)] +mod tests { +

Re: [PR] Snowflake: CREATE DYNAMIC TABLE [datafusion-sqlparser-rs]

iffyio commented on code in PR #1960: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1960#discussion_r2226064958 ## tests/sqlparser_snowflake.rs: ## @@ -1104,6 +1066,56 @@ fn parse_sf_create_table_or_view_with_dollar_quoted_comment() { ); } +#[test] +fn pa

Re: [D] Best practices for memory-efficient deduplication of pre-sorted Parquet files [datafusion]

GitHub user zheniasigayev added a comment to the discussion: Best practices for memory-efficient deduplication of pre-sorted Parquet files @alamb To clarify what you meant by "file a ticket", you're saying to create an issue within this GitHub repo, correct? GitHub link: https://github.com/a

Re: [I] CI: Check broken links in src doc comments [datafusion]

Adez017 commented on issue #16840: URL: https://github.com/apache/datafusion/issues/16840#issuecomment-3109285520 > [@Adez017](https://github.com/Adez017) Thank you. These are the commands I used. > > To generate rust docs (same as `ci/scripts/rust_docs.sh`): > > ``` > $ RUS

Re: [PR] Fix `next_up` and `next_down` behavior for zero float values [datafusion]

ozankabak commented on PR #16745: URL: https://github.com/apache/datafusion/pull/16745#issuecomment-3108909717 Well explained, @berkaysynnada. It seems like the `ScalarValue` code is deferring to `total_cmp` in the wrong place. -- This is an automated message from the Apache Git Service.

[I] Support `COLUMNS(* EXLCUDE(col1, col2))` syntax [datafusion-sqlparser-rs]

ryanschneider opened a new issue, #1972: URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1972 DuckDB supports excluding columns via this syntax: ``` SELECT COLUMNS(* EXCLUDE(col1, col2)) FROM tbl ``` Currently this format gives a syntax error: ``` P

Re: [PR] chore(deps): bump the proto group with 2 updates [datafusion]

dependabot[bot] commented on PR #16806: URL: https://github.com/apache/datafusion/pull/16806#issuecomment-3107778075 Looks like this PR is already up-to-date with main! If you'd still like to recreate it from scratch, overwriting any edits, you can request `@dependabot recreate`. -- This

Re: [I] Unable to read from ArrowFormat [datafusion]

colinmarc commented on issue #16874: URL: https://github.com/apache/datafusion/issues/16874#issuecomment-3108882467 Sorry, I failed to include the bit that was throwing the error. This is after round-tripping through protobuf. -- This is an automated message from the Apache Git Service. T

Re: [PR] MINOR: add unit tests for chr function [datafusion]

findepi commented on code in PR #16856: URL: https://github.com/apache/datafusion/pull/16856#discussion_r2225777623 ## datafusion/functions/src/string/chr.rs: ## @@ -132,3 +134,79 @@ impl ScalarUDFImpl for ChrFunc { self.doc() } } + +#[cfg(test)] +mod tests { +

Re: [PR] Ensure Substrait consumer can handle expressions in VirtualTable [datafusion]

lorenarosati commented on code in PR #16857: URL: https://github.com/apache/datafusion/pull/16857#discussion_r2225788037 ## datafusion/substrait/tests/testdata/test_plans/select_count_from_select_1_virtual_table_expressions.substrait.json: ## @@ -0,0 +1,94 @@ +{ +"extensionU

Re: [PR] Ensure Substrait consumer can handle expressions in VirtualTable [datafusion]

lorenarosati commented on code in PR #16857: URL: https://github.com/apache/datafusion/pull/16857#discussion_r2225787214 ## datafusion/substrait/src/logical_plan/consumer/rel/read_rel.rs: ## @@ -114,14 +114,37 @@ pub async fn from_read_rel( .await }

Re: [PR] Fix `next_up` and `next_down` behavior for zero float values [datafusion]

berkaysynnada commented on PR #16745: URL: https://github.com/apache/datafusion/pull/16745#issuecomment-3108950746 > Not sure whether you mean SQL spec, or what's implemented in DataFusion, or what DataFusion should be implementing? I mean SQL spec and what DataFusion should be implem

Re: [PR] feat: support datetime_field as expr for bigquery [datafusion-sqlparser-rs]

iffyio commented on code in PR #1971: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1971#discussion_r2226209387 ## tests/sqlparser_bigquery.rs: ## @@ -2566,3 +2566,101 @@ fn test_struct_trailing_and_nested_bracket() { ) ); } + +#[test] +fn test_date

Re: [PR] test: Fix flaky join tests [datafusion]

2010YOUY01 commented on code in PR #16860: URL: https://github.com/apache/datafusion/pull/16860#discussion_r2225641774 ## datafusion/sqllogictest/test_files/joins.slt: ## @@ -4164,23 +4164,40 @@ AS VALUES (3, 3, true), (3, 3, false); -query B -SELECT * FROM t0 FULL JOIN

Re: [PR] Fix `next_up` and `next_down` behavior for zero float values [datafusion]

findepi commented on PR #16745: URL: https://github.com/apache/datafusion/pull/16745#issuecomment-3108936064 You're absolutely right. The ORDER BY ordering and `<` operator are not the same thing. The ORDER BY places NaN as higher than +inf, while `<` operator _should likely_ return fal

Re: [PR] Snowflake: Add support for `CREATE USER` [datafusion-sqlparser-rs]

iffyio merged PR #1950: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1950 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] Postgres: Support parenthesized `SET` options for `ALTER TABLE` [datafusion-sqlparser-rs]

iffyio merged PR #1947: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [I] Postgres ALTER TABLE SET ( storage_parameter [= value] [, ... ] ) fails to parse [datafusion-sqlparser-rs]

iffyio closed issue #1946: Postgres ALTER TABLE SET ( storage_parameter [= value] [, ... ] ) fails to parse URL: https://github.com/apache/datafusion-sqlparser-rs/issues/1946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [PR] SGA-11419 Added snowflake ability for if not exists after create view… [datafusion-sqlparser-rs]

iffyio commented on code in PR #1961: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1961#discussion_r2226033548 ## tests/sqlparser_common.rs: ## @@ -16183,3 +16190,21 @@ fn test_identifier_unicode_start() { ]); let _ = dialects.verified_stmt(sql); } + +

Re: [PR] Snowflake: Support IDENTIFIER for GRANT ROLE [datafusion-sqlparser-rs]

iffyio commented on code in PR #1957: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1957#discussion_r2226043622 ## src/ast/mod.rs: ## @@ -6623,7 +6623,7 @@ pub enum Action { Replicate, ResolveAll, Role { -role: Ident, +role: ObjectNa

Re: [PR] Snowflake: Support IDENTIFIER for GRANT ROLE [datafusion-sqlparser-rs]

iffyio merged PR #1957: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1957 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] chore(deps): bump on-headers and compression in /datafusion/wasmtest/datafusion-wasm-app [datafusion]

findepi commented on PR #16812: URL: https://github.com/apache/datafusion/pull/16812#issuecomment-3107772965 @dependabot rebase flaky test likely fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] chore(deps): bump substrait from 0.58.0 to 0.59.0 [datafusion]

findepi commented on PR #16808: URL: https://github.com/apache/datafusion/pull/16808#issuecomment-3107776139 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] chore(deps): bump the proto group with 2 updates [datafusion]

findepi commented on PR #16806: URL: https://github.com/apache/datafusion/pull/16806#issuecomment-310610 @dependabot rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Chore: refactor Comparison out of QueryPlanSerde [datafusion-comet]

CuteChuanChuan commented on PR #2028: URL: https://github.com/apache/datafusion-comet/pull/2028#issuecomment-3108792235 Hi @andygrove , Could you please take a look at this PR and provide some guidance when you have a chance? I've refactored the comparison expressions (GreaterThan,

Re: [PR] Fixes 3 bugs during serialization and deserialization of physical plans [datafusion]

LiaCastaneda commented on code in PR #16858: URL: https://github.com/apache/datafusion/pull/16858#discussion_r2226105947 ## datafusion/proto/src/physical_plan/from_proto.rs: ## @@ -368,7 +368,12 @@ pub fn parse_physical_expr( e.name.as_str(),

Re: [PR] speedup `date_trunc` (~7x faster) in some cases [datafusion]

findepi commented on code in PR #16859: URL: https://github.com/apache/datafusion/pull/16859#discussion_r2225382275 ## datafusion/functions/src/datetime/date_trunc.rs: ## @@ -185,6 +187,21 @@ impl ScalarUDFImpl for DateTruncFunc { ) -> Result { let parsed_t

Re: [PR] Fix `next_up` and `next_down` behavior for zero float values [datafusion]

findepi commented on PR #16745: URL: https://github.com/apache/datafusion/pull/16745#issuecomment-3108568911 The SQL ordering of float values clearly distinguishes between 0 and -0 and is a total ordering. ``` $ cargo run --bin datafusion-cli > SELECT t, t::float AS f from (valu

Re: [PR] test: Fix flaky join tests [datafusion]

findepi commented on code in PR #16860: URL: https://github.com/apache/datafusion/pull/16860#discussion_r2225666844 ## datafusion/sqllogictest/test_files/joins.slt: ## @@ -4164,23 +4164,40 @@ AS VALUES (3, 3, true), (3, 3, false); -query B -SELECT * FROM t0 FULL JOIN t1

Re: [PR] feat: improve LiteralGuarantee for the case like `(a=1 AND b=1) OR (a=2 AND b=3)` [datafusion]

alamb merged PR #16762: URL: https://github.com/apache/datafusion/pull/16762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] [main] Update version to 49.0.0, add 49.0.0 changelog [datafusion]

alamb merged PR #16855: URL: https://github.com/apache/datafusion/pull/16855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] fix(build-wasm): put `arrow-ipc/zstd` dep under `compression` feature [datafusion]

alamb merged PR #16844: URL: https://github.com/apache/datafusion/pull/16844 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusi

Re: [PR] Fix `next_up` and `next_down` behavior for zero float values [datafusion]

berkaysynnada commented on PR #16745: URL: https://github.com/apache/datafusion/pull/16745#issuecomment-3108840433 What you give as an example is correct but missing. In SQL "ORDER BY" produces a total order and distinguishes `-0.0` from `+0.0`. However, SQL comparisons follow IEEE 754 orde

Re: [I] Bloom filters are unused for certain where clause patterns (improve LiteralGuarantee) [datafusion]

alamb closed issue #16697: Bloom filters are unused for certain where clause patterns (improve LiteralGuarantee) URL: https://github.com/apache/datafusion/issues/16697 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

Re: [I] Clang requirement when building for WebAssembly, `cc-rs` through `zstd`, fails [datafusion]

alamb closed issue #16843: Clang requirement when building for WebAssembly, `cc-rs` through `zstd`, fails URL: https://github.com/apache/datafusion/issues/16843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] MINOR: add unit tests for chr function [datafusion]

alamb commented on code in PR #16856: URL: https://github.com/apache/datafusion/pull/16856#discussion_r2225743598 ## datafusion/functions/src/string/chr.rs: ## @@ -132,3 +134,79 @@ impl ScalarUDFImpl for ChrFunc { self.doc() } } + +#[cfg(test)] +mod tests { +u

[I] Unable to read from ArrowFormat [datafusion]

colinmarc opened a new issue, #16874: URL: https://github.com/apache/datafusion/issues/16874 ### Describe the bug It's not in the list here: https://github.com/apache/datafusion/blob/a0ce581092430271e67552ec6370f100cbb80046/datafusion/proto/src/logical_plan/mod.rs#L1064

Re: [PR] feat: support multi value column unpivot & alias in unpivot [datafusion-sqlparser-rs]

chenkovsky commented on code in PR #1969: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1969#discussion_r2225764102 ## src/ast/query.rs: ## @@ -1351,9 +1392,9 @@ pub enum TableFactor { /// See .

Re: [PR] fix: clean up [iceberg] integration APIs [datafusion-comet]

parthchandra commented on code in PR #2032: URL: https://github.com/apache/datafusion-comet/pull/2032#discussion_r2226136383 ## common/src/main/java/org/apache/comet/parquet/BatchReader.java: ## @@ -143,7 +143,9 @@ public class BatchReader extends RecordReader implements Cl p

Re: [PR] Snowflake: Numeric prefix for stage name part [datafusion-sqlparser-rs]

iffyio merged PR #1966: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1966 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr

Re: [PR] MySQL: ALTER TABLE RENAME AS [datafusion-sqlparser-rs]

iffyio commented on code in PR #1965: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1965#discussion_r2226153975 ## src/ast/ddl.rs: ## @@ -264,6 +264,7 @@ pub enum AlterTableOperation { }, /// `RENAME TO ` RenameTable { +to_keyword: bool,

Re: [I] CI: Check broken links in src doc comments [datafusion]

jcsherin commented on issue #16840: URL: https://github.com/apache/datafusion/issues/16840#issuecomment-3109415899 @Adez017 Thank you for working on this! The standard practice is to introduce a check like this to validate new PRs. For rollout I think it involves two steps. First, we

Re: [PR] Redshift: Add support for IAM_ROLE and IGNOREHEADER COPY options [datafusion-sqlparser-rs]

iffyio commented on code in PR #1968: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1968#discussion_r2226163647 ## tests/sqlparser_common.rs: ## @@ -16256,3 +16256,25 @@ fn parse_notnull() { // for unsupported dialects, parsing should stop at `NOT NULL`

Re: [PR] MINOR: add unit tests for chr function [datafusion]

waynexia commented on code in PR #16856: URL: https://github.com/apache/datafusion/pull/16856#discussion_r2226369893 ## datafusion/functions/src/string/chr.rs: ## @@ -132,3 +134,79 @@ impl ScalarUDFImpl for ChrFunc { self.doc() } } + +#[cfg(test)] +mod tests { +

Re: [PR] feat: add multi level merge sort that will always fit in memory [datafusion]

rluvaton commented on code in PR #15700: URL: https://github.com/apache/datafusion/pull/15700#discussion_r2226402212 ## datafusion/physical-plan/src/spill/get_size.rs: ## @@ -0,0 +1,216 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor lic

Re: [PR] chore: migrate to DF 49.0.0 [datafusion-comet]

mbutrovich commented on PR #2040: URL: https://github.com/apache/datafusion-comet/pull/2040#issuecomment-3109797437 This PR https://github.com/apache/datafusion/pull/16290 changed the signature for several functions to return Utf8View. I can start bringing in changes to support Utf8View fr

Re: [PR] chore: migrate to DF 49.0.0 [datafusion-comet]

codecov-commenter commented on PR #2040: URL: https://github.com/apache/datafusion-comet/pull/2040#issuecomment-3109796192 ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/2040?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_ca

Re: [I] ScalarUDFImpl::equals default implementation is error-prone [datafusion]

findepi commented on issue #16677: URL: https://github.com/apache/datafusion/issues/16677#issuecomment-3106598537 I have defined sub tasks in this issue. I want to use `ScalarUDFImpl` as the design driver / place where review happens. The `AggregateUDFImpl` and `WindowUDFImpl` can be work

[PR] feat: support multi value column unpivot [datafusion-sqlparser-rs]

chenkovsky opened a new pull request, #1969: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1969 for example, spark supports ``` SELECT * FROM sales_quarterly UNPIVOT EXCLUDE NULLS ( (first_quarter, second_quarter) FOR half_of_the_year IN (

[PR] chore(deps): bump aws-config from 1.8.1 to 1.8.2 [datafusion]

dependabot[bot] opened a new pull request, #16864: URL: https://github.com/apache/datafusion/pull/16864 Bumps [aws-config](https://github.com/smithy-lang/smithy-rs) from 1.8.1 to 1.8.2. Commits See full diff in https://github.com/smithy-lang/smithy-rs/commits";>compare view

[PR] chore(deps): bump serde_json from 1.0.140 to 1.0.141 [datafusion]

dependabot[bot] opened a new pull request, #16863: URL: https://github.com/apache/datafusion/pull/16863 Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.140 to 1.0.141. Release notes Sourced from https://github.com/serde-rs/json/releases";>serde_json's releases.

[PR] Redshift: CREATE TABLE ... (LIKE ..) [datafusion-sqlparser-rs]

yoavcloud opened a new pull request, #1967: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1967 Add support for Redshift `CREATE TABLE new_table (LIKE parent_table [ { INCLUDING | EXCLUDING } DEFAULTS ])` -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Fixes 3 bugs during serialization and deserialization of physical plans [datafusion]

LiaCastaneda commented on code in PR #16858: URL: https://github.com/apache/datafusion/pull/16858#discussion_r2224790385 ## datafusion/proto/src/physical_plan/from_proto.rs: ## @@ -368,7 +368,12 @@ pub fn parse_physical_expr( e.name.as_str(),

Re: [PR] Derive UDF equality from PartialEq, Hash [datafusion]

findepi commented on PR #16842: URL: https://github.com/apache/datafusion/pull/16842#issuecomment-3106513147 Update. This PR covers `ScalarUDFImpl` that today have explicit equals/hash implementations. Next steps - `AggregateUDFImpl` that today have explicit equals/hash impleme

[I] Derive ScalarUDFImpl equality from PartialEq, Hash [datafusion]

findepi opened a new issue, #16865: URL: https://github.com/apache/datafusion/issues/16865 Provide a terse way to implement `ScalarUDFImpl::{equals,hash_code}` for function Foo if Foo implements `PartialEq` and `Hash` -- This is an automated message from the Apache Git Service. To respond

[I] Derive UDAF (`AggregateUDFImpl`) equality from PartialEq, Hash [datafusion]

findepi opened a new issue, #16866: URL: https://github.com/apache/datafusion/issues/16866 Same as https://github.com/apache/datafusion/issues/16865 but for UDAF (`AggregateUDFImpl`). - depends on https://github.com/apache/datafusion/issues/16865. That issue needs to be done first to

[I] Derive UDWF (`WindowUDFImpl `) equality from PartialEq, Hash [datafusion]

findepi opened a new issue, #16867: URL: https://github.com/apache/datafusion/issues/16867 Same as https://github.com/apache/datafusion/issues/16865 but for UDWF (`WindowUDFImpl`). - depends on https://github.com/apache/datafusion/issues/16865. That issue needs to be done first to de

[I] Implement PartialEq, Hash for all UDAFs (`ScalarUDFImpl`) [datafusion]

findepi opened a new issue, #16869: URL: https://github.com/apache/datafusion/issues/16869 Implement PartialEq, Hash for all UDAFs (`AggregateUDFImpl`) that do not implement PartialEq, Hash yet. This should boil down to adding `#[derive(PartialEq, Hash)]` to all of them. - depends

[I] Implement PartialEq, Hash for all UDFs (`ScalarUDFImpl`) [datafusion]

findepi opened a new issue, #16868: URL: https://github.com/apache/datafusion/issues/16868 Implement PartialEq, Hash for all UDFs (`ScalarUDFImpl`) that do not implement PartialEq, Hash yet. This should boil down to adding `#[derive(PartialEq, Hash)]` to all of them. - depends on

[I] Implement PartialEq, Hash for all UDWFs (`WindowUDFImpl`) [datafusion]

findepi opened a new issue, #16870: URL: https://github.com/apache/datafusion/issues/16870 Implement PartialEq, Hash for all UDWFs (`WindowUDFImpl`) that do not implement PartialEq, Hash yet. This should boil down to adding `#[derive(PartialEq, Hash)]` to all of them. - depends on

[I] Replace `ScalarUDFImpl::{equals,hash_value}` with `UdfHash`, `UdfEq` traits [datafusion]

findepi opened a new issue, #16871: URL: https://github.com/apache/datafusion/issues/16871 Implement proposal set forth in https://github.com/apache/datafusion/issues/16677#issuecomment-3092338265 - depends on https://github.com/apache/datafusion/issues/16865 - depends on https://g

[I] Replace `AggregateUDFImpl::{equals,hash_value}` with `UdfHash`, `UdfEq` traits [datafusion]

findepi opened a new issue, #16872: URL: https://github.com/apache/datafusion/issues/16872 Implement proposal set forth in https://github.com/apache/datafusion/issues/16677#issuecomment-3092338265 for `AggregateUDFImpl` - depends on https://github.com/apache/datafusion/issues/16871 f

[I] Replace `WindowUDFImpl::{equals,hash_value}` with `UdfHash`, `UdfEq` traits [datafusion]

findepi opened a new issue, #16873: URL: https://github.com/apache/datafusion/issues/16873 Implement proposal set forth in https://github.com/apache/datafusion/issues/16677#issuecomment-3092338265 for `WindowUDFImpl` (UDWF) - depends on https://github.com/apache/datafusion/issues/168

[PR] Redshift: Add support for IAM_ROLE and IGNOREHEADER COPY options [datafusion-sqlparser-rs]

yoavcloud opened a new pull request, #1968: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1968 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Memory accounting model discussion [datafusion]

fmonjalet commented on issue #16841: URL: https://github.com/apache/datafusion/issues/16841#issuecomment-3107483475 > I would argue that when accounting anything that is not owned (like ArrayRef, ScalarValue::List(...), ...), the memory reserved should be equal to the bytes occupied by the

Re: [PR] feat(spark): implement Spark datetime function last_day [datafusion]

2010YOUY01 commented on code in PR #16828: URL: https://github.com/apache/datafusion/pull/16828#discussion_r2227274568 ## datafusion/sqllogictest/test_files/spark/datetime/last_day.slt: ## @@ -21,7 +21,80 @@ # For more information, please see: # https://github.com/apache/dat

Re: [PR] dissallow pushdown of volatile PhysicalExprs [datafusion]

theirix commented on PR #16861: URL: https://github.com/apache/datafusion/pull/16861#issuecomment-3110119344 Thank you, @adriangb ! I can confirm that it works great with the table sampling, since I use `random` function (matched by name): ``` query TT EXPLAIN SELECT COUNT(*) from t

Re: [PR] Fix `next_up` and `next_down` behavior for zero float values [datafusion]

liamzwbao commented on PR #16745: URL: https://github.com/apache/datafusion/pull/16745#issuecomment-3110125410 Thanks for your insights, @berkaysynnada @findepi! It seems we haven't reached a consensus yet. Should I proceed with the `PartialOrd` fix, or wait until we have clearer doc to ali

Re: [I] Upgrade to sqlparser 0.56.0 [datafusion]

alamb commented on issue #16405: URL: https://github.com/apache/datafusion/issues/16405#issuecomment-3110129687 I think we are actually going to have to go directly to 0.58.0. See discussion on - https://github.com/apache/datafusion/pull/16456 -- This is an automated message from the

Re: [PR] feat: support datetime_field as expr for bigquery [datafusion-sqlparser-rs]

chenkovsky commented on code in PR #1971: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1971#discussion_r2226952924 ## tests/sqlparser_bigquery.rs: ## @@ -2566,3 +2566,101 @@ fn test_struct_trailing_and_nested_bracket() { ) ); } + +#[test] +fn test_

Re: [I] Add support for `MapSort` expression in Spark 4.0.0 [datafusion-comet]

rishvin commented on issue #1941: URL: https://github.com/apache/datafusion-comet/issues/1941#issuecomment-3111745019 > Some Updates: I have a simple test to start with, which will produce `_groupingmapsort`. > > ``` > val data = Seq( > | Map("a" -> 1, "b" -> 2), >

Re: [I] Do we have plans to support remote shuffle services, such as Apache Celeborn? [datafusion-comet]

SteNicholas commented on issue #1521: URL: https://github.com/apache/datafusion-comet/issues/1521#issuecomment-3111756711 This issue is duplicated by #1241. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[PR] Snowflake: DROP STREAM [datafusion-sqlparser-rs]

yoavcloud opened a new pull request, #1973: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1973 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] SGA-11419 Added snowflake ability for if not exists after create view… [datafusion-sqlparser-rs]

etgarperets commented on code in PR #1961: URL: https://github.com/apache/datafusion-sqlparser-rs/pull/1961#discussion_r2227579063 ## tests/sqlparser_common.rs: ## @@ -16183,3 +16190,21 @@ fn test_identifier_unicode_start() { ]); let _ = dialects.verified_stmt(sql);

Re: [I] Optimize concatenation of complex data type, such as list, struct [datafusion]

zhuqi-lucas commented on issue #16838: URL: https://github.com/apache/datafusion/issues/16838#issuecomment-3112047196 Thank you @xudong963 for the info, i tested: ```rust Field::new( "col3", DataType::Timestamp(TimeUnit::Nanosecond, None), false,

Re: [I] Optimize concatenation of complex data type, such as list, struct [datafusion]

zhuqi-lucas commented on issue #16838: URL: https://github.com/apache/datafusion/issues/16838#issuecomment-3112086083 Updated, i reproduced it now with a simple list cases, i will further investigate: ```rust let v1 = FixedSizeListArray::try_new( Arc::new(Field::new_li

Re: [PR] MINOR: add unit tests for chr function [datafusion]

waynexia commented on code in PR #16856: URL: https://github.com/apache/datafusion/pull/16856#discussion_r2227482405 ## datafusion/functions/src/string/chr.rs: ## @@ -132,3 +134,79 @@ impl ScalarUDFImpl for ChrFunc { self.doc() } } + +#[cfg(test)] +mod tests { +

Re: [PR] fix: clean up [iceberg] integration APIs [datafusion-comet]

huaxingao commented on code in PR #2032: URL: https://github.com/apache/datafusion-comet/pull/2032#discussion_r2226783940 ## common/src/main/java/org/apache/comet/parquet/BatchReader.java: ## @@ -143,7 +143,9 @@ public class BatchReader extends RecordReader implements Cl priv

Re: [PR] feat(spark): implement Spark datetime function last_day [datafusion]

Standing-Man commented on code in PR #16828: URL: https://github.com/apache/datafusion/pull/16828#discussion_r2227062667 ## datafusion/sqllogictest/test_files/spark/datetime/last_day.slt: ## @@ -21,7 +21,80 @@ # For more information, please see: # https://github.com/apache/d

Re: [PR] chore(deps): bump aws-config from 1.8.1 to 1.8.2 [datafusion]

comphead merged PR #16864: URL: https://github.com/apache/datafusion/pull/16864 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@dataf

  1   2   >