This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a change to branch branch-1.0
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
from e0de5820f8 fix: round on Int64 with scale <= -19 now overflows
correctly (#5082)
add cda9d29576 fix: match Spark's whitespace trim semantics for casts from
string to boolean, integral, float/double and decimal (#5150)
add d2a61bdf1c fix: disambiguate Iceberg scans that share a
metadata_location (#5180)
add 285a80385d chore: drop deprecated .toIterable from protobuf builder
calls (#5173)
add 1ce9df1e4f feat: native uuid() implementation compatible with Spark
(#5034)
add 0abe9667ba chore: bump Spark 3.5 to 3.5.9 (#5181)
add 60413eb52c feat: support Iceberg metadata columns _pos, _spec, _file,
and _partition (#4752)
add 064dcf64b9 chore: bump Spark 4.1 to 4.1.3 (#5183)
add 47f395c737 fix: use per-expression eval mode for decimal promotion
(#5171)
add ba21f02dec perf: use Arrow comparator for nested array equality (#5176)
add 2fa6461f4d chore: bump Spark 4.0 to 4.0.4 (#5182)
add 671a8dd9ee fix: make native cast from float/double to decimal
compatible with Spark (#5136)
add 3df58d5280 perf: intern QueryContext SQL text into a per-plan pool (up
to 20x smaller serialized plans for TPC-DS) (#5204)
add 42a48bbdf9 chore: remove unused spark.comet.exceptionOnDatetimeRebase
config (#5221)
add 936794d7f6 refactor: rename pyarrowUdf config to pyarrowUDF and fix
stale config docs (#5197)
add 41d887d07c perf: avoid rebuilding untouched operators in
PlanDataInjector.injectPlanData (#5220)
add 1008e63bd8 chore: cargo update for 1.0 release (#5226)
add 42528dbcd5 docs: drop compatibility notes for bugs that are now fixed
(#5154)
add 6acfe7dee1 feat: report native vs codegen-dispatch expression coverage
in extended explain (#5201)
add f13cc977ca fix: codegen dispatcher null short-circuit swallowed ANSI
errors, plus two latent TIME-type gaps (#5219)
add ee974a1fca fix: count ReusedSubquery and CometSubqueryBroadcast
correctly in extended explain (#5206)
add af534e0fa0 docs: document ReusedExchange caveat and both
CometSparkToColumnar names in operator-count exclusions (#5240)
No new revisions were added by this update.
Summary of changes:
.github/actions/setup-spark-builder/action.yaml | 2 +-
.github/workflows/ci.yml | 12 +-
.github/workflows/iceberg_spark_test_reusable.yml | 2 +-
.github/workflows/pr_build_linux.yml | 4 +
.github/workflows/pr_build_macos.yml | 4 +
.github/workflows/pyarrow_udf_test.yml | 4 +-
.github/workflows/spark_sql_test_reusable.yml | 2 +-
.github/workflows/spark_sql_writer_tests.yml | 4 +-
dev/ci/compute-changes.py | 6 +-
dev/diffs/{3.5.8.diff => 3.5.9.diff} | 68 +-
dev/diffs/{4.0.2.diff => 4.0.4.diff} | 100 +--
dev/diffs/{4.1.2.diff => 4.1.3.diff} | 78 +-
dev/diffs/iceberg/1.10.0.diff | 14 +
dev/diffs/iceberg/1.11.0.diff | 14 +
dev/diffs/iceberg/1.8.1.diff | 14 +
dev/diffs/iceberg/1.9.1.diff | 14 +
.../source/contributor-guide/benchmarking_macos.md | 10 +-
.../source/contributor-guide/config_conventions.md | 27 +-
.../expression-audits/conversion_funcs.md | 2 +-
.../expression-audits/misc_funcs.md | 7 +
.../contributor-guide/iceberg-spark-tests.md | 2 +-
.../expressions/_category_template/cast.md | 27 +-
.../user-guide/latest/compatibility/index.md | 9 +-
.../user-guide/latest/compatibility/scans.md | 4 +-
.../latest/compatibility/spark-versions.md | 6 +-
docs/source/user-guide/latest/expressions.md | 2 +-
docs/source/user-guide/latest/installation.md | 6 +-
docs/source/user-guide/latest/kubernetes.md | 8 +-
docs/source/user-guide/latest/pyarrow-udfs.md | 8 +-
.../user-guide/latest/understanding-comet-plans.md | 42 +-
kube/Dockerfile | 4 +-
native/Cargo.lock | 232 +++---
native/Cargo.toml | 4 +-
native/common/src/query_context.rs | 7 +-
native/core/src/execution/expressions/random.rs | 18 +-
native/core/src/execution/jni_api.rs | 1 +
.../core/src/execution/operators/iceberg_scan.rs | 1 +
native/core/src/execution/planner.rs | 390 +++++++++-
.../src/execution/planner/expression_registry.rs | 4 +
native/proto/src/proto/expr.proto | 20 +-
native/proto/src/proto/operator.proto | 22 +-
native/spark-expr/Cargo.toml | 8 +
native/spark-expr/benches/cast_from_string.rs | 199 ++---
native/spark-expr/benches/cast_string_to_date.rs | 18 +-
.../spark-expr/benches/cast_string_to_timestamp.rs | 204 +++++
native/spark-expr/benches/common/mod.rs | 43 ++
.../spark-expr/src/array_funcs/array_position.rs | 57 +-
.../spark-expr/src/array_funcs/arrays_overlap.rs | 159 ++--
native/spark-expr/src/conversion_funcs/mod.rs | 1 +
native/spark-expr/src/conversion_funcs/numeric.rs | 317 ++++++--
native/spark-expr/src/conversion_funcs/string.rs | 265 +++++--
native/spark-expr/src/conversion_funcs/trim.rs | 192 +++++
.../nondetermenistic_funcs/internal/mersenne.rs | 172 +++++
.../src/nondetermenistic_funcs/internal/mod.rs | 1 +
.../spark-expr/src/nondetermenistic_funcs/mod.rs | 2 +
.../src/nondetermenistic_funcs/shuffle.rs | 195 +----
.../spark-expr/src/nondetermenistic_funcs/uuid.rs | 290 +++++++
pom.xml | 8 +-
spark/pom.xml | 3 +-
.../main/scala/org/apache/comet/CometConf.scala | 14 +-
.../apache/comet/CometSparkSessionExtensions.scala | 49 +-
.../org/apache/comet/ExtendedExplainInfo.scala | 138 +++-
.../comet/codegen/CometBatchKernelCodegen.scala | 191 +++--
.../codegen/CometSpecializedGettersDispatch.scala | 12 +-
.../org/apache/comet/expressions/CometCast.scala | 15 +-
.../apache/comet/iceberg/IcebergReflection.scala | 84 ++
.../org/apache/comet/rules/CometExecRule.scala | 24 +-
.../org/apache/comet/rules/CometScanRule.scala | 87 ++-
.../rules/EliminateRedundantTransitions.scala | 2 +-
.../org/apache/comet/serde/CometScalaUDF.scala | 32 +-
.../apache/comet/serde/QueryContextInterner.scala | 137 ++++
.../org/apache/comet/serde/QueryPlanSerde.scala | 57 +-
.../scala/org/apache/comet/serde/arithmetic.scala | 12 -
.../apache/comet/serde/decimalExpressions.scala | 8 -
.../org/apache/comet/serde/nondetermenistic.scala | 28 +-
.../serde/operator/CometIcebergNativeScan.scala | 299 +++++---
.../comet/serde/operator/CometNativeScan.scala | 29 +-
.../org/apache/comet/serde/operator/package.scala | 2 +-
.../comet/udf/codegen/CometScalaUDFCodegen.scala | 10 +-
.../spark/sql/comet/CometCsvNativeScanExec.scala | 12 +-
.../sql/comet/CometIcebergNativeScanExec.scala | 8 +-
.../spark/sql/comet/CometNativeScanExec.scala | 11 +-
.../sql/comet/CometSubqueryBroadcastExec.scala | 3 +-
.../apache/spark/sql/comet/CometWindowExec.scala | 6 +-
.../apache/spark/sql/comet/DecimalPrecision.scala | 16 +-
.../shuffle/CometNativeShuffleWriter.scala | 1 -
.../org/apache/spark/sql/comet/operators.scala | 108 ++-
.../resources/pyspark/benchmark_pyarrow_udf.py | 6 +-
.../src/test/resources/pyspark/test_pyarrow_udf.py | 16 +-
.../expressions/cast/cast_string_trim.sql | 71 ++
.../resources/sql-tests/expressions/misc/uuid.sql | 45 ++
.../sql-tests/expressions/misc/uuid_with_seed.sql | 96 +++
.../iceberg/metadata_column_partition.sql | 854 +++++++++++++++++++++
.../approved-plans-v1_4-spark3_5/q44/extended.txt | 8 +-
.../approved-plans-v1_4-spark3_5/q58/extended.txt | 2 +-
.../approved-plans-v1_4-spark3_5/q67/extended.txt | 2 +-
.../approved-plans-v1_4-spark3_5/q70/extended.txt | 2 +-
.../approved-plans-v1_4-spark3_5/q83/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_0/q14b/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_0/q44/extended.txt | 8 +-
.../approved-plans-v1_4-spark4_0/q54/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_0/q58/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_0/q6/extended.txt | 4 +-
.../q83.ansi/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_1/q33/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_1/q49/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_1/q56/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_1/q60/extended.txt | 2 +-
.../approved-plans-v1_4-spark4_1/q66/extended.txt | 2 +-
.../approved-plans-v1_4/q1/extended.txt | 2 +-
.../approved-plans-v1_4/q10/extended.txt | 2 +-
.../approved-plans-v1_4/q11/extended.txt | 2 +-
.../approved-plans-v1_4/q12/extended.txt | 2 +-
.../approved-plans-v1_4/q13/extended.txt | 4 +-
.../approved-plans-v1_4/q14a/extended.txt | 2 +-
.../approved-plans-v1_4/q14b/extended.txt | 2 +-
.../approved-plans-v1_4/q15/extended.txt | 2 +-
.../approved-plans-v1_4/q16/extended.txt | 2 +-
.../approved-plans-v1_4/q17/extended.txt | 2 +-
.../approved-plans-v1_4/q18/extended.txt | 2 +-
.../approved-plans-v1_4/q19/extended.txt | 2 +-
.../approved-plans-v1_4/q2/extended.txt | 2 +-
.../approved-plans-v1_4/q20/extended.txt | 2 +-
.../approved-plans-v1_4/q21/extended.txt | 2 +-
.../approved-plans-v1_4/q22/extended.txt | 2 +-
.../approved-plans-v1_4/q23a/extended.txt | 2 +-
.../approved-plans-v1_4/q23b/extended.txt | 2 +-
.../approved-plans-v1_4/q24a/extended.txt | 6 +-
.../approved-plans-v1_4/q24b/extended.txt | 6 +-
.../approved-plans-v1_4/q25/extended.txt | 2 +-
.../approved-plans-v1_4/q26/extended.txt | 4 +-
.../approved-plans-v1_4/q27/extended.txt | 4 +-
.../approved-plans-v1_4/q28/extended.txt | 14 +-
.../approved-plans-v1_4/q29/extended.txt | 2 +-
.../approved-plans-v1_4/q3/extended.txt | 2 +-
.../approved-plans-v1_4/q30/extended.txt | 2 +-
.../approved-plans-v1_4/q31/extended.txt | 2 +-
.../approved-plans-v1_4/q32/extended.txt | 4 +-
.../approved-plans-v1_4/q33/extended.txt | 2 +-
.../approved-plans-v1_4/q34/extended.txt | 2 +-
.../approved-plans-v1_4/q35/extended.txt | 2 +-
.../approved-plans-v1_4/q36/extended.txt | 2 +-
.../approved-plans-v1_4/q37/extended.txt | 2 +-
.../approved-plans-v1_4/q38/extended.txt | 2 +-
.../approved-plans-v1_4/q39a/extended.txt | 2 +-
.../approved-plans-v1_4/q39b/extended.txt | 2 +-
.../approved-plans-v1_4/q4/extended.txt | 2 +-
.../approved-plans-v1_4/q40/extended.txt | 2 +-
.../approved-plans-v1_4/q41/extended.txt | 2 +-
.../approved-plans-v1_4/q42/extended.txt | 2 +-
.../approved-plans-v1_4/q43/extended.txt | 2 +-
.../approved-plans-v1_4/q44/extended.txt | 10 +-
.../approved-plans-v1_4/q45/extended.txt | 2 +-
.../approved-plans-v1_4/q46/extended.txt | 2 +-
.../approved-plans-v1_4/q47/extended.txt | 2 +-
.../approved-plans-v1_4/q48/extended.txt | 2 +-
.../approved-plans-v1_4/q49/extended.txt | 2 +-
.../approved-plans-v1_4/q5/extended.txt | 2 +-
.../approved-plans-v1_4/q50/extended.txt | 2 +-
.../approved-plans-v1_4/q51/extended.txt | 2 +-
.../approved-plans-v1_4/q52/extended.txt | 2 +-
.../approved-plans-v1_4/q53/extended.txt | 2 +-
.../approved-plans-v1_4/q54/extended.txt | 2 +-
.../approved-plans-v1_4/q55/extended.txt | 2 +-
.../approved-plans-v1_4/q56/extended.txt | 2 +-
.../approved-plans-v1_4/q57/extended.txt | 2 +-
.../approved-plans-v1_4/q58/extended.txt | 2 +-
.../approved-plans-v1_4/q59/extended.txt | 2 +-
.../approved-plans-v1_4/q6/extended.txt | 4 +-
.../approved-plans-v1_4/q60/extended.txt | 2 +-
.../approved-plans-v1_4/q61/extended.txt | 2 +-
.../approved-plans-v1_4/q62/extended.txt | 2 +-
.../approved-plans-v1_4/q63/extended.txt | 2 +-
.../approved-plans-v1_4/q64/extended.txt | 2 +-
.../approved-plans-v1_4/q65/extended.txt | 2 +-
.../approved-plans-v1_4/q66/extended.txt | 2 +-
.../approved-plans-v1_4/q67/extended.txt | 2 +-
.../approved-plans-v1_4/q68/extended.txt | 2 +-
.../approved-plans-v1_4/q69/extended.txt | 2 +-
.../approved-plans-v1_4/q7/extended.txt | 4 +-
.../approved-plans-v1_4/q70/extended.txt | 2 +-
.../approved-plans-v1_4/q71/extended.txt | 2 +-
.../approved-plans-v1_4/q72/extended.txt | 2 +-
.../approved-plans-v1_4/q73/extended.txt | 2 +-
.../approved-plans-v1_4/q74/extended.txt | 2 +-
.../approved-plans-v1_4/q75/extended.txt | 2 +-
.../approved-plans-v1_4/q76/extended.txt | 2 +-
.../approved-plans-v1_4/q77/extended.txt | 2 +-
.../approved-plans-v1_4/q78/extended.txt | 2 +-
.../approved-plans-v1_4/q79/extended.txt | 2 +-
.../approved-plans-v1_4/q8/extended.txt | 2 +-
.../approved-plans-v1_4/q80/extended.txt | 2 +-
.../approved-plans-v1_4/q81/extended.txt | 2 +-
.../approved-plans-v1_4/q82/extended.txt | 2 +-
.../approved-plans-v1_4/q83/extended.txt | 2 +-
.../approved-plans-v1_4/q84/extended.txt | 2 +-
.../approved-plans-v1_4/q85/extended.txt | 4 +-
.../approved-plans-v1_4/q86/extended.txt | 2 +-
.../approved-plans-v1_4/q87/extended.txt | 2 +-
.../approved-plans-v1_4/q88/extended.txt | 2 +-
.../approved-plans-v1_4/q89/extended.txt | 2 +-
.../approved-plans-v1_4/q9/extended.txt | 12 +-
.../approved-plans-v1_4/q90/extended.txt | 2 +-
.../approved-plans-v1_4/q91/extended.txt | 2 +-
.../approved-plans-v1_4/q92/extended.txt | 4 +-
.../approved-plans-v1_4/q93/extended.txt | 2 +-
.../approved-plans-v1_4/q94/extended.txt | 2 +-
.../approved-plans-v1_4/q95/extended.txt | 2 +-
.../approved-plans-v1_4/q96/extended.txt | 2 +-
.../approved-plans-v1_4/q97/extended.txt | 2 +-
.../approved-plans-v1_4/q98/extended.txt | 2 +-
.../approved-plans-v1_4/q99/extended.txt | 2 +-
.../approved-plans-v2_7-spark3_5/q67a/extended.txt | 2 +-
.../approved-plans-v2_7-spark3_5/q70a/extended.txt | 2 +-
.../approved-plans-v2_7-spark4_0/q14/extended.txt | 2 +-
.../approved-plans-v2_7-spark4_0/q6/extended.txt | 4 +-
.../approved-plans-v2_7-spark4_1/q14a/extended.txt | 2 +-
.../approved-plans-v2_7-spark4_1/q49/extended.txt | 2 +-
.../approved-plans-v2_7-spark4_2/q77a/extended.txt | 2 +-
.../approved-plans-v2_7/q10a/extended.txt | 2 +-
.../approved-plans-v2_7/q11/extended.txt | 2 +-
.../approved-plans-v2_7/q12/extended.txt | 2 +-
.../approved-plans-v2_7/q14/extended.txt | 2 +-
.../approved-plans-v2_7/q14a/extended.txt | 2 +-
.../approved-plans-v2_7/q18a/extended.txt | 2 +-
.../approved-plans-v2_7/q20/extended.txt | 2 +-
.../approved-plans-v2_7/q22/extended.txt | 2 +-
.../approved-plans-v2_7/q22a/extended.txt | 2 +-
.../approved-plans-v2_7/q24/extended.txt | 6 +-
.../approved-plans-v2_7/q27a/extended.txt | 8 +-
.../approved-plans-v2_7/q34/extended.txt | 2 +-
.../approved-plans-v2_7/q35/extended.txt | 2 +-
.../approved-plans-v2_7/q35a/extended.txt | 2 +-
.../approved-plans-v2_7/q36a/extended.txt | 2 +-
.../approved-plans-v2_7/q47/extended.txt | 2 +-
.../approved-plans-v2_7/q49/extended.txt | 2 +-
.../approved-plans-v2_7/q51a/extended.txt | 2 +-
.../approved-plans-v2_7/q57/extended.txt | 2 +-
.../approved-plans-v2_7/q5a/extended.txt | 2 +-
.../approved-plans-v2_7/q6/extended.txt | 4 +-
.../approved-plans-v2_7/q64/extended.txt | 2 +-
.../approved-plans-v2_7/q67a/extended.txt | 2 +-
.../approved-plans-v2_7/q70a/extended.txt | 2 +-
.../approved-plans-v2_7/q72/extended.txt | 2 +-
.../approved-plans-v2_7/q74/extended.txt | 2 +-
.../approved-plans-v2_7/q75/extended.txt | 2 +-
.../approved-plans-v2_7/q77a/extended.txt | 2 +-
.../approved-plans-v2_7/q78/extended.txt | 2 +-
.../approved-plans-v2_7/q80a/extended.txt | 2 +-
.../approved-plans-v2_7/q86a/extended.txt | 2 +-
.../approved-plans-v2_7/q98/extended.txt | 2 +-
.../scala/org/apache/comet/CometCastSuite.scala | 93 ++-
.../org/apache/comet/CometCodegenSourceSuite.scala | 131 +++-
.../scala/org/apache/comet/CometCodegenSuite.scala | 261 ++++++-
.../org/apache/comet/CometCoverageStatsSuite.scala | 53 ++
.../org/apache/comet/CometIcebergNativeSuite.scala | 670 +++++++++++++++-
.../apache/comet/CometUuidExpressionSuite.scala | 65 ++
.../apache/comet/QueryContextInternerSuite.scala | 163 ++++
.../CometSpecializedGettersDispatchSuite.scala | 74 ++
.../apache/comet/exec/CometAggregateSuite.scala | 8 +-
.../CometCastNumericToNumericBenchmark.scala | 12 +-
.../spark/sql/comet/CometPlanStabilitySuite.scala | 4 +
.../spark/sql/comet/PlanDataInjectorSuite.scala | 132 +++-
.../comet/CometDecimalArithmeticViewSuite.scala | 60 +-
264 files changed, 6443 insertions(+), 1503 deletions(-)
rename dev/diffs/{3.5.8.diff => 3.5.9.diff} (98%)
rename dev/diffs/{4.0.2.diff => 4.0.4.diff} (98%)
rename dev/diffs/{4.1.2.diff => 4.1.3.diff} (99%)
create mode 100644 native/spark-expr/benches/cast_string_to_timestamp.rs
create mode 100644 native/spark-expr/benches/common/mod.rs
create mode 100644 native/spark-expr/src/conversion_funcs/trim.rs
create mode 100644
native/spark-expr/src/nondetermenistic_funcs/internal/mersenne.rs
create mode 100644 native/spark-expr/src/nondetermenistic_funcs/uuid.rs
create mode 100644
spark/src/main/scala/org/apache/comet/serde/QueryContextInterner.scala
create mode 100644
spark/src/test/resources/sql-tests/expressions/cast/cast_string_trim.sql
create mode 100644 spark/src/test/resources/sql-tests/expressions/misc/uuid.sql
create mode 100644
spark/src/test/resources/sql-tests/expressions/misc/uuid_with_seed.sql
create mode 100644
spark/src/test/resources/sql-tests/iceberg/metadata_column_partition.sql
create mode 100644
spark/src/test/scala/org/apache/comet/CometCoverageStatsSuite.scala
create mode 100644
spark/src/test/scala/org/apache/comet/CometUuidExpressionSuite.scala
create mode 100644
spark/src/test/scala/org/apache/comet/QueryContextInternerSuite.scala
create mode 100644
spark/src/test/scala/org/apache/comet/codegen/CometSpecializedGettersDispatchSuite.scala
rename spark/src/test/{spark-4.1 =>
spark-4.1+}/org/apache/spark/sql/comet/CometDecimalArithmeticViewSuite.scala
(54%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
