This is an automated email from the ASF dual-hosted git repository.
dependabot[bot] pushed a change to branch
dependabot/cargo/native/main/object_store-0.14.1
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
omit 8e42ea6a81 chore(deps): bump object_store from 0.13.2 to 0.14.1 in
/native
add c65a5ee803 feat: support gzip compression in native Parquet writes
(#4930)
add e36a410a2f feat: support size() for MapType input (#4580)
add 83339d8891 feat: hint at native pyarrow UDF path when the feature is
disabled (#4892)
add bf18cec52f feat: Implement TimeType support - Infrastructure - shuffle
(#4398)
add a469fc6d49 Report unsupported scalar serde cases in support level
(#4745)
add bc84171769 chore(deps): bump actions/setup-node from 6 to 7 (#4955)
add 884e0a675a chore: bump spark-4.2 profile to the released 4.2.0 (#4960)
add c998e6f854 chore(deps): bump the all-other-cargo-deps group across 1
directory with 6 updates (#4956)
add c151a5771f chore(deps): bump lz4_flex from 0.13.1 to 0.14.0 in /native
(#4958)
add f480db1191 docs: add 0.17.1 changelog (#4961)
add cb71d700fb chore: codeql dependabot fix (#4962)
add 92f0f33e75 feat: fall back to Spark for collated predicate operands
(#4948)
add 415421bdb3 chore(deps): bump the codeql-actions group with 2 updates
(#4970)
add 9483db47f0 perf: vectorize `spark_unscaled_value` (9x faster) (#4972)
add f46719b46b test: promote `try_to_date`/`try_to_timestamp` SQL tests to
native coverage (#4973)
add b01655526d chore: remove dead Parquet parallel-IO configs (#4981)
add 957886db83 feat: add withAlternative alias mechanism for CometConf
(#4979)
add 32e2d7c4a8 refactor: remove untested async columnar shuffle (#4985)
add e3363e2cb6 deps: bump to datafusion 54.1.0 and latest iceberg-rust
(#4996)
add 86f50e9fd9 chore(deps): bump object_store from 0.13.2 to 0.14.1 in
/native
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (8e42ea6a81)
\
N -- N -- N
refs/heads/dependabot/cargo/native/main/object_store-0.14.1 (86f50e9fd9)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/dependabot.yml | 4 +
.github/workflows/ci.yml | 2 +-
.github/workflows/codeql.yml | 4 +-
.github/workflows/pr_build_linux.yml | 3 +-
.github/workflows/pr_build_macos.yml | 1 +
docs/source/changelog/0.17.1.md | 45 +++
docs/source/changelog/index.md | 1 +
.../contributor-guide/benchmark-results/tpc-h.md | 2 +-
.../source/contributor-guide/benchmarking_macos.md | 2 +-
.../source/contributor-guide/config_conventions.md | 115 ++++++
.../expression-audits/datetime_funcs.md | 8 +
docs/source/contributor-guide/index.md | 1 +
docs/source/contributor-guide/jvm_shuffle.md | 13 +-
.../contributor-guide/spark_configs_support.md | 44 ++-
docs/source/user-guide/latest/expressions.md | 8 +-
docs/source/user-guide/latest/tuning.md | 2 +-
native/Cargo.lock | 179 ++++-----
native/Cargo.toml | 12 +-
native/core/Cargo.toml | 4 +-
.../core/src/execution/operators/iceberg_scan.rs | 2 +
native/core/src/execution/operators/mod.rs | 2 +-
.../core/src/execution/operators/parquet_writer.rs | 71 +++-
native/core/src/execution/planner.rs | 16 +-
native/core/src/parquet/parquet_exec.rs | 32 +-
native/proto/src/proto/operator.proto | 2 +
native/shuffle/Cargo.toml | 2 +-
native/shuffle/src/schema_align.rs | 27 +-
native/shuffle/src/spark_unsafe/list.rs | 8 +-
native/shuffle/src/spark_unsafe/row.rs | 36 +-
native/spark-expr/Cargo.toml | 4 +
native/spark-expr/benches/unscaled_value.rs | 62 +++
native/spark-expr/src/array_funcs/size.rs | 141 +++++--
native/spark-expr/src/hash_funcs/utils.rs | 17 +
.../src/math_funcs/internal/unscaled_value.rs | 63 +++-
pom.xml | 14 +-
spark/pom.xml | 4 +-
.../shuffle/sort/CometShuffleExternalSorter.java | 317 +++++++++++++---
.../sort/CometShuffleExternalSorterAsync.java | 417 ---------------------
.../sort/CometShuffleExternalSorterSync.java | 346 -----------------
.../shuffle/CometBypassMergeSortShuffleWriter.java | 21 --
.../execution/shuffle/CometDiskBlockWriter.java | 91 +----
.../shuffle/CometUnsafeShuffleWriter.java | 2 +-
.../comet/execution/shuffle/ShuffleThreadPool.java | 71 ----
.../main/scala/org/apache/comet/CometConf.scala | 162 ++++----
.../org/apache/comet/codegen/CometArrayData.scala | 4 +-
.../comet/codegen/CometBatchKernelCodegen.scala | 6 +-
.../codegen/CometBatchKernelCodegenInput.scala | 15 +-
.../codegen/CometBatchKernelCodegenOutput.scala | 8 +-
.../org/apache/comet/rules/CometExecRule.scala | 2 +-
.../rules/EliminateRedundantTransitions.scala | 49 ++-
.../org/apache/comet/serde/QueryPlanSerde.scala | 6 +-
.../main/scala/org/apache/comet/serde/arrays.scala | 6 +-
.../apache/comet/serde/contraintExpressions.scala | 56 +--
.../main/scala/org/apache/comet/serde/maps.scala | 11 +-
.../org/apache/comet/serde/namedExpressions.scala | 76 ++--
.../serde/operator/CometDataWritingCommand.scala | 13 +-
.../scala/org/apache/comet/serde/predicates.scala | 104 ++++-
.../scala/org/apache/comet/serde/strings.scala | 36 +-
.../shuffle/CometShuffleExchangeExec.scala | 6 +
.../execution/shuffle/CometShuffleManager.scala | 5 +-
.../apache/comet/shims/CometInternalRowShim.scala | 5 +-
.../apache/comet/shims/CometInternalRowShim.scala | 4 +-
.../apache/comet/shims/CometInternalRowShim.scala | 0
.../apache/comet/shims/CometInternalRowShim.scala | 15 +-
.../sql-tests/expressions/array/posexplode.sql | 4 +-
.../resources/sql-tests/expressions/array/size.sql | 18 +-
.../sql-tests/expressions/datetime/make_time.sql | 13 +-
.../expressions/datetime/make_time_shuffle.sql | 64 ++++
...l => make_time_shuffle_range_partition_jvm.sql} | 20 +-
...> make_time_shuffle_range_partition_native.sql} | 34 +-
.../expressions/datetime/try_datetime.sql | 34 +-
.../try_to_timestamp_time_parser_policy.sql | 12 +-
...y_to_timestamp_time_parser_policy_corrected.sql | 4 +-
...y_to_timestamp_time_parser_policy_exception.sql | 12 +-
.../try_to_timestamp_time_parser_policy_legacy.sql | 4 +-
.../expressions/map/str_to_map_collation.sql | 9 +-
.../expressions/map/str_to_map_legacy_truncate.sql | 19 +-
.../sql-tests/expressions/string/collation.sql | 123 ++++++
.../approved-plans-v1_4-spark4_2/q2/extended.txt | 60 ---
.../approved-plans-v1_4-spark4_2/q5/extended.txt | 107 ------
.../approved-plans-v1_4-spark4_2/q54/extended.txt | 113 ------
.../approved-plans-v2_7-spark4_2/q5a/extended.txt | 320 ----------------
.../approved-plans-v2_7-spark4_2/q77a/extended.txt | 362 ++++++++++++++++++
.../scala/org/apache/comet/CometConfSuite.scala | 124 ++++++
.../org/apache/comet/CometExpressionSuite.scala | 17 +-
.../org/apache/comet/CometFuzzIcebergBase.scala | 25 +-
.../org/apache/comet/CometIcebergNativeSuite.scala | 1 +
.../org/apache/comet/CometIcebergTestBase.scala | 18 +-
.../org/apache/comet/CometMapExpressionSuite.scala | 51 ++-
.../scala/org/apache/comet/CometS3TestBase.scala | 17 +-
.../comet/exec/CometColumnarShuffleSuite.scala | 52 +--
.../comet/parquet/CometParquetWriterSuite.scala | 149 ++++++++
.../apache/comet/parquet/ParquetReadSuite.scala | 86 -----
.../spark/sql/benchmark/CometExecBenchmark.scala | 1 -
.../sql/benchmark/CometShuffleBenchmark.scala | 33 +-
.../apache/comet/exec/CometShuffle4_0Suite.scala | 2 -
.../spark/sql/comet/CometMapInBatchSuite.scala | 23 +-
97 files changed, 2429 insertions(+), 2309 deletions(-)
create mode 100644 docs/source/changelog/0.17.1.md
create mode 100644 docs/source/contributor-guide/config_conventions.md
create mode 100644 native/spark-expr/benches/unscaled_value.rs
delete mode 100644
spark/src/main/java/org/apache/spark/shuffle/sort/CometShuffleExternalSorterAsync.java
delete mode 100644
spark/src/main/java/org/apache/spark/shuffle/sort/CometShuffleExternalSorterSync.java
delete mode 100644
spark/src/main/java/org/apache/spark/sql/comet/execution/shuffle/ShuffleThreadPool.java
copy spark/src/main/{spark-4.1+ =>
spark-4.1}/org/apache/comet/shims/CometInternalRowShim.scala (100%)
rename spark/src/main/{spark-4.1+ =>
spark-4.2}/org/apache/comet/shims/CometInternalRowShim.scala (71%)
create mode 100644
spark/src/test/resources/sql-tests/expressions/datetime/make_time_shuffle.sql
copy
spark/src/test/resources/sql-tests/expressions/datetime/{to_unix_timestamp_time_parser_policy_exception.sql
=> make_time_shuffle_range_partition_jvm.sql} (66%)
copy
spark/src/test/resources/sql-tests/expressions/datetime/{current_timezone.sql
=> make_time_shuffle_range_partition_native.sql} (61%)
delete mode 100644
spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q2/extended.txt
delete mode 100644
spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q5/extended.txt
delete mode 100644
spark/src/test/resources/tpcds-plan-stability/approved-plans-v1_4-spark4_2/q54/extended.txt
delete mode 100644
spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q5a/extended.txt
create mode 100644
spark/src/test/resources/tpcds-plan-stability/approved-plans-v2_7-spark4_2/q77a/extended.txt
create mode 100644 spark/src/test/scala/org/apache/comet/CometConfSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
