This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/cargo/native/main/object_store-0.13.0
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
omit 407b818a7 chore(deps): bump object_store from 0.12.4 to 0.13.0 in
/native
add 937619f48 chore: Add microbenchmark for casting string to numeric
(#2979)
add 6ec70aeaa perf: Improve performance of normalize_nan (#2999)
add 2f1955165 chore: Skip some CI workflows for benchmark changes (#3030)
add 2738f0625 skip more workflows on benchmark PRs (#3034)
add 37cb5c90c chore: Improve microbenchmark for string expressions (#2964)
add 726c4b77a fix: format decimal to string when casting decimal with
overflow (#2916)
add be05e242d chore(deps): bump tokio from 1.48.0 to 1.49.0 in /native
(#3039)
add 10021fae7 chore(deps): bump libc from 0.2.178 to 0.2.179 in /native
(#3038)
add 394eb5db3 chore(deps): bump actions/cache from 4 to 5 (#3037)
add c08a3ac61 perf: Improve string expression microbenchmarks (#3012)
add af6d335b5 perf: Improve date/time microbenchmarks to avoid
redundant/duplicate benchmarks (#3020)
add 01a586f3a perf: Improve aggregate expression microbenchmarks (#3021)
add e041a1ef8 perf: Improve conditional expression microbenchmarks (#3024)
add 4dac7450c feat: Improve performance of date truncate (#2997)
add 1e525a4e2 perf: Add microbenchmark for comparison expressions (#3026)
add 89ebbd735 perf: Implement more microbenchmarks for cast expressions
(#3031)
add 092d88cee perf: Add microbenchmark for hash expressions (#3028)
add 069681a16 perf: Improve performance of CAST from string to int (#3017)
add edec4612a Chore: to_json unit/benchmark tests (#3011)
add 91f0f7b13 fix: reduce granularity of metrics updates in
IcebergFileStream. (#3050)
add 77899ee1a chore: Add checks to microbenchmarks for plan running
natively in Comet (#3045)
add af3bd81e1 perf: Improve criterion benchmarks for cast string to int
(#3049)
add 23f919463 chore: Refactor `CometScanRule` to improve scan selection
and fallback logic (#2978)
add c7aad6716 perf: Additional optimizations for cast from string to int
(#3048)
add 69b22601c perf: set DataFusion session context’s target_partitions to
match Spark's spark.task.cpus. (#3062)
add 3a6452b3a perf: don't busy-poll Tokio stream for plans without
CometScan. (#3063)
add fbde8b0fb perf: minor optimizations in `process_sorted_row_partition`
(#3059)
add b038ac531 chore: Respect to legacySizeOfNull option for size function
(#3036)
add c112f473d docs: Add documentation to contributor guide explaining
native + JVM shuffle implementation (#3055)
add cf8cba0bc chore: Add PySpark-based benchmarks, starting with ETL
example (#3065)
add b95d38205 chore(deps): bump the proto group in /native with 2 updates
(#3071)
add 757f9d8d2 chore: add MacOS file and event trace log to gitignore
(#3070)
add 629b7861e chore(deps): bump arrow from 57.1.0 to 57.2.0 in /native
(#3073)
add c3f22eb83 chore(deps): bump parquet from 57.1.0 to 57.2.0 in /native
(#3074)
add 009bf4773 chore(deps): bump cc from 1.2.50 to 1.2.52 in /native (#3072)
add c4da3c91d chore: improve cast documentation to add support per eval
mode (#3056)
add 33f514a41 feat: Expand `murmur3` hash support to complex types (#3077)
add 79b83d873 chore: Refactor JVM shuffle: Move `SpillSorter` to top level
class and add tests (#3081)
add aff07d088 feat: Comet Writer should respect object store settings
(#3042)
add 4bd664e5e perf: optimize complex-type hash implementations (#3140)
add 6c34c7a59 fix: native shuffle now reports spill metrics correctly
(#3197)
add cb49ff6ca chore: split CometShuffleExternalSorter into sync/async
implementations (#3192)
add 395f7de4c chore: Add pending PR shield (#3205)
add d19e1f24f docs: add guidance on disabling constant folding for literal
tests (#3200)
add 5fa1e6942 feat: add support for unix_date expression (#3141)
add 1009e9854 chore: deprecate native_comet scan in favor of
native_iceberg_compat (#2949)
add 68f127b75 feat: add partial support for date_format expression (#3201)
add 458917286 feat: add complex type support to native Parquet writer
(#3214)
add 57780bc6f perf: [iceberg] Remove IcebergFileStream, use iceberg-rust's
parallelization, bump iceberg-rust to latest, cache SchemaAdapter (#3051)
add 27d8ed6eb chore(deps): bump object_store from 0.12.4 to 0.13.0 in
/native
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (407b818a7)
\
N -- N -- N
refs/heads/dependabot/cargo/native/main/object_store-0.13.0 (27d8ed6eb)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/benchmark-tpcds.yml | 6 +
.github/workflows/benchmark-tpch.yml | 6 +
.github/workflows/iceberg_spark_test.yml | 6 +
.github/workflows/miri.yml | 6 +
.github/workflows/pr_benchmark_check.yml | 85 +++
.github/workflows/pr_build_linux.yml | 8 +
.github/workflows/pr_build_macos.yml | 8 +
.github/workflows/spark_sql_test.yml | 10 +
.gitignore | 2 +
README.md | 3 +
benchmarks/pyspark/README.md | 97 +++
benchmarks/pyspark/generate_data.py | 446 ++++++++++++
benchmarks/pyspark/run_all_benchmarks.sh | 120 ++++
benchmarks/pyspark/run_benchmark.py | 104 +++
.../main/scala/org/apache/comet/CometConf.scala | 17 +-
dev/ensure-jars-have-correct-contents.sh | 1 +
.../contributor-guide/adding_a_new_expression.md | 23 +
docs/source/contributor-guide/index.md | 2 +
docs/source/contributor-guide/jvm_shuffle.md | 199 ++++++
docs/source/contributor-guide/native_shuffle.md | 243 +++++++
docs/source/user-guide/latest/compatibility.md | 194 +++---
docs/source/user-guide/latest/configs.md | 2 +
native/Cargo.lock | 760 ++++++++++-----------
native/Cargo.toml | 8 +-
native/core/Cargo.toml | 3 +-
native/core/src/execution/jni_api.rs | 32 +-
.../core/src/execution/operators/iceberg_scan.rs | 330 +++------
.../core/src/execution/operators/parquet_writer.rs | 128 ++--
native/core/src/execution/planner.rs | 8 +
native/core/src/execution/shuffle/row.rs | 36 +-
.../core/src/execution/shuffle/shuffle_writer.rs | 2 +-
native/core/src/parquet/parquet_support.rs | 22 +-
native/proto/Cargo.toml | 4 +-
native/proto/src/proto/operator.proto | 7 +
native/spark-expr/Cargo.toml | 8 +
native/spark-expr/benches/cast_from_string.rs | 98 ++-
.../benches/date_trunc.rs} | 45 +-
native/spark-expr/benches/normalize_nan.rs | 88 +++
native/spark-expr/src/conversion_funcs/cast.rs | 459 ++++++++-----
native/spark-expr/src/hash_funcs/murmur3.rs | 3 +-
native/spark-expr/src/hash_funcs/utils.rs | 615 ++++++++++++++++-
native/spark-expr/src/hash_funcs/xxhash64.rs | 3 +-
native/spark-expr/src/kernels/mod.rs | 2 +-
native/spark-expr/src/kernels/temporal.rs | 197 +++---
native/spark-expr/src/lib.rs | 3 +-
.../src/math_funcs/internal/normalize_nan.rs | 80 +--
.../shuffle/sort/CometShuffleExternalSorter.java | 633 ++---------------
.../sort/CometShuffleExternalSorterAsync.java | 405 +++++++++++
.../sort/CometShuffleExternalSorterSync.java | 334 +++++++++
.../org/apache/spark/shuffle/sort/SpillSorter.java | 352 ++++++++++
.../shuffle/CometUnsafeShuffleWriter.java | 2 +-
.../scala/org/apache/comet/CometExecIterator.scala | 2 +
.../main/scala/org/apache/comet/GenerateDocs.scala | 106 ++-
spark/src/main/scala/org/apache/comet/Native.scala | 1 +
.../org/apache/comet/rules/CometScanRule.scala | 113 ++-
.../org/apache/comet/serde/QueryPlanSerde.scala | 2 +
.../main/scala/org/apache/comet/serde/arrays.scala | 37 +-
.../scala/org/apache/comet/serde/datetime.scala | 131 +++-
.../main/scala/org/apache/comet/serde/hash.scala | 35 +-
.../serde/operator/CometDataWritingCommand.scala | 26 +-
.../scala/org/apache/comet/serde/structs.scala | 39 +-
.../apache/comet/testing/FuzzDataGenerator.scala | 11 +-
.../sql/comet/CometIcebergNativeScanExec.scala | 15 +-
.../shuffle/CometNativeShuffleWriter.scala | 8 +
.../shuffle/CometShuffleExchangeExec.scala | 11 +-
.../apache/comet/CometArrayExpressionSuite.scala | 19 +
.../scala/org/apache/comet/CometCastSuite.scala | 20 +
.../apache/comet/CometHashExpressionSuite.scala | 463 +++++++++++++
.../org/apache/comet/CometIcebergNativeSuite.scala | 4 -
.../apache/comet/CometJsonExpressionSuite.scala | 39 +-
.../comet/CometTemporalExpressionSuite.scala | 141 +++-
.../comet/parquet/CometParquetWriterSuite.scala | 467 ++++++++++---
.../spark/shuffle/sort/SpillSorterSuite.scala | 262 +++++++
.../scala/org/apache/spark/sql/CometTestBase.scala | 33 +-
.../sql/benchmark/CometAggregateBenchmark.scala | 298 --------
.../CometAggregateExpressionBenchmark.scala | 137 ++++
.../spark/sql/benchmark/CometBenchmarkBase.scala | 52 +-
.../spark/sql/benchmark/CometCastBenchmark.scala | 96 ---
.../sql/benchmark/CometCastBooleanBenchmark.scala | 124 ++++
.../CometCastNumericToNumericBenchmark.scala | 147 ++++
.../CometCastNumericToStringBenchmark.scala | 104 +++
.../CometCastNumericToTemporalBenchmark.scala | 101 +++
.../CometCastStringToNumericBenchmark.scala | 98 +++
.../CometCastStringToTemporalBenchmark.scala | 10 +-
.../CometCastTemporalToNumericBenchmark.scala | 105 +++
.../CometCastTemporalToStringBenchmark.scala | 98 +++
.../CometCastTemporalToTemporalBenchmark.scala | 100 +++
.../CometComparisonExpressionBenchmark.scala | 94 +++
.../CometConditionalExpressionBenchmark.scala | 173 ++++-
.../CometDatetimeExpressionBenchmark.scala | 33 +-
.../benchmark/CometHashExpressionBenchmark.scala | 241 +++++++
.../benchmark/CometJsonExpressionBenchmark.scala | 64 +-
.../benchmark/CometStringExpressionBenchmark.scala | 75 +-
.../apache/spark/sql/comet/CometPlanChecker.scala | 68 ++
94 files changed, 7873 insertions(+), 2585 deletions(-)
create mode 100644 .github/workflows/pr_benchmark_check.yml
create mode 100644 benchmarks/pyspark/README.md
create mode 100755 benchmarks/pyspark/generate_data.py
create mode 100755 benchmarks/pyspark/run_all_benchmarks.sh
create mode 100755 benchmarks/pyspark/run_benchmark.py
create mode 100644 docs/source/contributor-guide/jvm_shuffle.md
create mode 100644 docs/source/contributor-guide/native_shuffle.md
copy native/{core/benches/parquet_decode.rs =>
spark-expr/benches/date_trunc.rs} (53%)
create mode 100644 native/spark-expr/benches/normalize_nan.rs
create mode 100644
spark/src/main/java/org/apache/spark/shuffle/sort/CometShuffleExternalSorterAsync.java
create mode 100644
spark/src/main/java/org/apache/spark/shuffle/sort/CometShuffleExternalSorterSync.java
create mode 100644
spark/src/main/java/org/apache/spark/shuffle/sort/SpillSorter.java
create mode 100644
spark/src/test/scala/org/apache/comet/CometHashExpressionSuite.scala
create mode 100644
spark/src/test/scala/org/apache/spark/shuffle/sort/SpillSorterSuite.scala
delete mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometAggregateBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometAggregateExpressionBenchmark.scala
delete mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastBooleanBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastNumericToNumericBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastNumericToStringBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastNumericToTemporalBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastStringToNumericBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastTemporalToNumericBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastTemporalToStringBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometCastTemporalToTemporalBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometComparisonExpressionBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometHashExpressionBenchmark.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/comet/CometPlanChecker.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
