This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
from 1182a1c614 fix: plan Iceberg writes with Spark's own operator when
Comet is disabled (#6151)
add 590b669a03 feat: positional round robin shuffle keyed on a row ordinal
(#6095)
No new revisions were added by this update.
Summary of changes:
.ai/skills/review-comet-shuffle-pr/SKILL.md | 25 +-
.github/workflows/pr_build_linux.yml | 1 +
.github/workflows/pr_build_macos.yml | 1 +
docs/source/contributor-guide/native_shuffle.md | 174 +++++-
docs/source/user-guide/latest/metrics.md | 2 +-
native/core/src/execution/planner.rs | 30 +-
native/proto/src/proto/partitioning.proto | 12 +
native/shuffle/benches/shuffle_writer.rs | 293 ++++++++--
native/shuffle/src/bench_support.rs | 121 ++++
native/shuffle/src/bin/shuffle_bench.rs | 8 +-
native/shuffle/src/comet_partitioning.rs | 99 +++-
native/shuffle/src/lib.rs | 5 +-
native/shuffle/src/partitioners/multi_partition.rs | 650 +++++++++++++++++----
.../src/partitioners/partitioned_batch_iterator.rs | 362 +++++++++++-
native/shuffle/src/rss_execution_tests.rs | 8 +-
native/shuffle/src/shuffle_writer.rs | 162 ++++-
.../main/scala/org/apache/comet/CometConf.scala | 40 ++
.../shuffle/CometNativeShuffleInputRDD.scala | 38 +-
.../shuffle/CometNativeShuffleWriter.scala | 16 +
.../execution/shuffle/CometShuffleDependency.scala | 19 +-
.../shuffle/CometShuffleExchangeExec.scala | 127 +++-
.../CometCelebornShufflePlanningSuite.scala | 24 +-
.../CometNativePositionalRoundRobinSuite.scala | 291 +++++++++
.../shuffle/CometNativeShuffleInputRDDSuite.scala | 31 +
24 files changed, 2286 insertions(+), 253 deletions(-)
create mode 100644 native/shuffle/src/bench_support.rs
create mode 100644
spark/src/test/scala/org/apache/spark/sql/comet/execution/shuffle/CometNativePositionalRoundRobinSuite.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
