This is an automated email from the ASF dual-hosted git repository.
agrove pushed a change to branch branch-0.13
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
from 6a2209d2d docs: Stop generating dynamic docs content in build (#3212)
add ec5df97b4 feat: Add support for round-robin partitioning in native
shuffle (#3076)
add 524eaa121 feat: Native columnar to row conversion (Phase 1) (#3221)
add 1b75777d3 fix: remove dead code for 0.8/0.9 docs that broke CI (#3264)
add 1a000abba refactor: rename scan.allowIncompatible to
scan.unsignedSmallIntSafetyCheck (#3238)
add deaec6f92 docs: document datetime rebasing and V2 API limitations for
DataFusion-based scans (#3259)
add f40a396e6 fix: Prevent native write when input is not Arrow format
(#3227)
add 5860dd2a0 Add JDK to Docker image for release build (#3262)
No new revisions were added by this update.
Summary of changes:
.github/workflows/pr_build_linux.yml | 1 +
.github/workflows/pr_build_macos.yml | 1 +
.gitignore | 1 +
.../main/scala/org/apache/comet/CometConf.scala | 56 +-
.../scala/org/apache/comet/vector/NativeUtil.scala | 20 +
.../org/apache/spark/sql/comet/util/Utils.scala | 10 +-
dev/release/comet-rm/Dockerfile | 5 +-
docs/build.sh | 6 -
docs/source/contributor-guide/jvm_shuffle.md | 8 +-
docs/source/contributor-guide/native_shuffle.md | 22 +-
docs/source/contributor-guide/parquet_scans.md | 22 +-
docs/source/user-guide/latest/compatibility.md | 26 +
native/core/src/execution/columnar_to_row.rs | 2751 ++++++++++++++++++++
native/core/src/execution/jni_api.rs | 118 +
native/core/src/execution/mod.rs | 1 +
native/core/src/execution/planner.rs | 12 +
.../src/execution/shuffle/comet_partitioning.rs | 6 +-
.../core/src/execution/shuffle/shuffle_writer.rs | 161 +-
native/proto/src/proto/partitioning.proto | 7 +
.../org/apache/comet/NativeColumnarToRowInfo.java | 75 +
.../org/apache/comet/ExtendedExplainInfo.scala | 4 +-
spark/src/main/scala/org/apache/comet/Native.scala | 45 +
.../comet/NativeColumnarToRowConverter.scala | 144 +
.../org/apache/comet/rules/CometExecRule.scala | 15 +
.../org/apache/comet/rules/CometScanRule.scala | 10 +-
.../rules/EliminateRedundantTransitions.scala | 26 +-
.../apache/comet/serde/CometOperatorSerde.scala | 7 +
.../serde/operator/CometDataWritingCommand.scala | 4 +
.../sql/comet/CometNativeColumnarToRowExec.scala | 138 +
.../shuffle/CometNativeShuffleWriter.scala | 12 +-
.../shuffle/CometShuffleExchangeExec.scala | 31 +-
.../scala/org/apache/comet/CometFuzzTestBase.scala | 2 +-
.../comet/exec/CometNativeColumnarToRowSuite.scala | 484 ++++
.../comet/exec/CometNativeShuffleSuite.scala | 49 +
.../comet/parquet/CometParquetWriterSuite.scala | 130 +-
.../apache/comet/parquet/ParquetReadSuite.scala | 2 +-
.../apache/comet/rules/CometScanRuleSuite.scala | 14 +-
.../scala/org/apache/spark/sql/CometTestBase.scala | 6 +-
.../benchmark/CometColumnarToRowBenchmark.scala | 393 +++
39 files changed, 4732 insertions(+), 93 deletions(-)
create mode 100644 native/core/src/execution/columnar_to_row.rs
create mode 100644
spark/src/main/java/org/apache/comet/NativeColumnarToRowInfo.java
create mode 100644
spark/src/main/scala/org/apache/comet/NativeColumnarToRowConverter.scala
create mode 100644
spark/src/main/scala/org/apache/spark/sql/comet/CometNativeColumnarToRowExec.scala
create mode 100644
spark/src/test/scala/org/apache/comet/exec/CometNativeColumnarToRowSuite.scala
create mode 100644
spark/src/test/scala/org/apache/spark/sql/benchmark/CometColumnarToRowBenchmark.scala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
