This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 2245c86 doc: Update README.md with shuffle configs (#208)
2245c86 is described below
commit 2245c86eb69f6c5a332748b5070c3d41aa58d8ad
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Thu Mar 14 14:28:10 2024 -0700
doc: Update README.md with shuffle configs (#208)
* doc: Update README.md with shuffle configs
* Update README.md
---
README.md | 19 ++++++++++++++++++-
1 file changed, 18 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 572a9d2..3b903b1 100644
--- a/README.md
+++ b/README.md
@@ -113,4 +113,21 @@ INFO src/lib.rs: Comet native library initialized
+- CometScan parquet [a#14] Batched: true, DataFilters:
[isnotnull(a#14), (a#14 > 5)],
Format: CometParquet, Location: InMemoryFileIndex(1
paths)[file:/tmp/test], PartitionFilters: [],
PushedFilters: [IsNotNull(a), GreaterThan(a,5)], ReadSchema:
struct<a:int>
-```
\ No newline at end of file
+```
+
+### Enable Comet shuffle
+
+Comet shuffle feature is disabled by default. To enable it, please add related
configs:
+
+```
+--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
+--conf spark.comet.exec.shuffle.enabled=true
+```
+
+Above configs enable Comet native shuffle which only supports hash partiting
and single partition.
+Comet native shuffle doesn't support complext types yet.
+
+To enable columnar shuffle which supports all partitioning and basic complex
types, one more config is required:
+```
+--conf spark.comet.columnar.shuffle.enabled=true
+```
