This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 5c47505c Publish built docs triggered by
fe071e0833a76686acee4d6478cf50037bb8008c
5c47505c is described below
commit 5c47505c05323e71f0b9175edf783133e0774abc
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri May 17 22:17:43 2024 +0000
Publish built docs triggered by fe071e0833a76686acee4d6478cf50037bb8008c
---
_sources/index.rst.txt | 1 +
_sources/user-guide/tuning.md.txt | 60 ++++++++++++
contributor-guide/contributing.html | 11 ++-
contributor-guide/debugging.html | 5 +
contributor-guide/development.html | 5 +
contributor-guide/plugin_overview.html | 5 +
contributor-guide/profiling_native_code.html | 5 +
genindex.html | 5 +
index.html | 6 ++
objects.inv | Bin 570 -> 588 bytes
search.html | 5 +
searchindex.js | 2 +-
user-guide/compatibility.html | 11 ++-
user-guide/configs.html | 5 +
user-guide/datatypes.html | 5 +
user-guide/expressions.html | 5 +
user-guide/installation.html | 5 +
user-guide/operators.html | 5 +
user-guide/overview.html | 5 +
user-guide/{overview.html => tuning.html} | 136 ++++++++++++++-------------
20 files changed, 214 insertions(+), 73 deletions(-)
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index 9066ce75..eb42950b 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -47,6 +47,7 @@ as a native runtime to achieve improvement in terms of query
efficiency and quer
Supported Data Types <user-guide/datatypes>
Configuration Settings <user-guide/configs>
Compatibility Guide <user-guide/compatibility>
+ Tuning Guide <user-guide/tuning>
.. _toc.contributor-guide-links:
.. toctree::
diff --git a/_sources/user-guide/tuning.md.txt
b/_sources/user-guide/tuning.md.txt
new file mode 100644
index 00000000..01fa7bdb
--- /dev/null
+++ b/_sources/user-guide/tuning.md.txt
@@ -0,0 +1,60 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Tuning Guide
+
+Comet provides some tuning options to help you get the best performance from
your queries.
+
+
+## Shuffle
+
+Comet provides Comet shuffle features that can be used to improve the
performance of your queries.
+The following sections describe the different shuffle options available in
Comet.
+
+To enable Comet shuffle, set the following configuration in your Spark
configuration:
+
+```
+spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
+spark.comet.exec.shuffle.enabled=true
+```
+
+`spark.shuffle.manager` is a Spark static configuration which cannot be
changed at runtime.
+It must be set before the Spark context is created. You can enable or disable
Comet shuffle
+at runtime by setting `spark.comet.exec.shuffle.enabled` to `true` or `false`.
+Once it is disabled, Comet will fallback to the default Spark shuffle manager.
+
+### Columnar Shuffle
+
+By default, once `spark.comet.exec.shuffle.enabled` is enabled, Comet uses
columnar shuffle
+to improve the performance of shuffle operations. Columnar shuffle supports
HashPartitioning,
+RoundRobinPartitioning, RangePartitioning and SinglePartitioning.
+
+Columnar shuffle can be disabled by setting
`spark.comet.columnar.shuffle.enabled` to `false`.
+
+### Native Shuffle
+
+Comet also provides a fully native shuffle implementation that can be used to
improve the performance.
+To enable native shuffle, just disable `spark.comet.columnar.shuffle.enabled`.
+
+Native shuffle only supports HashPartitioning and SinglePartitioning.
+
+
+
+
+
diff --git a/contributor-guide/contributing.html
b/contributor-guide/contributing.html
index 650e414e..6c7669c6 100644
--- a/contributor-guide/contributing.html
+++ b/contributor-guide/contributing.html
@@ -54,7 +54,7 @@ under the License.
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Comet Plugin Overview" href="plugin_overview.html"
/>
- <link rel="prev" title="Compatibility Guide"
href="../user-guide/compatibility.html" />
+ <link rel="prev" title="Tuning Guide" href="../user-guide/tuning.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -349,11 +354,11 @@ coordinate on issues that they are working on.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link"
href="../user-guide/compatibility.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="../user-guide/tuning.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Compatibility Guide</p>
+ <p class="prev-next-title">Tuning Guide</p>
</div>
</a>
<a class='right-next' id="next-link" href="plugin_overview.html"
title="next page">
diff --git a/contributor-guide/debugging.html b/contributor-guide/debugging.html
index 0912686b..2fd01106 100644
--- a/contributor-guide/debugging.html
+++ b/contributor-guide/debugging.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/contributor-guide/development.html
b/contributor-guide/development.html
index e87aaae8..2c6b5722 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/contributor-guide/plugin_overview.html
b/contributor-guide/plugin_overview.html
index b7a57303..de6d0c33 100644
--- a/contributor-guide/plugin_overview.html
+++ b/contributor-guide/plugin_overview.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/contributor-guide/profiling_native_code.html
b/contributor-guide/profiling_native_code.html
index dfd434dc..5934a64d 100644
--- a/contributor-guide/profiling_native_code.html
+++ b/contributor-guide/profiling_native_code.html
@@ -142,6 +142,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="../user-guide/tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/genindex.html b/genindex.html
index 82301075..921785d0 100644
--- a/genindex.html
+++ b/genindex.html
@@ -140,6 +140,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/index.html b/index.html
index 262ab537..4c90c937 100644
--- a/index.html
+++ b/index.html
@@ -142,6 +142,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -288,6 +293,7 @@ as a native runtime to achieve improvement in terms of
query efficiency and quer
<li class="toctree-l1"><a class="reference internal"
href="user-guide/datatypes.html">Supported Data Types</a></li>
<li class="toctree-l1"><a class="reference internal"
href="user-guide/configs.html">Configuration Settings</a></li>
<li class="toctree-l1"><a class="reference internal"
href="user-guide/compatibility.html">Compatibility Guide</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="user-guide/tuning.html">Tuning Guide</a></li>
</ul>
</div>
<div class="toctree-wrapper compound" id="toc-contributor-guide-links">
diff --git a/objects.inv b/objects.inv
index 1235694e..c2151557 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/search.html b/search.html
index 8f95d17d..5cc146d4 100644
--- a/search.html
+++ b/search.html
@@ -147,6 +147,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="user-guide/tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/searchindex.js b/searchindex.js
index cc6b5c56..f80e6a1e 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"ANSI mode": [[6, "ansi-mode"]], "ASF Links":
[[5, null]], "Additional Info": [[1, "additional-info"]], "After your debugging
is done": [[1, "after-your-debugging-is-done"]], "Apache DataFusion Comet":
[[5, "apache-datafusion-comet"]], "Architecture": [[12, "architecture"]],
"Asking for Help": [[0, "asking-for-help"]], "Benchmark": [[2, "benchmark"]],
"Build & Test": [[2, "build-test"]], "Building From Source": [[10,
"building-from-source"]], "CLion": [[2, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"ANSI mode": [[6, "ansi-mode"]], "ASF Links":
[[5, null]], "Additional Info": [[1, "additional-info"]], "After your debugging
is done": [[1, "after-your-debugging-is-done"]], "Apache DataFusion Comet":
[[5, "apache-datafusion-comet"]], "Architecture": [[12, "architecture"]],
"Asking for Help": [[0, "asking-for-help"]], "Benchmark": [[2, "benchmark"]],
"Build & Test": [[2, "build-test"]], "Building From Source": [[10,
"building-from-source"]], "CLion": [[2, [...]
\ No newline at end of file
diff --git a/user-guide/compatibility.html b/user-guide/compatibility.html
index 81765e90..706fca14 100644
--- a/user-guide/compatibility.html
+++ b/user-guide/compatibility.html
@@ -53,7 +53,7 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Contributing to Apache DataFusion Comet"
href="../contributor-guide/contributing.html" />
+ <link rel="next" title="Tuning Guide" href="tuning.html" />
<link rel="prev" title="Comet Configuration Settings" href="configs.html"
/>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -688,10 +693,10 @@ Spark.</p></li>
<p class="prev-next-title">Comet Configuration Settings</p>
</div>
</a>
- <a class='right-next' id="next-link"
href="../contributor-guide/contributing.html" title="next page">
+ <a class='right-next' id="next-link" href="tuning.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Contributing to Apache DataFusion Comet</p>
+ <p class="prev-next-title">Tuning Guide</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 129a6e58..ccbbb353 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/datatypes.html b/user-guide/datatypes.html
index dfbe5046..051a2726 100644
--- a/user-guide/datatypes.html
+++ b/user-guide/datatypes.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/expressions.html b/user-guide/expressions.html
index a6b5f549..153f74e0 100644
--- a/user-guide/expressions.html
+++ b/user-guide/expressions.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/installation.html b/user-guide/installation.html
index d342f80f..675c13ad 100644
--- a/user-guide/installation.html
+++ b/user-guide/installation.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/operators.html b/user-guide/operators.html
index ef7eef36..99b60cb1 100644
--- a/user-guide/operators.html
+++ b/user-guide/operators.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/overview.html b/user-guide/overview.html
index a6d1ff10..f466c163 100644
--- a/user-guide/overview.html
+++ b/user-guide/overview.html
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="tuning.html">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
diff --git a/user-guide/overview.html b/user-guide/tuning.html
similarity index 64%
copy from user-guide/overview.html
copy to user-guide/tuning.html
index a6d1ff10..d9434c72 100644
--- a/user-guide/overview.html
+++ b/user-guide/tuning.html
@@ -24,7 +24,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Comet Overview — Apache DataFusion Comet
documentation</title>
+ <title>Tuning Guide — Apache DataFusion Comet documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
@@ -53,8 +53,8 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Installing DataFusion Comet"
href="installation.html" />
- <link rel="prev" title="Apache DataFusion Comet" href="../index.html" />
+ <link rel="next" title="Contributing to Apache DataFusion Comet"
href="../contributor-guide/contributing.html" />
+ <link rel="prev" title="Compatibility Guide" href="compatibility.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -108,8 +108,8 @@ under the License.
</span>
</p>
<ul class="current nav bd-sidenav">
- <li class="toctree-l1 current active">
- <a class="current reference internal" href="#">
+ <li class="toctree-l1">
+ <a class="reference internal" href="overview.html">
Comet Overview
</a>
</li>
@@ -143,6 +143,11 @@ under the License.
Compatibility Guide
</a>
</li>
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Tuning Guide
+ </a>
+ </li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -247,19 +252,21 @@ under the License.
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#architecture">
- Architecture
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#current-status">
- Current Status
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#feature-parity-with-apache-spark">
- Feature Parity with Apache Spark
+ <a class="reference internal nav-link" href="#shuffle">
+ Shuffle
</a>
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#columnar-shuffle">
+ Columnar Shuffle
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#native-shuffle">
+ Native Shuffle
+ </a>
+ </li>
+ </ul>
</li>
</ul>
@@ -270,7 +277,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/overview.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/user-guide/tuning.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -290,55 +297,52 @@ under the License.
<div>
<!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
-->
-<section id="comet-overview">
-<h1>Comet Overview<a class="headerlink" href="#comet-overview" title="Link to
this heading">¶</a></h1>
-<p>Comet runs Spark SQL queries using the native Apache DataFusion runtime,
which is
-typically faster and more resource efficient than JVM based runtimes.</p>
-<p><img alt="Comet Overview" src="../_images/comet-overview.png" /></p>
-<p>Comet aims to support:</p>
-<ul class="simple">
-<li><p>a native Parquet implementation, including both reader and
writer</p></li>
-<li><p>full implementation of Spark operators, including
-Filter/Project/Aggregation/Join/Exchange etc.</p></li>
-<li><p>full implementation of Spark built-in expressions</p></li>
-<li><p>a UDF framework for users to migrate their existing UDF to
native</p></li>
-</ul>
-<section id="architecture">
-<h2>Architecture<a class="headerlink" href="#architecture" title="Link to this
heading">¶</a></h2>
-<p>The following diagram illustrates the architecture of Comet:</p>
-<p><img alt="Comet System Diagram" src="../_images/comet-system-diagram.png"
/></p>
+<section id="tuning-guide">
+<h1>Tuning Guide<a class="headerlink" href="#tuning-guide" title="Link to this
heading">¶</a></h1>
+<p>Comet provides some tuning options to help you get the best performance
from your queries.</p>
+<section id="shuffle">
+<h2>Shuffle<a class="headerlink" href="#shuffle" title="Link to this
heading">¶</a></h2>
+<p>Comet provides Comet shuffle features that can be used to improve the
performance of your queries.
+The following sections describe the different shuffle options available in
Comet.</p>
+<p>To enable Comet shuffle, set the following configuration in your Spark
configuration:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">spark</span><span
class="o">.</span><span class="n">shuffle</span><span class="o">.</span><span
class="n">manager</span><span class="o">=</span><span class="n">org</span><span
class="o">.</span><span class="n">apache</span><span class="o">.</span><span
class="n">spark</span><span class="o">.</span><span class="n">sql</span><span
class="o">.</span><span class="n">comet</span><span class="o" [...]
+<span class="n">spark</span><span class="o">.</span><span
class="n">comet</span><span class="o">.</span><span class="n">exec</span><span
class="o">.</span><span class="n">shuffle</span><span class="o">.</span><span
class="n">enabled</span><span class="o">=</span><span class="n">true</span>
+</pre></div>
+</div>
+<p><code class="docutils literal notranslate"><span
class="pre">spark.shuffle.manager</span></code> is a Spark static configuration
which cannot be changed at runtime.
+It must be set before the Spark context is created. You can enable or disable
Comet shuffle
+at runtime by setting <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.shuffle.enabled</span></code> to <code
class="docutils literal notranslate"><span class="pre">true</span></code> or
<code class="docutils literal notranslate"><span
class="pre">false</span></code>.
+Once it is disabled, Comet will fallback to the default Spark shuffle
manager.</p>
+<section id="columnar-shuffle">
+<h3>Columnar Shuffle<a class="headerlink" href="#columnar-shuffle" title="Link
to this heading">¶</a></h3>
+<p>By default, once <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.shuffle.enabled</span></code> is enabled, Comet
uses columnar shuffle
+to improve the performance of shuffle operations. Columnar shuffle supports
HashPartitioning,
+RoundRobinPartitioning, RangePartitioning and SinglePartitioning.</p>
+<p>Columnar shuffle can be disabled by setting <code class="docutils literal
notranslate"><span
class="pre">spark.comet.columnar.shuffle.enabled</span></code> to <code
class="docutils literal notranslate"><span class="pre">false</span></code>.</p>
</section>
-<section id="current-status">
-<h2>Current Status<a class="headerlink" href="#current-status" title="Link to
this heading">¶</a></h2>
-<p>The project is currently integrated into Apache Spark 3.2, 3.3, and 3.4.</p>
+<section id="native-shuffle">
+<h3>Native Shuffle<a class="headerlink" href="#native-shuffle" title="Link to
this heading">¶</a></h3>
+<p>Comet also provides a fully native shuffle implementation that can be used
to improve the performance.
+To enable native shuffle, just disable <code class="docutils literal
notranslate"><span
class="pre">spark.comet.columnar.shuffle.enabled</span></code>.</p>
+<p>Native shuffle only supports HashPartitioning and SinglePartitioning.</p>
</section>
-<section id="feature-parity-with-apache-spark">
-<h2>Feature Parity with Apache Spark<a class="headerlink"
href="#feature-parity-with-apache-spark" title="Link to this heading">¶</a></h2>
-<p>The project strives to keep feature parity with Apache Spark, that is,
-users should expect the same behavior (w.r.t features, configurations,
-query results, etc) with Comet turned on or turned off in their Spark
-jobs. In addition, Comet extension should automatically detect unsupported
-features and fallback to Spark engine.</p>
-<p>To achieve this, besides unit tests within Comet itself, we also re-use
-Spark SQL tests and make sure they all pass with Comet extension
-enabled.</p>
</section>
</section>
@@ -348,17 +352,17 @@ enabled.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="../index.html" title="previous
page">
+ <a class='left-prev' id="prev-link" href="compatibility.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Apache DataFusion Comet</p>
+ <p class="prev-next-title">Compatibility Guide</p>
</div>
</a>
- <a class='right-next' id="next-link" href="installation.html" title="next
page">
+ <a class='right-next' id="next-link"
href="../contributor-guide/contributing.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Installing DataFusion Comet</p>
+ <p class="prev-next-title">Contributing to Apache DataFusion Comet</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
