This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6d6dc7e4 Publish built docs triggered by
686c7adeb44944896981853ad3277291fcafe50b
6d6dc7e4 is described below
commit 6d6dc7e43c11ceb2e541bace358e2ec904a055a2
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Wed May 1 04:20:51 2024 +0000
Publish built docs triggered by 686c7adeb44944896981853ad3277291fcafe50b
---
_sources/contributor-guide/debugging.md.txt | 2 +-
_sources/contributor-guide/plugin_overview.md.txt | 59 +++++++++
_sources/index.rst.txt | 5 +-
contributor-guide/contributing.html | 19 +--
contributor-guide/debugging.html | 70 ++++++-----
contributor-guide/development.html | 19 +--
.../{contributing.html => plugin_overview.html} | 133 ++++++++++-----------
genindex.html | 13 +-
index.html | 19 +--
objects.inv | Bin 513 -> 530 bytes
search.html | 13 +-
searchindex.js | 2 +-
user-guide/compatibility.html | 13 +-
user-guide/configs-template.html | 13 +-
user-guide/configs.html | 13 +-
user-guide/datatypes.html | 13 +-
user-guide/expressions.html | 13 +-
user-guide/installation.html | 13 +-
user-guide/operators.html | 13 +-
user-guide/overview.html | 13 +-
20 files changed, 289 insertions(+), 169 deletions(-)
diff --git a/_sources/contributor-guide/debugging.md.txt
b/_sources/contributor-guide/debugging.md.txt
index 3b20ed0b..38c396c1 100644
--- a/_sources/contributor-guide/debugging.md.txt
+++ b/_sources/contributor-guide/debugging.md.txt
@@ -99,7 +99,7 @@
https://mail.openjdk.org/pipermail/hotspot-dev/2019-September/039429.html
Detecting the debugger
https://stackoverflow.com/questions/5393403/can-a-java-application-detect-that-a-debugger-is-attached#:~:text=No.,to%20let%20your%20app%20continue.&text=I%20know%20that%20those%20are,meant%20with%20my%20first%20phrase).
-# Verbose debug
+## Verbose debug
By default, Comet outputs the exception details specific for Comet.
diff --git a/_sources/contributor-guide/plugin_overview.md.txt
b/_sources/contributor-guide/plugin_overview.md.txt
new file mode 100644
index 00000000..8b48818e
--- /dev/null
+++ b/_sources/contributor-guide/plugin_overview.md.txt
@@ -0,0 +1,59 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet Plugin Overview
+
+The entry point to Comet is the `org.apache.comet.CometSparkSessionExtensions`
class, which can be registered with Spark by adding the following setting to
the Spark configuration when launching `spark-shell` or `spark-submit`:
+
+```
+--conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions
+```
+
+On initialization, this class registers two physical plan optimization rules
with Spark: `CometScanRule` and `CometExecRule`. These rules run whenever a
query stage is being planned.
+
+## CometScanRule
+
+`CometScanRule` replaces any Parquet scans with Comet Parquet scan classes.
+
+When the V1 data source API is being used, `FileSourceScanExec` is replaced
with `CometScanExec`.
+
+When the V2 data source API is being used, `BatchScanExec` is replaced with
`CometBatchScanExec`.
+
+## CometExecRule
+
+`CometExecRule` attempts to transform a Spark physical plan into a Comet plan.
This rule is executed against
+individual query stages when they are being prepared for execution.
+
+This rule traverses bottom-up from the original Spark plan and attempts to
replace each node with a Comet equivalent.
+For example, a `ProjectExec` will be replaced by `CometProjectExec`.
+
+When replacing a node, various checks are performed to determine if Comet can
support the operator and its expressions.
+If an operator, expression, or data type is not supported by Comet then the
reason will be stored in a tag on the
+underlying Spark node and the plan will not be converted.
+
+Comet does not support partially replacing subsets of the plan within a query
stage because this would involve adding
+transitions to convert between row-based and columnar data between Spark
operators and Comet operators and the overhead
+of this could outweigh the benefits of running parts of the query stage
natively in Comet.
+
+Once the plan has been transformed, it is serialized into Comet protocol
buffer format by the `QueryPlanSerde` class
+and this serialized plan is passed into the native code by `CometExecIterator`.
+
+In the native code there is a `PhysicalPlanner` struct (in `planner.rs`) which
converts the serialized plan into an
+Apache DataFusion physical plan. In some cases, Comet provides specialized
physical operators and expressions to
+override the DataFusion versions to ensure compatibility with Apache Spark.
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index dfd19e59..5759fcf4 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -54,9 +54,10 @@ as a native runtime to achieve improvement in terms of query
efficiency and quer
:caption: Contributor Guide
Getting Started <contributor-guide/contributing>
+ Comet Plugin Overview <contributor-guide/plugin_overview>
+ Development Guide <contributor-guide/development>
+ Debugging Guide <contributor-guide/debugging>
Github and Issue Tracker <https://github.com/apache/datafusion-comet>
- contributor-guide/development
- contributor-guide/debugging
.. _toc.asf-links:
.. toctree::
diff --git a/contributor-guide/contributing.html
b/contributor-guide/contributing.html
index b3271eaa..192fa543 100644
--- a/contributor-guide/contributing.html
+++ b/contributor-guide/contributing.html
@@ -53,7 +53,7 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="next" title="Comet Development Guide" href="development.html" />
+ <link rel="next" title="Comet Plugin Overview" href="plugin_overview.html"
/>
<link rel="prev" title="Compatibility Guide"
href="../user-guide/compatibility.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal" href="plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
@@ -346,10 +351,10 @@ coordinate on issues that they are working on.</p>
<p class="prev-next-title">Compatibility Guide</p>
</div>
</a>
- <a class='right-next' id="next-link" href="development.html" title="next
page">
+ <a class='right-next' id="next-link" href="plugin_overview.html"
title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
- <p class="prev-next-title">Comet Development Guide</p>
+ <p class="prev-next-title">Comet Plugin Overview</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
diff --git a/contributor-guide/debugging.html b/contributor-guide/debugging.html
index 565b2daf..c1b876ce 100644
--- a/contributor-guide/debugging.html
+++ b/contributor-guide/debugging.html
@@ -155,18 +155,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal" href="plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
@@ -235,41 +240,34 @@ under the License.
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
- <li class="toc-h1 nav-item toc-entry">
- <a class="reference internal nav-link" href="#">
- Comet Debugging Guide
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#debugging-for-advanced-developers">
+ Debugging for Advanced Developers
</a>
- <ul class="visible nav section-nav flex-column">
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#debugging-for-advanced-developers">
- Debugging for Advanced Developers
+ <ul class="nav section-nav flex-column">
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#in-intellij">
+ In Intellij
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#in-clion">
+ In CLion
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#after-your-debugging-is-done">
+ After your debugging is done,
+ </a>
+ </li>
+ <li class="toc-h3 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#additional-info">
+ Additional Info
</a>
- <ul class="nav section-nav flex-column">
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#in-intellij">
- In Intellij
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#in-clion">
- In CLion
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#after-your-debugging-is-done">
- After your debugging is done,
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link" href="#additional-info">
- Additional Info
- </a>
- </li>
- </ul>
</li>
</ul>
</li>
- <li class="toc-h1 nav-item toc-entry">
+ <li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#verbose-debug">
Verbose debug
</a>
@@ -394,9 +392,8 @@
https://mail.openjdk.org/pipermail/hotspot-dev/2019-September/039429.html</p>
https://stackoverflow.com/questions/5393403/can-a-java-application-detect-that-a-debugger-is-attached#:~:text=No.,to%20let%20your%20app%20continue.&text=I%20know%20that%20those%20are,meant%20with%20my%20first%20phrase).</p>
</section>
</section>
-</section>
<section id="verbose-debug">
-<h1>Verbose debug<a class="headerlink" href="#verbose-debug" title="Link to
this heading">¶</a></h1>
+<h2>Verbose debug<a class="headerlink" href="#verbose-debug" title="Link to
this heading">¶</a></h2>
<p>By default, Comet outputs the exception details specific for Comet.</p>
<div class="highlight-scala notranslate"><div
class="highlight"><pre><span></span><span class="n">scala</span><span
class="o">></span><span class="w"> </span><span class="n">spark</span><span
class="p">.</span><span class="n">sql</span><span class="p">(</span><span
class="s">"my_failing_query"</span><span class="p">).</span><span
class="n">show</span><span class="p">(</span><span class="kc">false</span><span
class="p">)</span>
@@ -448,6 +445,7 @@ To enable this option with Comet it is needed to include
<code class="docutils l
<li><p>The backtrace coverage in DataFusion is still improving. So there is a
chance the error still not covered, if so feel free to file a <a
class="reference external"
href="https://github.com/apache/arrow-datafusion/issues";>ticket</a></p></li>
<li><p>The backtrace evaluation comes with performance cost and intended
mostly for debugging purposes</p></li>
</ul>
+</section>
</section>
diff --git a/contributor-guide/development.html
b/contributor-guide/development.html
index aaacfdb8..d53fa2b2 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -54,7 +54,7 @@ under the License.
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Comet Debugging Guide" href="debugging.html" />
- <link rel="prev" title="Contributing to Apache DataFusion Comet"
href="contributing.html" />
+ <link rel="prev" title="Comet Plugin Overview" href="plugin_overview.html"
/>
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal" href="plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
@@ -420,11 +425,11 @@ automatically format the code. Before submitting a pull
request, you can simply
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="contributing.html"
title="previous page">
+ <a class='left-prev' id="prev-link" href="plugin_overview.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Contributing to Apache DataFusion
Comet</p>
+ <p class="prev-next-title">Comet Plugin Overview</p>
</div>
</a>
<a class='right-next' id="next-link" href="debugging.html" title="next
page">
diff --git a/contributor-guide/contributing.html
b/contributor-guide/plugin_overview.html
similarity index 62%
copy from contributor-guide/contributing.html
copy to contributor-guide/plugin_overview.html
index b3271eaa..8e48e2b9 100644
--- a/contributor-guide/contributing.html
+++ b/contributor-guide/plugin_overview.html
@@ -24,7 +24,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Contributing to Apache DataFusion Comet — Apache DataFusion
Comet documentation</title>
+ <title>Comet Plugin Overview — Apache DataFusion Comet
documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
@@ -54,7 +54,7 @@ under the License.
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="Comet Development Guide" href="development.html" />
- <link rel="prev" title="Compatibility Guide"
href="../user-guide/compatibility.html" />
+ <link rel="prev" title="Contributing to Apache DataFusion Comet"
href="contributing.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -150,24 +150,29 @@ under the License.
</span>
</p>
<ul class="current nav bd-sidenav">
- <li class="toctree-l1 current active">
- <a class="current reference internal" href="#">
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributing.html">
Getting Started
</a>
</li>
- <li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <li class="toctree-l1 current active">
+ <a class="current reference internal" href="#">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
@@ -237,23 +242,13 @@ under the License.
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#finding-issues-to-work-on">
- Finding issues to work on
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#reporting-issues">
- Reporting issues
- </a>
- </li>
- <li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#asking-for-help">
- Asking for Help
+ <a class="reference internal nav-link" href="#cometscanrule">
+ CometScanRule
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link" href="#regular-public-meetings">
- Regular public meetings
+ <a class="reference internal nav-link" href="#cometexecrule">
+ CometExecRule
</a>
</li>
</ul>
@@ -265,7 +260,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/contributor-guide/contributing.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/contributor-guide/plugin_overview.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -284,52 +279,54 @@ under the License.
<div>
- <!---
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
+ <!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
+ http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
-->
-<section id="contributing-to-apache-datafusion-comet">
-<h1>Contributing to Apache DataFusion Comet<a class="headerlink"
href="#contributing-to-apache-datafusion-comet" title="Link to this
heading">¶</a></h1>
-<p>We welcome contributions to Comet in many areas, and encourage new
contributors to get involved.</p>
-<p>Here are some areas where you can help:</p>
-<ul class="simple">
-<li><p>Testing Comet with existing Spark jobs and reporting issues for any
bugs or performance issues</p></li>
-<li><p>Contributing code to support Spark expressions, operators, and data
types that are not currently supported</p></li>
-<li><p>Reviewing pull requests and helping to test new features for
correctness and performance</p></li>
-<li><p>Improving documentation</p></li>
-</ul>
-<section id="finding-issues-to-work-on">
-<h2>Finding issues to work on<a class="headerlink"
href="#finding-issues-to-work-on" title="Link to this heading">¶</a></h2>
-<p>We maintain a list of good first issues in GitHub <a class="reference
external"
href="https://github.com/apache/datafusion-comet/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22";>here</a>.</p>
-</section>
-<section id="reporting-issues">
-<h2>Reporting issues<a class="headerlink" href="#reporting-issues" title="Link
to this heading">¶</a></h2>
-<p>We use <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues";>GitHub issues</a> for
bug reports and feature requests.</p>
-</section>
-<section id="asking-for-help">
-<h2>Asking for Help<a class="headerlink" href="#asking-for-help" title="Link
to this heading">¶</a></h2>
-<p>The Comet project uses the same Slack and Discord channels as the main
Apache DataFusion project. See details at
-<a class="reference external"
href="https://datafusion.apache.org/contributor-guide/communication.html";>Apache
DataFusion Communications</a>. There are dedicated Comet channels in both
Slack and Discord.</p>
+<section id="comet-plugin-overview">
+<h1>Comet Plugin Overview<a class="headerlink" href="#comet-plugin-overview"
title="Link to this heading">¶</a></h1>
+<p>The entry point to Comet is the <code class="docutils literal
notranslate"><span
class="pre">org.apache.comet.CometSparkSessionExtensions</span></code> class,
which can be registered with Spark by adding the following setting to the Spark
configuration when launching <code class="docutils literal notranslate"><span
class="pre">spark-shell</span></code> or <code class="docutils literal
notranslate"><span class="pre">spark-submit</span></code>:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="o">--</span><span
class="n">conf</span> <span class="n">spark</span><span class="o">.</span><span
class="n">sql</span><span class="o">.</span><span
class="n">extensions</span><span class="o">=</span><span
class="n">org</span><span class="o">.</span><span class="n">apache</span><span
class="o">.</span><span class="n">comet</span><span class="o">.</span><span
class="n">CometSparkSessionExtensio [...]
+</pre></div>
+</div>
+<p>On initialization, this class registers two physical plan optimization
rules with Spark: <code class="docutils literal notranslate"><span
class="pre">CometScanRule</span></code> and <code class="docutils literal
notranslate"><span class="pre">CometExecRule</span></code>. These rules run
whenever a query stage is being planned.</p>
+<section id="cometscanrule">
+<h2>CometScanRule<a class="headerlink" href="#cometscanrule" title="Link to
this heading">¶</a></h2>
+<p><code class="docutils literal notranslate"><span
class="pre">CometScanRule</span></code> replaces any Parquet scans with Comet
Parquet scan classes.</p>
+<p>When the V1 data source API is being used, <code class="docutils literal
notranslate"><span class="pre">FileSourceScanExec</span></code> is replaced
with <code class="docutils literal notranslate"><span
class="pre">CometScanExec</span></code>.</p>
+<p>When the V2 data source API is being used, <code class="docutils literal
notranslate"><span class="pre">BatchScanExec</span></code> is replaced with
<code class="docutils literal notranslate"><span
class="pre">CometBatchScanExec</span></code>.</p>
</section>
-<section id="regular-public-meetings">
-<h2>Regular public meetings<a class="headerlink"
href="#regular-public-meetings" title="Link to this heading">¶</a></h2>
-<p>The Comet contributors hold regular video calls where new and current
contributors are welcome to ask questions and
-coordinate on issues that they are working on.</p>
-<p>See the <a class="reference external"
href="https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit?usp=sharing";>Apache
DataFusion Comet community meeting</a> Google document for more
information.</p>
+<section id="cometexecrule">
+<h2>CometExecRule<a class="headerlink" href="#cometexecrule" title="Link to
this heading">¶</a></h2>
+<p><code class="docutils literal notranslate"><span
class="pre">CometExecRule</span></code> attempts to transform a Spark physical
plan into a Comet plan. This rule is executed against
+individual query stages when they are being prepared for execution.</p>
+<p>This rule traverses bottom-up from the original Spark plan and attempts to
replace each node with a Comet equivalent.
+For example, a <code class="docutils literal notranslate"><span
class="pre">ProjectExec</span></code> will be replaced by <code class="docutils
literal notranslate"><span class="pre">CometProjectExec</span></code>.</p>
+<p>When replacing a node, various checks are performed to determine if Comet
can support the operator and its expressions.
+If an operator, expression, or data type is not supported by Comet then the
reason will be stored in a tag on the
+underlying Spark node and the plan will not be converted.</p>
+<p>Comet does not support partially replacing subsets of the plan within a
query stage because this would involve adding
+transitions to convert between row-based and columnar data between Spark
operators and Comet operators and the overhead
+of this could outweigh the benefits of running parts of the query stage
natively in Comet.</p>
+<p>Once the plan has been transformed, it is serialized into Comet protocol
buffer format by the <code class="docutils literal notranslate"><span
class="pre">QueryPlanSerde</span></code> class
+and this serialized plan is passed into the native code by <code
class="docutils literal notranslate"><span
class="pre">CometExecIterator</span></code>.</p>
+<p>In the native code there is a <code class="docutils literal
notranslate"><span class="pre">PhysicalPlanner</span></code> struct (in <code
class="docutils literal notranslate"><span
class="pre">planner.rs</span></code>) which converts the serialized plan into an
+Apache DataFusion physical plan. In some cases, Comet provides specialized
physical operators and expressions to
+override the DataFusion versions to ensure compatibility with Apache Spark.</p>
</section>
</section>
@@ -339,11 +336,11 @@ coordinate on issues that they are working on.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link"
href="../user-guide/compatibility.html" title="previous page">
+ <a class='left-prev' id="prev-link" href="contributing.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Compatibility Guide</p>
+ <p class="prev-next-title">Contributing to Apache DataFusion
Comet</p>
</div>
</a>
<a class='right-next' id="next-link" href="development.html" title="next
page">
diff --git a/genindex.html b/genindex.html
index 05f1304b..d3fa65ec 100644
--- a/genindex.html
+++ b/genindex.html
@@ -153,18 +153,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal" href="contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/index.html b/index.html
index fed0583f..5c3ee7d5 100644
--- a/index.html
+++ b/index.html
@@ -155,18 +155,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal" href="contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
@@ -284,10 +289,10 @@ as a native runtime to achieve improvement in terms of
query efficiency and quer
<p aria-level="2" class="caption" role="heading"><span
class="caption-text">Contributor Guide</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/contributing.html">Getting Started</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/plugin_overview.html">Comet Plugin Overview</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/development.html">Development Guide</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/debugging.html">Debugging Guide</a></li>
<li class="toctree-l1"><a class="reference external"
href="https://github.com/apache/datafusion-comet";>Github and Issue
Tracker</a></li>
-<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/development.html">Comet Development Guide</a></li>
-<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/debugging.html">Comet Debugging Guide</a></li>
-<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/debugging.html#verbose-debug">Verbose debug</a></li>
</ul>
</div>
<div class="toctree-wrapper compound" id="toc-asf-links">
diff --git a/objects.inv b/objects.inv
index 9888be63..87342e48 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/search.html b/search.html
index 9566c2ec..2cd32852 100644
--- a/search.html
+++ b/search.html
@@ -160,18 +160,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal" href="contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/searchindex.js b/searchindex.js
index af6defb4..b911f68b 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"ANSI mode": [[4, "ansi-mode"]], "ASF Links":
[[3, null]], "Additional Info": [[1, "additional-info"]], "After your debugging
is done,": [[1, "after-your-debugging-is-done"]], "Apache DataFusion Comet":
[[3, "apache-datafusion-comet"]], "Architecture": [[11, "architecture"]],
"Asking for Help": [[0, "asking-for-help"]], "Benchmark": [[2, "benchmark"]],
"Build & Test": [[2, "build-test"]], "Building From Source": [[9,
"building-from-source"]], "CLion": [[2, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"ANSI mode": [[5, "ansi-mode"]], "ASF Links":
[[4, null]], "Additional Info": [[1, "additional-info"]], "After your debugging
is done,": [[1, "after-your-debugging-is-done"]], "Apache DataFusion Comet":
[[4, "apache-datafusion-comet"]], "Architecture": [[12, "architecture"]],
"Asking for Help": [[0, "asking-for-help"]], "Benchmark": [[2, "benchmark"]],
"Build & Test": [[2, "build-test"]], "Building From Source": [[10,
"building-from-source"]], "CLion": [[2, [...]
\ No newline at end of file
diff --git a/user-guide/compatibility.html b/user-guide/compatibility.html
index 6e02a071..09772510 100644
--- a/user-guide/compatibility.html
+++ b/user-guide/compatibility.html
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal"
href="../contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/user-guide/configs-template.html b/user-guide/configs-template.html
index 8f3d11f8..4cdc4049 100644
--- a/user-guide/configs-template.html
+++ b/user-guide/configs-template.html
@@ -154,18 +154,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal"
href="../contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 9c3cdf40..b1c6e825 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal"
href="../contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/user-guide/datatypes.html b/user-guide/datatypes.html
index bd44c2c6..81de18dc 100644
--- a/user-guide/datatypes.html
+++ b/user-guide/datatypes.html
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal"
href="../contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/user-guide/expressions.html b/user-guide/expressions.html
index 38157ca2..4be9d624 100644
--- a/user-guide/expressions.html
+++ b/user-guide/expressions.html
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal"
href="../contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/user-guide/installation.html b/user-guide/installation.html
index 2fc495b4..c8e00f1f 100644
--- a/user-guide/installation.html
+++ b/user-guide/installation.html
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal"
href="../contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/user-guide/operators.html b/user-guide/operators.html
index 7ab4f9fa..33386b26 100644
--- a/user-guide/operators.html
+++ b/user-guide/operators.html
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal"
href="../contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
diff --git a/user-guide/overview.html b/user-guide/overview.html
index 4cf5e647..8fb3d456 100644
--- a/user-guide/overview.html
+++ b/user-guide/overview.html
@@ -156,18 +156,23 @@ under the License.
</a>
</li>
<li class="toctree-l1">
- <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
- Github and Issue Tracker
+ <a class="reference internal"
href="../contributor-guide/plugin_overview.html">
+ Comet Plugin Overview
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/development.html">
- Comet Development Guide
+ Development Guide
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="../contributor-guide/debugging.html">
- Comet Debugging Guide
+ Debugging Guide
+ </a>
+ </li>
+ <li class="toctree-l1">
+ <a class="reference external"
href="https://github.com/apache/datafusion-comet";>
+ Github and Issue Tracker
</a>
</li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
