This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e7423736 Publish built docs triggered by
917fd4334ce0028946abbba84cd469ef6c9c8fd3
e7423736 is described below
commit e7423736742b188b15281d2f04297e814a077d45
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Jul 2 02:18:35 2024 +0000
Publish built docs triggered by 917fd4334ce0028946abbba84cd469ef6c9c8fd3
---
_sources/contributor-guide/spark-sql-tests.md.txt | 128 +++++++++++++++
_sources/index.rst.txt | 1 +
contributor-guide/adding_a_new_expression.html | 5 +
contributor-guide/benchmarking.html | 5 +
contributor-guide/contributing.html | 5 +
contributor-guide/debugging.html | 5 +
contributor-guide/development.html | 5 +
contributor-guide/plugin_overview.html | 5 +
contributor-guide/profiling_native_code.html | 13 ++
...iling_native_code.html => spark-sql-tests.html} | 173 ++++++++++++++-------
genindex.html | 5 +
index.html | 6 +
objects.inv | Bin 632 -> 662 bytes
search.html | 5 +
searchindex.js | 2 +-
user-guide/compatibility.html | 5 +
user-guide/configs.html | 5 +
user-guide/datatypes.html | 5 +
user-guide/expressions.html | 5 +
user-guide/installation.html | 5 +
user-guide/operators.html | 5 +
user-guide/overview.html | 5 +
user-guide/tuning.html | 5 +
23 files changed, 347 insertions(+), 56 deletions(-)
diff --git a/_sources/contributor-guide/spark-sql-tests.md.txt
b/_sources/contributor-guide/spark-sql-tests.md.txt
new file mode 100644
index 00000000..0cdef50a
--- /dev/null
+++ b/_sources/contributor-guide/spark-sql-tests.md.txt
@@ -0,0 +1,128 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Running Spark SQL Tests
+
+Running Apache Spark's SQL tests with Comet enabled is a good way to ensure
that Comet produces the same
+results as that version of Spark. To enable this, we apply some changes to the
Apache Spark source code so that
+Comet is enabled when we run the tests.
+
+Here is an overview of the changes that we need to make to Spark:
+
+- Update the pom.xml to add a dependency on Comet
+- Modify SparkSession to load the Comet extension
+- Modify TestHive to load Comet
+- Modify SQLTestUtilsBase to load Comet when `ENABLE_COMET` environment
variable exists
+
+Here are the steps involved in running the Spark SQL tests with Comet, using
Spark 3.4.3 for this example.
+
+## 1. Install Comet
+
+Run `make release` in Comet to install the Comet JAR into the local Maven
repository, specifying the Spark version.
+
+```shell
+PROFILES="-Pspark-3.4" make release
+```
+
+## 2. Clone Spark and Apply Diff
+
+Clone Apache Spark locally and apply the diff file from Comet.
+
+```shell
+git clone [email protected]:apache/spark.git apache-spark
+cd apache-spark
+git checkout v3.4.3
+git apply ../datafusion-comet/dev/diffs/3.4.3.diff
+```
+
+## 3. Run Spark SQL Tests
+
+Use the following commands to run the SQL tests locally.
+
+```shell
+ENABLE_COMET=true build/sbt catalyst/test
+ENABLE_COMET=true build/sbt "sql/testOnly * -- -l
org.apache.spark.tags.ExtendedSQLTest -l org.apache.spark.tags.SlowSQLTest"
+ENABLE_COMET=true build/sbt "sql/testOnly * -- -n
org.apache.spark.tags.ExtendedSQLTest"
+ENABLE_COMET=true build/sbt "sql/testOnly * -- -n
org.apache.spark.tags.SlowSQLTest"
+ENABLE_COMET=true build/sbt "hive/testOnly * -- -l
org.apache.spark.tags.ExtendedHiveTest -l org.apache.spark.tags.SlowHiveTest"
+ENABLE_COMET=true build/sbt "hive/testOnly * -- -n
org.apache.spark.tags.ExtendedHiveTest"
+ENABLE_COMET=true build/sbt "hive/testOnly * -- -n
org.apache.spark.tags.SlowHiveTest"
+```
+
+## Creating a diff file for a new Spark version
+
+Once Comet has support for a new Spark version, we need to create a diff file
that can be applied to that version
+of Apache Spark to enable Comet when running tests. This is a highly manual
process and the process can
+vary depending on the changes in the new version of Spark, but here is a
general guide to the process.
+
+We typically start by applying a patch from a previous version of Spark. For
example, when enabling the tests
+for Spark version 3.5.1 we may start by applying the existing diff for 3.4.3
first.
+
+```shell
+cd git/apache/spark
+git checkout v3.5.1
+git apply --reject --whitespace=fix ../datafusion-comet/dev/diffs/3.4.3.diff
+```
+
+Any changes that cannot be cleanly applied will instead be written out to
reject files. For example, the above
+command generated the following files.
+
+```shell
+find . -name "*.rej"
+./pom.xml.rej
+./sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormatSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRebaseDatetimeSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/sources/CreateTableAsSelectSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala.rej
+./sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala.rej
+```
+
+The changes in these reject files need to be applied manually.
+
+One method is to use the [wiggle](https://github.com/neilbrown/wiggle) command
(`brew install wiggle` on Mac).
+
+For example:
+
+```shell
+wiggle --replace
./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala.rej
+```
+
+## Generating The Diff File
+
+```shell
+git diff v3.5.1 > ../datafusion-comet/dev/diffs/3.5.1.diff
+```
+
+## Running Tests in CI
+
+The easiest way to run the tests is to create a PR against Comet and let CI
run the tests. When working with a
+new Spark version, the `spark_sql_test.yaml` and `spark_sql_test_ansi.yaml`
files will need updating with the
+new version.
\ No newline at end of file
diff --git a/_sources/index.rst.txt b/_sources/index.rst.txt
index 1e9eba21..100145a9 100644
--- a/_sources/index.rst.txt
+++ b/_sources/index.rst.txt
@@ -61,6 +61,7 @@ as a native runtime to achieve improvement in terms of query
efficiency and quer
Benchmarking Guide <contributor-guide/benchmarking>
Adding a New Expression <contributor-guide/adding_a_new_expression>
Profiling Native Code <contributor-guide/profiling_native_code>
+ Spark SQL Tests <contributor-guide/spark-sql-tests.md>
Github and Issue Tracker <https://github.com/apache/datafusion-comet>
.. _toc.asf-links:
diff --git a/contributor-guide/adding_a_new_expression.html
b/contributor-guide/adding_a_new_expression.html
index fcd012a6..dfaf119a 100644
--- a/contributor-guide/adding_a_new_expression.html
+++ b/contributor-guide/adding_a_new_expression.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/contributor-guide/benchmarking.html
b/contributor-guide/benchmarking.html
index 5dea0be7..f6c427ba 100644
--- a/contributor-guide/benchmarking.html
+++ b/contributor-guide/benchmarking.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/contributor-guide/contributing.html
b/contributor-guide/contributing.html
index bab8258f..ff05a021 100644
--- a/contributor-guide/contributing.html
+++ b/contributor-guide/contributing.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/contributor-guide/debugging.html b/contributor-guide/debugging.html
index 1afed7b9..74f80e03 100644
--- a/contributor-guide/debugging.html
+++ b/contributor-guide/debugging.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/contributor-guide/development.html
b/contributor-guide/development.html
index 4a2e604f..fb9f6e67 100644
--- a/contributor-guide/development.html
+++ b/contributor-guide/development.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/contributor-guide/plugin_overview.html
b/contributor-guide/plugin_overview.html
index f200baec..d9079676 100644
--- a/contributor-guide/plugin_overview.html
+++ b/contributor-guide/plugin_overview.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/contributor-guide/profiling_native_code.html
b/contributor-guide/profiling_native_code.html
index d8520ab4..e295818b 100644
--- a/contributor-guide/profiling_native_code.html
+++ b/contributor-guide/profiling_native_code.html
@@ -53,6 +53,7 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
+ <link rel="next" title="Running Spark SQL Tests"
href="spark-sql-tests.html" />
<link rel="prev" title="Adding a New Expression"
href="adding_a_new_expression.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -189,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
@@ -390,6 +396,13 @@ running flamegraph.</p>
<p class="prev-next-title">Adding a New Expression</p>
</div>
</a>
+ <a class='right-next' id="next-link" href="spark-sql-tests.html"
title="next page">
+ <div class="prev-next-info">
+ <p class="prev-next-subtitle">next</p>
+ <p class="prev-next-title">Running Spark SQL Tests</p>
+ </div>
+ <i class="fas fa-angle-right"></i>
+ </a>
</div>
</main>
diff --git a/contributor-guide/profiling_native_code.html
b/contributor-guide/spark-sql-tests.html
similarity index 52%
copy from contributor-guide/profiling_native_code.html
copy to contributor-guide/spark-sql-tests.html
index d8520ab4..1a72eae4 100644
--- a/contributor-guide/profiling_native_code.html
+++ b/contributor-guide/spark-sql-tests.html
@@ -24,7 +24,7 @@ under the License.
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0"
/><meta name="viewport" content="width=device-width, initial-scale=1" />
- <title>Profiling Native Code — Apache DataFusion Comet
documentation</title>
+ <title>Running Spark SQL Tests — Apache DataFusion Comet
documentation</title>
<link href="../_static/styles/theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
<link
href="../_static/styles/pydata-sphinx-theme.css?digest=1999514e3f237ded88cf"
rel="stylesheet">
@@ -53,7 +53,7 @@ under the License.
<script async="true" defer="true"
src="https://buttons.github.io/buttons.js";></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
- <link rel="prev" title="Adding a New Expression"
href="adding_a_new_expression.html" />
+ <link rel="prev" title="Profiling Native Code"
href="profiling_native_code.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en">
@@ -184,9 +184,14 @@ under the License.
Adding a New Expression
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="profiling_native_code.html">
+ Profiling Native Code
+ </a>
+ </li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
- Profiling Native Code
+ Spark SQL Tests
</a>
</li>
<li class="toctree-l1">
@@ -261,13 +266,33 @@ under the License.
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#running-micro-benchmarks-with-cargo-bench">
- Running micro benchmarks with cargo bench
+ <a class="reference internal nav-link" href="#install-comet">
+ 1. Install Comet
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#clone-spark-and-apply-diff">
+ 2. Clone Spark and Apply Diff
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#run-spark-sql-tests">
+ 3. Run Spark SQL Tests
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#creating-a-diff-file-for-a-new-spark-version">
+ Creating a diff file for a new Spark version
+ </a>
+ </li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#generating-the-diff-file">
+ Generating The Diff File
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#profiling-with-cargo-flamegraph">
- Profiling with cargo-flamegraph
+ <a class="reference internal nav-link" href="#running-tests-in-ci">
+ Running Tests in CI
</a>
</li>
</ul>
@@ -279,7 +304,7 @@ under the License.
<div class="tocsection editthispage">
- <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/contributor-guide/profiling_native_code.md";>
+ <a
href="https://github.com/apache/datafusion-comet/edit/main/docs/source/contributor-guide/spark-sql-tests.md";>
<i class="fas fa-pencil-alt"></i> Edit this page
</a>
</div>
@@ -316,64 +341,102 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
-<section id="profiling-native-code">
-<h1>Profiling Native Code<a class="headerlink" href="#profiling-native-code"
title="Link to this heading">¶</a></h1>
-<p>We use <code class="docutils literal notranslate"><span
class="pre">cargo</span> <span class="pre">bench</span></code> to run
benchmarks to measure the performance of individual operators and expressions
-and <a class="reference external"
href="https://github.com/flamegraph-rs/flamegraph";>cargo-flamegraph</a> for
profiling.</p>
-<section id="running-micro-benchmarks-with-cargo-bench">
-<h2>Running micro benchmarks with cargo bench<a class="headerlink"
href="#running-micro-benchmarks-with-cargo-bench" title="Link to this
heading">¶</a></h2>
-<p>When implementing a new operator or expression, it is good practice to add
a new microbenchmark under <code class="docutils literal notranslate"><span
class="pre">core/benches</span></code>.</p>
-<p>It is often easiest to copy an existing benchmark and modify it for the new
operator or expression. It is also
-necessary to add a new section to the <code class="docutils literal
notranslate"><span class="pre">Cargo.toml</span></code> file, such as:</p>
-<div class="highlight-toml notranslate"><div
class="highlight"><pre><span></span><span class="k">[[bench]]</span>
-<span class="n">name</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="s2">"shuffle_writer"</span>
-<span class="n">harness</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span class="kc">false</span>
+<section id="running-spark-sql-tests">
+<h1>Running Spark SQL Tests<a class="headerlink"
href="#running-spark-sql-tests" title="Link to this heading">¶</a></h1>
+<p>Running Apache Spark’s SQL tests with Comet enabled is a good way to ensure
that Comet produces the same
+results as that version of Spark. To enable this, we apply some changes to the
Apache Spark source code so that
+Comet is enabled when we run the tests.</p>
+<p>Here is an overview of the changes that we need to make to Spark:</p>
+<ul class="simple">
+<li><p>Update the pom.xml to add a dependency on Comet</p></li>
+<li><p>Modify SparkSession to load the Comet extension</p></li>
+<li><p>Modify TestHive to load Comet</p></li>
+<li><p>Modify SQLTestUtilsBase to load Comet when <code class="docutils
literal notranslate"><span class="pre">ENABLE_COMET</span></code> environment
variable exists</p></li>
+</ul>
+<p>Here are the steps involved in running the Spark SQL tests with Comet,
using Spark 3.4.3 for this example.</p>
+<section id="install-comet">
+<h2>1. Install Comet<a class="headerlink" href="#install-comet" title="Link to
this heading">¶</a></h2>
+<p>Run <code class="docutils literal notranslate"><span
class="pre">make</span> <span class="pre">release</span></code> in Comet to
install the Comet JAR into the local Maven repository, specifying the Spark
version.</p>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nv">PROFILES</span><span
class="o">=</span><span class="s2">"-Pspark-3.4"</span><span
class="w"> </span>make<span class="w"> </span>release
</pre></div>
</div>
-<p>These benchmarks are useful when for comparing performance between releases
or between feature branches and the
-main branch to help prevent regressions in performance when adding new
features or fixing bugs.</p>
-<p>Individual benchmarks can be run by name with the following command.</p>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>cargo<span class="w"> </span>bench<span
class="w"> </span>shuffle_writer
+</section>
+<section id="clone-spark-and-apply-diff">
+<h2>2. Clone Spark and Apply Diff<a class="headerlink"
href="#clone-spark-and-apply-diff" title="Link to this heading">¶</a></h2>
+<p>Clone Apache Spark locally and apply the diff file from Comet.</p>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>git<span class="w"> </span>clone<span
class="w"> </span>[email protected]:apache/spark.git<span class="w">
</span>apache-spark
+<span class="nb">cd</span><span class="w"> </span>apache-spark
+git<span class="w"> </span>checkout<span class="w"> </span>v3.4.3
+git<span class="w"> </span>apply<span class="w">
</span>../datafusion-comet/dev/diffs/3.4.3.diff
</pre></div>
</div>
-<p>Here is some sample output from running this command.</p>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span> <span class="n">Running</span> <span
class="n">benches</span><span class="o">/</span><span
class="n">shuffle_writer</span><span class="o">.</span><span
class="n">rs</span> <span class="p">(</span><span class="n">target</span><span
class="o">/</span><span class="n">release</span><span class="o">/</span><span
class="n">deps</span><span class="o">/</span><span
class="n">shuffle_writer</span><span class= [...]
-<span class="n">Gnuplot</span> <span class="ow">not</span> <span
class="n">found</span><span class="p">,</span> <span class="n">using</span>
<span class="n">plotters</span> <span class="n">backend</span>
-<span class="n">shuffle_writer</span><span class="o">/</span><span
class="n">shuffle_writer</span>
- <span class="n">time</span><span class="p">:</span>
<span class="p">[</span><span class="mf">2.0880</span> <span
class="n">ms</span> <span class="mf">2.0989</span> <span class="n">ms</span>
<span class="mf">2.1118</span> <span class="n">ms</span><span class="p">]</span>
-<span class="n">Found</span> <span class="mi">9</span> <span
class="n">outliers</span> <span class="n">among</span> <span
class="mi">100</span> <span class="n">measurements</span> <span
class="p">(</span><span class="mf">9.00</span><span class="o">%</span><span
class="p">)</span>
- <span class="mi">3</span> <span class="p">(</span><span
class="mf">3.00</span><span class="o">%</span><span class="p">)</span> <span
class="n">high</span> <span class="n">mild</span>
- <span class="mi">6</span> <span class="p">(</span><span
class="mf">6.00</span><span class="o">%</span><span class="p">)</span> <span
class="n">high</span> <span class="n">severe</span>
+</section>
+<section id="run-spark-sql-tests">
+<h2>3. Run Spark SQL Tests<a class="headerlink" href="#run-spark-sql-tests"
title="Link to this heading">¶</a></h2>
+<p>Use the following commands to run the SQL tests locally.</p>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nv">ENABLE_COMET</span><span
class="o">=</span><span class="nb">true</span><span class="w">
</span>build/sbt<span class="w"> </span>catalyst/test
+<span class="nv">ENABLE_COMET</span><span class="o">=</span><span
class="nb">true</span><span class="w"> </span>build/sbt<span class="w">
</span><span class="s2">"sql/testOnly * -- -l
org.apache.spark.tags.ExtendedSQLTest -l
org.apache.spark.tags.SlowSQLTest"</span>
+<span class="nv">ENABLE_COMET</span><span class="o">=</span><span
class="nb">true</span><span class="w"> </span>build/sbt<span class="w">
</span><span class="s2">"sql/testOnly * -- -n
org.apache.spark.tags.ExtendedSQLTest"</span>
+<span class="nv">ENABLE_COMET</span><span class="o">=</span><span
class="nb">true</span><span class="w"> </span>build/sbt<span class="w">
</span><span class="s2">"sql/testOnly * -- -n
org.apache.spark.tags.SlowSQLTest"</span>
+<span class="nv">ENABLE_COMET</span><span class="o">=</span><span
class="nb">true</span><span class="w"> </span>build/sbt<span class="w">
</span><span class="s2">"hive/testOnly * -- -l
org.apache.spark.tags.ExtendedHiveTest -l
org.apache.spark.tags.SlowHiveTest"</span>
+<span class="nv">ENABLE_COMET</span><span class="o">=</span><span
class="nb">true</span><span class="w"> </span>build/sbt<span class="w">
</span><span class="s2">"hive/testOnly * -- -n
org.apache.spark.tags.ExtendedHiveTest"</span>
+<span class="nv">ENABLE_COMET</span><span class="o">=</span><span
class="nb">true</span><span class="w"> </span>build/sbt<span class="w">
</span><span class="s2">"hive/testOnly * -- -n
org.apache.spark.tags.SlowHiveTest"</span>
</pre></div>
</div>
</section>
-<section id="profiling-with-cargo-flamegraph">
-<h2>Profiling with cargo-flamegraph<a class="headerlink"
href="#profiling-with-cargo-flamegraph" title="Link to this heading">¶</a></h2>
-<p>Install cargo-flamegraph:</p>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>cargo<span class="w"> </span>install<span
class="w"> </span>flamegraph
+<section id="creating-a-diff-file-for-a-new-spark-version">
+<h2>Creating a diff file for a new Spark version<a class="headerlink"
href="#creating-a-diff-file-for-a-new-spark-version" title="Link to this
heading">¶</a></h2>
+<p>Once Comet has support for a new Spark version, we need to create a diff
file that can be applied to that version
+of Apache Spark to enable Comet when running tests. This is a highly manual
process and the process can
+vary depending on the changes in the new version of Spark, but here is a
general guide to the process.</p>
+<p>We typically start by applying a patch from a previous version of Spark.
For example, when enabling the tests
+for Spark version 3.5.1 we may start by applying the existing diff for 3.4.3
first.</p>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nb">cd</span><span class="w">
</span>git/apache/spark
+git<span class="w"> </span>checkout<span class="w"> </span>v3.5.1
+git<span class="w"> </span>apply<span class="w"> </span>--reject<span
class="w"> </span>--whitespace<span class="o">=</span>fix<span class="w">
</span>../datafusion-comet/dev/diffs/3.4.3.diff
</pre></div>
</div>
-<p>Follow the instructions in <a class="reference external"
href="https://github.com/flamegraph-rs/flamegraph";>cargo-flamegraph</a> for
your platform for
-running flamegraph.</p>
-<p>Here is a sample command for running <code class="docutils literal
notranslate"><span class="pre">cargo-flamegraph</span></code> on MacOS.</p>
-<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>cargo<span class="w">
</span>flamegraph<span class="w"> </span>--root<span class="w">
</span>--bench<span class="w"> </span>shuffle_writer
+<p>Any changes that cannot be cleanly applied will instead be written out to
reject files. For example, the above
+command generated the following files.</p>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>find<span class="w"> </span>.<span
class="w"> </span>-name<span class="w"> </span><span
class="s2">"*.rej"</span>
+./pom.xml.rej
+./sql/core/src/test/scala/org/apache/spark/sql/FileBasedDataSourceSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SchemaPruningSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormatSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRebaseDatetimeSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/sources/CreateTableAsSelectSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/sources/DisableUnnecessaryBucketedScanSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala.rej
+./sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala.rej
+./sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala.rej
</pre></div>
</div>
-<p>This will produce output similar to the following.</p>
-<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">dtrace</span><span
class="p">:</span> <span class="n">system</span> <span
class="n">integrity</span> <span class="n">protection</span> <span
class="ow">is</span> <span class="n">on</span><span class="p">,</span> <span
class="n">some</span> <span class="n">features</span> <span
class="n">will</span> <span class="ow">not</span> <span class="n">be</span>
<span class="n">available</span>
-
-<span class="n">dtrace</span><span class="p">:</span> <span
class="n">description</span> <span class="s1">'profile-997 '</span>
<span class="n">matched</span> <span class="mi">1</span> <span
class="n">probe</span>
-<span class="n">Gnuplot</span> <span class="ow">not</span> <span
class="n">found</span><span class="p">,</span> <span class="n">using</span>
<span class="n">plotters</span> <span class="n">backend</span>
-<span class="n">Testing</span> <span class="n">shuffle_writer</span><span
class="o">/</span><span class="n">shuffle_writer</span>
-<span class="n">Success</span>
-
-<span class="n">dtrace</span><span class="p">:</span> <span
class="n">pid</span> <span class="mi">66402</span> <span class="n">has</span>
<span class="n">exited</span>
-<span class="n">writing</span> <span class="n">flamegraph</span> <span
class="n">to</span> <span class="s2">"flamegraph.svg"</span>
+<p>The changes in these reject files need to be applied manually.</p>
+<p>One method is to use the <a class="reference external"
href="https://github.com/neilbrown/wiggle";>wiggle</a> command (<code
class="docutils literal notranslate"><span class="pre">brew</span> <span
class="pre">install</span> <span class="pre">wiggle</span></code> on Mac).</p>
+<p>For example:</p>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>wiggle<span class="w">
</span>--replace<span class="w">
</span>./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala<span
class="w">
</span>./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala.rej
+</pre></div>
+</div>
+</section>
+<section id="generating-the-diff-file">
+<h2>Generating The Diff File<a class="headerlink"
href="#generating-the-diff-file" title="Link to this heading">¶</a></h2>
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>git<span class="w"> </span>diff<span
class="w"> </span>v3.5.1<span class="w"> </span>><span class="w">
</span>../datafusion-comet/dev/diffs/3.5.1.diff
</pre></div>
</div>
-<p>The generated flamegraph can now be opened in a browser that supports svg
format.</p>
-<p>Here is the flamegraph for this example:</p>
-<p><img alt="flamegraph" src="../_images/flamegraph.png" /></p>
+</section>
+<section id="running-tests-in-ci">
+<h2>Running Tests in CI<a class="headerlink" href="#running-tests-in-ci"
title="Link to this heading">¶</a></h2>
+<p>The easiest way to run the tests is to create a PR against Comet and let CI
run the tests. When working with a
+new Spark version, the <code class="docutils literal notranslate"><span
class="pre">spark_sql_test.yaml</span></code> and <code class="docutils literal
notranslate"><span class="pre">spark_sql_test_ansi.yaml</span></code> files
will need updating with the
+new version.</p>
</section>
</section>
@@ -383,11 +446,11 @@ running flamegraph.</p>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
- <a class='left-prev' id="prev-link" href="adding_a_new_expression.html"
title="previous page">
+ <a class='left-prev' id="prev-link" href="profiling_native_code.html"
title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
- <p class="prev-next-title">Adding a New Expression</p>
+ <p class="prev-next-title">Profiling Native Code</p>
</div>
</a>
</div>
diff --git a/genindex.html b/genindex.html
index cbb2e022..fa7d6093 100644
--- a/genindex.html
+++ b/genindex.html
@@ -187,6 +187,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/index.html b/index.html
index 1a8082c8..4a1a1927 100644
--- a/index.html
+++ b/index.html
@@ -189,6 +189,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
@@ -316,6 +321,7 @@ as a native runtime to achieve improvement in terms of
query efficiency and quer
<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/benchmarking.html">Benchmarking Guide</a></li>
<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/adding_a_new_expression.html">Adding a New
Expression</a></li>
<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/profiling_native_code.html">Profiling Native
Code</a></li>
+<li class="toctree-l1"><a class="reference internal"
href="contributor-guide/spark-sql-tests.html">Spark SQL Tests</a></li>
<li class="toctree-l1"><a class="reference external"
href="https://github.com/apache/datafusion-comet";>Github and Issue
Tracker</a></li>
</ul>
</div>
diff --git a/objects.inv b/objects.inv
index 72b9cbda..1be8f1ef 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/search.html b/search.html
index 043b51cd..1bbe7040 100644
--- a/search.html
+++ b/search.html
@@ -194,6 +194,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal" href="contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/searchindex.js b/searchindex.js
index 48f24555..c860de78 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"ANSI mode": [[8, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[7, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, "adding-a-new-expression"]], "Adding a New Scalar Function Expression":
[[0, "adding-a-new-scalar-function-expression"]], "Adding the Expression To the
Protobuf Definition" [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[7, "install-comet"]], "2.
Clone Spark and Apply Diff": [[7, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[7, "run-spark-sql-tests"]], "ANSI mode": [[9, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[8, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, "a [...]
\ No newline at end of file
diff --git a/user-guide/compatibility.html b/user-guide/compatibility.html
index cbc2c561..a3005d8b 100644
--- a/user-guide/compatibility.html
+++ b/user-guide/compatibility.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal"
href="../contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 4730e524..2b7774a7 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal"
href="../contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/user-guide/datatypes.html b/user-guide/datatypes.html
index 7d4fe428..61bf44e5 100644
--- a/user-guide/datatypes.html
+++ b/user-guide/datatypes.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal"
href="../contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/user-guide/expressions.html b/user-guide/expressions.html
index d149bb0b..71e8b367 100644
--- a/user-guide/expressions.html
+++ b/user-guide/expressions.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal"
href="../contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/user-guide/installation.html b/user-guide/installation.html
index 3d0b01a6..e5ebf26a 100644
--- a/user-guide/installation.html
+++ b/user-guide/installation.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal"
href="../contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/user-guide/operators.html b/user-guide/operators.html
index de5dec93..1a57f590 100644
--- a/user-guide/operators.html
+++ b/user-guide/operators.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal"
href="../contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/user-guide/overview.html b/user-guide/overview.html
index cd2b8747..b7f119ca 100644
--- a/user-guide/overview.html
+++ b/user-guide/overview.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal"
href="../contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
diff --git a/user-guide/tuning.html b/user-guide/tuning.html
index de9db92e..3a967d9e 100644
--- a/user-guide/tuning.html
+++ b/user-guide/tuning.html
@@ -190,6 +190,11 @@ under the License.
Profiling Native Code
</a>
</li>
+ <li class="toctree-l1">
+ <a class="reference internal"
href="../contributor-guide/spark-sql-tests.html">
+ Spark SQL Tests
+ </a>
+ </li>
<li class="toctree-l1">
<a class="reference external"
href="https://github.com/apache/datafusion-comet";>
Github and Issue Tracker
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
