This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 50cecd949 Publish built docs triggered by
ad873c2191cc9c917ccd15c195041c7bc7e25e3b
50cecd949 is described below
commit 50cecd949389f84a434816b8a1336691a80a2406
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu Jul 3 16:16:00 2025 +0000
Publish built docs triggered by ad873c2191cc9c917ccd15c195041c7bc7e25e3b
---
_sources/contributor-guide/spark-sql-tests.md.txt | 12 ++++++++----
contributor-guide/spark-sql-tests.html | 9 ++++++---
searchindex.js | 2 +-
3 files changed, 15 insertions(+), 8 deletions(-)
diff --git a/_sources/contributor-guide/spark-sql-tests.md.txt
b/_sources/contributor-guide/spark-sql-tests.md.txt
index 2252a6b13..1425b163b 100644
--- a/_sources/contributor-guide/spark-sql-tests.md.txt
+++ b/_sources/contributor-guide/spark-sql-tests.md.txt
@@ -67,16 +67,16 @@ ENABLE_COMET=true build/sbt "hive/testOnly * -- -n
org.apache.spark.tags.SlowHiv
```
#### Steps to run individual test suites through SBT
1. Open SBT with Comet enabled
-```sbt
-ENABLE_COMET=true sbt -Dspark.test.includeSlowTests=true
+```shell
+ENABLE_COMET=true sbt -J-Xmx4096m -Dspark.test.includeSlowTests=true
```
2. Run individual tests (Below code runs test named `SPARK-35568` in the
`spark-sql` module)
-```sbt
+```shell
sql/testOnly org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn -- -z
"SPARK-35568"
```
#### Steps to run individual test suites in IntelliJ IDE
1. Add below configuration in VM Options for your test case (apache-spark
repository)
-```sbt
+```shell
-Dspark.comet.enabled=true -Dspark.comet.debug.enabled=true
-Dspark.plugins=org.apache.spark.CometPlugin -DXmx4096m
-Dspark.executor.heartbeatInterval=20000 -Dspark.network.timeout=10000
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
```
2. Set `ENABLE_COMET=true` in environment variables
@@ -138,8 +138,12 @@ wiggle --replace
./sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.sc
The diff file can be generated using the `git diff` command. It may be
necessary to set the `core.abbrev`
configuration setting to use 11 digits hashes for consistency with existing
diff files.
+Note that there is an `IgnoreComet.scala` that is not part of the Spark
codebase, and therefore needs to be added
+using `git add` before generating the diff.
+
```shell
git config core.abbrev 11;
+git add sql/core/src/test/scala/org/apache/spark/sql/IgnoreComet.scala
git diff v3.5.6 > ../datafusion-comet/dev/diffs/3.5.6.diff
```
diff --git a/contributor-guide/spark-sql-tests.html
b/contributor-guide/spark-sql-tests.html
index f52e2bf11..44b4cfda7 100644
--- a/contributor-guide/spark-sql-tests.html
+++ b/contributor-guide/spark-sql-tests.html
@@ -432,13 +432,13 @@ git<span class="w"> </span>apply<span class="w">
</span>../datafusion-comet/dev/
<ol class="arabic simple">
<li><p>Open SBT with Comet enabled</p></li>
</ol>
-<div class="highlight-sbt notranslate"><div
class="highlight"><pre><span></span>ENABLE_COMET=true sbt
-Dspark.test.includeSlowTests=true
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="nv">ENABLE_COMET</span><span
class="o">=</span><span class="nb">true</span><span class="w"> </span>sbt<span
class="w"> </span>-J-Xmx4096m<span class="w">
</span>-Dspark.test.includeSlowTests<span class="o">=</span><span
class="nb">true</span><span class="w"> </span>
</pre></div>
</div>
<ol class="arabic simple" start="2">
<li><p>Run individual tests (Below code runs test named <code class="docutils
literal notranslate"><span class="pre">SPARK-35568</span></code> in the <code
class="docutils literal notranslate"><span class="pre">spark-sql</span></code>
module)</p></li>
</ol>
-<div class="highlight-sbt notranslate"><div
class="highlight"><pre><span></span> sql/testOnly
org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn -- -z
"SPARK-35568"
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span class="w"> </span>sql/testOnly<span
class="w"> </span>org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn<span
class="w"> </span>--<span class="w"> </span>-z<span class="w"> </span><span
class="s2">"SPARK-35568"</span>
</pre></div>
</div>
</section>
@@ -447,7 +447,7 @@ git<span class="w"> </span>apply<span class="w">
</span>../datafusion-comet/dev/
<ol class="arabic simple">
<li><p>Add below configuration in VM Options for your test case (apache-spark
repository)</p></li>
</ol>
-<div class="highlight-sbt notranslate"><div
class="highlight"><pre><span></span>-Dspark.comet.enabled=true
-Dspark.comet.debug.enabled=true -Dspark.plugins=org.apache.spark.CometPlugin
-DXmx4096m -Dspark.executor.heartbeatInterval=20000
-Dspark.network.timeout=10000 --add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
+<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>-Dspark.comet.enabled<span
class="o">=</span><span class="nb">true</span><span class="w">
</span>-Dspark.comet.debug.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span>-Dspark.plugins<span
class="o">=</span>org.apache.spark.CometPlugin<span class="w">
</span>-DXmx4096m<span class="w">
</span>-Dspark.executor.heartbeatInterval<span class="o">=</span><span
class="m">20000</span [...]
</pre></div>
</div>
<ol class="arabic simple" start="2">
@@ -503,7 +503,10 @@ command generated the following files.</p>
<h2>Generating The Diff File<a class="headerlink"
href="#generating-the-diff-file" title="Link to this heading">ΒΆ</a></h2>
<p>The diff file can be generated using the <code class="docutils literal
notranslate"><span class="pre">git</span> <span class="pre">diff</span></code>
command. It may be necessary to set the <code class="docutils literal
notranslate"><span class="pre">core.abbrev</span></code>
configuration setting to use 11 digits hashes for consistency with existing
diff files.</p>
+<p>Note that there is an <code class="docutils literal notranslate"><span
class="pre">IgnoreComet.scala</span></code> that is not part of the Spark
codebase, and therefore needs to be added
+using <code class="docutils literal notranslate"><span class="pre">git</span>
<span class="pre">add</span></code> before generating the diff.</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span>git<span class="w"> </span>config<span
class="w"> </span>core.abbrev<span class="w"> </span><span
class="m">11</span><span class="p">;</span>
+git<span class="w"> </span>add<span class="w">
</span>sql/core/src/test/scala/org/apache/spark/sql/IgnoreComet.scala
git<span class="w"> </span>diff<span class="w"> </span>v3.5.6<span class="w">
</span>><span class="w"> </span>../datafusion-comet/dev/diffs/3.5.6.diff
</pre></div>
</div>
diff --git a/searchindex.js b/searchindex.js
index 2cfcee58d..bdf7c0204 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[11, "install-comet"]],
"2. Clone Spark and Apply Diff": [[11, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[11, "run-spark-sql-tests"]], "ANSI mode": [[14,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[13, null]],
"Accelerating Apache Iceberg Parquet Scans using Comet (Experimental)": [[19,
null]], "Adding Spark-side Tests for the New Expression": [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[11, "install-comet"]],
"2. Clone Spark and Apply Diff": [[11, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[11, "run-spark-sql-tests"]], "ANSI mode": [[14,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[13, null]],
"Accelerating Apache Iceberg Parquet Scans using Comet (Experimental)": [[19,
null]], "Adding Spark-side Tests for the New Expression": [...]
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
