This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 38fef1754b Publish built docs triggered by
42a48bbdf9e14735a1bebeeb6545ab7979e6ca9b
38fef1754b is described below
commit 38fef1754bc79eb77e11e3ded5959e3a92a2c55b
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Sun Aug 2 20:26:07 2026 +0000
Publish built docs triggered by 42a48bbdf9e14735a1bebeeb6545ab7979e6ca9b
---
.../user-guide/latest/compatibility/scans.md.txt | 4 +--
_sources/user-guide/latest/configs.md.txt | 1 -
searchindex.js | 2 +-
user-guide/latest/compatibility/scans.html | 4 +--
user-guide/latest/configs.html | 32 ++++++++++------------
5 files changed, 17 insertions(+), 26 deletions(-)
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt
b/_sources/user-guide/latest/compatibility/scans.md.txt
index 0c07b5c237..00b3284e10 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -57,9 +57,7 @@ The following limitation may produce incorrect results
without falling back to S
if they were written using the Proleptic Gregorian calendar. This produces
silently-wrong
values for dates before October 15, 1582 in both projections and predicates.
Comet also
ignores `spark.sql.parquet.datetimeRebaseModeInRead` and the file-level
- `org.apache.spark.legacyDateTime` metadata that would tell it to rebase. The
- `spark.comet.exceptionOnDatetimeRebase` config is currently dead code and
does not raise on
- legacy-calendar data. Tracked by
+ `org.apache.spark.legacyDateTime` metadata that would tell it to rebase.
Tracked by
[#5010](https://github.com/apache/datafusion-comet/issues/5010).
The following limitations raise an error at scan time rather than falling back
to Spark:
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index 7853d03afc..8ac9f1eea9 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -57,7 +57,6 @@ Comet provides the following configuration settings.
| `spark.comet.convert.parquet.enabled` | When enabled, data from Spark
(non-native) Parquet v1 and v2 scans will be converted to Arrow format. | false
|
| `spark.comet.debug.enabled` | Whether to enable debug mode for Comet. When
enabled, Comet will do additional checks for debugging purpose. For example,
validating array when importing arrays from JVM at native side. Note that these
checks may be expensive in performance and should only be enabled for debugging
purpose. | false |
| `spark.comet.enabled` | Whether to enable Comet extension for Spark. When
this is turned on, Spark will use Comet to read Parquet data source. Note that
to enable native vectorized execution, both this config and
`spark.comet.exec.enabled` need to be enabled. It can be overridden by the
environment variable `ENABLE_COMET`. | true |
-| `spark.comet.exceptionOnDatetimeRebase` | Whether to throw exception when
seeing dates/timestamps from the legacy hybrid (Julian + Gregorian) calendar.
Since Spark 3, dates/timestamps were written according to the Proleptic
Gregorian calendar. When this is true, Comet will throw exceptions when seeing
these dates/timestamps that were written by Spark version before 3.0. If this
is false, these dates/timestamps will be read as if they were written to the
Proleptic Gregorian calendar and [...]
| `spark.comet.exec.columnarToRow.native.enabled` | Whether to enable native
columnar to row conversion. When enabled, Comet will use native Rust code to
convert Arrow columnar data to Spark UnsafeRow format instead of the JVM
implementation. The native conversion carries a fixed JNI cost per batch and is
slower than the JVM implementation for small batches. | false |
| `spark.comet.exec.enabled` | Whether to enable Comet native vectorized
execution for Spark. This controls whether Spark should convert operators into
their Comet counterparts and execute them in native space. Note: each operator
is associated with a separate config in the format of
`spark.comet.exec.<operator_name>.enabled` at the moment, and both the config
and this need to be turned on, in order for the operator to be executed in
native. | true |
| `spark.comet.exec.forceShuffledHashJoin` | Experimental feature to force
Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance.
This feature is not stable yet. For more information, refer to the [Comet
Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
false |
diff --git a/searchindex.js b/searchindex.js
index b5375dc6c4..45396097cb 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[54, "id1"]], "%": [[52, "id1"]], "&":
[[42, "id1"]], "*": [[52, "id2"]], "+": [[52, "id3"]], "-": [[52, "id4"]], "/":
[[52, "id5"]], "1. Format Your Code": [[39, "format-your-code"]], "1. Install
Comet": [[60, "install-comet"], [70, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[27, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[39, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[60,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[54, "id1"]], "%": [[52, "id1"]], "&":
[[42, "id1"]], "*": [[52, "id2"]], "+": [[52, "id3"]], "-": [[52, "id4"]], "/":
[[52, "id5"]], "1. Format Your Code": [[39, "format-your-code"]], "1. Install
Comet": [[60, "install-comet"], [70, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[27, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[39, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[60,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/scans.html
b/user-guide/latest/compatibility/scans.html
index cf1f1ed0eb..4b61dc5eec 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -779,9 +779,7 @@ data written before Spark 3.0, using the hybrid
Julian/Gregorian calendar), Come
if they were written using the Proleptic Gregorian calendar. This produces
silently-wrong
values for dates before October 15, 1582 in both projections and predicates.
Comet also
ignores <code class="docutils literal notranslate"><span
class="pre">spark.sql.parquet.datetimeRebaseModeInRead</span></code> and the
file-level
-<code class="docutils literal notranslate"><span
class="pre">org.apache.spark.legacyDateTime</span></code> metadata that would
tell it to rebase. The
-<code class="docutils literal notranslate"><span
class="pre">spark.comet.exceptionOnDatetimeRebase</span></code> config is
currently dead code and does not raise on
-legacy-calendar data. Tracked by
+<code class="docutils literal notranslate"><span
class="pre">org.apache.spark.legacyDateTime</span></code> metadata that would
tell it to rebase. Tracked by
<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/5010";>#5010</a>.</p></li>
</ul>
<p>The following limitations raise an error at scan time rather than falling
back to Spark:</p>
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index be7427d4aa..fbc0bc3df1 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -839,63 +839,59 @@ under the License.
<td><p>Whether to enable Comet extension for Spark. When this is turned on,
Spark will use Comet to read Parquet data source. Note that to enable native
vectorized execution, both this config and <code class="docutils literal
notranslate"><span class="pre">spark.comet.exec.enabled</span></code> need to
be enabled. It can be overridden by the environment variable <code
class="docutils literal notranslate"><span
class="pre">ENABLE_COMET</span></code>.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exceptionOnDatetimeRebase</span></code></p></td>
-<td><p>Whether to throw exception when seeing dates/timestamps from the legacy
hybrid (Julian + Gregorian) calendar. Since Spark 3, dates/timestamps were
written according to the Proleptic Gregorian calendar. When this is true, Comet
will throw exceptions when seeing these dates/timestamps that were written by
Spark version before 3.0. If this is false, these dates/timestamps will be read
as if they were written to the Proleptic Gregorian calendar and will not be
rebased.</p></td>
-<td><p>false</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.columnarToRow.native.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.columnarToRow.native.enabled</span></code></p></td>
<td><p>Whether to enable native columnar to row conversion. When enabled,
Comet will use native Rust code to convert Arrow columnar data to Spark
UnsafeRow format instead of the JVM implementation. The native conversion
carries a fixed JNI cost per batch and is slower than the JVM implementation
for small batches.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.enabled</span></code></p></td>
<td><p>Whether to enable Comet native vectorized execution for Spark. This
controls whether Spark should convert operators into their Comet counterparts
and execute them in native space. Note: each operator is associated with a
separate config in the format of <code class="docutils literal
notranslate"><span
class="pre">spark.comet.exec.<operator_name>.enabled</span></code> at the
moment, and both the config and this need to be turned on, in order for the
operator to be executed in [...]
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.forceShuffledHashJoin</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.forceShuffledHashJoin</span></code></p></td>
<td><p>Experimental feature to force Spark to replace SortMergeJoin with
ShuffledHashJoin for improved performance. This feature is not stable yet. For
more information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.pyarrowUdf.enabled</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.pyarrowUdf.enabled</span></code></p></td>
<td><p>Experimental: whether to enable optimized execution of PyArrow UDFs
(mapInArrow/mapInPandas). When enabled, Comet passes Arrow columnar data
directly to Python UDFs without the intermediate Arrow-to-Row-to-Arrow
conversion that Spark normally performs. Disabled by default while the feature
stabilizes.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
<td><p>Whether to route Spark <code class="docutils literal notranslate"><span
class="pre">ScalaUDF</span></code> expressions through Comet’s Arrow-direct
codegen dispatcher. When enabled, a supported ScalaUDF is compiled into a
per-batch kernel that reads and writes Arrow vectors directly from native
execution. When disabled, plans containing a ScalaUDF fall back to Spark for
the enclosing operator. The same dispatcher backs the regex family (<code
class="docutils literal notranslate">< [...]
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
<td><p>When enabled, fall back to Spark for floating-point operations that may
differ from Spark, such as when comparing or sorting -0.0 and 0.0. For more
information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html";>Comet
Compatibility Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
<td><p>When enabled, Comet reverts a query stage to Spark row-based execution
if the number of columnar-to-row (C2R) transitions in the stage exceeds the
configured threshold. This avoids the overhead of repeated format conversions
in stages where many operators fall back to row-based execution.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.transitionRevert.maxTransitions</span></code></p></td>
<td><p>The maximum number of columnar-to-row (C2R) transitions allowed in a
single query stage before Comet reverts the entire stage to Spark row-based
execution. When columnar shuffle is enabled, each such C2R typically implies a
corresponding row-to-columnar conversion to feed back into the columnar
shuffle, so each counted C2R is a useful proxy for the conversion overhead in
the stage. Set to 0 to revert any stage with transitions. Only effective when
spark.comet.exec.transitionRevert [...]
<td><p>2</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.metrics.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.metrics.enabled</span></code></p></td>
<td><p>Whether to enable Comet metrics reporting through Spark’s external
monitoring system. When enabled, Comet exposes metrics such as native
operators, Spark operators, queries planned, transitions, and acceleration
ratio. These metrics can be visualized through tools like Grafana when a
metrics sink (e.g., Prometheus) is configured. Disabled by default because
Spark plan traversal adds overhead and metrics require a sink to be useful.
This config must be set before the SparkSession i [...]
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.metrics.updateInterval</span></code></p></td>
<td><p>The interval in milliseconds to update metrics. If interval is
negative, metrics will be updated upon task completion.</p></td>
<td><p>3000</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.nativeLoadRequired</span></code></p></td>
<td><p>Whether to require Comet native library to load successfully when Comet
is enabled. If not, Comet will silently fallback to Spark when it fails to load
the native lib. Otherwise, an error will be thrown and the Spark job will be
aborted.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.operator.DataWritingCommandExec.allowIncompatible</span></code></p></td>
<td><p>Whether to allow incompatibility for operator: DataWritingCommandExec.
False by default. Can be overridden with
SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE env variable It
can be overridden by the environment variable <code class="docutils literal
notranslate"><span
class="pre">SPARK_COMET_OPERATOR_DATAWRITINGCOMMANDEXEC_ALLOWINCOMPATIBLE</span></code>.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code></p></td>
<td><p>Whether to enable Spark to Arrow columnar conversion. When this is
turned on, Comet will convert operators in <code class="docutils literal
notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code>
into Arrow columnar format before processing.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.supportedOperatorList</span></code></p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow
columnar format when <code class="docutils literal notranslate"><span
class="pre">spark.comet.sparkToColumnar.enabled</span></code> is true.</p></td>
<td><p>Range,InMemoryTableScan,RDDScan,OneRowRelation</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
