This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 8cf3c8dd25931459eeadf104bbad7500d3406c47
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Wed Jul 8 02:00:52 2026 +0000
Publish built docs triggered by 71758bcc77395bc1afd354ba2f1110baab1ea63e
---
_sources/user-guide/latest/configs.md.txt | 14 +++++++-------
user-guide/latest/configs.html | 14 +++++++-------
2 files changed, 14 insertions(+), 14 deletions(-)
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index b3f4a8740e..d24e70c341 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -27,10 +27,10 @@ Comet provides the following configuration settings.
<!-- prettier-ignore-start -->
| Config | Description | Default Value |
|--------|-------------|---------------|
-| `spark.comet.scan.allowDisabledParquetVectorizedReader` | Whether to allow
Comet's native scan to replace the Parquet scan when Spark's
spark.sql.parquet.enableVectorizedReader is set to false. By default (false),
Comet falls back to Spark in that case, because Comet's native readers mirror
Spark's vectorized reader semantics rather than Spark's parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information [...]
+| `spark.comet.scan.allowDisabledParquetVectorizedReader` | Whether to allow
Comet's native scan to replace the Parquet scan when Spark's
spark.sql.parquet.enableVectorizedReader is set to false. By default (false),
Comet falls back to Spark in that case, because Comet's native readers mirror
Spark's vectorized reader semantics rather than Spark's parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information [...]
| `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` | The number of
Iceberg data files to read concurrently within a single task. Higher values
improve throughput for tables with many small files by overlapping I/O latency,
but increase memory usage. Values between 2 and 8 are suggested. | 1 |
| `spark.comet.scan.icebergNative.enabled` | Whether to enable native Iceberg
table scan using iceberg-rust. When enabled, Iceberg tables are read directly
through native execution, bypassing Spark's DataSource V2 API for better
performance. | true |
-| `spark.comet.scan.unsignedSmallIntSafetyCheck` | Parquet files may contain
unsigned 8-bit integers (UINT_8) which Spark maps to ShortType. When this
config is true (default), Comet falls back to Spark for ShortType columns
because we cannot distinguish signed INT16 (safe) from unsigned UINT_8 (may
produce different results). Set to false to allow native execution of ShortType
columns if you know your data does not contain unsigned UINT_8 columns from
improperly encoded Parquet files. F [...]
+| `spark.comet.scan.unsignedSmallIntSafetyCheck` | Parquet files may contain
unsigned 8-bit integers (UINT_8) which Spark maps to ShortType. When this
config is true (default), Comet falls back to Spark for ShortType columns
because we cannot distinguish signed INT16 (safe) from unsigned UINT_8 (may
produce different results). Set to false to allow native execution of ShortType
columns if you know your data does not contain unsigned UINT_8 columns from
improperly encoded Parquet files. F [...]
| `spark.hadoop.fs.comet.libhdfs.schemes` | Defines filesystem schemes (e.g.,
hdfs, webhdfs) that the native side accesses via libhdfs, separated by commas.
Valid only when built with hdfs feature enabled. | |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
@@ -65,9 +65,9 @@ Comet provides the following configuration settings.
| `spark.comet.exceptionOnDatetimeRebase` | Whether to throw exception when
seeing dates/timestamps from the legacy hybrid (Julian + Gregorian) calendar.
Since Spark 3, dates/timestamps were written according to the Proleptic
Gregorian calendar. When this is true, Comet will throw exceptions when seeing
these dates/timestamps that were written by Spark version before 3.0. If this
is false, these dates/timestamps will be read as if they were written to the
Proleptic Gregorian calendar and [...]
| `spark.comet.exec.columnarToRow.native.enabled` | Whether to enable native
columnar to row conversion. When enabled, Comet will use native Rust code to
convert Arrow columnar data to Spark UnsafeRow format instead of the JVM
implementation. This can improve performance for queries that need to convert
between columnar and row formats. | true |
| `spark.comet.exec.enabled` | Whether to enable Comet native vectorized
execution for Spark. This controls whether Spark should convert operators into
their Comet counterparts and execute them in native space. Note: each operator
is associated with a separate config in the format of
`spark.comet.exec.<operator_name>.enabled` at the moment, and both the config
and this need to be turned on, in order for the operator to be executed in
native. | true |
-| `spark.comet.exec.replaceSortMergeJoin` | Experimental feature to force
Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance.
This feature is not stable yet. For more information, refer to the [Comet
Tuning Guide](https://datafusion.apache.org/comet/user-guide/tuning.html). |
false |
+| `spark.comet.exec.replaceSortMergeJoin` | Experimental feature to force
Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance.
This feature is not stable yet. For more information, refer to the [Comet
Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
false |
| `spark.comet.exec.scalaUDF.codegen.enabled` | Whether to route Spark
`ScalaUDF` expressions through Comet's Arrow-direct codegen dispatcher. When
enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads
and writes Arrow vectors directly from native execution. When disabled, plans
containing a ScalaUDF fall back to Spark for the enclosing operator. The same
dispatcher backs the regex family (`rlike`, `regexp_replace`, `split`,
`regexp_extract`, `regexp_extract_all`, [...]
-| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark
for floating-point operations that may differ from Spark, such as when
comparing or sorting -0.0 and 0.0. For more information, refer to the [Comet
Compatibility
Guide](https://datafusion.apache.org/comet/user-guide/compatibility.html). |
false |
+| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark
for floating-point operations that may differ from Spark, such as when
comparing or sorting -0.0 and 0.0. For more information, refer to the [Comet
Compatibility
Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html).
| false |
| `spark.comet.exec.transitionRevert.enabled` | When enabled, Comet reverts a
query stage to Spark row-based execution if the number of columnar-to-row (C2R)
transitions in the stage exceeds the configured threshold. This avoids the
overhead of repeated format conversions in stages where many operators fall
back to row-based execution. | false |
| `spark.comet.exec.transitionRevert.maxTransitions` | The maximum number of
columnar-to-row (C2R) transitions allowed in a single query stage before Comet
reverts the entire stage to Spark row-based execution. When columnar shuffle is
enabled, each such C2R typically implies a corresponding row-to-columnar
conversion to feed back into the columnar shuffle, so each counted C2R is a
useful proxy for the conversion overhead in the stage. Set to 0 to revert any
stage with transitions. Only [...]
| `spark.comet.maxTempDirectorySize` | The maximum amount of data (in bytes)
stored inside the temporary directories. | 107374182400b |
@@ -129,8 +129,8 @@ These settings can be used to determine which parts of the
plan are accelerated
| Config | Description | Default Value |
|--------|-------------|---------------|
| `spark.comet.batchSize` | The columnar batch size, i.e., the maximum number
of rows that a batch can contain. | 8192 |
-| `spark.comet.exec.memoryPool` | The type of memory pool to be used for Comet
native execution when running Spark in off-heap mode. Available pool types are
`greedy_unified` and `fair_unified`. For more information, refer to the [Comet
Tuning Guide](https://datafusion.apache.org/comet/user-guide/tuning.html). |
fair_unified |
-| `spark.comet.exec.memoryPool.fraction` | Fraction of off-heap memory pool
that is available to Comet. Only applies to off-heap mode. For more
information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/tuning.html). | 1.0 |
+| `spark.comet.exec.memoryPool` | The type of memory pool to be used for Comet
native execution when running Spark in off-heap mode. Available pool types are
`greedy_unified` and `fair_unified`. For more information, refer to the [Comet
Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
fair_unified |
+| `spark.comet.exec.memoryPool.fraction` | Fraction of off-heap memory pool
that is available to Comet. Only applies to off-heap mode. For more
information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
1.0 |
| `spark.comet.tracing.enabled` | Enable fine-grained tracing of events and
memory usage. For more information, refer to the [Comet Tracing
Guide](https://datafusion.apache.org/comet/contributor-guide/tracing.html). |
false |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
@@ -141,7 +141,7 @@ These settings can be used to determine which parts of the
plan are accelerated
<!-- prettier-ignore-start -->
| Config | Description | Default Value |
|--------|-------------|---------------|
-| `spark.comet.columnar.shuffle.memory.factor` | Fraction of Comet memory to
be allocated per executor process for columnar shuffle when running in on-heap
mode. For more information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/tuning.html). | 1.0 |
+| `spark.comet.columnar.shuffle.memory.factor` | Fraction of Comet memory to
be allocated per executor process for columnar shuffle when running in on-heap
mode. For more information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
1.0 |
| `spark.comet.debug.memory` | When enabled, log all native memory pool
interactions. For more information, refer to the Comet Debugging Guide
(https://datafusion.apache.org/comet/contributor-guide/debugging.html). | false
|
| `spark.comet.exec.onHeap.enabled` | Whether to allow Comet to run in on-heap
mode. Required for running Spark SQL tests. It can be overridden by the
environment variable `ENABLE_COMET_ONHEAP`. | false |
| `spark.comet.exec.onHeap.memoryPool` | The type of memory pool to be used
for Comet native execution when running Spark in on-heap mode. Available pool
types are `greedy`, `fair_spill`, `greedy_task_shared`,
`fair_spill_task_shared`, `greedy_global`, `fair_spill_global`, and
`unbounded`. | greedy_task_shared |
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 48ff9c902e..dbdb32cc6b 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -754,7 +754,7 @@ under the License.
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.allowDisabledParquetVectorizedReader</span></code></p></td>
-<td><p>Whether to allow Comet’s native scan to replace the Parquet scan when
Spark’s spark.sql.parquet.enableVectorizedReader is set to false. By default
(false), Comet falls back to Spark in that case, because Comet’s native readers
mirror Spark’s vectorized reader semantics rather than Spark’s parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information, refer to the <a
class="reference external" href="ht [...]
+<td><p>Whether to allow Comet’s native scan to replace the Parquet scan when
Spark’s spark.sql.parquet.enableVectorizedReader is set to false. By default
(false), Comet falls back to Spark in that case, because Comet’s native readers
mirror Spark’s vectorized reader semantics rather than Spark’s parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information, refer to the <a
class="reference external" href="ht [...]
<td><p>false</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.icebergNative.dataFileConcurrencyLimit</span></code></p></td>
@@ -766,7 +766,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.unsignedSmallIntSafetyCheck</span></code></p></td>
-<td><p>Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark
maps to ShortType. When this config is true (default), Comet falls back to
Spark for ShortType columns because we cannot distinguish signed INT16 (safe)
from unsigned UINT_8 (may produce different results). Set to false to allow
native execution of ShortType columns if you know your data does not contain
unsigned UINT_8 columns from improperly encoded Parquet files. For more
information, refer to the <a class=" [...]
+<td><p>Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark
maps to ShortType. When this config is true (default), Comet falls back to
Spark for ShortType columns because we cannot distinguish signed INT16 (safe)
from unsigned UINT_8 (may produce different results). Set to false to allow
native execution of ShortType columns if you know your data does not contain
unsigned UINT_8 columns from improperly encoded Parquet files. For more
information, refer to the <a class=" [...]
<td><p>true</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.hadoop.fs.comet.libhdfs.schemes</span></code></p></td>
@@ -870,7 +870,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.replaceSortMergeJoin</span></code></p></td>
-<td><p>Experimental feature to force Spark to replace SortMergeJoin with
ShuffledHashJoin for improved performance. This feature is not stable yet. For
more information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/tuning.html";>Comet Tuning
Guide</a>.</p></td>
+<td><p>Experimental feature to force Spark to replace SortMergeJoin with
ShuffledHashJoin for improved performance. This feature is not stable yet. For
more information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
@@ -878,7 +878,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
-<td><p>When enabled, fall back to Spark for floating-point operations that may
differ from Spark, such as when comparing or sorting -0.0 and 0.0. For more
information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/compatibility.html";>Comet
Compatibility Guide</a>.</p></td>
+<td><p>When enabled, fall back to Spark for floating-point operations that may
differ from Spark, such as when comparing or sorting -0.0 and 0.0. For more
information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html";>Comet
Compatibility Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
@@ -1065,11 +1065,11 @@ under the License.
<td><p>8192</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool</span></code></p></td>
-<td><p>The type of memory pool to be used for Comet native execution when
running Spark in off-heap mode. Available pool types are <code class="docutils
literal notranslate"><span class="pre">greedy_unified</span></code> and <code
class="docutils literal notranslate"><span
class="pre">fair_unified</span></code>. For more information, refer to the <a
class="reference external"
href="https://datafusion.apache.org/comet/user-guide/tuning.html";>Comet Tuning
Guide</a>.</p></td>
+<td><p>The type of memory pool to be used for Comet native execution when
running Spark in off-heap mode. Available pool types are <code class="docutils
literal notranslate"><span class="pre">greedy_unified</span></code> and <code
class="docutils literal notranslate"><span
class="pre">fair_unified</span></code>. For more information, refer to the <a
class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>fair_unified</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool.fraction</span></code></p></td>
-<td><p>Fraction of off-heap memory pool that is available to Comet. Only
applies to off-heap mode. For more information, refer to the <a
class="reference external"
href="https://datafusion.apache.org/comet/user-guide/tuning.html";>Comet Tuning
Guide</a>.</p></td>
+<td><p>Fraction of off-heap memory pool that is available to Comet. Only
applies to off-heap mode. For more information, refer to the <a
class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>1.0</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.tracing.enabled</span></code></p></td>
@@ -1095,7 +1095,7 @@ under the License.
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.memory.factor</span></code></p></td>
-<td><p>Fraction of Comet memory to be allocated per executor process for
columnar shuffle when running in on-heap mode. For more information, refer to
the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/tuning.html";>Comet Tuning
Guide</a>.</p></td>
+<td><p>Fraction of Comet memory to be allocated per executor process for
columnar shuffle when running in on-heap mode. For more information, refer to
the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>1.0</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.debug.memory</span></code></p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
