This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 75110f59d0 Publish built docs triggered by
0cec8f8a084b9a15f0ccec0eab7b0bd3494cee4e
75110f59d0 is described below
commit 75110f59d09ee06e26f45a8235738129561bb125
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon May 18 17:28:58 2026 +0000
Publish built docs triggered by 0cec8f8a084b9a15f0ccec0eab7b0bd3494cee4e
---
_sources/user-guide/latest/compatibility/scans.md.txt | 6 ++++++
_sources/user-guide/latest/configs.md.txt | 1 +
searchindex.js | 2 +-
user-guide/latest/compatibility/scans.html | 6 ++++++
user-guide/latest/configs.html | 12 ++++++++----
5 files changed, 22 insertions(+), 5 deletions(-)
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt
b/_sources/user-guide/latest/compatibility/scans.md.txt
index de5e273df1..f784a4f467 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -79,6 +79,12 @@ requires `spark.comet.exec.enabled=true` because the scan
node must be wrapped b
- Duplicate field names in case-insensitive mode (e.g., a Parquet file with
both `B` and `b` columns)
are detected at read time and raise a `SparkRuntimeException` with error
class `_LEGACY_ERROR_TEMP_2093`,
matching Spark's behavior.
+- `spark.sql.parquet.enableVectorizedReader=false`. Disabling the vectorized
reader opts into
+ Spark's parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet's native reader
+ does not replicate. By default Comet falls back to Spark in this case. Set
+ `spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to
running the
+ `native_datafusion` scan regardless. See
+ [#4352](https://github.com/apache/datafusion-comet/issues/4352).
The following `native_datafusion` limitations may produce incorrect results on
Spark versions prior to 4.0
without falling back to Spark:
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index 747a517ab5..85f06a149b 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -27,6 +27,7 @@ Comet provides the following configuration settings.
<!-- prettier-ignore-start -->
| Config | Description | Default Value |
|--------|-------------|---------------|
+| `spark.comet.scan.allowDisabledParquetVectorizedReader` | Whether to allow
Comet's native scan to replace the Parquet scan when Spark's
spark.sql.parquet.enableVectorizedReader is set to false. By default (false),
Comet falls back to Spark in that case, because Comet's native readers mirror
Spark's vectorized reader semantics rather than Spark's parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information [...]
| `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` | The number of
Iceberg data files to read concurrently within a single task. Higher values
improve throughput for tables with many small files by overlapping I/O latency,
but increase memory usage. Values between 2 and 8 are suggested. | 1 |
| `spark.comet.scan.icebergNative.enabled` | Whether to enable native Iceberg
table scan using iceberg-rust. When enabled, Iceberg tables are read directly
through native execution, bypassing Spark's DataSource V2 API for better
performance. | true |
| `spark.comet.scan.unsignedSmallIntSafetyCheck` | Parquet files may contain
unsigned 8-bit integers (UINT_8) which Spark maps to ShortType. When this
config is true (default), Comet falls back to Spark for ShortType columns
because we cannot distinguish signed INT16 (safe) from unsigned UINT_8 (may
produce different results). Set to false to allow native execution of ShortType
columns if you know your data does not contain unsigned UINT_8 columns from
improperly encoded Parquet files. F [...]
diff --git a/searchindex.js b/searchindex.js
index 61de897579..134e781c05 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[34,
"format-your-code"]], "1. Install Comet": [[36, "install-comet"], [44,
"install-comet"]], "1. Native Operators (nativeExecs map)": [[23,
"native-operators-nativeexecs-map"]], "2. Build and Verify": [[34,
"build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36,
"clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[44,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23,
"sink-operators-sinks [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[34,
"format-your-code"]], "1. Install Comet": [[36, "install-comet"], [44,
"install-comet"]], "1. Native Operators (nativeExecs map)": [[23,
"native-operators-nativeexecs-map"]], "2. Build and Verify": [[34,
"build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36,
"clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[44,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23,
"sink-operators-sinks [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/scans.html
b/user-guide/latest/compatibility/scans.html
index 36f28ec8d8..b90acbff60 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -582,6 +582,12 @@ The <code class="docutils literal notranslate"><span
class="pre">native_datafusi
<li><p>Duplicate field names in case-insensitive mode (e.g., a Parquet file
with both <code class="docutils literal notranslate"><span
class="pre">B</span></code> and <code class="docutils literal
notranslate"><span class="pre">b</span></code> columns)
are detected at read time and raise a <code class="docutils literal
notranslate"><span class="pre">SparkRuntimeException</span></code> with error
class <code class="docutils literal notranslate"><span
class="pre">_LEGACY_ERROR_TEMP_2093</span></code>,
matching Spark’s behavior.</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">spark.sql.parquet.enableVectorizedReader=false</span></code>.
Disabling the vectorized reader opts into
+Spark’s parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet’s native reader
+does not replicate. By default Comet falls back to Spark in this case. Set
+<code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.allowDisabledParquetVectorizedReader=true</span></code>
to opt in to running the
+<code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan regardless. See
+<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4352";>#4352</a>.</p></li>
</ul>
<p>The following <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> limitations may produce incorrect
results on Spark versions prior to 4.0
without falling back to Spark:</p>
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index ede2772209..24063b0620 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -514,19 +514,23 @@ under the License.
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.icebergNative.dataFileConcurrencyLimit</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.allowDisabledParquetVectorizedReader</span></code></p></td>
+<td><p>Whether to allow Comet’s native scan to replace the Parquet scan when
Spark’s spark.sql.parquet.enableVectorizedReader is set to false. By default
(false), Comet falls back to Spark in that case, because Comet’s native readers
mirror Spark’s vectorized reader semantics rather than Spark’s parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information, refer to the <a
class="reference external" href="ht [...]
+<td><p>false</p></td>
+</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.icebergNative.dataFileConcurrencyLimit</span></code></p></td>
<td><p>The number of Iceberg data files to read concurrently within a single
task. Higher values improve throughput for tables with many small files by
overlapping I/O latency, but increase memory usage. Values between 2 and 8 are
suggested.</p></td>
<td><p>1</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.icebergNative.enabled</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.icebergNative.enabled</span></code></p></td>
<td><p>Whether to enable native Iceberg table scan using iceberg-rust. When
enabled, Iceberg tables are read directly through native execution, bypassing
Spark’s DataSource V2 API for better performance.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.unsignedSmallIntSafetyCheck</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.unsignedSmallIntSafetyCheck</span></code></p></td>
<td><p>Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark
maps to ShortType. When this config is true (default), Comet falls back to
Spark for ShortType columns because we cannot distinguish signed INT16 (safe)
from unsigned UINT_8 (may produce different results). Set to false to allow
native execution of ShortType columns if you know your data does not contain
unsigned UINT_8 columns from improperly encoded Parquet files. For more
information, refer to the <a class=" [...]
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.hadoop.fs.comet.libhdfs.schemes</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.hadoop.fs.comet.libhdfs.schemes</span></code></p></td>
<td><p>Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side
accesses via libhdfs, separated by commas. Valid only when built with hdfs
feature enabled.</p></td>
<td><p></p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
