This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new cba3d4413b Publish built docs triggered by
fbc3d2f1b18c24c47a2d30adc03b0b5920cc9574
cba3d4413b is described below
commit cba3d4413b637bb9416bc00ed810e7a31ca07180
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue May 19 13:53:48 2026 +0000
Publish built docs triggered by fbc3d2f1b18c24c47a2d30adc03b0b5920cc9574
---
.../adding_a_new_spark_version.md.txt | 12 +--
.../user-guide/latest/compatibility/index.md.txt | 2 +-
.../user-guide/latest/compatibility/scans.md.txt | 76 ++++++------------
.../latest/compatibility/spark-versions.md.txt | 16 ++--
_sources/user-guide/latest/datasources.md.txt | 19 ++---
.../latest/understanding-comet-plans.md.txt | 13 ++--
contributor-guide/adding_a_new_spark_version.html | 12 +--
searchindex.js | 2 +-
user-guide/latest/compatibility/index.html | 2 +-
user-guide/latest/compatibility/scans.html | 90 ++++++----------------
.../latest/compatibility/spark-versions.html | 16 ++--
user-guide/latest/datasources.html | 19 ++---
user-guide/latest/understanding-comet-plans.html | 13 ++--
13 files changed, 103 insertions(+), 189 deletions(-)
diff --git a/_sources/contributor-guide/adding_a_new_spark_version.md.txt
b/_sources/contributor-guide/adding_a_new_spark_version.md.txt
index 1f16e10153..65109e4d0c 100644
--- a/_sources/contributor-guide/adding_a_new_spark_version.md.txt
+++ b/_sources/contributor-guide/adding_a_new_spark_version.md.txt
@@ -138,9 +138,7 @@ own test suites under the new profile.
Promote the new Spark version from the compile-only job to the main test
jobs in `.github/workflows/pr_build_linux.yml` (and `pr_build_macos.yml` if
-capacity allows). Use `scan_impl: "auto"` so both `native_datafusion` and
-`native_iceberg_compat` get exercised, matching how earlier versions are
-configured.
+capacity allows). Match how earlier versions are configured.
### Run the Suite Locally First
@@ -256,14 +254,12 @@ new-version bring-up are:
### CI for the Spark SQL Tests
Spark SQL tests do not run from the main PR build workflows. They have
-their own dedicated workflow files:
+their own dedicated workflow file:
- `.github/workflows/spark_sql_test.yml`
-- `.github/workflows/spark_sql_test_native_iceberg_compat.yml`
-Add the new version to the matrix in each of these files (`spark-short`,
-`spark-full`, `java`, `scan-impl`). Use the closest existing entry as a
-template.
+Add the new version to the matrix (`spark-short`, `spark-full`, `java`).
+Use the closest existing entry as a template.
Before merging, run `make format`, run clippy
(`cd native && cargo clippy --all-targets --workspace -- -D warnings`), and
diff --git a/_sources/user-guide/latest/compatibility/index.md.txt
b/_sources/user-guide/latest/compatibility/index.md.txt
index 1ba9d9e181..59c6a906f1 100644
--- a/_sources/user-guide/latest/compatibility/index.md.txt
+++ b/_sources/user-guide/latest/compatibility/index.md.txt
@@ -23,7 +23,7 @@ Comet aims to provide consistent results with the version of
Apache Spark that i
This guide documents areas where Comet's behavior is known to differ from
Spark. Topics are grouped by subsystem:
-- **Parquet**: limitations when reading Parquet files (both scan
implementations, shared and per-implementation).
+- **Parquet**: limitations when reading Parquet files.
- **Floating-point comparison**: NaN and signed-zero handling in comparisons.
- **Regular expressions**: differences between the Rust regexp crate and
Java's regex engine.
- **Operators**: operator-level compatibility notes, including window
functions and round-robin partitioning.
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt
b/_sources/user-guide/latest/compatibility/scans.md.txt
index f784a4f467..37524a829e 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -19,22 +19,13 @@ under the License.
# Parquet Compatibility
-Comet currently has two distinct implementations of the Parquet scan operator.
+Comet's Parquet scan offloads decoding to native code and produces Arrow
batches for the rest of
+the plan. Comet falls back to Spark when the scan cannot be converted (for
example, due to one of
+the unsupported features listed below).
-| Scan Implementation | Notes |
-| ----------------------- | ---------------------- |
-| `native_datafusion` | Fully native scan |
-| `native_iceberg_compat` | Hybrid JVM/native scan |
+## Parquet Scan Limitations
-The configuration property `spark.comet.scan.impl` is used to select an
implementation. The default setting is
-`spark.comet.scan.impl=auto`, which attempts to use `native_datafusion` first,
and falls back to Spark if the scan
-cannot be converted (e.g., due to unsupported features). Most users should not
need to change this setting. However,
-it is possible to force Comet to use a particular implementation for all scan
operations by setting this
-configuration property to one of the following implementations. For example:
`--conf spark.comet.scan.impl=native_datafusion`.
-
-## Shared Limitations
-
-The following features are not supported by either scan implementation, and
Comet will fall back to Spark in these scenarios:
+The following features are not supported and cause Comet to fall back to Spark:
- Decimals encoded in binary format.
- `ShortType` columns, by default. When reading Parquet files written by
systems other than Spark that contain
@@ -46,17 +37,30 @@ The following features are not supported by either scan
implementation, and Come
columns are always safe because they can only come from signed `INT8`, where
truncation preserves the signed value.
- Default values that are nested types (e.g., maps, arrays, structs). Literal
default values are supported.
- Spark's Datasource V2 API. When `spark.sql.sources.useV1SourceList` does not
include `parquet`, Spark uses the
- V2 API for Parquet scans. The DataFusion-based implementations only support
the V1 API.
+ V2 API for Parquet scans. Comet's Parquet scan only supports the V1 API.
- Spark metadata columns (e.g., `_metadata.file_path`)
+- No support for row indexes
+- No support for `input_file_name()`, `input_file_block_start()`, or
`input_file_block_length()` SQL functions.
+ Comet's Parquet scan does not use Spark's `FileScanRDD`, so these functions
cannot populate their values.
+- No support for `ignoreMissingFiles` or `ignoreCorruptFiles` being set to
`true`
+- Duplicate field names in case-insensitive mode (e.g., a Parquet file with
both `B` and `b` columns)
+ are detected at read time and raise a `SparkRuntimeException` with error
class `_LEGACY_ERROR_TEMP_2093`,
+ matching Spark's behavior.
+- `spark.sql.parquet.enableVectorizedReader=false`. Disabling the vectorized
reader opts into
+ Spark's parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet's native reader
+ does not replicate. By default Comet falls back to Spark in this case. Set
+ `spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to
running the
+ Comet Parquet scan regardless. See
+ [#4352](https://github.com/apache/datafusion-comet/issues/4352).
-The following shared limitation may produce incorrect results without falling
back to Spark:
+The following limitation may produce incorrect results without falling back to
Spark:
- No support for datetime rebasing. When reading Parquet files containing
dates or timestamps written before
Spark 3.0 (which used a hybrid Julian/Gregorian calendar), dates/timestamps
will be read as if they were
written using the Proleptic Gregorian calendar. This may produce incorrect
results for dates before
October 15, 1582.
-The following shared limitation raises an error at scan time rather than
falling back to Spark:
+The following limitation raises an error at scan time rather than falling back
to Spark:
- Invalid UTF-8 bytes in `STRING` columns. Spark permits arbitrary byte
sequences in a `STRING`
column (for example from `CAST(X'C1' AS STRING)`), but Comet's native
execution path is built on
@@ -65,28 +69,7 @@ The following shared limitation raises an error at scan time
rather than falling
query, or cast the column to `BINARY` before persisting, if you need to
preserve non-UTF-8 bytes.
See [#4121](https://github.com/apache/datafusion-comet/issues/4121).
-## `native_datafusion` Limitations
-
-The `native_datafusion` scan has some additional limitations, mostly related
to Parquet metadata. All of these
-cause Comet to fall back to Spark (including when using `auto` mode). Note
that the `native_datafusion` scan
-requires `spark.comet.exec.enabled=true` because the scan node must be wrapped
by `CometExecRule`.
-
-- No support for row indexes
-- No support for reading Parquet field IDs
-- No support for `input_file_name()`, `input_file_block_start()`, or
`input_file_block_length()` SQL functions.
- The `native_datafusion` scan does not use Spark's `FileScanRDD`, so these
functions cannot populate their values.
-- No support for `ignoreMissingFiles` or `ignoreCorruptFiles` being set to
`true`
-- Duplicate field names in case-insensitive mode (e.g., a Parquet file with
both `B` and `b` columns)
- are detected at read time and raise a `SparkRuntimeException` with error
class `_LEGACY_ERROR_TEMP_2093`,
- matching Spark's behavior.
-- `spark.sql.parquet.enableVectorizedReader=false`. Disabling the vectorized
reader opts into
- Spark's parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet's native reader
- does not replicate. By default Comet falls back to Spark in this case. Set
- `spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to
running the
- `native_datafusion` scan regardless. See
- [#4352](https://github.com/apache/datafusion-comet/issues/4352).
-
-The following `native_datafusion` limitations may produce incorrect results on
Spark versions prior to 4.0
+The following limitation may produce incorrect results on Spark versions prior
to 4.0
without falling back to Spark:
- Reading `TimestampLTZ` as `TimestampNTZ`. On Spark 3.x, Spark raises an
error per
@@ -112,8 +95,8 @@ Schema mismatch happens in two real-world scenarios:
table types at read time.
Spark's vectorized Parquet reader fully validates these conversions in
`ParquetVectorUpdaterFactory.getUpdater`
-and throws `SchemaColumnConvertNotSupportedException` for unsupported pairs.
`native_datafusion` mirrors
-that validation in its schema adapter; the entries below are the remaining
gaps.
+and throws `SchemaColumnConvertNotSupportedException` for unsupported pairs.
Comet's Parquet scan
+mirrors that validation in its schema adapter; the entries below are the
remaining gaps.
Note that the exact set of accepted conversions has changed between Spark
versions
(for example, Spark 3.x's `schemaEvolution.enabled` flag gates `INT32 →
INT64`, `FLOAT → DOUBLE`,
@@ -136,14 +119,3 @@ SchemaColumnConvertNotSupportedException`) instead of the
one-level chain Spark'
`SparkException` instead. Walk the cause chain to recover the
`SchemaColumnConvertNotSupportedException`. Spark 4.0+ produces a
single-level chain, matching
vanilla Spark. See
[#4354](https://github.com/apache/datafusion-comet/issues/4354).
-
-## `native_iceberg_compat` Limitations
-
-The `native_iceberg_compat` scan has the following additional limitation that
may produce incorrect results
-without falling back to Spark:
-
-- Some Spark configuration values are hard-coded to their defaults rather than
respecting user-specified values.
- This may produce incorrect results when non-default values are set. The
affected configurations are
- `spark.sql.parquet.binaryAsString`, `spark.sql.parquet.int96AsTimestamp`,
`spark.sql.caseSensitive`,
- `spark.sql.parquet.inferTimestampNTZ.enabled`, and
`spark.sql.legacy.parquet.nanosAsLong`. See
- [issue #1816](https://github.com/apache/datafusion-comet/issues/1816) for
more details.
diff --git a/_sources/user-guide/latest/compatibility/spark-versions.md.txt
b/_sources/user-guide/latest/compatibility/spark-versions.md.txt
index 8c17fd6b13..4856cf5a1b 100644
--- a/_sources/user-guide/latest/compatibility/spark-versions.md.txt
+++ b/_sources/user-guide/latest/compatibility/spark-versions.md.txt
@@ -31,13 +31,13 @@ Spark 3.4.3 is supported with Java 11/17 and Scala
2.12/2.13.
### Known Limitations
- **Reading `TimestampLTZ` as `TimestampNTZ`**: Spark 3.4 raises an error for
this operation
- (SPARK-36182), but Comet's `native_datafusion` scan silently returns the raw
UTC value instead.
- See [Parquet Compatibility](scans.md#native_datafusion-limitations) for
details.
+ (SPARK-36182), but Comet's Parquet scan silently returns the raw UTC value
instead.
+ See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
- **Unsupported Parquet type conversions**: Spark 3.4 raises schema
incompatibility errors for
certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision
changes), but Comet's
- `native_datafusion` scan may not detect these and could return unexpected
values.
- See [Parquet Compatibility](scans.md#native_datafusion-limitations) for
details.
+ Comet's Parquet scan may not detect these and could return unexpected values.
+ See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
## Spark 3.5
@@ -46,13 +46,13 @@ Spark 3.5.8 is supported with Java 11/17 and Scala
2.12/2.13.
### Known Limitations
- **Reading `TimestampLTZ` as `TimestampNTZ`**: Spark 3.5 raises an error for
this operation
- (SPARK-36182), but Comet's `native_datafusion` scan silently returns the raw
UTC value instead.
- See [Parquet Compatibility](scans.md#native_datafusion-limitations) for
details.
+ (SPARK-36182), but Comet's Parquet scan silently returns the raw UTC value
instead.
+ See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
- **Unsupported Parquet type conversions**: Spark 3.5 raises schema
incompatibility errors for
certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision
changes), but Comet's
- `native_datafusion` scan may not detect these and could return unexpected
values.
- See [Parquet Compatibility](scans.md#native_datafusion-limitations) for
details.
+ Comet's Parquet scan may not detect these and could return unexpected values.
+ See [Parquet Compatibility](scans.md#parquet-scan-limitations) for details.
## Spark 4.0
diff --git a/_sources/user-guide/latest/datasources.md.txt
b/_sources/user-guide/latest/datasources.md.txt
index 9e1896141c..f38917819a 100644
--- a/_sources/user-guide/latest/datasources.md.txt
+++ b/_sources/user-guide/latest/datasources.md.txt
@@ -169,10 +169,11 @@ Or use `spark-shell` with HDFS support as described
[above](#building-comet-with
## S3
-The `native_datafusion` and `native_iceberg_compat` Parquet scan
implementations completely offload data loading
-to native code. They use the [`object_store`
crate](https://crates.io/crates/object_store) to read data from S3 and
-support configuring S3 access using standard [Hadoop S3A
configurations](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration)
by translating them to
-the `object_store` crate's format.
+Comet's Parquet scan completely offloads data loading to native code. It uses
the
+[`object_store` crate](https://crates.io/crates/object_store) to read data
from S3 and supports
+configuring S3 access using standard
+[Hadoop S3A
configurations](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration)
+by translating them to the `object_store` crate's format.
This implementation maintains compatibility with existing Hadoop S3A
configurations, so existing code will
continue to work as long as the configurations are supported and can be
translated without loss of functionality.
@@ -206,8 +207,7 @@ Multiple credential providers can be specified in a
comma-separated list using t
### Additional S3 Configuration Options
-Beyond credential providers, the `native_datafusion` and
`native_iceberg_compat` implementations support additional
-S3 configuration options:
+Beyond credential providers, Comet's Parquet scan supports additional S3
configuration options:
| Option | Description
|
| ------------------------------- |
--------------------------------------------------------------------------------------------------
|
@@ -220,8 +220,7 @@ All configuration options support bucket-specific overrides
using the pattern `f
### Examples
-The following examples demonstrate how to configure S3 access with the
`native_datafusion` and `native_iceberg_compat`
-Parquet scan implementations using different authentication methods.
+The following examples demonstrate how to configure S3 access using different
authentication methods.
**Example 1: Simple Credentials**
@@ -230,7 +229,6 @@ This example shows how to access a private S3 bucket using
an access key and sec
```shell
$SPARK_HOME/bin/spark-shell \
...
---conf spark.comet.scan.impl=native_datafusion \
--conf spark.hadoop.fs.s3a.access.key=my-access-key \
--conf spark.hadoop.fs.s3a.secret.key=my-secret-key
...
@@ -243,7 +241,6 @@ This example demonstrates using an assumed role credential
to access a private S
```shell
$SPARK_HOME/bin/spark-shell \
...
---conf spark.comet.scan.impl=native_datafusion \
--conf
spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
\
--conf
spark.hadoop.fs.s3a.assumed.role.arn=arn:aws:iam::123456789012:role/my-role \
--conf spark.hadoop.fs.s3a.assumed.role.session.name=my-session \
@@ -253,7 +250,7 @@ $SPARK_HOME/bin/spark-shell \
### Limitations
-The S3 support of `native_datafusion` and `native_iceberg_compat` has the
following limitations:
+Comet's S3 support has the following limitations:
1. **Partial Hadoop S3A configuration support**: Not all Hadoop S3A
configurations are currently supported. Only the configurations listed in the
tables above are translated and applied to the underlying `object_store` crate.
diff --git a/_sources/user-guide/latest/understanding-comet-plans.md.txt
b/_sources/user-guide/latest/understanding-comet-plans.md.txt
index 791f4bf766..7fb93c8f53 100644
--- a/_sources/user-guide/latest/understanding-comet-plans.md.txt
+++ b/_sources/user-guide/latest/understanding-comet-plans.md.txt
@@ -145,13 +145,12 @@ by role. Names match what is shown in the plan output.
### Scans
-| Node | Description
|
-| ------------------------ |
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| `CometScan` | V1 Parquet scan driven by Spark's file-source
path through Comet's Parquet reader. Decoding runs in native code; the
resulting Arrow batches cross JNI into the native plan. The active scan
implementation is shown in brackets, e.g. `CometScan [native_iceberg_compat]`. |
-| `CometBatchScan` | DataSource V2 scan, including Iceberg Parquet,
that produces Arrow batches consumed by Comet.
|
-| `CometNativeScan` | Fully native Parquet scan that runs entirely in
DataFusion (no JVM Parquet reader involvement).
|
-| `CometIcebergNativeScan` | Fully native Iceberg Parquet scan.
|
-| `CometCsvNativeScan` | Fully native CSV scan (experimental).
|
+| Node | Description
|
+| ------------------------ |
---------------------------------------------------------------------------------------------
|
+| `CometBatchScan` | DataSource V2 scan, including Iceberg Parquet,
that produces Arrow batches consumed by Comet. |
+| `CometNativeScan` | Fully native Parquet scan that runs entirely in
DataFusion. |
+| `CometIcebergNativeScan` | Fully native Iceberg Parquet scan.
|
+| `CometCsvNativeScan` | Fully native CSV scan (experimental).
|
### Native Execution Operators
diff --git a/contributor-guide/adding_a_new_spark_version.html
b/contributor-guide/adding_a_new_spark_version.html
index 52f39a4758..babee320e2 100644
--- a/contributor-guide/adding_a_new_spark_version.html
+++ b/contributor-guide/adding_a_new_spark_version.html
@@ -619,9 +619,7 @@ own test suites under the new profile.</p>
<h3>Add the New Profile to the Test Matrix<a class="headerlink"
href="#add-the-new-profile-to-the-test-matrix" title="Link to this
heading">#</a></h3>
<p>Promote the new Spark version from the compile-only job to the main test
jobs in <code class="docutils literal notranslate"><span
class="pre">.github/workflows/pr_build_linux.yml</span></code> (and <code
class="docutils literal notranslate"><span
class="pre">pr_build_macos.yml</span></code> if
-capacity allows). Use <code class="docutils literal notranslate"><span
class="pre">scan_impl:</span> <span class="pre">"auto"</span></code>
so both <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> and
-<code class="docutils literal notranslate"><span
class="pre">native_iceberg_compat</span></code> get exercised, matching how
earlier versions are
-configured.</p>
+capacity allows). Match how earlier versions are configured.</p>
</section>
<section id="run-the-suite-locally-first">
<h3>Run the Suite Locally First<a class="headerlink"
href="#run-the-suite-locally-first" title="Link to this heading">#</a></h3>
@@ -732,14 +730,12 @@ test or cluster, and do not disable whole suites.</p></li>
<section id="ci-for-the-spark-sql-tests">
<h3>CI for the Spark SQL Tests<a class="headerlink"
href="#ci-for-the-spark-sql-tests" title="Link to this heading">#</a></h3>
<p>Spark SQL tests do not run from the main PR build workflows. They have
-their own dedicated workflow files:</p>
+their own dedicated workflow file:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span
class="pre">.github/workflows/spark_sql_test.yml</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">.github/workflows/spark_sql_test_native_iceberg_compat.yml</span></code></p></li>
</ul>
-<p>Add the new version to the matrix in each of these files (<code
class="docutils literal notranslate"><span
class="pre">spark-short</span></code>,
-<code class="docutils literal notranslate"><span
class="pre">spark-full</span></code>, <code class="docutils literal
notranslate"><span class="pre">java</span></code>, <code class="docutils
literal notranslate"><span class="pre">scan-impl</span></code>). Use the
closest existing entry as a
-template.</p>
+<p>Add the new version to the matrix (<code class="docutils literal
notranslate"><span class="pre">spark-short</span></code>, <code class="docutils
literal notranslate"><span class="pre">spark-full</span></code>, <code
class="docutils literal notranslate"><span class="pre">java</span></code>).
+Use the closest existing entry as a template.</p>
<p>Before merging, run <code class="docutils literal notranslate"><span
class="pre">make</span> <span class="pre">format</span></code>, run clippy
(<code class="docutils literal notranslate"><span class="pre">cd</span> <span
class="pre">native</span> <span class="pre">&&</span> <span
class="pre">cargo</span> <span class="pre">clippy</span> <span
class="pre">--all-targets</span> <span class="pre">--workspace</span> <span
class="pre">--</span> <span class="pre">-D</span> <span
class="pre">warnings</span></code>), and
confirm every skip introduced in this PR has a linked GitHub issue.</p>
diff --git a/searchindex.js b/searchindex.js
index 63e34e9baf..bea4a099ad 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Format Your Code": [[34,
"format-your-code"]], "1. Install Comet": [[36, "install-comet"], [44,
"install-comet"]], "1. Native Operators (nativeExecs map)": [[23,
"native-operators-nativeexecs-map"]], "2. Build and Verify": [[34,
"build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36,
"clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[44,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23,
"sink-operators-sinks [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Format Your Code": [[34,
"format-your-code"]], "1. Install Comet": [[36, "install-comet"], [44,
"install-comet"]], "1. Native Operators (nativeExecs map)": [[23,
"native-operators-nativeexecs-map"]], "2. Build and Verify": [[34,
"build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[36,
"clone-iceberg-and-apply-diff"]], "2. Clone Spark and Apply Diff": [[44,
"clone-spark-and-apply-diff"]], "2. Sink Operators (sinks map)": [[23,
"sink-operators-sinks [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/index.html
b/user-guide/latest/compatibility/index.html
index eca82a2cc0..9496aeb44c 100644
--- a/user-guide/latest/compatibility/index.html
+++ b/user-guide/latest/compatibility/index.html
@@ -512,7 +512,7 @@ under the License.
<p>Comet aims to provide consistent results with the version of Apache Spark
that is being used.</p>
<p>This guide documents areas where Comet’s behavior is known to differ from
Spark. Topics are grouped by subsystem:</p>
<ul class="simple">
-<li><p><strong>Parquet</strong>: limitations when reading Parquet files (both
scan implementations, shared and per-implementation).</p></li>
+<li><p><strong>Parquet</strong>: limitations when reading Parquet
files.</p></li>
<li><p><strong>Floating-point comparison</strong>: NaN and signed-zero
handling in comparisons.</p></li>
<li><p><strong>Regular expressions</strong>: differences between the Rust
regexp crate and Java’s regex engine.</p></li>
<li><p><strong>Operators</strong>: operator-level compatibility notes,
including window functions and round-robin partitioning.</p></li>
diff --git a/user-guide/latest/compatibility/scans.html
b/user-guide/latest/compatibility/scans.html
index b90acbff60..528f28a236 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -512,31 +512,12 @@ under the License.
-->
<section id="parquet-compatibility">
<h1>Parquet Compatibility<a class="headerlink" href="#parquet-compatibility"
title="Link to this heading">#</a></h1>
-<p>Comet currently has two distinct implementations of the Parquet scan
operator.</p>
-<div class="pst-scrollable-table-container"><table class="table">
-<thead>
-<tr class="row-odd"><th class="head"><p>Scan Implementation</p></th>
-<th class="head"><p>Notes</p></th>
-</tr>
-</thead>
-<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code></p></td>
-<td><p>Fully native scan</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">native_iceberg_compat</span></code></p></td>
-<td><p>Hybrid JVM/native scan</p></td>
-</tr>
-</tbody>
-</table>
-</div>
-<p>The configuration property <code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.impl</span></code> is used to select an
implementation. The default setting is
-<code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.impl=auto</span></code>, which attempts to use
<code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> first, and falls back to Spark if
the scan
-cannot be converted (e.g., due to unsupported features). Most users should not
need to change this setting. However,
-it is possible to force Comet to use a particular implementation for all scan
operations by setting this
-configuration property to one of the following implementations. For example:
<code class="docutils literal notranslate"><span class="pre">--conf</span>
<span class="pre">spark.comet.scan.impl=native_datafusion</span></code>.</p>
-<section id="shared-limitations">
-<h2>Shared Limitations<a class="headerlink" href="#shared-limitations"
title="Link to this heading">#</a></h2>
-<p>The following features are not supported by either scan implementation, and
Comet will fall back to Spark in these scenarios:</p>
+<p>Comet’s Parquet scan offloads decoding to native code and produces Arrow
batches for the rest of
+the plan. Comet falls back to Spark when the scan cannot be converted (for
example, due to one of
+the unsupported features listed below).</p>
+<section id="parquet-scan-limitations">
+<h2>Parquet Scan Limitations<a class="headerlink"
href="#parquet-scan-limitations" title="Link to this heading">#</a></h2>
+<p>The following features are not supported and cause Comet to fall back to
Spark:</p>
<ul class="simple">
<li><p>Decimals encoded in binary format.</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">ShortType</span></code> columns, by default. When reading Parquet
files written by systems other than Spark that contain
@@ -548,17 +529,30 @@ This behavior can be disabled by setting <code
class="docutils literal notransla
columns are always safe because they can only come from signed <code
class="docutils literal notranslate"><span class="pre">INT8</span></code>,
where truncation preserves the signed value.</p></li>
<li><p>Default values that are nested types (e.g., maps, arrays, structs).
Literal default values are supported.</p></li>
<li><p>Spark’s Datasource V2 API. When <code class="docutils literal
notranslate"><span class="pre">spark.sql.sources.useV1SourceList</span></code>
does not include <code class="docutils literal notranslate"><span
class="pre">parquet</span></code>, Spark uses the
-V2 API for Parquet scans. The DataFusion-based implementations only support
the V1 API.</p></li>
+V2 API for Parquet scans. Comet’s Parquet scan only supports the V1
API.</p></li>
<li><p>Spark metadata columns (e.g., <code class="docutils literal
notranslate"><span class="pre">_metadata.file_path</span></code>)</p></li>
+<li><p>No support for row indexes</p></li>
+<li><p>No support for <code class="docutils literal notranslate"><span
class="pre">input_file_name()</span></code>, <code class="docutils literal
notranslate"><span class="pre">input_file_block_start()</span></code>, or <code
class="docutils literal notranslate"><span
class="pre">input_file_block_length()</span></code> SQL functions.
+Comet’s Parquet scan does not use Spark’s <code class="docutils literal
notranslate"><span class="pre">FileScanRDD</span></code>, so these functions
cannot populate their values.</p></li>
+<li><p>No support for <code class="docutils literal notranslate"><span
class="pre">ignoreMissingFiles</span></code> or <code class="docutils literal
notranslate"><span class="pre">ignoreCorruptFiles</span></code> being set to
<code class="docutils literal notranslate"><span
class="pre">true</span></code></p></li>
+<li><p>Duplicate field names in case-insensitive mode (e.g., a Parquet file
with both <code class="docutils literal notranslate"><span
class="pre">B</span></code> and <code class="docutils literal
notranslate"><span class="pre">b</span></code> columns)
+are detected at read time and raise a <code class="docutils literal
notranslate"><span class="pre">SparkRuntimeException</span></code> with error
class <code class="docutils literal notranslate"><span
class="pre">_LEGACY_ERROR_TEMP_2093</span></code>,
+matching Spark’s behavior.</p></li>
+<li><p><code class="docutils literal notranslate"><span
class="pre">spark.sql.parquet.enableVectorizedReader=false</span></code>.
Disabling the vectorized reader opts into
+Spark’s parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet’s native reader
+does not replicate. By default Comet falls back to Spark in this case. Set
+<code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.allowDisabledParquetVectorizedReader=true</span></code>
to opt in to running the
+Comet Parquet scan regardless. See
+<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4352";>#4352</a>.</p></li>
</ul>
-<p>The following shared limitation may produce incorrect results without
falling back to Spark:</p>
+<p>The following limitation may produce incorrect results without falling back
to Spark:</p>
<ul class="simple">
<li><p>No support for datetime rebasing. When reading Parquet files containing
dates or timestamps written before
Spark 3.0 (which used a hybrid Julian/Gregorian calendar), dates/timestamps
will be read as if they were
written using the Proleptic Gregorian calendar. This may produce incorrect
results for dates before
October 15, 1582.</p></li>
</ul>
-<p>The following shared limitation raises an error at scan time rather than
falling back to Spark:</p>
+<p>The following limitation raises an error at scan time rather than falling
back to Spark:</p>
<ul class="simple">
<li><p>Invalid UTF-8 bytes in <code class="docutils literal notranslate"><span
class="pre">STRING</span></code> columns. Spark permits arbitrary byte
sequences in a <code class="docutils literal notranslate"><span
class="pre">STRING</span></code>
column (for example from <code class="docutils literal notranslate"><span
class="pre">CAST(X'C1'</span> <span class="pre">AS</span> <span
class="pre">STRING)</span></code>), but Comet’s native execution path is built
on
@@ -567,29 +561,7 @@ non-UTF-8 bytes fails with <code class="docutils literal
notranslate"><span clas
query, or cast the column to <code class="docutils literal notranslate"><span
class="pre">BINARY</span></code> before persisting, if you need to preserve
non-UTF-8 bytes.
See <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4121";>#4121</a>.</p></li>
</ul>
-</section>
-<section id="native-datafusion-limitations">
-<h2><code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> Limitations<a class="headerlink"
href="#native-datafusion-limitations" title="Link to this heading">#</a></h2>
-<p>The <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan has some additional
limitations, mostly related to Parquet metadata. All of these
-cause Comet to fall back to Spark (including when using <code class="docutils
literal notranslate"><span class="pre">auto</span></code> mode). Note that the
<code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan
-requires <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.enabled=true</span></code> because the scan node
must be wrapped by <code class="docutils literal notranslate"><span
class="pre">CometExecRule</span></code>.</p>
-<ul class="simple">
-<li><p>No support for row indexes</p></li>
-<li><p>No support for reading Parquet field IDs</p></li>
-<li><p>No support for <code class="docutils literal notranslate"><span
class="pre">input_file_name()</span></code>, <code class="docutils literal
notranslate"><span class="pre">input_file_block_start()</span></code>, or <code
class="docutils literal notranslate"><span
class="pre">input_file_block_length()</span></code> SQL functions.
-The <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan does not use Spark’s <code
class="docutils literal notranslate"><span
class="pre">FileScanRDD</span></code>, so these functions cannot populate their
values.</p></li>
-<li><p>No support for <code class="docutils literal notranslate"><span
class="pre">ignoreMissingFiles</span></code> or <code class="docutils literal
notranslate"><span class="pre">ignoreCorruptFiles</span></code> being set to
<code class="docutils literal notranslate"><span
class="pre">true</span></code></p></li>
-<li><p>Duplicate field names in case-insensitive mode (e.g., a Parquet file
with both <code class="docutils literal notranslate"><span
class="pre">B</span></code> and <code class="docutils literal
notranslate"><span class="pre">b</span></code> columns)
-are detected at read time and raise a <code class="docutils literal
notranslate"><span class="pre">SparkRuntimeException</span></code> with error
class <code class="docutils literal notranslate"><span
class="pre">_LEGACY_ERROR_TEMP_2093</span></code>,
-matching Spark’s behavior.</p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">spark.sql.parquet.enableVectorizedReader=false</span></code>.
Disabling the vectorized reader opts into
-Spark’s parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet’s native reader
-does not replicate. By default Comet falls back to Spark in this case. Set
-<code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.allowDisabledParquetVectorizedReader=true</span></code>
to opt in to running the
-<code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan regardless. See
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4352";>#4352</a>.</p></li>
-</ul>
-<p>The following <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> limitations may produce incorrect
results on Spark versions prior to 4.0
+<p>The following limitation may produce incorrect results on Spark versions
prior to 4.0
without falling back to Spark:</p>
<ul class="simple">
<li><p>Reading <code class="docutils literal notranslate"><span
class="pre">TimestampLTZ</span></code> as <code class="docutils literal
notranslate"><span class="pre">TimestampNTZ</span></code>. On Spark 3.x, Spark
raises an error per
@@ -615,8 +587,8 @@ that has evolved past one or more underlying Parquet files.
Spark coerces the fi
table types at read time.</p></li>
</ol>
<p>Spark’s vectorized Parquet reader fully validates these conversions in
<code class="docutils literal notranslate"><span
class="pre">ParquetVectorUpdaterFactory.getUpdater</span></code>
-and throws <code class="docutils literal notranslate"><span
class="pre">SchemaColumnConvertNotSupportedException</span></code> for
unsupported pairs. <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> mirrors
-that validation in its schema adapter; the entries below are the remaining
gaps.</p>
+and throws <code class="docutils literal notranslate"><span
class="pre">SchemaColumnConvertNotSupportedException</span></code> for
unsupported pairs. Comet’s Parquet scan
+mirrors that validation in its schema adapter; the entries below are the
remaining gaps.</p>
<p>Note that the exact set of accepted conversions has changed between Spark
versions
(for example, Spark 3.x’s <code class="docutils literal notranslate"><span
class="pre">schemaEvolution.enabled</span></code> flag gates <code
class="docutils literal notranslate"><span class="pre">INT32</span> <span
class="pre">→</span> <span class="pre">INT64</span></code>, <code
class="docutils literal notranslate"><span class="pre">FLOAT</span> <span
class="pre">→</span> <span class="pre">DOUBLE</span></code>,
and <code class="docutils literal notranslate"><span class="pre">INT32</span>
<span class="pre">→</span> <span class="pre">DOUBLE</span></code> widening that
Spark 4.0+ accepts unconditionally; <code class="docutils literal
notranslate"><span class="pre">TimestampLTZ</span> <span class="pre">→</span>
<span class="pre">TimestampNTZ</span></code>
@@ -640,18 +612,6 @@ vanilla Spark. See <a class="reference external"
href="https://github.com/apache
</ul>
</section>
</section>
-<section id="native-iceberg-compat-limitations">
-<h2><code class="docutils literal notranslate"><span
class="pre">native_iceberg_compat</span></code> Limitations<a
class="headerlink" href="#native-iceberg-compat-limitations" title="Link to
this heading">#</a></h2>
-<p>The <code class="docutils literal notranslate"><span
class="pre">native_iceberg_compat</span></code> scan has the following
additional limitation that may produce incorrect results
-without falling back to Spark:</p>
-<ul class="simple">
-<li><p>Some Spark configuration values are hard-coded to their defaults rather
than respecting user-specified values.
-This may produce incorrect results when non-default values are set. The
affected configurations are
-<code class="docutils literal notranslate"><span
class="pre">spark.sql.parquet.binaryAsString</span></code>, <code
class="docutils literal notranslate"><span
class="pre">spark.sql.parquet.int96AsTimestamp</span></code>, <code
class="docutils literal notranslate"><span
class="pre">spark.sql.caseSensitive</span></code>,
-<code class="docutils literal notranslate"><span
class="pre">spark.sql.parquet.inferTimestampNTZ.enabled</span></code>, and
<code class="docutils literal notranslate"><span
class="pre">spark.sql.legacy.parquet.nanosAsLong</span></code>. See
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/1816";>issue #1816</a>
for more details.</p></li>
-</ul>
-</section>
</section>
diff --git a/user-guide/latest/compatibility/spark-versions.html
b/user-guide/latest/compatibility/spark-versions.html
index ae0d66d1a1..b5c04b1c57 100644
--- a/user-guide/latest/compatibility/spark-versions.html
+++ b/user-guide/latest/compatibility/spark-versions.html
@@ -522,12 +522,12 @@ compatibility guide.</p>
<h3>Known Limitations<a class="headerlink" href="#known-limitations"
title="Link to this heading">#</a></h3>
<ul class="simple">
<li><p><strong>Reading <code class="docutils literal notranslate"><span
class="pre">TimestampLTZ</span></code> as <code class="docutils literal
notranslate"><span class="pre">TimestampNTZ</span></code></strong>: Spark 3.4
raises an error for this operation
-(SPARK-36182), but Comet’s <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan silently returns the raw UTC
value instead.
-See <a class="reference internal"
href="scans.html#native-datafusion-limitations"><span class="std
std-ref">Parquet Compatibility</span></a> for details.</p></li>
+(SPARK-36182), but Comet’s Parquet scan silently returns the raw UTC value
instead.
+See <a class="reference internal"
href="scans.html#parquet-scan-limitations"><span class="std std-ref">Parquet
Compatibility</span></a> for details.</p></li>
<li><p><strong>Unsupported Parquet type conversions</strong>: Spark 3.4 raises
schema incompatibility errors for
certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision
changes), but Comet’s
-<code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan may not detect these and could
return unexpected values.
-See <a class="reference internal"
href="scans.html#native-datafusion-limitations"><span class="std
std-ref">Parquet Compatibility</span></a> for details.</p></li>
+Comet’s Parquet scan may not detect these and could return unexpected values.
+See <a class="reference internal"
href="scans.html#parquet-scan-limitations"><span class="std std-ref">Parquet
Compatibility</span></a> for details.</p></li>
</ul>
</section>
</section>
@@ -538,12 +538,12 @@ See <a class="reference internal"
href="scans.html#native-datafusion-limitations
<h3>Known Limitations<a class="headerlink" href="#id1" title="Link to this
heading">#</a></h3>
<ul class="simple">
<li><p><strong>Reading <code class="docutils literal notranslate"><span
class="pre">TimestampLTZ</span></code> as <code class="docutils literal
notranslate"><span class="pre">TimestampNTZ</span></code></strong>: Spark 3.5
raises an error for this operation
-(SPARK-36182), but Comet’s <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan silently returns the raw UTC
value instead.
-See <a class="reference internal"
href="scans.html#native-datafusion-limitations"><span class="std
std-ref">Parquet Compatibility</span></a> for details.</p></li>
+(SPARK-36182), but Comet’s Parquet scan silently returns the raw UTC value
instead.
+See <a class="reference internal"
href="scans.html#parquet-scan-limitations"><span class="std std-ref">Parquet
Compatibility</span></a> for details.</p></li>
<li><p><strong>Unsupported Parquet type conversions</strong>: Spark 3.5 raises
schema incompatibility errors for
certain type mismatches (e.g., reading INT32 as BIGINT, decimal precision
changes), but Comet’s
-<code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> scan may not detect these and could
return unexpected values.
-See <a class="reference internal"
href="scans.html#native-datafusion-limitations"><span class="std
std-ref">Parquet Compatibility</span></a> for details.</p></li>
+Comet’s Parquet scan may not detect these and could return unexpected values.
+See <a class="reference internal"
href="scans.html#parquet-scan-limitations"><span class="std std-ref">Parquet
Compatibility</span></a> for details.</p></li>
</ul>
</section>
</section>
diff --git a/user-guide/latest/datasources.html
b/user-guide/latest/datasources.html
index 0656c40808..d46ede0291 100644
--- a/user-guide/latest/datasources.html
+++ b/user-guide/latest/datasources.html
@@ -638,10 +638,11 @@ Input<span class="w"> </span><span
class="o">[</span><span class="m">3</span><sp
</section>
<section id="s3">
<h2>S3<a class="headerlink" href="#s3" title="Link to this heading">#</a></h2>
-<p>The <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> and <code class="docutils literal
notranslate"><span class="pre">native_iceberg_compat</span></code> Parquet scan
implementations completely offload data loading
-to native code. They use the <a class="reference external"
href="https://crates.io/crates/object_store";><code class="docutils literal
notranslate"><span class="pre">object_store</span></code> crate</a> to read
data from S3 and
-support configuring S3 access using standard <a class="reference external"
href="https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration";>Hadoop
S3A configurations</a> by translating them to
-the <code class="docutils literal notranslate"><span
class="pre">object_store</span></code> crate’s format.</p>
+<p>Comet’s Parquet scan completely offloads data loading to native code. It
uses the
+<a class="reference external"
href="https://crates.io/crates/object_store";><code class="docutils literal
notranslate"><span class="pre">object_store</span></code> crate</a> to read
data from S3 and supports
+configuring S3 access using standard
+<a class="reference external"
href="https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration";>Hadoop
S3A configurations</a>
+by translating them to the <code class="docutils literal notranslate"><span
class="pre">object_store</span></code> crate’s format.</p>
<p>This implementation maintains compatibility with existing Hadoop S3A
configurations, so existing code will
continue to work as long as the configurations are supported and can be
translated without loss of functionality.</p>
<section id="root-ca-certificates">
@@ -710,8 +711,7 @@ installed.</p>
</section>
<section id="additional-s3-configuration-options">
<h3>Additional S3 Configuration Options<a class="headerlink"
href="#additional-s3-configuration-options" title="Link to this
heading">#</a></h3>
-<p>Beyond credential providers, the <code class="docutils literal
notranslate"><span class="pre">native_datafusion</span></code> and <code
class="docutils literal notranslate"><span
class="pre">native_iceberg_compat</span></code> implementations support
additional
-S3 configuration options:</p>
+<p>Beyond credential providers, Comet’s Parquet scan supports additional S3
configuration options:</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>Option</p></th>
@@ -738,13 +738,11 @@ S3 configuration options:</p>
</section>
<section id="examples">
<h3>Examples<a class="headerlink" href="#examples" title="Link to this
heading">#</a></h3>
-<p>The following examples demonstrate how to configure S3 access with the
<code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> and <code class="docutils literal
notranslate"><span class="pre">native_iceberg_compat</span></code>
-Parquet scan implementations using different authentication methods.</p>
+<p>The following examples demonstrate how to configure S3 access using
different authentication methods.</p>
<p><strong>Example 1: Simple Credentials</strong></p>
<p>This example shows how to access a private S3 bucket using an access key
and secret key. The <code class="docutils literal notranslate"><span
class="pre">fs.s3a.aws.credentials.provider</span></code> configuration can be
omitted since <code class="docutils literal notranslate"><span
class="pre">org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</span></code>
is included in Hadoop S3A’s default credential provider chain.</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span
class="nv">$SPARK_HOME</span>/bin/spark-shell<span class="w"> </span><span
class="se">\</span>
...
---conf<span class="w"> </span>spark.comet.scan.impl<span
class="o">=</span>native_datafusion<span class="w"> </span><span
class="se">\</span>
--conf<span class="w"> </span>spark.hadoop.fs.s3a.access.key<span
class="o">=</span>my-access-key<span class="w"> </span><span class="se">\</span>
--conf<span class="w"> </span>spark.hadoop.fs.s3a.secret.key<span
class="o">=</span>my-secret-key
...
@@ -754,7 +752,6 @@ Parquet scan implementations using different authentication
methods.</p>
<p>This example demonstrates using an assumed role credential to access a
private S3 bucket, where the base credential for assuming the role is provided
by a web identity token credentials provider.</p>
<div class="highlight-shell notranslate"><div
class="highlight"><pre><span></span><span
class="nv">$SPARK_HOME</span>/bin/spark-shell<span class="w"> </span><span
class="se">\</span>
...
---conf<span class="w"> </span>spark.comet.scan.impl<span
class="o">=</span>native_datafusion<span class="w"> </span><span
class="se">\</span>
--conf<span class="w">
</span>spark.hadoop.fs.s3a.aws.credentials.provider<span
class="o">=</span>org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider<span
class="w"> </span><span class="se">\</span>
--conf<span class="w"> </span>spark.hadoop.fs.s3a.assumed.role.arn<span
class="o">=</span>arn:aws:iam::123456789012:role/my-role<span class="w">
</span><span class="se">\</span>
--conf<span class="w">
</span>spark.hadoop.fs.s3a.assumed.role.session.name<span
class="o">=</span>my-session<span class="w"> </span><span class="se">\</span>
@@ -765,7 +762,7 @@ Parquet scan implementations using different authentication
methods.</p>
</section>
<section id="limitations">
<h3>Limitations<a class="headerlink" href="#limitations" title="Link to this
heading">#</a></h3>
-<p>The S3 support of <code class="docutils literal notranslate"><span
class="pre">native_datafusion</span></code> and <code class="docutils literal
notranslate"><span class="pre">native_iceberg_compat</span></code> has the
following limitations:</p>
+<p>Comet’s S3 support has the following limitations:</p>
<ol class="arabic simple">
<li><p><strong>Partial Hadoop S3A configuration support</strong>: Not all
Hadoop S3A configurations are currently supported. Only the configurations
listed in the tables above are translated and applied to the underlying <code
class="docutils literal notranslate"><span
class="pre">object_store</span></code> crate.</p></li>
<li><p><strong>Custom credential providers</strong>: Custom implementations of
AWS credential providers are not supported. The implementation only supports
the standard credential providers listed in the table above. We are planning to
add support for custom credential providers through a JNI-based adapter that
will allow calling Java credential providers from native code. See <a
class="reference external"
href="https://github.com/apache/datafusion-comet/issues/1829";>issue #1829</a>
for [...]
diff --git a/user-guide/latest/understanding-comet-plans.html
b/user-guide/latest/understanding-comet-plans.html
index 8dbbcfe2cd..5bd26b3458 100644
--- a/user-guide/latest/understanding-comet-plans.html
+++ b/user-guide/latest/understanding-comet-plans.html
@@ -649,19 +649,16 @@ by role. Names match what is shown in the plan output.</p>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">CometScan</span></code></p></td>
-<td><p>V1 Parquet scan driven by Spark’s file-source path through Comet’s
Parquet reader. Decoding runs in native code; the resulting Arrow batches cross
JNI into the native plan. The active scan implementation is shown in brackets,
e.g. <code class="docutils literal notranslate"><span
class="pre">CometScan</span> <span
class="pre">[native_iceberg_compat]</span></code>.</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">CometBatchScan</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">CometBatchScan</span></code></p></td>
<td><p>DataSource V2 scan, including Iceberg Parquet, that produces Arrow
batches consumed by Comet.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">CometNativeScan</span></code></p></td>
-<td><p>Fully native Parquet scan that runs entirely in DataFusion (no JVM
Parquet reader involvement).</p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">CometNativeScan</span></code></p></td>
+<td><p>Fully native Parquet scan that runs entirely in DataFusion.</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">CometIcebergNativeScan</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">CometIcebergNativeScan</span></code></p></td>
<td><p>Fully native Iceberg Parquet scan.</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">CometCsvNativeScan</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">CometCsvNativeScan</span></code></p></td>
<td><p>Fully native CSV scan (experimental).</p></td>
</tr>
</tbody>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
