This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8443d39283 Publish built docs triggered by
d953fc0168cc6ab51e95920ee32fd531ce4b4a89
8443d39283 is described below
commit 8443d3928373e8139d95f4981cd60f91f6595444
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Wed Jul 8 02:00:10 2026 +0000
Publish built docs triggered by d953fc0168cc6ab51e95920ee32fd531ce4b4a89
---
.../latest/compatibility/expressions/spark-3.4/cast.md.txt | 4 ----
.../compatibility/expressions/spark-3.4/datetime.md.txt | 4 ----
.../latest/compatibility/expressions/spark-3.5/cast.md.txt | 4 ----
.../compatibility/expressions/spark-3.5/datetime.md.txt | 4 ----
.../latest/compatibility/expressions/spark-4.0/cast.md.txt | 4 ----
.../compatibility/expressions/spark-4.0/datetime.md.txt | 4 ----
.../latest/compatibility/expressions/spark-4.1/cast.md.txt | 4 ----
.../compatibility/expressions/spark-4.1/datetime.md.txt | 4 ----
_sources/user-guide/latest/compatibility/operators.md.txt | 2 +-
_sources/user-guide/latest/compatibility/scans.md.txt | 13 +------------
.../user-guide/latest/compatibility/spark-versions.md.txt | 3 +--
_sources/user-guide/latest/configs.md.txt | 14 +++++++-------
searchindex.js | 2 +-
.../latest/compatibility/expressions/spark-3.4/cast.html | 2 --
.../compatibility/expressions/spark-3.4/datetime.html | 6 +-----
.../latest/compatibility/expressions/spark-3.5/cast.html | 2 --
.../compatibility/expressions/spark-3.5/datetime.html | 6 +-----
.../latest/compatibility/expressions/spark-4.0/cast.html | 2 --
.../compatibility/expressions/spark-4.0/datetime.html | 6 +-----
.../latest/compatibility/expressions/spark-4.1/cast.html | 2 --
.../compatibility/expressions/spark-4.1/datetime.html | 6 +-----
user-guide/latest/compatibility/operators.html | 2 +-
user-guide/latest/compatibility/scans.html | 13 +------------
user-guide/latest/compatibility/spark-versions.html | 3 +--
user-guide/latest/configs.html | 14 +++++++-------
25 files changed, 25 insertions(+), 105 deletions(-)
diff --git
a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/cast.md.txt
b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/cast.md.txt
index 318fce7f56..25cf6ccd83 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/cast.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/cast.md.txt
@@ -34,8 +34,6 @@ Cast operations in Comet fall into three levels of support:
Cast will fall back to Spark in some cases when ANSI mode is enabled. This can
be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See
the [Comet Supported Expressions Guide](../../../expressions.md) for more
information on this configuration setting.
-There is an [epic](https://github.com/apache/datafusion-comet/issues/313)
where we are tracking the work to fully implement ANSI support.
-
## String to Decimal
Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's
behavior,
@@ -232,5 +230,3 @@ as `"1.23E+4"`).
- **string -> date**: Only supports years between 262143 BC and 262142 AD
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
-
-See the [tracking
issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
diff --git
a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md.txt
b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md.txt
index 12cb9d0acb..7b0ea3efc1 100644
---
a/_sources/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md.txt
+++
b/_sources/user-guide/latest/compatibility/expressions/spark-3.4/datetime.md.txt
@@ -19,16 +19,12 @@ under the License.
# Date/Time Expressions
-- **Hour, Minute, Second**: Incorrectly apply timezone conversion to
TimestampNTZ inputs. TimestampNTZ stores local
- time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
- [#3180](https://github.com/apache/datafusion-comet/issues/3180)
- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is
marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz's DST horizon
(approximately year 2100;
see "Date and Time Functions" below) and can be enabled by setting
`spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ
inputs are handled
correctly regardless of session timezone (timezone-independent truncation).
- [#2649](https://github.com/apache/datafusion-comet/issues/2649)
## Date and Time Functions
diff --git
a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/cast.md.txt
b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/cast.md.txt
index 318fce7f56..25cf6ccd83 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/cast.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/cast.md.txt
@@ -34,8 +34,6 @@ Cast operations in Comet fall into three levels of support:
Cast will fall back to Spark in some cases when ANSI mode is enabled. This can
be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See
the [Comet Supported Expressions Guide](../../../expressions.md) for more
information on this configuration setting.
-There is an [epic](https://github.com/apache/datafusion-comet/issues/313)
where we are tracking the work to fully implement ANSI support.
-
## String to Decimal
Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's
behavior,
@@ -232,5 +230,3 @@ as `"1.23E+4"`).
- **string -> date**: Only supports years between 262143 BC and 262142 AD
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
-
-See the [tracking
issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
diff --git
a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md.txt
b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md.txt
index 12cb9d0acb..7b0ea3efc1 100644
---
a/_sources/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md.txt
+++
b/_sources/user-guide/latest/compatibility/expressions/spark-3.5/datetime.md.txt
@@ -19,16 +19,12 @@ under the License.
# Date/Time Expressions
-- **Hour, Minute, Second**: Incorrectly apply timezone conversion to
TimestampNTZ inputs. TimestampNTZ stores local
- time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
- [#3180](https://github.com/apache/datafusion-comet/issues/3180)
- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is
marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz's DST horizon
(approximately year 2100;
see "Date and Time Functions" below) and can be enabled by setting
`spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ
inputs are handled
correctly regardless of session timezone (timezone-independent truncation).
- [#2649](https://github.com/apache/datafusion-comet/issues/2649)
## Date and Time Functions
diff --git
a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/cast.md.txt
b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/cast.md.txt
index 318fce7f56..25cf6ccd83 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/cast.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/cast.md.txt
@@ -34,8 +34,6 @@ Cast operations in Comet fall into three levels of support:
Cast will fall back to Spark in some cases when ANSI mode is enabled. This can
be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See
the [Comet Supported Expressions Guide](../../../expressions.md) for more
information on this configuration setting.
-There is an [epic](https://github.com/apache/datafusion-comet/issues/313)
where we are tracking the work to fully implement ANSI support.
-
## String to Decimal
Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's
behavior,
@@ -232,5 +230,3 @@ as `"1.23E+4"`).
- **string -> date**: Only supports years between 262143 BC and 262142 AD
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
-
-See the [tracking
issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
diff --git
a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md.txt
b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md.txt
index becbb44035..055b3021c1 100644
---
a/_sources/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md.txt
+++
b/_sources/user-guide/latest/compatibility/expressions/spark-4.0/datetime.md.txt
@@ -19,16 +19,12 @@ under the License.
# Date/Time Expressions
-- **Hour, Minute, Second**: Incorrectly apply timezone conversion to
TimestampNTZ inputs. TimestampNTZ stores local
- time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
- [#3180](https://github.com/apache/datafusion-comet/issues/3180)
- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is
marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz's DST horizon
(approximately year 2100;
see "Date and Time Functions" below) and can be enabled by setting
`spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ
inputs are handled
correctly regardless of session timezone (timezone-independent truncation).
- [#2649](https://github.com/apache/datafusion-comet/issues/2649)
## Date and Time Functions
diff --git
a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/cast.md.txt
b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/cast.md.txt
index 318fce7f56..25cf6ccd83 100644
--- a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/cast.md.txt
+++ b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/cast.md.txt
@@ -34,8 +34,6 @@ Cast operations in Comet fall into three levels of support:
Cast will fall back to Spark in some cases when ANSI mode is enabled. This can
be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See
the [Comet Supported Expressions Guide](../../../expressions.md) for more
information on this configuration setting.
-There is an [epic](https://github.com/apache/datafusion-comet/issues/313)
where we are tracking the work to fully implement ANSI support.
-
## String to Decimal
Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's
behavior,
@@ -232,5 +230,3 @@ as `"1.23E+4"`).
- **string -> date**: Only supports years between 262143 BC and 262142 AD
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
-
-See the [tracking
issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
diff --git
a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md.txt
b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md.txt
index becbb44035..055b3021c1 100644
---
a/_sources/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md.txt
+++
b/_sources/user-guide/latest/compatibility/expressions/spark-4.1/datetime.md.txt
@@ -19,16 +19,12 @@ under the License.
# Date/Time Expressions
-- **Hour, Minute, Second**: Incorrectly apply timezone conversion to
TimestampNTZ inputs. TimestampNTZ stores local
- time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
- [#3180](https://github.com/apache/datafusion-comet/issues/3180)
- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is
marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz's DST horizon
(approximately year 2100;
see "Date and Time Functions" below) and can be enabled by setting
`spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ
inputs are handled
correctly regardless of session timezone (timezone-independent truncation).
- [#2649](https://github.com/apache/datafusion-comet/issues/2649)
## Date and Time Functions
diff --git a/_sources/user-guide/latest/compatibility/operators.md.txt
b/_sources/user-guide/latest/compatibility/operators.md.txt
index 28c0954047..eec21d283b 100644
--- a/_sources/user-guide/latest/compatibility/operators.md.txt
+++ b/_sources/user-guide/latest/compatibility/operators.md.txt
@@ -44,7 +44,7 @@ incorrect result. When any single window expression in a
`WindowExec` falls back
and `sum` / `avg` on year-month or day-time interval input types. Windowed
aggregates inherit the same input-type
support as the batch aggregates, so these fall back in both contexts.
- `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame,
because the sliding path would wrap on
- overflow instead of returning Spark's `NULL`
([#4729](https://github.com/apache/datafusion-comet/issues/4729)).
+ overflow instead of returning Spark's `NULL`.
- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE`
or `DECIMAL`
([#4834](https://github.com/apache/datafusion-comet/issues/4834)).
- `first_value` / `last_value` on a `RANGE` frame with a literal offset
diff --git a/_sources/user-guide/latest/compatibility/scans.md.txt
b/_sources/user-guide/latest/compatibility/scans.md.txt
index 4f2f14a944..184ae9f06b 100644
--- a/_sources/user-guide/latest/compatibility/scans.md.txt
+++ b/_sources/user-guide/latest/compatibility/scans.md.txt
@@ -43,15 +43,11 @@ The following features are not supported and cause Comet to
fall back to Spark:
- No support for `input_file_name()`, `input_file_block_start()`, or
`input_file_block_length()` SQL functions.
Comet's Parquet scan does not use Spark's `FileScanRDD`, so these functions
cannot populate their values.
- No support for `ignoreMissingFiles` or `ignoreCorruptFiles` being set to
`true`
-- Duplicate field names in case-insensitive mode (e.g., a Parquet file with
both `B` and `b` columns)
- are detected at read time and raise a `SparkRuntimeException` with error
class `_LEGACY_ERROR_TEMP_2093`,
- matching Spark's behavior.
- `spark.sql.parquet.enableVectorizedReader=false`. Disabling the vectorized
reader opts into
Spark's parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet's native reader
does not replicate. By default Comet falls back to Spark in this case. Set
`spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to
running the
- Comet Parquet scan regardless. See
- [#4352](https://github.com/apache/datafusion-comet/issues/4352).
+ Comet Parquet scan regardless.
The following limitation may produce incorrect results without falling back to
Spark:
@@ -68,13 +64,6 @@ The following limitations raise an error at scan time rather
than falling back t
non-UTF-8 bytes fails with `Parquet error: encountered non UTF-8 data`.
Disable Comet for the
query, or cast the column to `BINARY` before persisting, if you need to
preserve non-UTF-8 bytes.
See [#4121](https://github.com/apache/datafusion-comet/issues/4121).
-- Reading `TimestampLTZ` as `TimestampNTZ` on Spark 3.x. Spark raises an error
per
- [SPARK-36182](https://issues.apache.org/jira/browse/SPARK-36182) because LTZ
encodes UTC-adjusted
- instants that cannot be safely reinterpreted as timezone-free values, and
Comet matches this by
- rejecting the read. This applies to all LTZ physical encodings (INT96,
TIMESTAMP_MICROS,
- TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted
- ([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet
matches Spark's
- behavior. See
[#4219](https://github.com/apache/datafusion-comet/issues/4219).
### Schema Mismatch Handling
diff --git a/_sources/user-guide/latest/compatibility/spark-versions.md.txt
b/_sources/user-guide/latest/compatibility/spark-versions.md.txt
index a6ad5bcf16..5715fef765 100644
--- a/_sources/user-guide/latest/compatibility/spark-versions.md.txt
+++ b/_sources/user-guide/latest/compatibility/spark-versions.md.txt
@@ -60,8 +60,7 @@ Spark 4.0.2 is supported with Java 17 and Scala 2.13.
### Known Limitations
-- **Collation support**
([#1947](https://github.com/apache/datafusion-comet/issues/1947),
- [#4051](https://github.com/apache/datafusion-comet/issues/4051)): Spark 4.0
introduced collation
+- **Collation support**: Spark 4.0 introduced collation
support. Non-default collated strings are not yet supported by Comet and
will fall back to Spark.
## Spark 4.1
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index b3f4a8740e..d24e70c341 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -27,10 +27,10 @@ Comet provides the following configuration settings.
<!-- prettier-ignore-start -->
| Config | Description | Default Value |
|--------|-------------|---------------|
-| `spark.comet.scan.allowDisabledParquetVectorizedReader` | Whether to allow
Comet's native scan to replace the Parquet scan when Spark's
spark.sql.parquet.enableVectorizedReader is set to false. By default (false),
Comet falls back to Spark in that case, because Comet's native readers mirror
Spark's vectorized reader semantics rather than Spark's parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information [...]
+| `spark.comet.scan.allowDisabledParquetVectorizedReader` | Whether to allow
Comet's native scan to replace the Parquet scan when Spark's
spark.sql.parquet.enableVectorizedReader is set to false. By default (false),
Comet falls back to Spark in that case, because Comet's native readers mirror
Spark's vectorized reader semantics rather than Spark's parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information [...]
| `spark.comet.scan.icebergNative.dataFileConcurrencyLimit` | The number of
Iceberg data files to read concurrently within a single task. Higher values
improve throughput for tables with many small files by overlapping I/O latency,
but increase memory usage. Values between 2 and 8 are suggested. | 1 |
| `spark.comet.scan.icebergNative.enabled` | Whether to enable native Iceberg
table scan using iceberg-rust. When enabled, Iceberg tables are read directly
through native execution, bypassing Spark's DataSource V2 API for better
performance. | true |
-| `spark.comet.scan.unsignedSmallIntSafetyCheck` | Parquet files may contain
unsigned 8-bit integers (UINT_8) which Spark maps to ShortType. When this
config is true (default), Comet falls back to Spark for ShortType columns
because we cannot distinguish signed INT16 (safe) from unsigned UINT_8 (may
produce different results). Set to false to allow native execution of ShortType
columns if you know your data does not contain unsigned UINT_8 columns from
improperly encoded Parquet files. F [...]
+| `spark.comet.scan.unsignedSmallIntSafetyCheck` | Parquet files may contain
unsigned 8-bit integers (UINT_8) which Spark maps to ShortType. When this
config is true (default), Comet falls back to Spark for ShortType columns
because we cannot distinguish signed INT16 (safe) from unsigned UINT_8 (may
produce different results). Set to false to allow native execution of ShortType
columns if you know your data does not contain unsigned UINT_8 columns from
improperly encoded Parquet files. F [...]
| `spark.hadoop.fs.comet.libhdfs.schemes` | Defines filesystem schemes (e.g.,
hdfs, webhdfs) that the native side accesses via libhdfs, separated by commas.
Valid only when built with hdfs feature enabled. | |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
@@ -65,9 +65,9 @@ Comet provides the following configuration settings.
| `spark.comet.exceptionOnDatetimeRebase` | Whether to throw exception when
seeing dates/timestamps from the legacy hybrid (Julian + Gregorian) calendar.
Since Spark 3, dates/timestamps were written according to the Proleptic
Gregorian calendar. When this is true, Comet will throw exceptions when seeing
these dates/timestamps that were written by Spark version before 3.0. If this
is false, these dates/timestamps will be read as if they were written to the
Proleptic Gregorian calendar and [...]
| `spark.comet.exec.columnarToRow.native.enabled` | Whether to enable native
columnar to row conversion. When enabled, Comet will use native Rust code to
convert Arrow columnar data to Spark UnsafeRow format instead of the JVM
implementation. This can improve performance for queries that need to convert
between columnar and row formats. | true |
| `spark.comet.exec.enabled` | Whether to enable Comet native vectorized
execution for Spark. This controls whether Spark should convert operators into
their Comet counterparts and execute them in native space. Note: each operator
is associated with a separate config in the format of
`spark.comet.exec.<operator_name>.enabled` at the moment, and both the config
and this need to be turned on, in order for the operator to be executed in
native. | true |
-| `spark.comet.exec.replaceSortMergeJoin` | Experimental feature to force
Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance.
This feature is not stable yet. For more information, refer to the [Comet
Tuning Guide](https://datafusion.apache.org/comet/user-guide/tuning.html). |
false |
+| `spark.comet.exec.replaceSortMergeJoin` | Experimental feature to force
Spark to replace SortMergeJoin with ShuffledHashJoin for improved performance.
This feature is not stable yet. For more information, refer to the [Comet
Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
false |
| `spark.comet.exec.scalaUDF.codegen.enabled` | Whether to route Spark
`ScalaUDF` expressions through Comet's Arrow-direct codegen dispatcher. When
enabled, a supported ScalaUDF is compiled into a per-batch kernel that reads
and writes Arrow vectors directly from native execution. When disabled, plans
containing a ScalaUDF fall back to Spark for the enclosing operator. The same
dispatcher backs the regex family (`rlike`, `regexp_replace`, `split`,
`regexp_extract`, `regexp_extract_all`, [...]
-| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark
for floating-point operations that may differ from Spark, such as when
comparing or sorting -0.0 and 0.0. For more information, refer to the [Comet
Compatibility
Guide](https://datafusion.apache.org/comet/user-guide/compatibility.html). |
false |
+| `spark.comet.exec.strictFloatingPoint` | When enabled, fall back to Spark
for floating-point operations that may differ from Spark, such as when
comparing or sorting -0.0 and 0.0. For more information, refer to the [Comet
Compatibility
Guide](https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html).
| false |
| `spark.comet.exec.transitionRevert.enabled` | When enabled, Comet reverts a
query stage to Spark row-based execution if the number of columnar-to-row (C2R)
transitions in the stage exceeds the configured threshold. This avoids the
overhead of repeated format conversions in stages where many operators fall
back to row-based execution. | false |
| `spark.comet.exec.transitionRevert.maxTransitions` | The maximum number of
columnar-to-row (C2R) transitions allowed in a single query stage before Comet
reverts the entire stage to Spark row-based execution. When columnar shuffle is
enabled, each such C2R typically implies a corresponding row-to-columnar
conversion to feed back into the columnar shuffle, so each counted C2R is a
useful proxy for the conversion overhead in the stage. Set to 0 to revert any
stage with transitions. Only [...]
| `spark.comet.maxTempDirectorySize` | The maximum amount of data (in bytes)
stored inside the temporary directories. | 107374182400b |
@@ -129,8 +129,8 @@ These settings can be used to determine which parts of the
plan are accelerated
| Config | Description | Default Value |
|--------|-------------|---------------|
| `spark.comet.batchSize` | The columnar batch size, i.e., the maximum number
of rows that a batch can contain. | 8192 |
-| `spark.comet.exec.memoryPool` | The type of memory pool to be used for Comet
native execution when running Spark in off-heap mode. Available pool types are
`greedy_unified` and `fair_unified`. For more information, refer to the [Comet
Tuning Guide](https://datafusion.apache.org/comet/user-guide/tuning.html). |
fair_unified |
-| `spark.comet.exec.memoryPool.fraction` | Fraction of off-heap memory pool
that is available to Comet. Only applies to off-heap mode. For more
information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/tuning.html). | 1.0 |
+| `spark.comet.exec.memoryPool` | The type of memory pool to be used for Comet
native execution when running Spark in off-heap mode. Available pool types are
`greedy_unified` and `fair_unified`. For more information, refer to the [Comet
Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
fair_unified |
+| `spark.comet.exec.memoryPool.fraction` | Fraction of off-heap memory pool
that is available to Comet. Only applies to off-heap mode. For more
information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
1.0 |
| `spark.comet.tracing.enabled` | Enable fine-grained tracing of events and
memory usage. For more information, refer to the [Comet Tracing
Guide](https://datafusion.apache.org/comet/contributor-guide/tracing.html). |
false |
<!-- prettier-ignore-end -->
<!--END:CONFIG_TABLE-->
@@ -141,7 +141,7 @@ These settings can be used to determine which parts of the
plan are accelerated
<!-- prettier-ignore-start -->
| Config | Description | Default Value |
|--------|-------------|---------------|
-| `spark.comet.columnar.shuffle.memory.factor` | Fraction of Comet memory to
be allocated per executor process for columnar shuffle when running in on-heap
mode. For more information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/tuning.html). | 1.0 |
+| `spark.comet.columnar.shuffle.memory.factor` | Fraction of Comet memory to
be allocated per executor process for columnar shuffle when running in on-heap
mode. For more information, refer to the [Comet Tuning
Guide](https://datafusion.apache.org/comet/user-guide/latest/tuning.html). |
1.0 |
| `spark.comet.debug.memory` | When enabled, log all native memory pool
interactions. For more information, refer to the Comet Debugging Guide
(https://datafusion.apache.org/comet/contributor-guide/debugging.html). | false
|
| `spark.comet.exec.onHeap.enabled` | Whether to allow Comet to run in on-heap
mode. Required for running Spark SQL tests. It can be overridden by the
environment variable `ENABLE_COMET_ONHEAP`. | false |
| `spark.comet.exec.onHeap.memoryPool` | The type of memory pool to be used
for Comet native execution when running Spark in on-heap mode. Available pool
types are `greedy`, `fair_spill`, `greedy_task_shared`,
`fair_spill_task_shared`, `greedy_global`, `fair_spill_global`, and
`unbounded`. | greedy_task_shared |
diff --git a/searchindex.js b/searchindex.js
index 71588ead18..126eb1fd97 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[51, "id1"]], "%": [[49, "id1"]], "&":
[[39, "id1"]], "*": [[49, "id2"]], "+": [[49, "id3"]], "-": [[49, "id4"]], "/":
[[49, "id5"]], "1. Format Your Code": [[36, "format-your-code"]], "1. Install
Comet": [[57, "install-comet"], [66, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[25, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[36, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[57,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[51, "id1"]], "%": [[49, "id1"]], "&":
[[39, "id1"]], "*": [[49, "id2"]], "+": [[49, "id3"]], "-": [[49, "id4"]], "/":
[[49, "id5"]], "1. Format Your Code": [[36, "format-your-code"]], "1. Install
Comet": [[57, "install-comet"], [66, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[25, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[36, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[57,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/compatibility/expressions/spark-3.4/cast.html
b/user-guide/latest/compatibility/expressions/spark-3.4/cast.html
index d0ab4a40ec..58a94ea798 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/cast.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/cast.html
@@ -760,7 +760,6 @@ Spark.</p></li>
<section id="ansi-mode-fallback">
<h2>ANSI Mode Fallback<a class="headerlink" href="#ansi-mode-fallback"
title="Link to this heading">#</a></h2>
<p>Cast will fall back to Spark in some cases when ANSI mode is enabled. This
can be enabled by setting <code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>.
See the <a class="reference internal" href="../../../expressions.html"><span
class="std std-doc">Comet Supported Expressions Guide</span></a> for more
information on this configuration setting.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/313";>epic</a> where we
are tracking the work to fully implement ANSI support.</p>
</section>
<section id="string-to-decimal">
<h2>String to Decimal<a class="headerlink" href="#string-to-decimal"
title="Link to this heading">#</a></h2>
@@ -1592,7 +1591,6 @@ as <code class="docutils literal notranslate"><span
class="pre">"1.23E+4&qu
</ul>
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
-<p>See the <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/286";>tracking issue</a>
for more details.</p>
</section>
</section>
diff --git
a/user-guide/latest/compatibility/expressions/spark-3.4/datetime.html
b/user-guide/latest/compatibility/expressions/spark-3.4/datetime.html
index 41df4ce4e5..171a427fd5 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.4/datetime.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.4/datetime.html
@@ -747,16 +747,12 @@ under the License.
<section id="date-time-expressions">
<h1>Date/Time Expressions<a class="headerlink" href="#date-time-expressions"
title="Link to this heading">#</a></h1>
<ul class="simple">
-<li><p><strong>Hour, Minute, Second</strong>: Incorrectly apply timezone
conversion to TimestampNTZ inputs. TimestampNTZ stores local
-time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/3180";>#3180</a></p></li>
<li><p><strong>TruncTimestamp (date_trunc)</strong>: In non-UTC sessions the
native path is marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz’s DST horizon
(approximately year 2100;
see “Date and Time Functions” below) and can be enabled by setting
<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.TruncTimestamp.allowIncompatible=true</span></code>.
TimestampNTZ inputs are handled
-correctly regardless of session timezone (timezone-independent truncation).
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/2649";>#2649</a></p></li>
+correctly regardless of session timezone (timezone-independent
truncation).</p></li>
</ul>
<section id="date-and-time-functions">
<h2>Date and Time Functions<a class="headerlink"
href="#date-and-time-functions" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/compatibility/expressions/spark-3.5/cast.html
b/user-guide/latest/compatibility/expressions/spark-3.5/cast.html
index f53f6c7d51..2ecf9b74d8 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/cast.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/cast.html
@@ -760,7 +760,6 @@ Spark.</p></li>
<section id="ansi-mode-fallback">
<h2>ANSI Mode Fallback<a class="headerlink" href="#ansi-mode-fallback"
title="Link to this heading">#</a></h2>
<p>Cast will fall back to Spark in some cases when ANSI mode is enabled. This
can be enabled by setting <code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>.
See the <a class="reference internal" href="../../../expressions.html"><span
class="std std-doc">Comet Supported Expressions Guide</span></a> for more
information on this configuration setting.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/313";>epic</a> where we
are tracking the work to fully implement ANSI support.</p>
</section>
<section id="string-to-decimal">
<h2>String to Decimal<a class="headerlink" href="#string-to-decimal"
title="Link to this heading">#</a></h2>
@@ -1592,7 +1591,6 @@ as <code class="docutils literal notranslate"><span
class="pre">"1.23E+4&qu
</ul>
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
-<p>See the <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/286";>tracking issue</a>
for more details.</p>
</section>
</section>
diff --git
a/user-guide/latest/compatibility/expressions/spark-3.5/datetime.html
b/user-guide/latest/compatibility/expressions/spark-3.5/datetime.html
index 05d279ed01..451cf2e6b5 100644
--- a/user-guide/latest/compatibility/expressions/spark-3.5/datetime.html
+++ b/user-guide/latest/compatibility/expressions/spark-3.5/datetime.html
@@ -747,16 +747,12 @@ under the License.
<section id="date-time-expressions">
<h1>Date/Time Expressions<a class="headerlink" href="#date-time-expressions"
title="Link to this heading">#</a></h1>
<ul class="simple">
-<li><p><strong>Hour, Minute, Second</strong>: Incorrectly apply timezone
conversion to TimestampNTZ inputs. TimestampNTZ stores local
-time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/3180";>#3180</a></p></li>
<li><p><strong>TruncTimestamp (date_trunc)</strong>: In non-UTC sessions the
native path is marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz’s DST horizon
(approximately year 2100;
see “Date and Time Functions” below) and can be enabled by setting
<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.TruncTimestamp.allowIncompatible=true</span></code>.
TimestampNTZ inputs are handled
-correctly regardless of session timezone (timezone-independent truncation).
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/2649";>#2649</a></p></li>
+correctly regardless of session timezone (timezone-independent
truncation).</p></li>
</ul>
<section id="date-and-time-functions">
<h2>Date and Time Functions<a class="headerlink"
href="#date-and-time-functions" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.0/cast.html
b/user-guide/latest/compatibility/expressions/spark-4.0/cast.html
index 74375f6331..fc04d09bf3 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/cast.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/cast.html
@@ -760,7 +760,6 @@ Spark.</p></li>
<section id="ansi-mode-fallback">
<h2>ANSI Mode Fallback<a class="headerlink" href="#ansi-mode-fallback"
title="Link to this heading">#</a></h2>
<p>Cast will fall back to Spark in some cases when ANSI mode is enabled. This
can be enabled by setting <code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>.
See the <a class="reference internal" href="../../../expressions.html"><span
class="std std-doc">Comet Supported Expressions Guide</span></a> for more
information on this configuration setting.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/313";>epic</a> where we
are tracking the work to fully implement ANSI support.</p>
</section>
<section id="string-to-decimal">
<h2>String to Decimal<a class="headerlink" href="#string-to-decimal"
title="Link to this heading">#</a></h2>
@@ -1592,7 +1591,6 @@ as <code class="docutils literal notranslate"><span
class="pre">"1.23E+4&qu
</ul>
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
-<p>See the <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/286";>tracking issue</a>
for more details.</p>
</section>
</section>
diff --git
a/user-guide/latest/compatibility/expressions/spark-4.0/datetime.html
b/user-guide/latest/compatibility/expressions/spark-4.0/datetime.html
index 52898d6d70..3d036bf2c6 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.0/datetime.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.0/datetime.html
@@ -747,16 +747,12 @@ under the License.
<section id="date-time-expressions">
<h1>Date/Time Expressions<a class="headerlink" href="#date-time-expressions"
title="Link to this heading">#</a></h1>
<ul class="simple">
-<li><p><strong>Hour, Minute, Second</strong>: Incorrectly apply timezone
conversion to TimestampNTZ inputs. TimestampNTZ stores local
-time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/3180";>#3180</a></p></li>
<li><p><strong>TruncTimestamp (date_trunc)</strong>: In non-UTC sessions the
native path is marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz’s DST horizon
(approximately year 2100;
see “Date and Time Functions” below) and can be enabled by setting
<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.TruncTimestamp.allowIncompatible=true</span></code>.
TimestampNTZ inputs are handled
-correctly regardless of session timezone (timezone-independent truncation).
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/2649";>#2649</a></p></li>
+correctly regardless of session timezone (timezone-independent
truncation).</p></li>
</ul>
<section id="date-and-time-functions">
<h2>Date and Time Functions<a class="headerlink"
href="#date-and-time-functions" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/compatibility/expressions/spark-4.1/cast.html
b/user-guide/latest/compatibility/expressions/spark-4.1/cast.html
index d03a7f5500..b2157a8ced 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/cast.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/cast.html
@@ -760,7 +760,6 @@ Spark.</p></li>
<section id="ansi-mode-fallback">
<h2>ANSI Mode Fallback<a class="headerlink" href="#ansi-mode-fallback"
title="Link to this heading">#</a></h2>
<p>Cast will fall back to Spark in some cases when ANSI mode is enabled. This
can be enabled by setting <code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.Cast.allowIncompatible=true</span></code>.
See the <a class="reference internal" href="../../../expressions.html"><span
class="std std-doc">Comet Supported Expressions Guide</span></a> for more
information on this configuration setting.</p>
-<p>There is an <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/313";>epic</a> where we
are tracking the work to fully implement ANSI support.</p>
</section>
<section id="string-to-decimal">
<h2>String to Decimal<a class="headerlink" href="#string-to-decimal"
title="Link to this heading">#</a></h2>
@@ -1592,7 +1591,6 @@ as <code class="docutils literal notranslate"><span
class="pre">"1.23E+4&qu
</ul>
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
-<p>See the <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/286";>tracking issue</a>
for more details.</p>
</section>
</section>
diff --git
a/user-guide/latest/compatibility/expressions/spark-4.1/datetime.html
b/user-guide/latest/compatibility/expressions/spark-4.1/datetime.html
index 416b104475..bc9ae6b3a3 100644
--- a/user-guide/latest/compatibility/expressions/spark-4.1/datetime.html
+++ b/user-guide/latest/compatibility/expressions/spark-4.1/datetime.html
@@ -747,16 +747,12 @@ under the License.
<section id="date-time-expressions">
<h1>Date/Time Expressions<a class="headerlink" href="#date-time-expressions"
title="Link to this heading">#</a></h1>
<ul class="simple">
-<li><p><strong>Hour, Minute, Second</strong>: Incorrectly apply timezone
conversion to TimestampNTZ inputs. TimestampNTZ stores local
-time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/3180";>#3180</a></p></li>
<li><p><strong>TruncTimestamp (date_trunc)</strong>: In non-UTC sessions the
native path is marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz’s DST horizon
(approximately year 2100;
see “Date and Time Functions” below) and can be enabled by setting
<code class="docutils literal notranslate"><span
class="pre">spark.comet.expression.TruncTimestamp.allowIncompatible=true</span></code>.
TimestampNTZ inputs are handled
-correctly regardless of session timezone (timezone-independent truncation).
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/2649";>#2649</a></p></li>
+correctly regardless of session timezone (timezone-independent
truncation).</p></li>
</ul>
<section id="date-and-time-functions">
<h2>Date and Time Functions<a class="headerlink"
href="#date-and-time-functions" title="Link to this heading">#</a></h2>
diff --git a/user-guide/latest/compatibility/operators.html
b/user-guide/latest/compatibility/operators.html
index cdc3c5d91f..b26f956d88 100644
--- a/user-guide/latest/compatibility/operators.html
+++ b/user-guide/latest/compatibility/operators.html
@@ -764,7 +764,7 @@ natively as plain aggregations but not as window functions
and <code class="docutils literal notranslate"><span
class="pre">sum</span></code> / <code class="docutils literal
notranslate"><span class="pre">avg</span></code> on year-month or day-time
interval input types. Windowed aggregates inherit the same input-type
support as the batch aggregates, so these fall back in both contexts.</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">sum</span></code> or <code class="docutils literal
notranslate"><span class="pre">avg</span></code> on <code class="docutils
literal notranslate"><span class="pre">DECIMAL</span></code> with a sliding
(non ever-expanding) frame, because the sliding path would wrap on
-overflow instead of returning Spark’s <code class="docutils literal
notranslate"><span class="pre">NULL</span></code> (<a class="reference
external"
href="https://github.com/apache/datafusion-comet/issues/4729";>#4729</a>).</p></li>
+overflow instead of returning Spark’s <code class="docutils literal
notranslate"><span class="pre">NULL</span></code>.</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">RANGE</span></code> frame with an explicit offset when the <code
class="docutils literal notranslate"><span class="pre">ORDER</span> <span
class="pre">BY</span></code> column is <code class="docutils literal
notranslate"><span class="pre">DATE</span></code> or <code class="docutils
literal notranslate"><span class="pre">DECIMAL</span></code>
(<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4834";>#4834</a>).</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">first_value</span></code> / <code class="docutils literal
notranslate"><span class="pre">last_value</span></code> on a <code
class="docutils literal notranslate"><span class="pre">RANGE</span></code>
frame with a literal offset
diff --git a/user-guide/latest/compatibility/scans.html
b/user-guide/latest/compatibility/scans.html
index 719925c160..518c45b996 100644
--- a/user-guide/latest/compatibility/scans.html
+++ b/user-guide/latest/compatibility/scans.html
@@ -763,15 +763,11 @@ V2 API for Parquet scans. Comet’s Parquet scan only
supports the V1 API.</p></
<li><p>No support for <code class="docutils literal notranslate"><span
class="pre">input_file_name()</span></code>, <code class="docutils literal
notranslate"><span class="pre">input_file_block_start()</span></code>, or <code
class="docutils literal notranslate"><span
class="pre">input_file_block_length()</span></code> SQL functions.
Comet’s Parquet scan does not use Spark’s <code class="docutils literal
notranslate"><span class="pre">FileScanRDD</span></code>, so these functions
cannot populate their values.</p></li>
<li><p>No support for <code class="docutils literal notranslate"><span
class="pre">ignoreMissingFiles</span></code> or <code class="docutils literal
notranslate"><span class="pre">ignoreCorruptFiles</span></code> being set to
<code class="docutils literal notranslate"><span
class="pre">true</span></code></p></li>
-<li><p>Duplicate field names in case-insensitive mode (e.g., a Parquet file
with both <code class="docutils literal notranslate"><span
class="pre">B</span></code> and <code class="docutils literal
notranslate"><span class="pre">b</span></code> columns)
-are detected at read time and raise a <code class="docutils literal
notranslate"><span class="pre">SparkRuntimeException</span></code> with error
class <code class="docutils literal notranslate"><span
class="pre">_LEGACY_ERROR_TEMP_2093</span></code>,
-matching Spark’s behavior.</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">spark.sql.parquet.enableVectorizedReader=false</span></code>.
Disabling the vectorized reader opts into
Spark’s parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet’s native reader
does not replicate. By default Comet falls back to Spark in this case. Set
<code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.allowDisabledParquetVectorizedReader=true</span></code>
to opt in to running the
-Comet Parquet scan regardless. See
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4352";>#4352</a>.</p></li>
+Comet Parquet scan regardless.</p></li>
</ul>
<p>The following limitation may produce incorrect results without falling back
to Spark:</p>
<ul class="simple">
@@ -788,13 +784,6 @@ Arrow, whose string type is strictly UTF-8. Reading a
Parquet file whose <code c
non-UTF-8 bytes fails with <code class="docutils literal notranslate"><span
class="pre">Parquet</span> <span class="pre">error:</span> <span
class="pre">encountered</span> <span class="pre">non</span> <span
class="pre">UTF-8</span> <span class="pre">data</span></code>. Disable Comet
for the
query, or cast the column to <code class="docutils literal notranslate"><span
class="pre">BINARY</span></code> before persisting, if you need to preserve
non-UTF-8 bytes.
See <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4121";>#4121</a>.</p></li>
-<li><p>Reading <code class="docutils literal notranslate"><span
class="pre">TimestampLTZ</span></code> as <code class="docutils literal
notranslate"><span class="pre">TimestampNTZ</span></code> on Spark 3.x. Spark
raises an error per
-<a class="reference external"
href="https://issues.apache.org/jira/browse/SPARK-36182";>SPARK-36182</a>
because LTZ encodes UTC-adjusted
-instants that cannot be safely reinterpreted as timezone-free values, and
Comet matches this by
-rejecting the read. This applies to all LTZ physical encodings (INT96,
TIMESTAMP_MICROS,
-TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted
-(<a class="reference external"
href="https://issues.apache.org/jira/browse/SPARK-47447";>SPARK-47447</a>) and
Comet matches Spark’s
-behavior. See <a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4219";>#4219</a>.</p></li>
</ul>
<section id="schema-mismatch-handling">
<h3>Schema Mismatch Handling<a class="headerlink"
href="#schema-mismatch-handling" title="Link to this heading">#</a></h3>
diff --git a/user-guide/latest/compatibility/spark-versions.html
b/user-guide/latest/compatibility/spark-versions.html
index 755bdc784d..47183c2729 100644
--- a/user-guide/latest/compatibility/spark-versions.html
+++ b/user-guide/latest/compatibility/spark-versions.html
@@ -781,8 +781,7 @@ See <a class="reference internal"
href="scans.html#parquet-scan-limitations"><sp
<section id="id2">
<h3>Known Limitations<a class="headerlink" href="#id2" title="Link to this
heading">#</a></h3>
<ul class="simple">
-<li><p><strong>Collation support</strong> (<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/1947";>#1947</a>,
-<a class="reference external"
href="https://github.com/apache/datafusion-comet/issues/4051";>#4051</a>): Spark
4.0 introduced collation
+<li><p><strong>Collation support</strong>: Spark 4.0 introduced collation
support. Non-default collated strings are not yet supported by Comet and will
fall back to Spark.</p></li>
</ul>
</section>
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 48ff9c902e..dbdb32cc6b 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -754,7 +754,7 @@ under the License.
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.allowDisabledParquetVectorizedReader</span></code></p></td>
-<td><p>Whether to allow Comet’s native scan to replace the Parquet scan when
Spark’s spark.sql.parquet.enableVectorizedReader is set to false. By default
(false), Comet falls back to Spark in that case, because Comet’s native readers
mirror Spark’s vectorized reader semantics rather than Spark’s parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information, refer to the <a
class="reference external" href="ht [...]
+<td><p>Whether to allow Comet’s native scan to replace the Parquet scan when
Spark’s spark.sql.parquet.enableVectorizedReader is set to false. By default
(false), Comet falls back to Spark in that case, because Comet’s native readers
mirror Spark’s vectorized reader semantics rather than Spark’s parquet-mr
(non-vectorized) semantics, which permit silent overflow / null-on-narrowing
that Comet has no equivalent for. For more information, refer to the <a
class="reference external" href="ht [...]
<td><p>false</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.icebergNative.dataFileConcurrencyLimit</span></code></p></td>
@@ -766,7 +766,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.scan.unsignedSmallIntSafetyCheck</span></code></p></td>
-<td><p>Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark
maps to ShortType. When this config is true (default), Comet falls back to
Spark for ShortType columns because we cannot distinguish signed INT16 (safe)
from unsigned UINT_8 (may produce different results). Set to false to allow
native execution of ShortType columns if you know your data does not contain
unsigned UINT_8 columns from improperly encoded Parquet files. For more
information, refer to the <a class=" [...]
+<td><p>Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark
maps to ShortType. When this config is true (default), Comet falls back to
Spark for ShortType columns because we cannot distinguish signed INT16 (safe)
from unsigned UINT_8 (may produce different results). Set to false to allow
native execution of ShortType columns if you know your data does not contain
unsigned UINT_8 columns from improperly encoded Parquet files. For more
information, refer to the <a class=" [...]
<td><p>true</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.hadoop.fs.comet.libhdfs.schemes</span></code></p></td>
@@ -870,7 +870,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.replaceSortMergeJoin</span></code></p></td>
-<td><p>Experimental feature to force Spark to replace SortMergeJoin with
ShuffledHashJoin for improved performance. This feature is not stable yet. For
more information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/tuning.html";>Comet Tuning
Guide</a>.</p></td>
+<td><p>Experimental feature to force Spark to replace SortMergeJoin with
ShuffledHashJoin for improved performance. This feature is not stable yet. For
more information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.scalaUDF.codegen.enabled</span></code></p></td>
@@ -878,7 +878,7 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.strictFloatingPoint</span></code></p></td>
-<td><p>When enabled, fall back to Spark for floating-point operations that may
differ from Spark, such as when comparing or sorting -0.0 and 0.0. For more
information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/compatibility.html";>Comet
Compatibility Guide</a>.</p></td>
+<td><p>When enabled, fall back to Spark for floating-point operations that may
differ from Spark, such as when comparing or sorting -0.0 and 0.0. For more
information, refer to the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/compatibility/index.html";>Comet
Compatibility Guide</a>.</p></td>
<td><p>false</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.transitionRevert.enabled</span></code></p></td>
@@ -1065,11 +1065,11 @@ under the License.
<td><p>8192</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool</span></code></p></td>
-<td><p>The type of memory pool to be used for Comet native execution when
running Spark in off-heap mode. Available pool types are <code class="docutils
literal notranslate"><span class="pre">greedy_unified</span></code> and <code
class="docutils literal notranslate"><span
class="pre">fair_unified</span></code>. For more information, refer to the <a
class="reference external"
href="https://datafusion.apache.org/comet/user-guide/tuning.html";>Comet Tuning
Guide</a>.</p></td>
+<td><p>The type of memory pool to be used for Comet native execution when
running Spark in off-heap mode. Available pool types are <code class="docutils
literal notranslate"><span class="pre">greedy_unified</span></code> and <code
class="docutils literal notranslate"><span
class="pre">fair_unified</span></code>. For more information, refer to the <a
class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>fair_unified</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool.fraction</span></code></p></td>
-<td><p>Fraction of off-heap memory pool that is available to Comet. Only
applies to off-heap mode. For more information, refer to the <a
class="reference external"
href="https://datafusion.apache.org/comet/user-guide/tuning.html";>Comet Tuning
Guide</a>.</p></td>
+<td><p>Fraction of off-heap memory pool that is available to Comet. Only
applies to off-heap mode. For more information, refer to the <a
class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>1.0</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.tracing.enabled</span></code></p></td>
@@ -1095,7 +1095,7 @@ under the License.
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.memory.factor</span></code></p></td>
-<td><p>Fraction of Comet memory to be allocated per executor process for
columnar shuffle when running in on-heap mode. For more information, refer to
the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/tuning.html";>Comet Tuning
Guide</a>.</p></td>
+<td><p>Fraction of Comet memory to be allocated per executor process for
columnar shuffle when running in on-heap mode. For more information, refer to
the <a class="reference external"
href="https://datafusion.apache.org/comet/user-guide/latest/tuning.html";>Comet
Tuning Guide</a>.</p></td>
<td><p>1.0</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.debug.memory</span></code></p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
