This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new d953fc0168 docs: remove references to closed and fixed issues from
compatibility guide (#4856)
d953fc0168 is described below
commit d953fc0168cc6ab51e95920ee32fd531ce4b4a89
Author: Andy Grove <[email protected]>
AuthorDate: Tue Jul 7 19:50:28 2026 -0600
docs: remove references to closed and fixed issues from compatibility guide
(#4856)
---
.../compatibility/expressions/_category_template/cast.md | 4 ----
.../expressions/_category_template/datetime.md | 4 ----
docs/source/user-guide/latest/compatibility/operators.md | 2 +-
docs/source/user-guide/latest/compatibility/scans.md | 13 +------------
.../user-guide/latest/compatibility/spark-versions.md | 3 +--
5 files changed, 3 insertions(+), 23 deletions(-)
diff --git
a/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md
b/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md
index db2b1dff8b..b9fea8998f 100644
---
a/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md
+++
b/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md
@@ -34,8 +34,6 @@ Cast operations in Comet fall into three levels of support:
Cast will fall back to Spark in some cases when ANSI mode is enabled. This can
be enabled by setting `spark.comet.expression.Cast.allowIncompatible=true`. See
the [Comet Supported Expressions Guide](../../../expressions.md) for more
information on this configuration setting.
-There is an [epic](https://github.com/apache/datafusion-comet/issues/313)
where we are tracking the work to fully implement ANSI support.
-
## String to Decimal
Comet's native `CAST(string AS DECIMAL)` implementation matches Apache Spark's
behavior,
@@ -160,5 +158,3 @@ as `"1.23E+4"`).
<!--BEGIN:CAST_ANSI_TABLE-->
<!--END:CAST_ANSI_TABLE-->
-
-See the [tracking
issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
diff --git
a/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md
b/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md
index 4c5d5c9db0..ea9eb10e37 100644
---
a/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md
+++
b/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md
@@ -19,16 +19,12 @@ under the License.
# Date/Time Expressions
-- **Hour, Minute, Second**: Incorrectly apply timezone conversion to
TimestampNTZ inputs. TimestampNTZ stores local
- time without timezone, so no conversion should be applied. These expressions
work correctly with Timestamp inputs.
- [#3180](https://github.com/apache/datafusion-comet/issues/3180)
- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is
marked Incompatible and
routes through the JVM codegen dispatcher by default, producing
Spark-identical results. The
native path is itself correct for dates within chrono-tz's DST horizon
(approximately year 2100;
see "Date and Time Functions" below) and can be enabled by setting
`spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ
inputs are handled
correctly regardless of session timezone (timezone-independent truncation).
- [#2649](https://github.com/apache/datafusion-comet/issues/2649)
## Date and Time Functions
diff --git a/docs/source/user-guide/latest/compatibility/operators.md
b/docs/source/user-guide/latest/compatibility/operators.md
index 28c0954047..eec21d283b 100644
--- a/docs/source/user-guide/latest/compatibility/operators.md
+++ b/docs/source/user-guide/latest/compatibility/operators.md
@@ -44,7 +44,7 @@ incorrect result. When any single window expression in a
`WindowExec` falls back
and `sum` / `avg` on year-month or day-time interval input types. Windowed
aggregates inherit the same input-type
support as the batch aggregates, so these fall back in both contexts.
- `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame,
because the sliding path would wrap on
- overflow instead of returning Spark's `NULL`
([#4729](https://github.com/apache/datafusion-comet/issues/4729)).
+ overflow instead of returning Spark's `NULL`.
- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE`
or `DECIMAL`
([#4834](https://github.com/apache/datafusion-comet/issues/4834)).
- `first_value` / `last_value` on a `RANGE` frame with a literal offset
diff --git a/docs/source/user-guide/latest/compatibility/scans.md
b/docs/source/user-guide/latest/compatibility/scans.md
index 4f2f14a944..184ae9f06b 100644
--- a/docs/source/user-guide/latest/compatibility/scans.md
+++ b/docs/source/user-guide/latest/compatibility/scans.md
@@ -43,15 +43,11 @@ The following features are not supported and cause Comet to
fall back to Spark:
- No support for `input_file_name()`, `input_file_block_start()`, or
`input_file_block_length()` SQL functions.
Comet's Parquet scan does not use Spark's `FileScanRDD`, so these functions
cannot populate their values.
- No support for `ignoreMissingFiles` or `ignoreCorruptFiles` being set to
`true`
-- Duplicate field names in case-insensitive mode (e.g., a Parquet file with
both `B` and `b` columns)
- are detected at read time and raise a `SparkRuntimeException` with error
class `_LEGACY_ERROR_TEMP_2093`,
- matching Spark's behavior.
- `spark.sql.parquet.enableVectorizedReader=false`. Disabling the vectorized
reader opts into
Spark's parquet-mr semantics (silent overflow, null-on-narrowing), which
Comet's native reader
does not replicate. By default Comet falls back to Spark in this case. Set
`spark.comet.scan.allowDisabledParquetVectorizedReader=true` to opt in to
running the
- Comet Parquet scan regardless. See
- [#4352](https://github.com/apache/datafusion-comet/issues/4352).
+ Comet Parquet scan regardless.
The following limitation may produce incorrect results without falling back to
Spark:
@@ -68,13 +64,6 @@ The following limitations raise an error at scan time rather
than falling back t
non-UTF-8 bytes fails with `Parquet error: encountered non UTF-8 data`.
Disable Comet for the
query, or cast the column to `BINARY` before persisting, if you need to
preserve non-UTF-8 bytes.
See [#4121](https://github.com/apache/datafusion-comet/issues/4121).
-- Reading `TimestampLTZ` as `TimestampNTZ` on Spark 3.x. Spark raises an error
per
- [SPARK-36182](https://issues.apache.org/jira/browse/SPARK-36182) because LTZ
encodes UTC-adjusted
- instants that cannot be safely reinterpreted as timezone-free values, and
Comet matches this by
- rejecting the read. This applies to all LTZ physical encodings (INT96,
TIMESTAMP_MICROS,
- TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted
- ([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet
matches Spark's
- behavior. See
[#4219](https://github.com/apache/datafusion-comet/issues/4219).
### Schema Mismatch Handling
diff --git a/docs/source/user-guide/latest/compatibility/spark-versions.md
b/docs/source/user-guide/latest/compatibility/spark-versions.md
index a6ad5bcf16..5715fef765 100644
--- a/docs/source/user-guide/latest/compatibility/spark-versions.md
+++ b/docs/source/user-guide/latest/compatibility/spark-versions.md
@@ -60,8 +60,7 @@ Spark 4.0.2 is supported with Java 17 and Scala 2.13.
### Known Limitations
-- **Collation support**
([#1947](https://github.com/apache/datafusion-comet/issues/1947),
- [#4051](https://github.com/apache/datafusion-comet/issues/4051)): Spark 4.0
introduced collation
+- **Collation support**: Spark 4.0 introduced collation
support. Non-default collated strings are not yet supported by Comet and
will fall back to Spark.
## Spark 4.1
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
