This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new c76dd73aff docs: note invalid UTF-8 scan limitation and FFI import
risk (#4846)
c76dd73aff is described below
commit c76dd73affdee9adaeb498a05991b4138434b2c5
Author: Andy Grove <[email protected]>
AuthorDate: Wed Jul 8 13:13:40 2026 -0600
docs: note invalid UTF-8 scan limitation and FFI import risk (#4846)
---
docs/source/user-guide/latest/compatibility/scans.md | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/docs/source/user-guide/latest/compatibility/scans.md
b/docs/source/user-guide/latest/compatibility/scans.md
index 184ae9f06b..aa2be5b3ce 100644
--- a/docs/source/user-guide/latest/compatibility/scans.md
+++ b/docs/source/user-guide/latest/compatibility/scans.md
@@ -63,7 +63,16 @@ The following limitations raise an error at scan time rather
than falling back t
Arrow, whose string type is strictly UTF-8. Reading a Parquet file whose
`STRING` column contains
non-UTF-8 bytes fails with `Parquet error: encountered non UTF-8 data`.
Disable Comet for the
query, or cast the column to `BINARY` before persisting, if you need to
preserve non-UTF-8 bytes.
- See [#4121](https://github.com/apache/datafusion-comet/issues/4121).
+ Separately, non-UTF-8 bytes that reach native execution from a JVM-side
columnar source are not
+ currently validated at the Arrow FFI import boundary. See
[#4121](https://github.com/apache/datafusion-comet/issues/4121)
+ and the tracking issue
[#4764](https://github.com/apache/datafusion-comet/issues/4764).
+- Reading `TimestampLTZ` as `TimestampNTZ` on Spark 3.x. Spark raises an error
per
+ [SPARK-36182](https://issues.apache.org/jira/browse/SPARK-36182) because LTZ
encodes UTC-adjusted
+ instants that cannot be safely reinterpreted as timezone-free values, and
Comet matches this by
+ rejecting the read. This applies to all LTZ physical encodings (INT96,
TIMESTAMP_MICROS,
+ TIMESTAMP_MILLIS). On Spark 4.0+, this read is permitted
+ ([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet
matches Spark's
+ behavior. See
[#4219](https://github.com/apache/datafusion-comet/issues/4219).
### Schema Mismatch Handling
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
