comphead commented on code in PR #7052: URL: https://github.com/apache/datafusion-comet/pull/7052#discussion_r4300329677
########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | Review Comment: `spark.comet.scan.enabled` is in the `testing` category in `CometConf.scala` and is documented as not intended for production. Showing it as the Parquet scan switch could invite users to toggle it. A short note under the table would help. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the Review Comment: "That is Spark's only behavior ... and its behavior on Spark 4.1" is hard to parse. Suggested: > **A struct or array with none of its requested fields in the file.** Spark returns NULL for the column. Comet returns a non-null struct or array that holds NULLs. This applies to Spark 3.4, 3.5 and 4.0, and to Spark 4.1 and later with `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true`. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to Review Comment: Suggested intro, one fact per sentence: > This page lists open issues in Comet's Parquet and Iceberg reads and writes where Comet does not fall back to Spark. Each entry links to its tracking issue. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and Review Comment: Two things here. First, `native/core/src/parquet/datetime_rebase.rs` says per-file rebase is enabled only for the Delta scan (`rebase_from_file_metadata`), and `delta.md` documents it. "Comet does not rebase" is true only for the plain native Parquet scan, so please scope the entry. Second, a shorter wording: > **Dates and timestamps written with the legacy calendar.** Spark before 3.0, and later versions with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`, write these values in the hybrid Julian/Gregorian calendar. The native Parquet scan reads them as proleptic Gregorian, so values before October 15, 1582 are wrong in projections and predicates. It ignores `spark.sql.parquet.datetimeRebaseModeInRead` and the file's `org.apache.spark.legacyDateTime` metadata (#5010). The Delta scan rebases. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With Review Comment: Suggested: > **Field-id reads where one file column matches a field by id and a sibling by name.** With `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top level. Inside a struct, it returns the values of the field matched by id. Spark's own readers disagree here (#6994). ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without Review Comment: Suggested: > **Nested BINARY values that are not valid UTF-8, read as STRING.** This applies to a BINARY column without the UTF8 annotation that is read as `string`, through the read schema or `spark.sql.parquet.binaryAsString=true`. Spark returns the bytes. Comet returns NULL for each such nested value. The current last sentence says top-level values "skip UTF-8 validation". Please say whether they match Spark, since a reader will ask. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are + read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` + procedures set, the native scan returns NULL for nested fields (struct fields, list elements and + map values) whenever the query also reads a top-level primitive column, so filters on them select + the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes + with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +### Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the Review Comment: Suggested: > **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** Comet skips the steps Spark runs after a write: file index refresh, cache refresh, partition refresh and statistics update. Later queries can return the table's previous contents until `REFRESH TABLE` (#6315). ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic Review Comment: #6264 was closed today by #6270, which is on `main` (e242944ec4). This entry is no longer an open issue, so please drop it. The `data-source-audits` pages do not mention it, so nothing else should need to change. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are + read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` + procedures set, the native scan returns NULL for nested fields (struct fields, list elements and + map values) whenever the query also reads a top-level primitive column, so filters on them select + the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes + with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +### Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the + write (file index refresh, cache refresh, partition refresh and statistics update) are skipped, + so later queries can read the table's previous contents until `REFRESH TABLE` + ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **An input with no partitions writes no file.** Spark writes one file that holds the schema + ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema + ([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR` + ([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union + ([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default + columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after + `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)), + `INSERT INTO` from non-projection queries + ([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output + schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)), multiple inserts + ([#3430](https://github.com/apache/datafusion-comet/issues/3430)), + `INSERT OVERWRITE LOCAL DIRECTORY` + ([#3420](https://github.com/apache/datafusion-comet/issues/3420)), empty files + ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit` + ([#3428](https://github.com/apache/datafusion-comet/issues/3428)). + +### Iceberg writes + +The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: + +- **Brotli tables.** A native write commits brotli-compressed files that no JVM Parquet reader can + read unless Hadoop's `BrotliCodec` is on its classpath, where iceberg-java fails the write + ([#6960](https://github.com/apache/datafusion-comet/issues/6960)). +- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, Review Comment: Suggested: > **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, and iceberg-java rejects the value. The native writer also trims or lowercases a few other settings that iceberg-java reads as written (#7008). ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are + read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` + procedures set, the native scan returns NULL for nested fields (struct fields, list elements and + map values) whenever the query also reads a top-level primitive column, so filters on them select + the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes + with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +### Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the + write (file index refresh, cache refresh, partition refresh and statistics update) are skipped, + so later queries can read the table's previous contents until `REFRESH TABLE` + ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **An input with no partitions writes no file.** Spark writes one file that holds the schema + ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema Review Comment: Eleven issues in one sentence are hard to scan. A nested list with one bullet per issue would read better, for example: > - **Spark SQL tests that still fail with the native writer**, not yet classified: > - CTAS schema (#3418) > - CTAS with `CHAR_AS_VARCHAR` (#3419) > - ... ########## docs/source/user-guide/latest/iceberg-writes.md: ########## @@ -494,9 +494,14 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed change pruning decisions. - On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for float/double columns nested under a nullable struct count rows whose parent struct is null (they come from the - parquet footer), while iceberg-java's writer-tracked counts do not. Both counts inflate by - the same amount, so the derived null ratios and `IS NULL` / `IS NOT NULL` pruning decisions - are unaffected. + parquet footer), while iceberg-java's writer-tracked counts do not. The native counts are the + correct ones. iceberg-java's leave out the rows where the struct is null, so when every Review Comment: "iceberg-java's leave out" reads like a typo. Suggested wording for the whole bullet tail: > The native counts are the correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null struct in a file holds a value, iceberg-java records a null count of 0, and `IS NULL` prunes a file that contains matching rows ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The native counts never prune such a file. `IS NOT NULL` pruning is the same for both. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are + read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` + procedures set, the native scan returns NULL for nested fields (struct fields, list elements and + map values) whenever the query also reads a top-level primitive column, so filters on them select + the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes + with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +### Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the + write (file index refresh, cache refresh, partition refresh and statistics update) are skipped, + so later queries can read the table's previous contents until `REFRESH TABLE` + ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **An input with no partitions writes no file.** Spark writes one file that holds the schema + ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema + ([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR` + ([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union + ([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default + columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after + `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)), + `INSERT INTO` from non-projection queries + ([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output + schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)), multiple inserts + ([#3430](https://github.com/apache/datafusion-comet/issues/3430)), + `INSERT OVERWRITE LOCAL DIRECTORY` + ([#3420](https://github.com/apache/datafusion-comet/issues/3420)), empty files + ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit` + ([#3428](https://github.com/apache/datafusion-comet/issues/3428)). + +### Iceberg writes + +The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: + +- **Brotli tables.** A native write commits brotli-compressed files that no JVM Parquet reader can Review Comment: "where iceberg-java fails the write" attaches to the wrong clause. Suggested for this entry and the next: > **Brotli tables.** The native writer commits brotli-compressed files. JVM Parquet readers can read them only if Hadoop's `BrotliCodec` is on the classpath. iceberg-java fails the write instead (#6960). ########## docs/source/user-guide/latest/compatibility/spark-versions.md: ########## @@ -69,17 +69,6 @@ Spark 4.0.4 is supported with Java 17/21 and Scala 2.13. Spark 4.1.3 is supported with Java 17/21 and Scala 2.13. Review Comment: Removing the `NullType` limitation matches #4199 being closed. `dev/diffs/4.1.3.diff` (around line 3750) and `dev/diffs/4.2.0.diff` (around line 3831) still tag the `SPARK-54220` test with `IgnoreComet(".../issues/4199")`. I did not run that test. Either the tag is stale or the test still fails for another reason. If it is stale, it should be regenerated through the documented Spark SQL tests workflow, not edited by hand. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are Review Comment: Suggested, with the trigger first and the consequence second: > **Migrated tables that use a name mapping.** The `migrate`, `snapshot` and `add_files` procedures set `schema.name-mapping.default` for files without Iceberg field ids. When a query also reads a top-level primitive column, the native scan returns NULL for nested fields (struct fields, list elements and map values). Filters on those fields then select the wrong rows (#6790). The fix needs an iceberg-rust upgrade (#6934). ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are + read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` + procedures set, the native scan returns NULL for nested fields (struct fields, list elements and + map values) whenever the query also reads a top-level primitive column, so filters on them select + the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes + with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +### Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the + write (file index refresh, cache refresh, partition refresh and statistics update) are skipped, + so later queries can read the table's previous contents until `REFRESH TABLE` + ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **An input with no partitions writes no file.** Spark writes one file that holds the schema + ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema + ([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR` + ([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union + ([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default + columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after + `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)), + `INSERT INTO` from non-projection queries + ([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output + schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)), multiple inserts + ([#3430](https://github.com/apache/datafusion-comet/issues/3430)), + `INSERT OVERWRITE LOCAL DIRECTORY` + ([#3420](https://github.com/apache/datafusion-comet/issues/3420)), empty files + ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit` + ([#3428](https://github.com/apache/datafusion-comet/issues/3428)). + +### Iceberg writes + +The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: + +- **Brotli tables.** A native write commits brotli-compressed files that no JVM Parquet reader can + read unless Hadoop's `BrotliCodec` is on its classpath, where iceberg-java fails the write + ([#6960](https://github.com/apache/datafusion-comet/issues/6960)). +- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, + where iceberg-java rejects the value. It also trims or lowercases a few other settings that + iceberg-java reads as written ([#7008](https://github.com/apache/datafusion-comet/issues/7008)). + +The physical differences that every native data file has, and why they are safe, are listed in +[Accepted divergences](../iceberg-writes.md#accepted-divergences). + +## Queries that fail where Spark succeeds + +In these cases a Comet read or write fails with an error where Spark, or iceberg-java, succeeds. +Comet does not fall back to Spark for them. + +### Parquet reads that fail + +- **STRING columns that hold invalid UTF-8.** The read fails with + `Parquet error: encountered non UTF-8 data`, where Spark returns the bytes + ([#4121](https://github.com/apache/datafusion-comet/issues/4121), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Duplicate field names.** Selecting a field by name when several physical siblings match it, + including inside structs, arrays and maps, fails with a duplicate-field error + ([#5884](https://github.com/apache/datafusion-comet/issues/5884), + [#5964](https://github.com/apache/datafusion-comet/issues/5964)). + [Parquet Compatibility](scans.md) describes the reads this affects. +- **A duplicate field id in a column the query does not read.** With Review Comment: This reads like the `scans.md` fallback for a read schema that repeats a field id. This entry is about a file that repeats a field id, so please say the read schema is valid, for example "With a valid read schema and `spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same field id fails the scan ...". ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are + read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` + procedures set, the native scan returns NULL for nested fields (struct fields, list elements and + map values) whenever the query also reads a top-level primitive column, so filters on them select + the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes + with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +### Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the + write (file index refresh, cache refresh, partition refresh and statistics update) are skipped, + so later queries can read the table's previous contents until `REFRESH TABLE` + ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **An input with no partitions writes no file.** Spark writes one file that holds the schema + ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema + ([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR` + ([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union + ([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default + columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after + `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)), + `INSERT INTO` from non-projection queries + ([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output + schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)), multiple inserts + ([#3430](https://github.com/apache/datafusion-comet/issues/3430)), + `INSERT OVERWRITE LOCAL DIRECTORY` + ([#3420](https://github.com/apache/datafusion-comet/issues/3420)), empty files + ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit` + ([#3428](https://github.com/apache/datafusion-comet/issues/3428)). + +### Iceberg writes + +The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: + +- **Brotli tables.** A native write commits brotli-compressed files that no JVM Parquet reader can + read unless Hadoop's `BrotliCodec` is on its classpath, where iceberg-java fails the write + ([#6960](https://github.com/apache/datafusion-comet/issues/6960)). +- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, + where iceberg-java rejects the value. It also trims or lowercases a few other settings that + iceberg-java reads as written ([#7008](https://github.com/apache/datafusion-comet/issues/7008)). + +The physical differences that every native data file has, and why they are safe, are listed in +[Accepted divergences](../iceberg-writes.md#accepted-divergences). + +## Queries that fail where Spark succeeds + +In these cases a Comet read or write fails with an error where Spark, or iceberg-java, succeeds. +Comet does not fall back to Spark for them. + +### Parquet reads that fail + +- **STRING columns that hold invalid UTF-8.** The read fails with + `Parquet error: encountered non UTF-8 data`, where Spark returns the bytes + ([#4121](https://github.com/apache/datafusion-comet/issues/4121), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Duplicate field names.** Selecting a field by name when several physical siblings match it, + including inside structs, arrays and maps, fails with a duplicate-field error + ([#5884](https://github.com/apache/datafusion-comet/issues/5884), + [#5964](https://github.com/apache/datafusion-comet/issues/5964)). + [Parquet Compatibility](scans.md) describes the reads this affects. +- **A duplicate field id in a column the query does not read.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same + field id fails the scan whenever the table schema has a field with that id, even when the query + does not read that field ([#7006](https://github.com/apache/datafusion-comet/issues/7006)). +- **A nested encoding that arrow-rs rejects.** A file fails to read when any column in its schema + uses such an encoding, even a column the query does not read + ([#7035](https://github.com/apache/datafusion-comet/issues/7035)). +- **A `DATE` column read as `INTERVAL YEAR TO MONTH`.** Spark reads it, and Comet fails the read + ([#5188](https://github.com/apache/datafusion-comet/issues/5188)). + +### Iceberg reads that fail + +- **A data file that omits an optional nested field.** The native scan fails on a data file that + lacks an optional struct field, which a write with `write.spark.accept-any-schema=true` can + produce without any schema change + ([#7033](https://github.com/apache/datafusion-comet/issues/7033)). +- **`VERSION AS OF` reads of a column whose type was promoted after the snapshot** + ([#6993](https://github.com/apache/datafusion-comet/issues/6993)). +- **An equality delete keyed on a nested field.** Every query the native scan runs on the table + fails with `field not found`, even one that reads the key + ([#6782](https://github.com/apache/datafusion-comet/issues/6782)). +- **Column names that only newer Unicode case data folds together**, such as a pair of Vithkuqi + letters, which Unicode 14 added. Queries on such a table fail with a duplicate-field error + ([#5540](https://github.com/apache/datafusion-comet/issues/5540)). + +### Object store and credential settings + +The native Parquet and Iceberg scans resolve storage settings with their own clients rather than +with Hadoop's. Where the two disagree, a native read can use a different identity or route than +Spark, and fail where Spark succeeds: + +- **S3 settings that Hadoop's S3A client resolves differently**, such as JCEKS-backed secrets and + the precedence of per-bucket settings. The native Parquet scan forwards them without checking + for a difference ([#5662](https://github.com/apache/datafusion-comet/issues/5662)). +- **AWS profile credential providers.** The native scans ignore `fs.s3a.auth.profile.name` and + `fs.s3a.auth.profile.file` ([#4245](https://github.com/apache/datafusion-comet/issues/4245)), + and resolve some profiles differently from Hadoop 3.4 and later, for example profiles that use + `credential_process` or chains of assumed roles + ([#6575](https://github.com/apache/datafusion-comet/issues/6575)). +- **Azure credentials.** Container-scoped OAuth settings, Fabric hosts and a few other + configurations resolve differently from Hadoop's ABFS driver + ([#6605](https://github.com/apache/datafusion-comet/issues/6605)). +- **`fs.comet.libhdfs.schemes` set only in the Hadoop configuration.** Comet's planner and its Review Comment: #6837 was closed today by #6838, which is on `main` (3a4a159409). Please drop this entry. ########## docs/source/user-guide/latest/compatibility/parquet-iceberg-known-issues.md: ########## @@ -0,0 +1,264 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Known Issues in Parquet and Iceberg Reads and Writes + +This page lists the open issues in Comet's Parquet and Iceberg reads and writes that a fallback to +Spark does not cover, with a link to the issue that tracks each one: + +- [Results that differ from Spark](#results-that-differ-from-spark): wrong values, a NULL where + Spark returns a value or the reverse, rows returned for a read that Spark rejects, and written + files that Spark or other engines read differently from the files Spark writes. +- [Queries that fail where Spark succeeds](#queries-that-fail-where-spark-succeeds). + +The cases where Comet falls back to Spark are documented in [Parquet Compatibility](scans.md), in +the [current limitations](../iceberg.md#current-limitations) of the Iceberg guide and in +[Iceberg Writes](../iceberg-writes.md). + +Which entries apply depends on which native paths are enabled: + +| Native path | Setting | Default | +| -------------- | ---------------------------------------- | ---------------------- | +| Parquet scan | `spark.comet.scan.enabled` | `true` | +| Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | +| Iceberg writer | `spark.comet.write.iceberg.enabled` | `false` | +| Parquet writer | `spark.comet.write.parquet.enabled` | `false` (testing only) | + +## Results that differ from Spark + +### Parquet reads + +- **Dates and timestamps written with the legacy calendar.** Comet does not rebase dates and + timestamps written with the hybrid Julian/Gregorian calendar, which Spark versions before 3.0 + wrote, and later versions write with `spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY`. Comet + reads them as proleptic Gregorian values, so values before October 15, 1582 are wrong in both + projections and predicates. Comet also ignores `spark.sql.parquet.datetimeRebaseModeInRead` and + the file's `org.apache.spark.legacyDateTime` metadata, which would tell it to rebase + ([#5010](https://github.com/apache/datafusion-comet/issues/5010)). +- **INT96 timestamps inside legacy-format lists.** INT96 timestamps outside the years 1677 to 2262 + read back wrong inside lists written in a legacy layout, as Hive and + `spark.sql.parquet.writeLegacyFormat=true` write them + ([#7029](https://github.com/apache/datafusion-comet/issues/7029)). +- **A struct or array with none of its requested fields in the file.** Spark returns NULL for the + column, and Comet returns a non-null struct or array that holds NULLs. That is Spark's only + behavior on Spark 3.4, 3.5 and 4.0, and its behavior on Spark 4.1 and later with + `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing=true` + ([#7030](https://github.com/apache/datafusion-comet/issues/7030)). +- **Nested BINARY values that are not valid UTF-8, read as STRING.** When a BINARY column without + the UTF8 annotation is read as STRING, because the read schema says `string` or because of + `spark.sql.parquet.binaryAsString=true`, Comet returns NULL for each nested value that is not + valid UTF-8 where Spark returns the bytes. Top-level values read this way skip UTF-8 validation + ([#7032](https://github.com/apache/datafusion-comet/issues/7032), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Spark 3.4: a struct matched by field id under a different name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a struct column that the read schema matches to + the file by field id but under another name reads as NULL when Spark prunes its nested fields + ([#7031](https://github.com/apache/datafusion-comet/issues/7031)). +- **Field-id reads in which one file column matches a field by id and a sibling by name.** With + `spark.sql.parquet.fieldId.read.enabled=true`, Comet returns NULL for the sibling at the top + level, and the values of the field matched by id inside a struct. Spark's own readers disagree + with each other here ([#6994](https://github.com/apache/datafusion-comet/issues/6994)). +- **`TIMESTAMP_MILLIS` values that overflow when converted to microseconds.** In nested fields, and + in every column of a scan that has data filters, Comet returns NULL where Spark fails with + `ArithmeticException: long overflow` + ([#5553](https://github.com/apache/datafusion-comet/issues/5553), + [#5739](https://github.com/apache/datafusion-comet/issues/5739)). +- **Reads that Spark rejects.** Comet returns rows where Spark fails the read for + `TIMESTAMP(NANOS)` columns read as `timestamp` or `timestamp_ntz` + ([#7034](https://github.com/apache/datafusion-comet/issues/7034)), `INT` columns read as + `INTERVAL DAY TO SECOND` ([#5188](https://github.com/apache/datafusion-comet/issues/5188)), and + Variant-annotated fields read as a plain struct + ([#5741](https://github.com/apache/datafusion-comet/issues/5741)). +- **Dynamic partition pruning under AQE.** Two exchanges whose scans differ only in their dynamic + partition pruning filter can compare equal, so AQE reuses the first for the second and the rows of + the second branch are lost. It shows when no coalesced shuffle read sits between the stages, as + with large shuffles or with partition coalescing off, and it applies to the native Iceberg scan + as well ([#6264](https://github.com/apache/datafusion-comet/issues/6264)). + +### Iceberg reads + +- **Migrated tables that use a name mapping.** For data files without Iceberg field ids that are + read through `schema.name-mapping.default`, which the `migrate`, `snapshot` and `add_files` + procedures set, the native scan returns NULL for nested fields (struct fields, list elements and + map values) whenever the query also reads a top-level primitive column, so filters on them select + the wrong rows ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The fix comes + with an iceberg-rust upgrade ([#6934](https://github.com/apache/datafusion-comet/issues/6934)). +- **Data files with neither field ids nor a name mapping.** Spark reads every nested field of such + a file as NULL, and the native scan returns the values stored in the file + ([#6794](https://github.com/apache/datafusion-comet/issues/6794)). +- **`VERSION AS OF` reads of a column renamed after the snapshot.** The renamed column reads as + NULL, and two columns that swapped names return each other's values + ([#6822](https://github.com/apache/datafusion-comet/issues/6822)). +- **Reflection failures while the scan is planned.** If Comet fails to read the table's schema or + metadata through reflection, it continues with an empty or partial result instead of falling + back, which can resolve a column to a different field id + ([#5257](https://github.com/apache/datafusion-comet/issues/5257)). + +### Parquet writes + +The native Parquet writer is experimental and disabled by default. With +`spark.comet.write.parquet.enabled=true`: + +- **The datetime rebase modes are ignored.** On Spark 3.4 and 3.5, whose default rebase mode is + `EXCEPTION`, the native writer writes dates and timestamps before October 15, 1582 that Spark + refuses to write, and Spark then fails to read them back under its default read mode. With + `spark.sql.parquet.datetimeRebaseModeInRead=LEGACY`, Spark reads them shifted by several days, on + every version. The files lack the `org.apache.spark.version` footer key that tells Spark they need + no rebasing ([#7047](https://github.com/apache/datafusion-comet/issues/7047), + [#3427](https://github.com/apache/datafusion-comet/issues/3427)). +- **`spark.sql.parquet.outputTimestampType` and `spark.sql.parquet.writeLegacyFormat` are + ignored.** Timestamps are always written as `INT64` microseconds, never as Spark's default + `INT96`, so with `TIMESTAMP_MILLIS` they keep the sub-millisecond digits that Spark drops. Spark's + schema footer key is not written either, so a path-based read loses column metadata such as + comments, and with `spark.sql.parquet.inferTimestampNTZ.enabled=false` it reads `TIMESTAMP_NTZ` + columns as `TIMESTAMP` ([#7048](https://github.com/apache/datafusion-comet/issues/7048), + [#3425](https://github.com/apache/datafusion-comet/issues/3425)). +- **Spark 3.4 and 3.5: `INSERT INTO` leaves the target stale.** The steps Spark runs after the + write (file index refresh, cache refresh, partition refresh and statistics update) are skipped, + so later queries can read the table's previous contents until `REFRESH TABLE` + ([#6315](https://github.com/apache/datafusion-comet/issues/6315)). +- **An input with no partitions writes no file.** Spark writes one file that holds the schema + ([#5303](https://github.com/apache/datafusion-comet/issues/5303)). +- **Spark SQL tests that still fail with the native writer**, not yet classified: CTAS schema + ([#3418](https://github.com/apache/datafusion-comet/issues/3418)), CTAS with `CHAR_AS_VARCHAR` + ([#3419](https://github.com/apache/datafusion-comet/issues/3419)), CTAS with a union + ([#3429](https://github.com/apache/datafusion-comet/issues/3429)), `INSERT INTO` with default + columns ([#3421](https://github.com/apache/datafusion-comet/issues/3421)) and after + `ALTER TABLE ADD COLUMNS` ([#3422](https://github.com/apache/datafusion-comet/issues/3422)), + `INSERT INTO` from non-projection queries + ([#3423](https://github.com/apache/datafusion-comet/issues/3423)), the `INSERT OVERWRITE` output + schema ([#3424](https://github.com/apache/datafusion-comet/issues/3424)), multiple inserts + ([#3430](https://github.com/apache/datafusion-comet/issues/3430)), + `INSERT OVERWRITE LOCAL DIRECTORY` + ([#3420](https://github.com/apache/datafusion-comet/issues/3420)), empty files + ([#3417](https://github.com/apache/datafusion-comet/issues/3417)) and `onTaskCommit` + ([#3428](https://github.com/apache/datafusion-comet/issues/3428)). + +### Iceberg writes + +The native Iceberg writer is disabled by default. With `spark.comet.write.iceberg.enabled=true`: + +- **Brotli tables.** A native write commits brotli-compressed files that no JVM Parquet reader can + read unless Hadoop's `BrotliCodec` is on its classpath, where iceberg-java fails the write + ([#6960](https://github.com/apache/datafusion-comet/issues/6960)). +- **`write.parquet.compression-codec=none`.** The native writer writes the table uncompressed, + where iceberg-java rejects the value. It also trims or lowercases a few other settings that + iceberg-java reads as written ([#7008](https://github.com/apache/datafusion-comet/issues/7008)). + +The physical differences that every native data file has, and why they are safe, are listed in +[Accepted divergences](../iceberg-writes.md#accepted-divergences). + +## Queries that fail where Spark succeeds + +In these cases a Comet read or write fails with an error where Spark, or iceberg-java, succeeds. +Comet does not fall back to Spark for them. + +### Parquet reads that fail + +- **STRING columns that hold invalid UTF-8.** The read fails with + `Parquet error: encountered non UTF-8 data`, where Spark returns the bytes + ([#4121](https://github.com/apache/datafusion-comet/issues/4121), part of + [#4764](https://github.com/apache/datafusion-comet/issues/4764)). +- **Duplicate field names.** Selecting a field by name when several physical siblings match it, + including inside structs, arrays and maps, fails with a duplicate-field error + ([#5884](https://github.com/apache/datafusion-comet/issues/5884), + [#5964](https://github.com/apache/datafusion-comet/issues/5964)). + [Parquet Compatibility](scans.md) describes the reads this affects. +- **A duplicate field id in a column the query does not read.** With + `spark.sql.parquet.fieldId.read.enabled=true`, a file in which two root columns carry the same + field id fails the scan whenever the table schema has a field with that id, even when the query + does not read that field ([#7006](https://github.com/apache/datafusion-comet/issues/7006)). +- **A nested encoding that arrow-rs rejects.** A file fails to read when any column in its schema + uses such an encoding, even a column the query does not read + ([#7035](https://github.com/apache/datafusion-comet/issues/7035)). +- **A `DATE` column read as `INTERVAL YEAR TO MONTH`.** Spark reads it, and Comet fails the read + ([#5188](https://github.com/apache/datafusion-comet/issues/5188)). + +### Iceberg reads that fail + +- **A data file that omits an optional nested field.** The native scan fails on a data file that + lacks an optional struct field, which a write with `write.spark.accept-any-schema=true` can + produce without any schema change + ([#7033](https://github.com/apache/datafusion-comet/issues/7033)). +- **`VERSION AS OF` reads of a column whose type was promoted after the snapshot** + ([#6993](https://github.com/apache/datafusion-comet/issues/6993)). +- **An equality delete keyed on a nested field.** Every query the native scan runs on the table + fails with `field not found`, even one that reads the key + ([#6782](https://github.com/apache/datafusion-comet/issues/6782)). +- **Column names that only newer Unicode case data folds together**, such as a pair of Vithkuqi + letters, which Unicode 14 added. Queries on such a table fail with a duplicate-field error + ([#5540](https://github.com/apache/datafusion-comet/issues/5540)). + +### Object store and credential settings + +The native Parquet and Iceberg scans resolve storage settings with their own clients rather than +with Hadoop's. Where the two disagree, a native read can use a different identity or route than +Spark, and fail where Spark succeeds: + +- **S3 settings that Hadoop's S3A client resolves differently**, such as JCEKS-backed secrets and + the precedence of per-bucket settings. The native Parquet scan forwards them without checking + for a difference ([#5662](https://github.com/apache/datafusion-comet/issues/5662)). +- **AWS profile credential providers.** The native scans ignore `fs.s3a.auth.profile.name` and + `fs.s3a.auth.profile.file` ([#4245](https://github.com/apache/datafusion-comet/issues/4245)), + and resolve some profiles differently from Hadoop 3.4 and later, for example profiles that use + `credential_process` or chains of assumed roles + ([#6575](https://github.com/apache/datafusion-comet/issues/6575)). +- **Azure credentials.** Container-scoped OAuth settings, Fabric hosts and a few other + configurations resolve differently from Hadoop's ABFS driver + ([#6605](https://github.com/apache/datafusion-comet/issues/6605)). +- **`fs.comet.libhdfs.schemes` set only in the Hadoop configuration.** Comet's planner and its + native scan read the setting from different places, so they can route a scheme differently + ([#6837](https://github.com/apache/datafusion-comet/issues/6837)). +- **Iceberg tables whose metadata location has no host.** The native Iceberg scan skips the + credential provider configured with `s3.comet.credential.provider.class` + ([#6536](https://github.com/apache/datafusion-comet/issues/6536)). + +### Parquet writes that fail + +With `spark.comet.write.parquet.enabled=true`: + +- **Row groups that outgrow the memory pool.** The native writer caps row groups by row count Review Comment: Two points. First, `data-source-audits/parquet.md` records that this failure did not reproduce (a 239 MB row group fit in a 256 MB pool), yet this page lists it under failures. Suggested wording: > **Row groups that outgrow the memory pool.** The native writer cuts row groups by row count only, where Spark uses `parquet.block.size`, and it cannot spill. With wide or nested rows, the open row group can outgrow the off-heap pool and fail the task. The audit did not reproduce a failure, because a 239 MB row group fit in a 256 MB pool (#5304). Second, #5304 is mainly about the writer ignoring Spark's Parquet writer properties. That part is missing from "Parquet writes" under "Results that differ from Spark". Suggested entry there: > **Spark's Parquet writer properties are ignored.** `parquet.block.size`, page size, dictionary, writer version and statistics settings never reach the native writer (#5304). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
