andygrove opened a new issue, #7040: URL: https://github.com/apache/datafusion-comet/issues/7040
### What is the problem the feature request solves? Nested types (structs, arrays and maps) in the native Parquet scan and the native Iceberg scan were audited on `main` at 66c63bd307, with a focus on NULL structs, arrays and maps, NULLs inside them, Parquet field ids and Iceberg name mappings. About 3,000 queries were compared against Spark with Comet disabled: Parquet on Spark 3.4.3, 3.5.9 and 4.1.3, and Iceberg 1.11 on Spark 4.1.3. A code review of DataFusion 55.2, arrow-rs 59.3 and iceberg-rust at the current pin ran alongside, and each finding below was reproduced before it was listed. This epic collects the new issues and the related ones that were already open, so they can be worked and tracked in one place. Items are listed by priority within each section. ### Correctness: native Parquet scan - [ ] #7029 **INT96 timestamps inside legacy-format lists read wrong** outside 1677 to 2262 (`writeLegacyFormat=true`, Hive `bag` / `array_element`, two-level lists). DataFusion's `Int96Coercer` assumes a `list` group. - [ ] #7030 **A requested nested column with no field in the file reads as a non-null array or struct of NULLs** where Spark returns NULL. Default behavior on Spark 3.4 to 4.0, legacy mode on 4.1+. - [ ] #7032 **Nested BINARY values that are not valid UTF-8 read as NULL** when read as STRING. Top-level values bypass UTF-8 validation. Part of #4764. - [ ] #7031 **Spark 3.4 only: a struct matched by field id under a new name reads as NULL** when Spark prunes its fields. - [ ] #5553 and #5739: checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans. - [ ] #6994: nested field-id reads give one file column to both an id match and a name match. - [ ] #7006: a duplicate field id in a column the query does not read fails the scan. - [ ] #5884 and #5964: Spark-compatible resolution of duplicate Parquet field names. - [ ] #4121: STRING columns holding invalid UTF-8 are rejected (part of #4764). - [ ] #7034: `TIMESTAMP(NANOS)` read as `timestamp` returns values where Spark rejects the read. - [ ] #7035: a file with a nested encoding arrow-rs rejects cannot be read at all, even for other columns. ### Correctness: native Iceberg scan - [ ] #6790 **Name-mapped (migrated) tables read every nested column as NULL**: struct fields, list elements and map values, including whole `array<int>` and `map<string,int>` columns, so nested filters return the wrong rows. - [ ] #6794 **Files with neither field ids nor a name mapping return nested values that Spark reads as NULL.** - [ ] #6822 **`VERSION AS OF` returns wrong values for a column renamed after the snapshot.** - [ ] #7033: a data file that omits an optional nested field fails the scan, with no schema change involved. - [ ] #6993: `VERSION AS OF` fails for a column whose type was promoted after the snapshot. - [ ] #6782: an equality delete keyed on a nested field fails with `field not found`. - [ ] #5540: column names that are case-distinct to Java but not to Rust fail the scan. - [ ] #5257: the schema and metadata reflection helpers return empty collections on failure. - [ ] #6934: bump iceberg-rust past apache/iceberg-rust#3255 and narrow the nested schema evolution fallback from #6543. The bump also fixes #6790, and by code reading should fix #7033. ### Performance: native Parquet scan - [ ] #7036: the whole nested column is read for maps inside structs, map value structs, case-insensitive names, field-id reads, requested fields missing from the file, and leaf type changes (TIMESTAMP_MILLIS, widening, LTZ to NTZ and others). The remaining shapes after #4859. - [ ] #7037: no row-group or page pruning with predicates on struct fields (apache/datafusion#20871). - [ ] #5355: scan plus shuffle write is slower than Spark on deeply nested schemas. ### Performance: native Iceberg scan - [ ] #6724: every field of a nested column is read even when Spark prunes it (fix in #6725). - [ ] #7019: the full schema is read when a partition source is nested in a struct the query prunes. - [ ] #7038: residual predicates on struct fields are never pushed into iceberg-rust. - [ ] #5883: an unsupported complex null conjunct drops the pruning of the whole residual. - [ ] #7039: a file is read in full when the query needs none of its columns (`count(*)` with deletes, late-added columns). Not nested-specific. - [ ] #6921: faster nested Parquet decoding in parquet-rs. ### Cleanup - [ ] #6978: share field-id and name resolution between the root and nested fields (#6990). - [ ] #5413: empty struct columns fall back to Spark. ### Related epics - #4764: invalid UTF-8 in native `StringType` (ingress policy). - #6855: nested-type conversion performance in C2R, R2C and JVM shuffle. - #5438: native Variant support, including Variant nested in structs, arrays and maps (#5435). - #3736: native_datafusion scan improvements. ### Describe the potential solution Each item is a linked issue. Check it here when it closes. ### Additional context **Already fixed, for reference:** #4136, #4859, #5707, #6131, #6136, #6192, #6506, and #6504 and #6546 (fallback in #6543). **Differences that are not Comet bugs**, listed so they are not refiled: - On an Iceberg table identity-partitioned by a struct field, iceberg-java before 1.12 reads a NULL struct as a struct of NULLs when the partition source is the struct's first field, because its constant reader reports a fixed definition level. apache/iceberg#17320 fixed it in 1.12.0. The native scan returns NULL, which matches 1.12. Comet's Iceberg profiles pin 1.5.2 to 1.11.0, so a comparison against Spark shows the difference there. - With `spark.sql.parquet.fieldId.read.enabled=true` and a read schema that swaps two nested field names, Spark's vectorized reader pairs the clipped Parquet columns with the read fields by the file's names, so it fails with a type mismatch or returns no rows. Comet resolves the fields by id. - On Spark 3.5 and later, nested schema pruning drops a root field id, so Spark itself reads a renamed root as NULL. Comet matches it there. #7031 covers 3.4, where Spark keeps the id. - A map with NULL keys from a non-Spark writer fails in both engines. **What matched Spark**: NULL structs, arrays and maps, empty arrays and maps, and NULLs inside them at every level, including deep (eight levels) and wide (300 fields) structs. Spark-written files in five layouts (default, tiny row groups and pages, the legacy format, v2 pages without dictionaries, v2 tiny pages), with batch sizes 1 to 3, row-filter pushdown on and off, and filter pushdown off. Avro, Thrift, Hive and protobuf style list and map encodings, maps with struct and array keys, nested type promotion, nested timestamp, decimal, boolean and binary leaves (micros, millis, and INT96 in standard layouts), ENUM, JSON, BSON, unsigned and binary-decimal annotations, files written by arrow-rs, partitioned tables, nested schemas that differ across the files of one scan, case-insensitive nested names, nested field ids (renamed, missing, partly present), and both SPARK-53535 modes on 4.1. For Iceberg, the same data written by iceberg-java, merge-on-read position deletes across three -row pages (format v2 and v3 deletion vectors), equality deletes including a NULL key, required nested fields under NULL parents, nested UUID, fixed, timestamptz and timestamp_ntz leaves, nested type promotion inside structs, lists and maps, the nested evolution fallback for lists and maps, `_partition` with NULL values and spec evolution, and nested partition sources. **Not covered:** Variant (#5438), Delta, ORC, the V2 Parquet source, encrypted files and the JVM-side conversion paths. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
