sunchao commented on issue #25433: URL: https://github.com/apache/datafusion/issues/25433#issuecomment-7051344107
**The malformed file is the result of a bug in Spark 4.1.3’s Parquet write path. Spark accepts duplicate nested field names and writes six values into each child chunk where each should contain exactly three.** On the file in #25433, **adding `SUM` to the `SELECT` list changes the result of `COUNT(*)`**. I reproduced both queries on DataFusion 55.2.0. ```sql SELECT count(*) FROM t; -- count = 3 SELECT count(*), sum(s['dup']) FROM t; -- count = 6, sum = 306 ``` Both queries count the same table with no filter. Adding `SUM` forces DataFusion to read the values. That exposes a contradiction between the row count Spark recorded and the data Spark actually wrote. [Issue and fixture](https://github.com/apache/datafusion/issues/25433) ### How three rows became six Spark started with three rows. Each row contained a struct with two integer fields, both named `dup`. The first row was `(0, 100)`, the second was `(1, 101)`, and the third was `(2, 102)`. Parquet stores each child of a struct in its own column chunk. **Six integers across two chunks is correct. This file has six integers in each chunk.** I decoded both chunks from the attachment. | Physical column | Expected values | Actual values on disk | | ------------------- | --------------- | ------------------------ | | First child of `s` | `0, 1, 2` | `0, 100, 1, 101, 2, 102` | | Second child of `s` | `100, 101, 102` | `0, 100, 1, 101, 2, 102` | The corruption starts on the **first write**. Spark distinguishes the two children by position, so it emits `0` for the first child and `100` for the second. Parquet-java looks up the destination writer by field path. Both lookups resolve to `s.dup`, so both values go into the same buffer. After that first row, the row counter is `1` and the buffer already contains `[0, 100]`. After three rows, the counter is `3` and the buffer contains all six integers. Flushing writes that same sequence to both child chunks. [Spark’s field write loop](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala#L172-L188), [parquet-java’s writer lookup](https://github.com/apache/parquet-java/blob/apache-parquet-1.16.0/parquet-column/src/main/java/org/apache/parquet/column/impl/ColumnWriteStoreBase.java#L96-L121), [chunk flush](https://github.com/apache/parquet-java/blob/apache-parquet-1.16.0/parquet-hadoop/sr c/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java#L691-L695) ### Why DataFusion gives two answers - **`COUNT(*)` alone** can be answered entirely from the footer. DataFusion reads `row_group.num_rows = 3`, marks it exact, and replaces the aggregate and scan with the literal `3`. [Footer statistics](https://github.com/apache/datafusion/blob/55.0.0/datafusion/datasource-parquet/src/metadata.rs#L449-L464), [aggregate rewrite](https://github.com/apache/datafusion/blob/55.0.0/datafusion/physical-optimizer/src/aggregate_statistics.rs#L57-L94) - **`COUNT(*)` alongside `SUM(s['dup'])`** needs a scan because the footer cannot supply the sum. Both aggregates then consume the same decoded batches. The child is a required scalar, so the decoder treats each successive integer as belonging to the next row. It emits `0` as row one and `100` as row two, even though those values originally belonged to the **same** row. Reading just one child is enough to produce six rows. [Parquet record decoding](https://github.com/apache/arrow-rs/blob/59.2.0/parquet/src/column/reader.rs#L235-L265) **The writer corrupted the field boundaries, and the reader has enough information to detect it.** The footer itself says “three rows” and “six values in this required scalar child.” Rejecting that contradiction when loading the footer would make both queries report the same corrupt-file error. Stopping the scan after three values, as Spark and DuckDB do on this fixture, yields `[0, 100, 1]`. The count looks right, but the values still belong to the wrong rows. The check applies to leaves with no repeated ancestor, since a list can legitimately contribute several elements per row. [Reader comparison](https://github.com/apache/datafusion/issues/25433), [Parquet nested encoding](https://parquet.apache.org/docs/file-format/nestedencoding/) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
