sunchao commented on issue #25433:
URL: https://github.com/apache/datafusion/issues/25433#issuecomment-7051344107

   **The malformed file is the result of a bug in Spark 4.1.3’s Parquet write 
path. Spark accepts duplicate nested field names and writes six values into 
each child chunk where each should contain exactly three.**
   
   On the file in #25433, **adding `SUM` to the `SELECT` list changes the 
result of `COUNT(*)`**. I reproduced both queries on DataFusion 55.2.0.
   ```sql
   SELECT count(*) FROM t;                  -- count = 3
   SELECT count(*), sum(s['dup']) FROM t;   -- count = 6, sum = 306
   ```
   
   Both queries count the same table with no filter. Adding `SUM` forces 
DataFusion to read the values. That exposes a contradiction between the row 
count Spark recorded and the data Spark actually wrote. [Issue and 
fixture](https://github.com/apache/datafusion/issues/25433)
   
   ### How three rows became six
   
   Spark started with three rows. Each row contained a struct with two integer 
fields, both named `dup`. The first row was `(0, 100)`, the second was `(1, 
101)`, and the third was `(2, 102)`. Parquet stores each child of a struct in 
its own column chunk. **Six integers across two chunks is correct. This file 
has six integers in each chunk.** I decoded both chunks from the attachment.
   
   | Physical column     | Expected values | Actual values on disk    |
   | ------------------- | --------------- | ------------------------ |
   | First child of `s`  | `0, 1, 2`       | `0, 100, 1, 101, 2, 102` |
   | Second child of `s` | `100, 101, 102` | `0, 100, 1, 101, 2, 102` |
   
   The corruption starts on the **first write**. Spark distinguishes the two 
children by position, so it emits `0` for the first child and `100` for the 
second. Parquet-java looks up the destination writer by field path. Both 
lookups resolve to `s.dup`, so both values go into the same buffer. After that 
first row, the row counter is `1` and the buffer already contains `[0, 100]`. 
After three rows, the counter is `3` and the buffer contains all six integers. 
Flushing writes that same sequence to both child chunks. [Spark’s field write 
loop](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala#L172-L188),
 [parquet-java’s writer 
lookup](https://github.com/apache/parquet-java/blob/apache-parquet-1.16.0/parquet-column/src/main/java/org/apache/parquet/column/impl/ColumnWriteStoreBase.java#L96-L121),
 [chunk 
flush](https://github.com/apache/parquet-java/blob/apache-parquet-1.16.0/parquet-hadoop/sr
 c/main/java/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java#L691-L695)
   
   ### Why DataFusion gives two answers
   
   - **`COUNT(*)` alone** can be answered entirely from the footer. DataFusion 
reads `row_group.num_rows = 3`, marks it exact, and replaces the aggregate and 
scan with the literal `3`. [Footer 
statistics](https://github.com/apache/datafusion/blob/55.0.0/datafusion/datasource-parquet/src/metadata.rs#L449-L464),
 [aggregate 
rewrite](https://github.com/apache/datafusion/blob/55.0.0/datafusion/physical-optimizer/src/aggregate_statistics.rs#L57-L94)
   - **`COUNT(*)` alongside `SUM(s['dup'])`** needs a scan because the footer 
cannot supply the sum. Both aggregates then consume the same decoded batches. 
The child is a required scalar, so the decoder treats each successive integer 
as belonging to the next row. It emits `0` as row one and `100` as row two, 
even though those values originally belonged to the **same** row. Reading just 
one child is enough to produce six rows. [Parquet record 
decoding](https://github.com/apache/arrow-rs/blob/59.2.0/parquet/src/column/reader.rs#L235-L265)
   
   **The writer corrupted the field boundaries, and the reader has enough 
information to detect it.** The footer itself says “three rows” and “six values 
in this required scalar child.” Rejecting that contradiction when loading the 
footer would make both queries report the same corrupt-file error. Stopping the 
scan after three values, as Spark and DuckDB do on this fixture, yields `[0, 
100, 1]`. The count looks right, but the values still belong to the wrong rows. 
The check applies to leaves with no repeated ancestor, since a list can 
legitimately contribute several elements per row. [Reader 
comparison](https://github.com/apache/datafusion/issues/25433), [Parquet nested 
encoding](https://parquet.apache.org/docs/file-format/nestedencoding/)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to