rustyconover opened a new issue, #15042:
URL: https://github.com/apache/arrow/issues/15042

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   When writing a column that is a 
   
   `pa.dictionary(pa.int32(), pa.string())` to a Parquet file incorrect 
statistics are produced.  I believe the statistics are calculated from the 
contents of the first chunk rather than all chunks.  Since the CSV parser 
produces chunks that may not contain all dictionary rows, this results in 
incorrect statistics to be produced.
   
   A test case is below that demonstrates the problem without using the CSV 
parser:
   
   ```python
   import pyarrow as pa
   import pyarrow.parquet as pq
   
   schema = pa.schema({"field_1": pa.dictionary(pa.int32(), pa.string())})
   
   # The ordering of the values don't matter, but they must
   # be stored in seperate chunks (as they would be if they are parsed from CSV)
   arr_1 = pa.array(["rusty", "sean", "aa"]).dictionary_encode()
   arr_2 = pa.array(["zzz", "frank"]).dictionary_encode()
   
   t = pa.Table.from_batches(
       [
           pa.record_batch([arr_1], names=["field_1"]),
           pa.record_batch([arr_2], names=["field_1"]),
       ]
   )
   
   # If this is commented the bug does not occur.
   t = t.unify_dictionaries()
   
   with pq.ParquetWriter("example.parquet", schema) as writer:
       writer.write_table(t)
   
   # The Parquet stats will be:
   # ┌─────────────────┬─────────────────┐
   # │ stats_min_value │ stats_max_value │
   # ├─────────────────┼─────────────────┤
   # │ aa              │ sean            │
   # └─────────────────┴─────────────────┘
   #
   # The stats should be:
   #
   # ┌─────────────────┬─────────────────┐
   # │ stats_min_value │ stats_max_value │
   # ├─────────────────┼─────────────────┤
   # │ aa              │ zzz             │
   # └─────────────────┴─────────────────┘
   #
   ```
   
   Workaround: if you call
   
   `t = t.combine_chunks()`
   
   Before calling `write_table()` the proper column statistics are written.
   
   The difficulty with having improper column statistics is that query engines 
(Athena, Trino) use column statistics to create predicate pushdowns as part of 
their query execution.  If these query plans are incorrect resulting in data 
that exists in the Parquet file not being returned as part of the query result.
   
   ### Component(s)
   
   C++, Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to