rustyconover opened a new issue, #15042:
URL: https://github.com/apache/arrow/issues/15042
### Describe the bug, including details regarding any error messages,
version, and platform.
When writing a column that is a
`pa.dictionary(pa.int32(), pa.string())` to a Parquet file incorrect
statistics are produced. I believe the statistics are calculated from the
contents of the first chunk rather than all chunks. Since the CSV parser
produces chunks that may not contain all dictionary rows, this results in
incorrect statistics to be produced.
A test case is below that demonstrates the problem without using the CSV
parser:
```python
import pyarrow as pa
import pyarrow.parquet as pq
schema = pa.schema({"field_1": pa.dictionary(pa.int32(), pa.string())})
# The ordering of the values don't matter, but they must
# be stored in seperate chunks (as they would be if they are parsed from CSV)
arr_1 = pa.array(["rusty", "sean", "aa"]).dictionary_encode()
arr_2 = pa.array(["zzz", "frank"]).dictionary_encode()
t = pa.Table.from_batches(
[
pa.record_batch([arr_1], names=["field_1"]),
pa.record_batch([arr_2], names=["field_1"]),
]
)
# If this is commented the bug does not occur.
t = t.unify_dictionaries()
with pq.ParquetWriter("example.parquet", schema) as writer:
writer.write_table(t)
# The Parquet stats will be:
# ┌─────────────────┬─────────────────┐
# │ stats_min_value │ stats_max_value │
# ├─────────────────┼─────────────────┤
# │ aa │ sean │
# └─────────────────┴─────────────────┘
#
# The stats should be:
#
# ┌─────────────────┬─────────────────┐
# │ stats_min_value │ stats_max_value │
# ├─────────────────┼─────────────────┤
# │ aa │ zzz │
# └─────────────────┴─────────────────┘
#
```
Workaround: if you call
`t = t.combine_chunks()`
Before calling `write_table()` the proper column statistics are written.
The difficulty with having improper column statistics is that query engines
(Athena, Trino) use column statistics to create predicate pushdowns as part of
their query execution. If these query plans are incorrect resulting in data
that exists in the Parquet file not being returned as part of the query result.
### Component(s)
C++, Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]