Rich-T-kid opened a new issue, #26530:
URL: https://github.com/apache/datafusion/issues/26530
### Problem
When both of the following Parquet options are enabled:
```text
datafusion.execution.parquet.binary_as_string = true
datafusion.execution.parquet.enable_rle_to_dictionary = true
```
an unannotated Parquet `BYTE_ARRAY` column with dictionary pages is exposed
by
the table schema as `Dictionary<Int32, Utf8>`, but the physical per-file
schema
still has binary values.
The per-file schema-coercion path does not preserve the physical dictionary
value type while requesting the logical string dictionary. The reader
therefore expands the Parquet dictionary to a plain binary array, after which
`DecoderProjection` casts the values to strings and rebuilds a dictionary by
hashing rows.
The effective path is:
```text
Parquet RLE dictionary
-> plain Binary rows
-> Utf8 rows
-> rebuild Dictionary<Int32, Utf8>
```
This is not a missing Arrow cast capability. Arrow can represent and cast
`Dictionary<Int32, Binary>` to `Dictionary<Int32, Utf8>` without first
expanding the dictionary.
### Expected behavior
For this combination of options, DataFusion should ask the Parquet reader for
`Dictionary<K, Binary>`, cast the distinct dictionary values to `Utf8` once,
and preserve the keys:
```text
Parquet RLE dictionary
-> Dictionary<Int32, Binary>
-> Dictionary<Int32, Utf8>
```
When the Parquet reader shares a dictionary values buffer across output
batches, the cast values should also be reusable across those batches.
### Impact
A 1,000,000-row scan benchmark over an unannotated binary column showed the
following results on a Neoverse-V2 benchmark runner:
| NDV | Current path | Dictionary-preserving path | Speedup |
|---:|---:|---:|---:|
| 1 | 18.4 ms | 492 us | 37.4x |
| 10 | 18.7 ms | 785 us | 23.9x |
| 100 | 18.8 ms | 875 us | 21.5x |
| 1,000 | 20.6 ms | 991 us | 20.8x |
| 10,000 | 34.5 ms | 1.194 ms | 28.9x |
| 100,000 | 49.5 ms | 2.9 ms | 16.9x |
| 500,000 | 57.4 ms | 7.1 ms | 8.1x |
The annotated UTF-8 control is unchanged, which isolates the regression to
`binary_as_string` and per-file schema coercion rather than general
dictionary
decoding.
Benchmark result:
https://github.com/apache/datafusion/pull/26250#issuecomment-6280763379
### Reproduction and proposed regression coverage
1. Write an unannotated Arrow `Binary` column to Parquet with dictionary
encoding enabled.
2. Configure a small output batch size so the same Parquet dictionary is used
by multiple batches.
3. Enable `binary_as_string` and `enable_rle_to_dictionary`.
4. Scan through the normal per-file opener path.
5. Verify the output is `Dictionary<Int32, Utf8>`, its keys are preserved,
and
the converted values buffer is reused across batches.
PR #26250 contains a benchmark, a per-file opener regression, and a candidate
fix: https://github.com/apache/datafusion/pull/26250
This issue is intentionally limited to the reader/schema-coercion expansion.
Downstream dictionary hashing, grouping, and repartitioning costs are
separate
performance concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
