brgr-s opened a new issue, #10524:
URL: https://github.com/apache/arrow-rs/issues/10524
### Describe the bug
`WriterProperties::set_dictionary_enabled(true)` and
`set_column_dictionary_enabled(col, true)` are both silently discarded for
`FIXED_LEN_BYTE_ARRAY` columns when the writer version is `PARQUET_1_0`.
`parquet/src/column/writer/encoder.rs`:
```
let dict_supported = props.dictionary_enabled(descr.path())
&& has_dictionary_support(T::get_physical_type(), props);
```
`parquet/src/column/writer/mod.rs`
```
fn has_dictionary_support(kind: Type, props: &WriterProperties) -> bool {
match (kind, props.writer_version()) {
(Type::BOOLEAN, _) => false,
(Type::FIXED_LEN_BYTE_ARRAY, WriterVersion::PARQUET_1_0) => false,
(Type::FIXED_LEN_BYTE_ARRAY, WriterVersion::PARQUET_2_0) => true,
_ => true,
}
}
```
Please note that I am not asking to change the default; it conforms with
Java and f. ex. Trinos' own writer also implements the same behaviour. I found
one outlier (parquet-cpp), but that is besides the point.
I am just suggesting to either honor the Clients' `set_dictionary_enabled`
request or "fail" at setting the value in a more obvious way.
### To Reproduce
Two columns, same file, 100k rows, 1000 distinct values each, no
compression. `dictionary_enabled` set to `true` both globally and per-column.
```
let props = WriterProperties::builder()
.set_dictionary_enabled(true)
.set_column_dictionary_enabled("d".into(), true)
.set_writer_version(version)
.build();
```
```
PARQUET_1_0 (default)
col d FIXED_LEN_BYTE_ARRAY dict_page=NONE encodings=[PLAIN, RLE]
col i INT64 dict_page=present encodings=[PLAIN, RLE,
RLE_DICTIONARY]
file: 1,734,680 bytes
```
```
PARQUET_2_0
col d FIXED_LEN_BYTE_ARRAY dict_page=present encodings=[PLAIN, RLE,
RLE_DICTIONARY]
col i INT64 dict_page=present encodings=[PLAIN, RLE,
RLE_DICTIONARY]
file: 275,932 bytes
```
### Expected behavior
Either the explicit setting is honoured, or it is rejected with an
error/warning.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
