ryanvickr opened a new pull request, #55572: URL: https://github.com/apache/arrow/pull/55572
### Rationale for this change When dictionary encoding is enabled on a column (`enable_dictionary()`, the default), `WriterProperties::Builder::encoding(...)` rejects `PLAIN_DICTIONARY` and `RLE_DICTIONARY` with `"Can't use dictionary encoding as fallback encoding"`, indicating that the configured encoding is intended as the fallback encoding. However: 1. `ColumnWriter::Make` overwrote `encoding` with `properties->dictionary_index_encoding()` when `use_dictionary` was true without preserving the resolved non-dictionary encoding, and `TypedColumnWriterImpl::FallbackToPlainEncoding()` hardcoded `Encoding::PLAIN` (see also #40636). As a result, configuring e.g. `DELTA_LENGTH_BYTE_ARRAY`, `DELTA_BYTE_ARRAY`, `DELTA_BINARY_PACKED`, or `BYTE_STREAM_SPLIT` on a dictionary-enabled column was ignored whenever the column overflowed `dictionary_pagesize_limit` or fell back on varying `arrow::DictionaryArray` chunks. 2. Incompatible `(physical_type, encoding)` pairs on dictionary-enabled columns were not validated upfront on `ColumnWriter` creation, only failing at runtime if the column happened to overflow its dictionary. 3. `CheckDictionarySizeLimit()` was called after every batch in `WriteBatchInternal`, `WriteBatchSpacedInternal`, and `TypedColumnWriterImpl<ByteArrayType>::WriteArrowDense` even when `check_page == false` (mid-record on a repeated column when `pages_change_on_record_boundaries()` is active), which could flush buffered dictionary pages mid-record. ### What changes are included in this PR? - Preserve `fallback_encoding` in `ColumnWriter::Make` before switching `encoding` to `properties->dictionary_index_encoding()`, and use `fallback_encoding_` in `TypedColumnWriterImpl::FallbackToPlainEncoding()`. - Validate `fallback_encoding` upfront in `TypedColumnWriterImpl` when `use_dictionary` is true so unsupported `(physical_type, fallback_encoding)` combinations throw `ParquetException` immediately upon `ColumnWriter` creation. - Gate `CheckDictionarySizeLimit()` on `check_page || descr_->max_repetition_level() == 0 || rep_levels == nullptr` in `WriteBatchInternal`, `WriteBatchSpacedInternal`, and `TypedColumnWriterImpl<ByteArrayType>::WriteArrowDense` so dictionary fallback on repeated columns only flushes at record boundaries when `pages_change_on_record_boundaries()` is true. - Update `WriterProperties::Builder::encoding(...)` docstrings in `cpp/src/parquet/properties.h`. ### Are these changes tested? Yes, added unit tests in `cpp/src/parquet/column_writer_test.cc` and `cpp/src/parquet/properties_test.cc` covering: - `PARQUET_1_0` (`PLAIN_DICTIONARY` -> `PLAIN`) and `PARQUET_2_6` (`RLE_DICTIONARY` -> configured fallback encoding) dictionary page and data page encodings. - Dictionary fallback to `DELTA_LENGTH_BYTE_ARRAY`, `DELTA_BYTE_ARRAY`, `DELTA_BINARY_PACKED`, and `BYTE_STREAM_SPLIT` across `WriteTable` (including varying `DictionaryArray` chunks and `SizeStatistics`), `WriteBatch`, and `WriteBatchSpaced`. - Repeated `list<utf8>` dictionary overflow across mid-record batch splits with `enable_write_page_index()` verifying record-aligned `OffsetIndex` page locations. - Upfront rejection of incompatible fallback encodings on dictionary-enabled columns. ### Are there any user-facing changes? Yes: columns with both dictionary encoding enabled and an explicit non-dictionary `WriterProperties::Builder::encoding(...)` now fall back to that configured encoding instead of `Encoding::PLAIN` when the dictionary page size limit is reached, and incompatible fallback encodings on dictionary-enabled columns now fail immediately when creating the `ColumnWriter`. * GitHub Issue: #55571 --- ### AI Transparency Statement - [x] I have used AI tools to help generate this pull request. - [x] I have reviewed, understood, and verified all AI-generated code and text in this pull request, and I take full responsibility for the contribution. - Tool(s) used: Gemini - How it was used: Used to help draft unit tests and the PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
