bhaskargurram-ai opened a new pull request, #11333:
URL: https://github.com/apache/arrow-rs/pull/11333
# Which issue does this PR close?
- Closes #10643.
# Rationale for this change
`parquet-rewrite --encoding` applies one encoding to every column. That
makes type-specific encodings such as ALP (FLOAT and DOUBLE only) hard to use
on files with mixed physical types. An unsupported combination was only
detected when the column writer was created. By then the output file already
existed, and `GenericColumnWriter::new` unwraps the encoder error, so the tool
panicked and left an incomplete output file.
# What changes are included in this PR?
All changes are in `parquet/src/bin/parquet-rewrite.rs`:
- A new repeatable option `--column-encoding <COLUMN>=<ENCODING>`.
`<COLUMN>` is the dot-separated path of a leaf column (as `parquet-schema`
shows it), and `<ENCODING>` takes the same values as `--encoding`. It takes
precedence over `--encoding` for that column, and `--encoding` stays the
default for all other columns. The value is split at the last `=`, so column
names that contain `=` still work.
- Validation before the output file is created. The encodings are checked
against the physical types of the schema the writer will actually use. That
schema comes from `ArrowSchemaConverter` and honours `--coerce-types`, so the
check uses the physical types that are actually written, which can differ from
the input's. The validation reports:
- unknown columns (listing the leaf columns) and columns given more than
once;
- `PLAIN_DICTIONARY` / `RLE_DICTIONARY` (which panic in
`WriterPropertiesBuilder::set_encoding` today) and `BIT_PACKED`;
- unsupported encoding/type pairs, from either option, with messages such
as `--column-encoding for column 'name' (BYTE_ARRAY): ALP supports only FLOAT
and DOUBLE`. For `--encoding`, the message also suggests overriding the column
with `--column-encoding`.
All problems are reported together, as a clap error (exit code 2):
```
$ parquet-rewrite -i bigdelta.parquet -o out.parquet --encoding alp
error: --encoding for column 'int_col' (INT32): ALP supports only FLOAT
and DOUBLE (set another encoding for this column with --column-encoding
'int_col=<ENCODING>')
--encoding for column 'bigint_col' (INT64): ALP supports only FLOAT and
DOUBLE (set another encoding for this column with --column-encoding
'bigint_col=<ENCODING>')
```
- `main` now calls a new `rewrite(args)` function so the rewrite can be
tested end to end. Other failures still panic as before.
The table of supported encodings per physical type is kept in the binary
because the crate has no public API for it yet. #10964 asks for such an API;
once it exists, `check_encoding` could be replaced by it. Until then, a test
keeps the table in sync with the writer (see below).
# Are these changes tested?
Yes. These are unit tests in the binary, following `parquet-fromcsv`:
- parsing of `--column-encoding` values: valid values, case-insensitive
encodings, `=` in column names, and malformed values;
- `test_check_encoding_matches_writer`: for every encoding and every
physical type the `ArrowWriter` can produce, `check_encoding` must agree with
whether the writer actually writes the column (the writer is run, and a panic
counts as unsupported);
- resolution against a schema with nested columns: overrides, global plus
per-column, unknown and duplicate columns, dictionary encodings, and reporting
several problems at once;
- end to end: a file rewritten with `--column-encoding price=alp
--column-encoding id=delta-binary-packed` has those encodings in its column
chunk metadata and reads back unchanged. A rewrite with `--encoding alp` on a
mixed schema returns the error and leaves no output file.
Without the validation of `--encoding`, the last test fails because the
writer panics after the output file has been created.
Commands run: `cargo test -p parquet --features cli --bin parquet-rewrite`
(7 passed), `cargo clippy -p parquet --features cli --all-targets -- -D
warnings` and `cargo fmt --all -- --check`. I also ran the built binary by hand
on `parquet/tests/arrow_reader/bigdelta.parquet`.
# Are there any user-facing changes?
Yes, in the `parquet-rewrite` CLI: the new `--column-encoding` option.
Invalid encoding settings now fail with a clear error before any output is
written, instead of panicking part-way through the rewrite. There are no
library API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
