sundapeng opened a new pull request, #1072:
URL: https://github.com/apache/paimon-rust/pull/1072
### Purpose
Java Paimon passes `parquet.*` table options to parquet-java, so users size
row groups and pages with `parquet.block.size`, `parquet.page.size`,
`parquet.page.row.count.limit` and friends, and `file.block-size` overrides the
row group size (`ParquetFileFormat#getParquetConfiguration`). The native writer
ignores all of them: every file gets the parquet-rs defaults, one row group of
up to 1,048,576 rows and page limits checked only every 1,024 rows.
For tables with wide rows or wide shredded schemas this decides how much a
point read fetches. Example: a VARIANT column with about 5,500 keys, 200 of
them shredded into DOUBLE. Reading 84 consecutive rows needs only the 200 typed
leaves, but each typed leaf spans the whole row group, and nearby ranges are
coalesced. With one row group per file, a batch of 16 such reads fetched 37.7
MB; with `parquet.block.size` set to 32 MB it fetched 23.5 MB, with identical
output.
### Brief change log
- `parquet_writer_properties` maps the keys Java reads from its Hadoop
configuration:
- `parquet.block.size` -> `max_row_group_bytes`; `file.block-size`
(MemorySize) takes precedence, as in Java;
- `parquet.page.size` -> `data_page_size_limit`;
- `parquet.dictionary.page.size` -> `dictionary_page_size_limit`;
- `parquet.page.row.count.limit` -> `data_page_row_count_limit`;
- `parquet.page.size.row.check.min` -> `write_batch_size`, the interval at
which parquet-rs checks page limits. It is capped at the page row limit,
otherwise that limit could not take effect.
- Values are positive integers like Java's `Configuration#getLong`/`getInt`;
invalid values fail with `ConfigInvalid` before the file is created. Unset keys
keep today's defaults.
- Document the options next to the dictionary options in
`getting-started.md`.
### Tests
- New tests: the properties reflect each option, `file.block-size` wins over
`parquet.block.size`, and the row check bounds the write batch; invalid values
for every key create no file; a 4,000-row write with `parquet.block.size=16384`
and `parquet.page.row.count.limit=100` produces several row groups, each within
the byte limit, whose offset indexes show pages within one write batch of the
row limit.
- `cargo test -p paimon --lib` (3,785 passed), `cargo fmt --all -- --check`,
`cargo clippy --locked --all-targets --workspace --features fulltext,vortex --
-D warnings`.
### API and Format
No API change. Existing tables are unaffected unless they already set these
options, in which case new files follow them.
### Documentation
Added a "Parquet Row Groups and Pages" section to
`docs/src/getting-started.md`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
