sundapeng commented on PR #864:
URL: https://github.com/apache/paimon-rust/pull/864#issuecomment-5731192120
Done - the budget is now one type, `ReadBudget` in `arrow/read_budget.rs`,
and both formats take it.
I kept the two option pairs separate, because they are not the same resource:
| | `read.parquet.row-group.*`
| `mosaic.read.prefetch-*` |
| -------------- |
------------------------------------------------------------- |
---------------------------------------------------------- |
| defined by | paimon-rust only (#633)
| Java `MosaicFileFormat` (apache/paimon#9740), 8 / 64 MiB |
| scope | one budget per `TableRead`, shared by every file of the
scan | one budget per open file |
| bytes | footer *uncompressed* size of the projected chunks
| *estimated decoded* Arrow size of the row group |
| value `0` | rejected (`1` disables concurrency)
| valid, "read row groups on demand" - Java's semantics |
| active | only on the predicate-free, selection-free path
| always |
| acquired from | async tasks
| the blocking Mosaic decode threads |
The Mosaic keys are Java's keys with Java's defaults, so a table tuned on
the Java side has to read the same
way here - that is what this PR fixes. If Rust honoured only a generic key,
a Java-tuned table would
silently read differently.
So the sharing is in the mechanism: `ParquetReadBudget` is now `ReadBudget`,
it grew `try_acquire` and
`acquire_blocking` next to the async `acquire` (the Mosaic reader decodes on
its own threads, so it cannot
await), plus a byte granularity - Parquet keeps MiB permits for a scan-wide
budget, Mosaic charges the exact
estimated bytes of a row group. Mosaic's private `MosaicPrefetchBudget` and
its permit are deleted; the
reader builds a per-file `ReadBudget` from the two Java options.
`paimon::arrow::ParquetReadBudget` is
renamed accordingly, no other public signature changes, and no behaviour
change on either format.
The branch is also rebased on main (it had conflicted with the BLOB read
parallelism change).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
