sundapeng opened a new pull request, #864: URL: https://github.com/apache/paimon-rust/pull/864
### Purpose apache/paimon-rust#851 added Mosaic row-group prefetch with the depth and decoded-byte budget hardcoded to Java's defaults (8 row groups, 64 MiB), because the format-reader factory did not receive format-scoped table options. This change reads the same two table options Java uses, so the setting a table carries applies to both engines: - `mosaic.read.prefetch-row-groups` (default 8; `0` reads row groups on demand) - `mosaic.read.prefetch-max-bytes` (default 64 MiB; estimated decoded bytes allowed in flight beyond the head row group) Both are parsed in `CoreOptions` next to the Parquet row-group budget options and reach `MosaicFormatReader` through `DataFileReader`, so every read path that opens Mosaic files (plain append, key-value merge, data evolution, format tables, audit log) honours them. ### Brief change log - `CoreOptions::mosaic_read_prefetch_row_groups()` and `mosaic_read_prefetch_max_bytes()` with the same validation style as `read.parquet.row-group.*` (`0` is allowed for the depth, the byte budget must be positive). - `MosaicPrefetchOptions` on `MosaicFormatReader`; `create_format_reader_with_budget` takes it; `DataFileReader::with_mosaic_prefetch`; `configured_mosaic_prefetch(table)` plumbs it from the table schema in `TableRead`, `KeyValueReadConfig`, `DataEvolutionReader`, `FormatTableRead` and the audit-log read. - No behaviour change without the options: the defaults equal the values #851 hardcoded. ### Tests - `spec::core_options::tests::test_mosaic_read_prefetch_options`: defaults, custom values, invalid values. - `arrow::format::mosaic::tests::test_format_reader_prefetch_options_control_look_ahead`: depth `0` reads on demand (one storage read in flight), the default overlaps reads, both return the same rows. - `table::table_read::tests::test_mosaic_prefetch_options_come_from_table_options`: the values come from the table options and invalid values fail the read. - Ran `cargo fmt --all -- --check`, `cargo clippy --locked -p paimon --all-targets --features fulltext,vortex -- -D warnings`, and `cargo test --locked -p paimon --lib` for the `spec::core_options`, `arrow::format::mosaic`, `table::data_file_reader`, `table::kv_file_reader`, `table::data_evolution_reader`, `table::format_table_read`, `table::audit_log_table` and `table::table_read` modules. Measured on a StarRocks-derived engine reading an 11,374-column Mosaic table through the C bindings: with #851 the Rust scan of the same 54 splits went from 7.1 s to 3.4 s (the Java reader with the same prefetch reads them in 1.2 s), so exposing the knobs lets a table tune depth and budget per workload. ### API and Format Two new table options, `mosaic.read.prefetch-row-groups` and `mosaic.read.prefetch-max-bytes`, matching the keys and defaults of the Java implementation. No format change. ### Documentation None; the option keys mirror `MosaicFileFormat` in Apache Paimon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
