jianguotian opened a new pull request, #76: URL: https://github.com/apache/paimon-mosaic/pull/76
## Summary Add `visit_encoded_columns` for reading non-nested columns directly from Mosaic row-group data without first creating Arrow arrays. The new API: - exposes the column encoding, row count, null bitmap, and `CONST` value; - provides access to column values; - supports `ALL_NULL`, `CONST`, `DICT`, and `PLAIN`; - works with both monolithic and paged buckets. The existing `read_columns()` path and projection behavior are unchanged. List and Map columns are not supported by this API. This adds a public Core API. It does not change the file format or existing readers. ## Performance Tested with a customer sample containing: - 9,837 rows and 8,690 columns; - 88.72% `CONST` or `ALL_NULL` columns; - 100 monolithic buckets using Zstd. Release build, three independent runs with 11 iterations each, reporting the median: | Path | Processing time | Allocated memory | |---|---:|---:| | Existing `read_columns()` path | 155.97 ms | 261.1 MiB | | New `visit_encoded_columns()` path | 81.00 ms | 8.5 KiB | The new API was used by reading `ALL_NULL` and `CONST` columns directly and iterating the values of `DICT` and `PLAIN` columns. It reduced column processing time by 48.1% and avoided creating temporary Arrow arrays for this sample. This benchmark measures column processing after the row group has been loaded. It does not include file I/O, bucket decompression, JSON generation, or final output compression. ## Tests - Compare values returned by the new API with `read_columns()`. - Cover all supported non-nested column types and encodings. - Cover monolithic and paged buckets and projection behavior. - Verify errors for invalid data and unsupported nested columns. ## Validation - `cargo +1.97.1 fmt --all -- --check` - `cargo +1.97.1 test -p paimon-mosaic-core` — 371 passed - `cargo +1.97.1 clippy --all-targets --workspace -- -D warnings` - `cargo +1.86 check -p paimon-mosaic-core --all-targets` - `cargo deny check licenses` - `python3 tools/validate_asf_yaml.py` - `git diff --check` ## Scope This PR adds the Core API and tests. Consumer changes are handled separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
