JingsongLi opened a new pull request, #8365: URL: https://github.com/apache/paimon/pull/8365
## Summary Add opt-in row-store sidecar files for data evolution tables so sparse row-id reads can use `.row` files while normal data files keep their configured columnar format. The implementation covers both Java and Python read/write paths and keeps dedicated blob/vector writers sidecar-free. ## Changes - Add explicit table options for row sidecar generation and sparse-read thresholds. - Reuse the data-file auxiliary writer path to write file indexes and row sidecars from `RowDataFileWriter`. - Route sparse row-id reads to a single `.row` sidecar when the selected row count and ratio are below the configured limits. - Keep dedicated format writers from generating `.row` sidecars, and handle bundle writes through the same row path so sidecars and sequence ranges stay aligned. - Mirror row sidecar write/read support in Python, including row-format reads with schema-evolution missing-field handling. - Update generated option docs and add focused Java/Python tests. ## Testing - [x] `mvn -pl paimon-api,paimon-core -am -Pfast-build -DskipTests compile` - [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=DataFileIndexWriterTest,DataEvolutionSplitReadTest,RollingFileWriterTest,DedicatedFormatRollingFileWriterTest,DataEvolutionTableTest#testRowSidecarDisabledByDefault+testRowSidecarEnabledByTableOption test` - [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=RollingFileWriterTest,DataEvolutionSplitReadTest test` - [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=AppendOnlySimpleTableTest#testBatchRecordsWrite,PrimaryKeySimpleTableTest#testBatchRecordsWrite test` - [x] `python3 -m py_compile paimon-python/pypaimon/common/options/core_options.py paimon-python/pypaimon/read/split_read.py paimon-python/pypaimon/tests/data_evolution_formats_test.py` - [x] `python3 -m pytest paimon-python/pypaimon/tests/data_evolution_formats_test.py -k row_sidecar -q` - [x] `git diff --check` ## Notes Row sidecar generation is disabled by default through `data-evolution.row-sidecar.enabled=false`. The default read-side guardrails are `data-evolution.row-sidecar.max-selected-rows=4096` and `data-evolution.row-sidecar.max-selection-ratio=0.05`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
