lxy-9602 opened a new pull request, #8355: URL: https://github.com/apache/paimon/pull/8355
### Purpose This is a subtask of [MAP shared-shredding support](https://cwiki.apache.org/confluence/display/PAIMON/PIP-43%3A+Columnar+Storage+Optimization+for+MAP+Type+in+Paimon). This PR adds the append write path for MAP shared-shredding. When a MAP column is configured with `map.storage-layout=shared-shredding`, append writers now build a per-file physical write plan, convert logical rows to the shared-shredding physical layout, and write field-level shared-shredding metadata into file metadata before closing the format writer. Main changes: - Add `RowDataFileWritePlan`, `RowDataFileWritePlanFactory`, `RowDataTransform`, and `FileMetadataFinalizer` to support per-file physical row conversion and metadata finalization. - Add `MapSharedShreddingWritePlanFactory` for append shared-shredding writes. - Add `MapSharedShreddingCoreUtils` to create and restore shared-shredding write context from recent data files. - Wire shared-shredding append writes into `BaseAppendFileStoreWrite`, `AppendOnlyWriter`, `RowDataRollingFileWriter`, `RowDataFileWriter`, and `SingleFileWriter`. - Keep file index and sequence tracking based on logical rows while writing physical rows to data files. - Restore adaptive shared-shredding stats from existing file metadata using each data file's actual file format. - Reject unsupported blob/vector append writes for shared-shredding for now. - Reject nested MAP file index when shared-shredding is enabled. - Extend format metadata field-id support for ORC via `paimon.id`. This PR currently focuses on append tables. The design keeps the row write plan generic so other table models can reuse the same logical-to-physical write flow later. ### Tests - `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=AppendOnlyWriterTest,MapSharedShreddingCoreUtilsTest,SchemaValidationTest test` - `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=KeyValueFileReadWriteTest test` - `mvn -pl paimon-format -Pfast-build -DfailIfNoTests=false -Dtest=FormatMetadataUtilsTest,OrcFormatReadWriteTest test` - `mvn -pl paimon-arrow -am -Pfast-build -DfailIfNoTests=false -Dtest=ArrowSchemaMetadataCompatibilityTest test` - `mvn -pl paimon-core -DskipTests spotless:check` - `mvn -pl paimon-format -DskipTests spotless:check` - `mvn -pl paimon-arrow -DskipTests spotless:check` - `git diff --cached --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
