lxy-9602 opened a new pull request, #8347: URL: https://github.com/apache/paimon/pull/8347
### Purpose This is a sub-PR for [shared-shredding MAP write support](https://cwiki.apache.org/confluence/display/PAIMON/PIP-43%3A+Columnar+Storage+Optimization+for+MAP+Type+in+Paimon). Shared-shredding needs to attach file-level and per-field metadata, such as field dictionaries and physical column mappings, before ORC/Parquet files are finalized. This PR adds a format-level metadata write/read path so upper layers can pass metadata to format writers and read back per-field metadata from file schemas. The per-field metadata is stored through Arrow IPC schema metadata under the standard `ARROW:schema` key, matching the Arrow schema metadata representation used by Arrow Parquet/ORC integrations. To avoid introducing Arrow runtime dependencies into `paimon-format` and `paimon-bundle`, this PR implements a small Arrow schema metadata encoder/decoder in `paimon-format` based on the Arrow IPC FlatBuffers layout. RowType-to-Arrow field conversion is kept in sync with `paimon-arrow`'s `ArrowUtils`, and compatibility is covered in `paimon-arrow` tests. ### Brief change log - Add `SupportsWriterMetadata` for format writers that can accept metadata before close. - Add `SupportsReaderFieldMetadata` for format readers to expose field metadata parsed from `ARROW:schema`. - Add `FormatMetadataUtils` for: - base64 encoding/decoding format metadata values; - building Arrow schema metadata from `RowType` and field metadata; - reading top-level field metadata from serialized Arrow schema metadata. - Add a minimal `ArrowSchemaMetadata` implementation in `paimon-format` without depending on Arrow Java runtime. - Support metadata write/read in ORC writers/readers. - Support metadata write/read in Parquet writers/readers. - Add `MetadataParquetBuilder` so only builders that explicitly wire metadata into Parquet `WriteSupport.finalizeWrite()` expose writer metadata support. - Shade and relocate `flatbuffers-java` in `paimon-format`, and verify it is included through `paimon-bundle`. ### Tests - Add format tests for ORC/Parquet metadata roundtrip. - Add Arrow compatibility tests in `paimon-arrow` to verify: - metadata bytes generated by `paimon-format` can be read by Arrow Java; - metadata bytes generated by Arrow Java can be parsed by `paimon-format`. - `mvn -pl paimon-format -DskipTests spotless:check` - `mvn -pl paimon-arrow -DskipTests spotless:check` - `mvn -pl paimon-format -am -Pfast-build -DfailIfNoTests=false -Dtest=FormatMetadataUtilsTest,ParquetFormatReadWriteTest,OrcFormatReadWriteTest test` - `mvn -pl paimon-arrow -am -Pfast-build -DfailIfNoTests=false -Dtest=ArrowSchemaMetadataCompatibilityTest test` - `mvn -pl paimon-format -DskipTests dependency:tree -Dincludes=org.apache.arrow,org.apache.paimon:paimon-arrow` - `mvn -pl paimon-format,paimon-bundle -am -Pfast-build -DskipTests package` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
