lxy-9602 opened a new pull request, #8827: URL: https://github.com/apache/paimon/pull/8827
### Purpose This PR introduces a shared history mechanism for restoring per-file shredding write plans from recent data files. - Add format-independent `ShreddingFileMetadata`, `ShreddingWritePlanHistory`, and `ShreddingWritePlanPreparation` abstractions. - Lazily load physical schemas and field metadata from recent restored files, while preserving schema evolution and column renames through field IDs. - Update history after each rolling file so subsequent files can use the latest metadata. - Restore MAP shared-shredding column counts from historical maximum row widths. Files without valid history continue to use the configured `max-columns`. - Add the opt-in `variant.restoreShreddingSchemaFromHistory` option. Existing Variant inference behavior remains unchanged by default; missing Variant fields still fall back to row sampling. - Propagate history through append-only, primary-key, changelog, thin-mode, and compaction/rewrite writers. Rewrite outputs derive history from their actual input files. - Support metadata recovery through both ORC and Parquet, including mixed-format history. ### Tests Over 1,900 lines of this PR are dedicated to comprehensive test coverage. Added or updated coverage for: - Shredding history ordering, file limit, schema-ID normalization, column renames, and recoverable metadata IO failures. - MAP adaptive column-count restoration, multiple independent MAP fields, rolling files, mixed ORC/Parquet history, and partition/bucket isolation. - Variant schema restoration, nested Variant fields, partial restoration, and fallback inference for newly added Variant fields. - Append-only and primary-key tables, including thin mode, aggregation, changelog files, external spill, deletion vectors, and regular/partial/clustering compaction. - ORC and Parquet metadata round trips and per-file history updates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
