JingsongLi opened a new pull request, #9574: URL: https://github.com/apache/paimon/pull/9574
### Purpose Reduce DataFileMeta size for wide data-evolution tables with dedicated BLOB or vector fields. This adds the disabled-by-default `data-evolution.write-cols-optimization.enabled` option. When enabled in a persisted table schema, normal files containing every non-dedicated field omit the redundant `writeCols` list, while dedicated files keep explicit columns. Readers in Java and Python resolve omitted metadata from the historical table schema. The change also covers stats evolution, conflict detection and rewrite, streaming column discovery, global-index refresh, Daft reads, and Spark copy-files schema rebinding. For compatibility, dynamic enablement or disablement is rejected. The option must be persisted with CREATE TABLE or ALTER TABLE, and all readers should be upgraded before enabling it. ### Tests - `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=TableSchemaTest,BlobTableTest#testOmitWriteColsForAllNonDedicatedColumns,VectorStoreTableTest#testOmitWriteColsForAllNonDedicatedColumns+testRejectDynamicWriteColsOptimizationEnablement,RowIdColumnConflictCheckerTest,FilesTableTest#testReadStatsWithOmittedNonDedicatedWriteCols,DataEvolutionUtilsTest test` - `mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -Pspark3 -DfailIfNoTests=false -DwildcardSuites=none -Dtest=CopyFilesUtilTest test` - `python3 -m pytest -q pypaimon/tests/table_schema_test.py pypaimon/tests/blob_table_test.py pypaimon/tests/write/conflict_detection_test.py pypaimon/tests/ray_row_id_conflict_rewriter_test.py pypaimon/tests/vector_table_test.py::VectorTableWriteReadTest::test_vector_table_partial_update_non_vector_column` - `python3 -m compileall -q pypaimon` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
