SteNicholas opened a new pull request, #1447: URL: https://github.com/apache/paimon-rust/pull/1447
### Purpose Linked issue: #1442 Loading a Java schema discards `generationExpression`, so rewriting it can erase generated-column definitions and allow schema changes that invalidate stored values. This preserves generated-column metadata through schema round-trips and protects generated columns and their dependencies during schema evolution. ### Brief change log - Preserve the optional `generationExpression` property in `DataField`, omitting it when absent or null. Retain field metadata through copies, schema builders, nested field ID assignment, type changes, read/write projections and Map/Variant transformations. - Validate final fields by stable ID after nullability normalization and before publishing a schema. Reject changes to generated columns or their dependencies and reject new top-level generated columns after table creation. For dependency syntax that cannot be resolved, conservatively reject renames, drops and type changes of existing columns while allowing metadata changes. - Carry generation expressions through Arrow field metadata and DataFusion Variant extraction pushdown, and expose them through the Python `DataField` wrapper. ### Tests - Share `crates/paimon/tests/fixtures/generated_column_schema.json` across schema, catalog and Python binding tests. Cover Java-compatible JSON round-trips, absent and explicit-null expressions, unchanged expression text, nested field IDs, field copies and Python metadata access. - Add schema-evolution coverage in `crates/paimon/src/spec/generated_column.rs` and the existing schema tests for generated columns, dependencies, nullability normalization, escaped identifiers, unsupported syntax and parser depth limits. Extend the filesystem catalog tests to cover table creation, rejected changes without schema publication, allowed metadata and unrelated field changes, and reads of explicitly stored generated values, including NULL. - Extend existing Arrow, Map/Variant, read projection and key-value writer tests to cover metadata preservation. Extend `crates/integrations/datafusion/tests/variant_pushdown.rs` to assert expression metadata in the pushed-down scan schema alongside the existing query-result assertions. - Validation: Static source review only. Compilation, builds and tests were not run, as requested. ### API and Format Add Rust `DataField::generation_expression()`, `with_generation_expression(...)` and `with_data_type(...)`, plus Python `DataField.generation_expression()`. Schema JSON gains the optional Java-compatible `generationExpression` property, and Arrow fields carry it under `paimon:generationExpression`. Missing and null expressions serialize without the property. Data file layouts remain unchanged. Generated-column metadata is preserved; expression evaluation and SQL `GENERATED ALWAYS AS` authoring remain unsupported. ### Documentation Update `docs/src/getting-started.md` with the JSON representation, Rust and Python accessors, Arrow metadata, supported schema evolution and conservative handling of unknown dependency syntax. Document that writers must explicitly supply generated values, including NULL for nullable fields, and that stored values are not automatically recomputed. 🤖 Assisted with Codex. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
