SteNicholas opened a new pull request, #1447:
URL: https://github.com/apache/paimon-rust/pull/1447

   ### Purpose
   
   Linked issue: #1442
   
   Loading a Java schema discards `generationExpression`, so rewriting it can 
erase generated-column definitions and allow schema changes that invalidate 
stored values. This preserves generated-column metadata through schema 
round-trips and protects generated columns and their dependencies during schema 
evolution.
   
   ### Brief change log
   
   - Preserve the optional `generationExpression` property in `DataField`, 
omitting it when absent or null. Retain field metadata through copies, schema 
builders, nested field ID assignment, type changes, read/write projections and 
Map/Variant transformations.
   - Validate final fields by stable ID after nullability normalization and 
before publishing a schema. Reject changes to generated columns or their 
dependencies and reject new top-level generated columns after table creation. 
For dependency syntax that cannot be resolved, conservatively reject renames, 
drops and type changes of existing columns while allowing metadata changes.
   - Carry generation expressions through Arrow field metadata and DataFusion 
Variant extraction pushdown, and expose them through the Python `DataField` 
wrapper.
   
   ### Tests
   
   - Share `crates/paimon/tests/fixtures/generated_column_schema.json` across 
schema, catalog and Python binding tests. Cover Java-compatible JSON 
round-trips, absent and explicit-null expressions, unchanged expression text, 
nested field IDs, field copies and Python metadata access.
   - Add schema-evolution coverage in 
`crates/paimon/src/spec/generated_column.rs` and the existing schema tests for 
generated columns, dependencies, nullability normalization, escaped 
identifiers, unsupported syntax and parser depth limits. Extend the filesystem 
catalog tests to cover table creation, rejected changes without schema 
publication, allowed metadata and unrelated field changes, and reads of 
explicitly stored generated values, including NULL.
   - Extend existing Arrow, Map/Variant, read projection and key-value writer 
tests to cover metadata preservation. Extend 
`crates/integrations/datafusion/tests/variant_pushdown.rs` to assert expression 
metadata in the pushed-down scan schema alongside the existing query-result 
assertions.
   
   - Validation: Static source review only. Compilation, builds and tests were 
not run, as requested.
   
   ### API and Format
   
   Add Rust `DataField::generation_expression()`, 
`with_generation_expression(...)` and `with_data_type(...)`, plus Python 
`DataField.generation_expression()`. Schema JSON gains the optional 
Java-compatible `generationExpression` property, and Arrow fields carry it 
under `paimon:generationExpression`. Missing and null expressions serialize 
without the property. Data file layouts remain unchanged. Generated-column 
metadata is preserved; expression evaluation and SQL `GENERATED ALWAYS AS` 
authoring remain unsupported.
   
   ### Documentation
   
   Update `docs/src/getting-started.md` with the JSON representation, Rust and 
Python accessors, Arrow metadata, supported schema evolution and conservative 
handling of unknown dependency syntax. Document that writers must explicitly 
supply generated values, including NULL for nullable fields, and that stored 
values are not automatically recomputed.
   
   🤖 Assisted with Codex.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to