SteNicholas opened a new issue, #1442: URL: https://github.com/apache/paimon-rust/issues/1442
### Search before asking - [x] I searched existing issues and open pull requests. No issue or active PR covering this specific scope was found on November 18, 2026. ### Please describe the bug 🐞 Java Paimon's [DataField](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-api/src/main/java/org/apache/paimon/types/DataField.java#L40-L81) stores a generated physical column's expression as `generationExpression` in schema JSON. Rust's [DataField](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/spec/schema.rs#L1276-L1287) has no corresponding field. The Rust type derives Serde deserialization and serialization without retaining unknown fields. A Java-generated field therefore loses its expression when decoded into this model and serialized again. This is a metadata compatibility problem even before Rust implements expression evaluation: loading and subsequently rewriting a schema must not turn a generated column into an ordinary column. For example, the following field metadata should preserve the expression through a Rust round-trip: ```json { "id": 3, "name": "event_year", "type": "INT", "generationExpression": "YEAR(event_time)" } ``` The physical values of existing generated columns remain readable; the problem here is preserving their schema definition and preventing incompatible schema changes. ### Solution Start with metadata preservation and schema-evolution protection: - Add an optional generation-expression field using Java's exact JSON name, `generationExpression`. Missing or null expressions remain compatible with ordinary fields, and absent expressions should not add unnecessary JSON properties. - Preserve the expression through field cloning/copying, schema builders and conversions, catalog loading, and schema serialization. Changes to unrelated fields, table options or comments must not discard it. - Align schema-change protection with Java's [validateGeneratedColumnEvolution](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java#L1466-L1500): do not silently rename, drop or change the type/expression of a generated field, or rename/drop/change the type of a field referenced by its expression. Unsupported mutations must fail before publishing a new schema. - Keep the metadata change separate from implementing the expression evaluator and SQL `GENERATED ALWAYS AS` authoring. Preserving an expression must not be presented as complete generated-column write support. Java's generation semantics should guide later work: omitted generated fields are filled before partition routing, while explicitly supplied values, including NULL, are preserved. This issue does not introduce automatic recomputation or a stricter computed-column constraint. #### Acceptance criteria - [ ] A Java-compatible field/schema fixture retains `generationExpression` through deserialize/serialize and catalog schema round-trips. - [ ] Schemas with absent and explicit-null expressions still load and serialize compatibly. - [ ] Field copies and unrelated schema/options/comment changes preserve the expression unchanged. - [ ] Invalid changes to a generated field or its dependencies are rejected without publishing a damaged schema; unrelated changes remain usable. - [ ] Existing stored generated values can still be read, and documentation distinguishes metadata support from write-time generation. ### References - [Java JSON field parser](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-api/src/main/java/org/apache/paimon/types/DataTypeJsonParser.java#L65-L82). - [Java generated-column writer mapping](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-common/src/main/java/org/apache/paimon/utils/GeneratedColumnRow.java#L37-L101). - [Rust field copy/accessor methods](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/spec/schema.rs#L1289-L1339). Source baseline: paimon-rust `650d26277b5e5bc656d9667c535d42e00aa4b9aa`, compared with Java Paimon `2bf10c194b9213e406b8cf0732b9e947f835cebd`. These findings come from source inspection; no new runtime reproduction or performance measurement is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
