SteNicholas opened a new issue, #1442:
URL: https://github.com/apache/paimon-rust/issues/1442

   ### Search before asking
   
   - [x] I searched existing issues and open pull requests. No issue or active 
PR covering this specific scope was found on November 18, 2026.
   
   ### Please describe the bug 🐞
   
   Java Paimon's 
[DataField](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-api/src/main/java/org/apache/paimon/types/DataField.java#L40-L81)
 stores a generated physical column's expression as `generationExpression` in 
schema JSON. Rust's 
[DataField](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/spec/schema.rs#L1276-L1287)
 has no corresponding field.
   
   The Rust type derives Serde deserialization and serialization without 
retaining unknown fields. A Java-generated field therefore loses its expression 
when decoded into this model and serialized again. This is a metadata 
compatibility problem even before Rust implements expression evaluation: 
loading and subsequently rewriting a schema must not turn a generated column 
into an ordinary column.
   
   For example, the following field metadata should preserve the expression 
through a Rust round-trip:
   
   ```json
   {
     "id": 3,
     "name": "event_year",
     "type": "INT",
     "generationExpression": "YEAR(event_time)"
   }
   ```
   
   The physical values of existing generated columns remain readable; the 
problem here is preserving their schema definition and preventing incompatible 
schema changes.
   
   ### Solution
   
   Start with metadata preservation and schema-evolution protection:
   
   - Add an optional generation-expression field using Java's exact JSON name, 
`generationExpression`. Missing or null expressions remain compatible with 
ordinary fields, and absent expressions should not add unnecessary JSON 
properties.
   - Preserve the expression through field cloning/copying, schema builders and 
conversions, catalog loading, and schema serialization. Changes to unrelated 
fields, table options or comments must not discard it.
   - Align schema-change protection with Java's 
[validateGeneratedColumnEvolution](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java#L1466-L1500):
 do not silently rename, drop or change the type/expression of a generated 
field, or rename/drop/change the type of a field referenced by its expression. 
Unsupported mutations must fail before publishing a new schema.
   - Keep the metadata change separate from implementing the expression 
evaluator and SQL `GENERATED ALWAYS AS` authoring. Preserving an expression 
must not be presented as complete generated-column write support.
   
   Java's generation semantics should guide later work: omitted generated 
fields are filled before partition routing, while explicitly supplied values, 
including NULL, are preserved. This issue does not introduce automatic 
recomputation or a stricter computed-column constraint.
   
   #### Acceptance criteria
   
   - [ ] A Java-compatible field/schema fixture retains `generationExpression` 
through deserialize/serialize and catalog schema round-trips.
   - [ ] Schemas with absent and explicit-null expressions still load and 
serialize compatibly.
   - [ ] Field copies and unrelated schema/options/comment changes preserve the 
expression unchanged.
   - [ ] Invalid changes to a generated field or its dependencies are rejected 
without publishing a damaged schema; unrelated changes remain usable.
   - [ ] Existing stored generated values can still be read, and documentation 
distinguishes metadata support from write-time generation.
   
   ### References
   
   - [Java JSON field 
parser](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-api/src/main/java/org/apache/paimon/types/DataTypeJsonParser.java#L65-L82).
   - [Java generated-column writer 
mapping](https://github.com/apache/paimon/blob/2bf10c194b9213e406b8cf0732b9e947f835cebd/paimon-common/src/main/java/org/apache/paimon/utils/GeneratedColumnRow.java#L37-L101).
   - [Rust field copy/accessor 
methods](https://github.com/apache/paimon-rust/blob/650d26277b5e5bc656d9667c535d42e00aa4b9aa/crates/paimon/src/spec/schema.rs#L1289-L1339).
   
   Source baseline: paimon-rust `650d26277b5e5bc656d9667c535d42e00aa4b9aa`, 
compared with Java Paimon `2bf10c194b9213e406b8cf0732b9e947f835cebd`. These 
findings come from source inspection; no new runtime reproduction or 
performance measurement is claimed.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to