JingsongLi opened a new pull request, #10912: URL: https://github.com/apache/paimon/pull/10912
### Purpose Add physical generated partition columns whose expressions are part of the column schema. For example, a `dt DATE GENERATED ALWAYS AS (CAST(event_time AS DATE))` partition column can be omitted from an insert; the writer computes it before partition routing. Explicitly supplied values, including NULL, must match the expression using NULL-safe equality. - Persist an optional `generationExpression` attribute on `DataField` and preserve it through schema serialization, copying, catalog conversion, and cloning. - Generate and validate partition values in the core writer, with a separate input schema for omitted generated columns. - Support Spark 3.5 `GENERATED ALWAYS AS` DDL and insert alignment through `SparkCatalog`. Preserve the metadata through the Flink catalog without changing Flink virtual computed-column semantics. - Validate definitions and reject changes to generated definitions or the names/types of their dependencies. Add schema specification and Spark SQL documentation. The initial scope is top-level DATE partition columns generated by `CAST(timestamp_column AS DATE)`, with a top-level TIMESTAMP WITHOUT TIME ZONE source. It supports ordinary append inserts, primary-key deduplicate writes, and Data Evolution full-row inserts. Partial-column writes and other primary-key merge engines are rejected. Key-only deletes may provide the existing partition value without the timestamp source. Compatibility limitation: this change does not introduce a writer-protocol feature gate. Older writers may ignore the new attribute, so writes to these tables require clients that support it. Expression definitions and dependencies are immutable after table creation. ### Tests All of the following passed: - API/common/core: `GeneratedDataFieldTest`, `GeneratedColumnRowTest`, `GeneratedPartitionColumnTest`, `SchemaValidationTest`, `SchemaManagerTest`, and `TableWriteTest`. Includes legacy/NULL JSON metadata, pre-epoch timestamps, explicit NULL and mismatched values, schema evolution, append/primary-key/Data Evolution writes, commits, and reads. - Spark 3.5 (`-Pspark3`): `GeneratedPartitionColumnTest` and `DefaultValueTest` (10 tests), covering SQL creation, omitted and explicit values, quoted/case-insensitive source names, and rejected definitions. - Flink (`-Pflink1`): generated-schema catalog round trip, `FlinkCatalogPropertiesUtilTest`, and the existing batch virtual computed-column test (5 tests). - Spark 3.3: `RefreshTableTest` smoke test using the Spark 3.3 runtime. - Final Maven verification included Checkstyle, Spotless, Enforcer, and RAT without `fast-build`; `git diff --check` also passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
