JingsongLi opened a new pull request, #10946:
URL: https://github.com/apache/paimon/pull/10946
### Purpose
Flink SQL currently fills omitted physical generated columns with NULL
instead of evaluating their stored generation expressions. This prevents using
generated business-date partitions or primary keys when an INSERT supplies only
the ordinary columns.
Fill omitted generated columns before partition routing and bucket
assignment on Flink 1.18+. Reuse Paimon's transforms and source-column
defaults, preserve explicitly supplied values (including NULL), and enforce
stored NOT NULL constraints after generation. Support DATE and STRING
business-date partitions, generated primary keys, static partitions, INSERT
OVERWRITE, and mixed virtual/physical computed columns.
Keep UPDATE and merge-engine semantics intact. Deduplicate sinks request
before images for updating inputs so changed generated keys or partitions
retract the old records. Generated-primary-key sources reconstruct full
changelogs using checkpointed keyed state before projection/filtering;
first-row and partial-update retain their existing write behavior. Preserve
catalog schema attributes and support generated-key managed-BLOB tables.
### Tests
- Flink 1.20.1 / JDK 8: 45 focused tests passed, with Checkstyle, Spotless,
and dependency checks enabled.
- GeneratedColumnITCase, GeneratedPrimaryKeyChangelogNormalizerTest,
ChangelogModeTest, ComputedColumnAndWatermarkTableITCase, and focused
catalog/nullable-primary-key regressions.
- Flink 2.2.0 / JDK 17: 32 generated-column and changelog-normalizer tests
passed.
- Coverage includes DATE/STRING partitions across append, primary-key, and
Data Evolution tables; late records; explicit NULL versus omitted columns; NOT
NULL rejection; updating generated keys/partitions; merge-engine behavior;
streaming filtering/projection; object reuse; and checkpoint restore.
Physical generated-column definitions continue to use Spark DDL or the Java
schema builder; this change does not add Flink DDL syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]