JingsongLi opened a new pull request, #10946:
URL: https://github.com/apache/paimon/pull/10946

   ### Purpose
   
   Flink SQL currently fills omitted physical generated columns with NULL 
instead of evaluating their stored generation expressions. This prevents using 
generated business-date partitions or primary keys when an INSERT supplies only 
the ordinary columns.
   
   Fill omitted generated columns before partition routing and bucket 
assignment on Flink 1.18+. Reuse Paimon's transforms and source-column 
defaults, preserve explicitly supplied values (including NULL), and enforce 
stored NOT NULL constraints after generation. Support DATE and STRING 
business-date partitions, generated primary keys, static partitions, INSERT 
OVERWRITE, and mixed virtual/physical computed columns.
   
   Keep UPDATE and merge-engine semantics intact. Deduplicate sinks request 
before images for updating inputs so changed generated keys or partitions 
retract the old records. Generated-primary-key sources reconstruct full 
changelogs using checkpointed keyed state before projection/filtering; 
first-row and partial-update retain their existing write behavior. Preserve 
catalog schema attributes and support generated-key managed-BLOB tables.
   
   ### Tests
   
   - Flink 1.20.1 / JDK 8: 45 focused tests passed, with Checkstyle, Spotless, 
and dependency checks enabled.
     - GeneratedColumnITCase, GeneratedPrimaryKeyChangelogNormalizerTest, 
ChangelogModeTest, ComputedColumnAndWatermarkTableITCase, and focused 
catalog/nullable-primary-key regressions.
   - Flink 2.2.0 / JDK 17: 32 generated-column and changelog-normalizer tests 
passed.
   - Coverage includes DATE/STRING partitions across append, primary-key, and 
Data Evolution tables; late records; explicit NULL versus omitted columns; NOT 
NULL rejection; updating generated keys/partitions; merge-engine behavior; 
streaming filtering/projection; object reuse; and checkpoint restore.
   
   Physical generated-column definitions continue to use Spark DDL or the Java 
schema builder; this change does not add Flink DDL syntax.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to