JingsongLi commented on code in PR #10109:
URL: https://github.com/apache/paimon/pull/10109#discussion_r4492560058


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkConnectorOptions.java:
##########
@@ -481,18 +486,64 @@ public class FlinkConnectorOptions {
                     .stringType()
                     .noDefaultValue()
                     .withDescription(
-                            "Set the uid suffix for the source operators. 
After setting, the uid format is "
-                                    + 
"${UID_PREFIX}_${TABLE_NAME}_${USER_UID_SUFFIX}. If the uid suffix is not set, 
flink will "
-                                    + "automatically generate the operator 
uid, which may be incompatible when the topology changes.");
+                            Description.builder()
+                                    .text(
+                                            "Set the uid suffix for the source 
operators. The uid format is "
+                                                    + 
"${UID_PREFIX}_${TABLE_NAME}_${USER_UID_SUFFIX}. Without a suffix Flink "
+                                                    + "generates these ids 
from the shape of the stream graph, so a topology "
+                                                    + "change can orphan their 
state. The suffix alone does not prevent that: "
+                                                    + "the other operators the 
table source adds keep generated ids, and a "
+                                                    + "recovery from the HA 
checkpoint store fails when one of them moves. "
+                                                    + "For stable ids also set 
'source.operator-uid.cover-table-source-operators'; existing jobs must follow 
the %s first.",
+                                            link(OPERATOR_UID_GUIDE, 
"migration steps"))
+                                    .build());
+
+    public static final ConfigOption<Boolean> 
SOURCE_OPERATOR_UID_COVER_TABLE_SOURCE_OPERATORS =
+            key("source.operator-uid.cover-table-source-operators")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            Description.builder()
+                                    .text(
+                                            "If true, 
'source.operator-uid.suffix' also names every other operator the table source 
adds in a streaming job: the split monitor, the split reader, the watermark 
assigner and the DataStream row conversion. "

Review Comment:
   [P2] Cover the bounded postpone merge-on-read source topology
   
   The new cover option still misses the early `buildPostponeMergeSource()` 
return in `FlinkSourceBuilder.build()`. With a real primary-key POSTPONE table 
(`bucket=-2`, `postpone.merge-on-read=true`), both source UID options enabled, 
and `sourceBounded(true)` in a STREAMING environment, all three internal 
operators retain generated IDs: the merge-input source, `Postpone merge input`, 
and `Postpone merge on read`.
   
   I first executed that public DataStream read and verified its merged row, so 
this is an operational bounded STREAMING route rather than a BATCH-only graph. 
With `pipeline.auto-generate-uids=false`, constructing the same graph then 
fails with `Auto generated UIDs have been disabled but no UID or hash has been 
assigned to operator ... Postpone merge inputs`. The ordinary bounded source 
control passes. Assigning UIDs to just these three operators makes the 
identical read/strict-graph test pass.
   
   Please apply the source UID assigner to this branch as well, including its 
empty-plan source, and add this source shape to coverage. This concerns the new 
option's startup-coverage promise; it does not establish an offset or recovery 
data-loss regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to