bube commented on code in PR #10109:
URL: https://github.com/apache/paimon/pull/10109#discussion_r4497873895


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkConnectorOptions.java:
##########
@@ -481,18 +486,64 @@ public class FlinkConnectorOptions {
                     .stringType()
                     .noDefaultValue()
                     .withDescription(
-                            "Set the uid suffix for the source operators. 
After setting, the uid format is "
-                                    + 
"${UID_PREFIX}_${TABLE_NAME}_${USER_UID_SUFFIX}. If the uid suffix is not set, 
flink will "
-                                    + "automatically generate the operator 
uid, which may be incompatible when the topology changes.");
+                            Description.builder()
+                                    .text(
+                                            "Set the uid suffix for the source 
operators. The uid format is "
+                                                    + 
"${UID_PREFIX}_${TABLE_NAME}_${USER_UID_SUFFIX}. Without a suffix Flink "
+                                                    + "generates these ids 
from the shape of the stream graph, so a topology "
+                                                    + "change can orphan their 
state. The suffix alone does not prevent that: "
+                                                    + "the other operators the 
table source adds keep generated ids, and a "
+                                                    + "recovery from the HA 
checkpoint store fails when one of them moves. "
+                                                    + "For stable ids also set 
'source.operator-uid.cover-table-source-operators'; existing jobs must follow 
the %s first.",
+                                            link(OPERATOR_UID_GUIDE, 
"migration steps"))
+                                    .build());
+
+    public static final ConfigOption<Boolean> 
SOURCE_OPERATOR_UID_COVER_TABLE_SOURCE_OPERATORS =
+            key("source.operator-uid.cover-table-source-operators")
+                    .booleanType()
+                    .defaultValue(false)
+                    .withDescription(
+                            Description.builder()
+                                    .text(
+                                            "If true, 
'source.operator-uid.suffix' also names every other operator the table source 
adds in a streaming job: the split monitor, the split reader, the watermark 
assigner and the DataStream row conversion. "

Review Comment:
   Thanks, you are right. In the first version I took this route to be 
batch-only, but a bounded source takes it in STREAMING too. Fixed in eb6f630121:
   - the merge-input source, `Postpone merge input` and `Postpone merge on 
read` take uids from the source cover option;
   - the source built when the table has no snapshot gets its own prefix 
(`Postpone merge empty source`). Its state differs from the merge-input source, 
so the two must not share an id.
   
   `SourceOperatorUidSuffixTest` gains both branches as shapes, built in a 
STREAMING env with auto-generated uids off, and a pin of the four uids. With 
the assignments removed, exactly those shapes fail. The existing `toDataStream` 
uid is unchanged: this route returned before it on master too.
   
   The route holds no state that survives a restore (the source finishes after 
its first poll; the merge buffer is not checkpointed), so the migration steps 
need no change.
   
   I also rebased onto current master, which starts a new CI run. Locally the 
uid suites (39 tests) pass on Flink 1.20 and 2.2.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to