bube commented on code in PR #10109:
URL: https://github.com/apache/paimon/pull/10109#discussion_r4497873895
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/FlinkConnectorOptions.java:
##########
@@ -481,18 +486,64 @@ public class FlinkConnectorOptions {
.stringType()
.noDefaultValue()
.withDescription(
- "Set the uid suffix for the source operators.
After setting, the uid format is "
- +
"${UID_PREFIX}_${TABLE_NAME}_${USER_UID_SUFFIX}. If the uid suffix is not set,
flink will "
- + "automatically generate the operator
uid, which may be incompatible when the topology changes.");
+ Description.builder()
+ .text(
+ "Set the uid suffix for the source
operators. The uid format is "
+ +
"${UID_PREFIX}_${TABLE_NAME}_${USER_UID_SUFFIX}. Without a suffix Flink "
+ + "generates these ids
from the shape of the stream graph, so a topology "
+ + "change can orphan their
state. The suffix alone does not prevent that: "
+ + "the other operators the
table source adds keep generated ids, and a "
+ + "recovery from the HA
checkpoint store fails when one of them moves. "
+ + "For stable ids also set
'source.operator-uid.cover-table-source-operators'; existing jobs must follow
the %s first.",
+ link(OPERATOR_UID_GUIDE,
"migration steps"))
+ .build());
+
+ public static final ConfigOption<Boolean>
SOURCE_OPERATOR_UID_COVER_TABLE_SOURCE_OPERATORS =
+ key("source.operator-uid.cover-table-source-operators")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription(
+ Description.builder()
+ .text(
+ "If true,
'source.operator-uid.suffix' also names every other operator the table source
adds in a streaming job: the split monitor, the split reader, the watermark
assigner and the DataStream row conversion. "
Review Comment:
Thanks, you are right. In the first version I took this route to be
batch-only, but a bounded source takes it in STREAMING too. Fixed in eb6f630121:
- the merge-input source, `Postpone merge input` and `Postpone merge on
read` take uids from the source cover option;
- the source built when the table has no snapshot gets its own prefix
(`Postpone merge empty source`). Its state differs from the merge-input source,
so the two must not share an id.
`SourceOperatorUidSuffixTest` gains both branches as shapes, built in a
STREAMING env with auto-generated uids off, and a pin of the four uids. With
the assignments removed, exactly those shapes fail. The existing `toDataStream`
uid is unchanged: this route returned before it on master too.
The route holds no state that survives a restore (the source finishes after
its first poll; the merge buffer is not checkpointed), so the migration steps
need no change.
I also rebased onto current master, which starts a new CI run. Locally the
uid suites (39 tests) pass on Flink 1.20 and 2.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]