danny0405 commented on code in PR #19376:
URL: https://github.com/apache/hudi/pull/19376#discussion_r3670328573
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/enumerator/HoodieSplitEnumeratorState.java:
##########
@@ -30,9 +31,35 @@
* State of Hoodie split enumerator. Mainly include the states of pending
splits of split provider.
*/
@Value
+@AllArgsConstructor
public class HoodieSplitEnumeratorState implements Serializable {
Collection<HoodieSourceSplitState> pendingSplitStates;
Option<String> lastEnumeratedInstant;
Option<String> lastEnumeratedInstantOffset;
+ /**
+ * The {@code read.start-commit} / {@code read.end-commit} bounds configured
when this checkpoint
+ * was taken, recorded only for bounded reads. Both hold {@code
Option.of("")} when the option was
+ * not configured, so that "recorded but unset" stays distinguishable from
"not recorded at all";
+ * both are {@link Option#empty()} for streaming reads and for checkpoints
written by serializer
+ * VERSION 1, which predates this field.
+ *
+ * <p>A bounded read's split set is frozen at enumeration time and is NOT
re-derived on restore, so
+ * {@code HoodieSource} compares these against the configured bounds and
fails fast when they
+ * differ. See {@code HoodieSource#checkBoundedCommitRangeUnchanged}.
+ */
+ Option<String> readStartCommit;
Review Comment:
since we already have `lastEnumeratedInstant` and
`lastEnumeratedInstantOffset` so that we can infer the start commit from the
state recovery and we can just compare it agains the options from write config
and throw if necessary, why snapshot the static read options every time in the
state?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
