stevenzwu commented on code in PR #7571:
URL: https://github.com/apache/iceberg/pull/7571#discussion_r1191304024
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java:
##########
@@ -115,7 +116,12 @@ private ContinuousEnumerationResult discoverSplits() {
return new ContinuousEnumerationResult(
Collections.emptyList(), enumeratorPosition.get(),
enumeratorPosition.get());
} else {
- return splitPlanner.planSplits(enumeratorPosition.get());
+ AtomicReference<ContinuousEnumerationResult> result = new
AtomicReference<>();
+ Tasks.foreach(enumeratorPosition.get())
Review Comment:
I was thinking about sth in the same line as Flink's tolerable checkpoint
failures.
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#execution-checkpointing-tolerable-failed-checkpoints
We don't do any retry here.
##########
flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java:
##########
@@ -161,7 +167,12 @@ private void
processDiscoveredSplits(ContinuousEnumerationResult result, Throwab
LOG.info("Update enumerator position to {}", result.toPosition());
}
} else {
- LOG.error("Failed to discover new splits", error);
+ if (scanContext.maxAllowedPlanningFailures() == -1) {
Review Comment:
Here, we check the consecutive failures. if it is over the threshold,
exception is throw. otherwise, just log an error and move on.
if it is a successful planning, the consecutive failure counter is reset to 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
