pvary commented on code in PR #18717:
URL: https://github.com/apache/iceberg/pull/18717#discussion_r4492896514
##########
flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/EqualityConvertPlanner.java:
##########
@@ -350,14 +351,17 @@ private void ensureIndexCurrent(Snapshot mainSnapshot) {
return;
}
- if (!indexReflects(mainSnapshot)) {
+ List<Snapshot> externalCommits = externalCommitsSinceIndex(mainSnapshot);
+ if (externalCommits == null || deltaReadsMoreThanRebuild(mainSnapshot,
externalCommits)) {
boolean bootstrap = indexSnapshotId == null;
LOG.info(
"{} worker index from main snapshot {} for field IDs {}.",
bootstrap ? "Bootstrapping" : "Reindexing",
mainSnapshot.snapshotId(),
eqFieldIds);
rebuildIndex(mainSnapshot, !bootstrap);
+ } else {
+ emitIncrementalReindex(mainSnapshot, externalCommits);
Review Comment:
I have asked an AI to collect the possible failure scenarios, and
distributed them based on whether they can be reried-or-not.
Transient: retrying the same delta works
- Storage throttling or outages: S3/GCS/ADLS 503 or 429 errors last longer
than the FileIO retries.
- Credential refresh hiccup: an STS/IMDS timeout, or the REST catalog's
refresh endpoint is briefly down.
- Replication lag: a replica doesn't have a just-committed manifest list
yet, so the read returns 404.
- Catalog outage: table.refresh() fails before the delta even starts.
Already ends in a rebuild
- Snapshot expiry by another engine: it deletes manifest lists mid-read, and
on the next trigger the walk breaks and the planner rebuilds.
Never succeeds, though a rebuild at the head would (the delta is stuck)
- Leaked equality delete, later cleaned up: validateTargetDeleteFiles fails
on every trigger.
- Lost metadata of an older snapshot: a lifecycle rule, cleanup script or
orphan-file removal deleted a manifest list the head no longer uses.
- Corrupt manifest from a buggy writer: it was later replaced by a manifest
merge or rewrite, but the delta still reads it.
- Delta too big for the planner's heap: the OutOfMemoryError bypasses the
catch, and every restore replays the same delta, so the job crash-loops.
- Metadata from a writer that doesn't follow the spec: a malformed summary
count or a deleted entry without a data sequence number fails on every trigger.
Never succeeds, and a rebuild fails too (needs an operator)
- Permissions: an object ACL from another account, a KMS key the converter's
role can't use, or expired vended credentials.
- Metadata from a newer Iceberg version: the converter's version can't parse
it until it is upgraded.
- Equality delete still live on the target: a separate-branch setup that is
really misconfigured.
We should go though the list and throw out the garbage, but there are some
interesting ones here.
Or alternatively we can do this in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
