voonhous commented on code in PR #18816:
URL: https://github.com/apache/hudi/pull/18816#discussion_r3911139609
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/RepairsCommand.java:
##########
@@ -201,7 +202,12 @@ public void removeCorruptedPendingCleanAction() {
TimelineUtils.deleteInstantFile(client.getStorage(),
client.getTimelinePath(),
instant, client.getInstantFileNameGenerator());
} catch (IOException ioe) {
- if (ioe.getMessage().contains("Not an Avro data file")) {
+ // An empty or truncated instant file does not reach Avro's magic-byte
check; the timeline
+ // reader reports it as "unable to read commit metadata" instead,
capitalised by the v1
+ // serde and lowercase by the v2 one, hence the case-insensitive match.
+ if (ioe.getMessage() != null
+ && (ioe.getMessage().contains("Not an Avro data file")
+ || ioe.getMessage().toLowerCase(Locale.ROOT).contains("unable
to read commit metadata"))) {
Review Comment:
Agreed: the message match could not tell a stalled read from a bad file, and
the earlier bot reply on this thread was wrong about it. In 4a468c1fa2f4 the
command reads the plan bytes in full first, so a failure of the read itself
(your `SocketTimeoutException` case) propagates as `HoodieIOException` and the
instant stays. Only the in-memory decode of those bytes, which no I/O can
disturb, decides that a plan is corrupt. Both layouts decode a clean plan
through `TimelineMetadataUtils.deserializeAvroMetadata`, so the
`CleanerUtils.getCleanerPlan(metaClient, InputStream)` overload runs on the
bytes and the plan migrator still applies.
Tests: `testRemoveCorruptedPendingCleanActionKeepsPlanOnReadFailure` runs
the command against a storage whose stream of a valid `.clean.requested` throws
`SocketTimeoutException` after 32 bytes, asserts the `HoodieIOException`
carries that cause and the file is still there, then runs it through a healthy
storage and checks the plan is kept. `testRemoveCorruptedPendingCleanAction`
now also holds a plan truncated inside the Avro header and a readable one next
to the four empty files: the empty and truncated ones go, the readable one
stays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
