[
https://issues.apache.org/jira/browse/HDFS-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520617#comment-14520617
]
Aaron T. Myers commented on HDFS-8178:
--------------------------------------
Hi Zhe, thanks a lot for posting this patch. It looks pretty good to me. I have
one comment, and one question for you:
Comment: I'm a tad concerned that the current patch be be over aggressive in
terms of purging edit log segments, if somehow we ended up with two in-progress
segments, both of which were required to form the complete history of edit log
transactions. Unless we're somehow guaranteed that that's not a state we could
end up in, I think we should do something to guarantee that the in-progress
edits file(s) that we're considering purging definitely overlap with finalized
edit log segments so that they don't contain any edits that we can't afford to
lose. Or, perhaps we should move them aside with a different name until such
time as we can be sure that we don't need their transactions anymore, i.e.
their transaction is less than minTxIdToKeep, and so we can definitely safely
discard them.
Question: In the patch for HDFS-5919, we introduced a new regex for identifying
stale in-progress files. I'm not familiar with why that was necessary in that
patch, but can you please comment on why it's not necessary in this case?
Naively, I'd expect either both or neither the FJM and JN to require accounting
for that.
> QJM doesn't purge empty and corrupt inprogress edits files
> ----------------------------------------------------------
>
> Key: HDFS-8178
> URL: https://issues.apache.org/jira/browse/HDFS-8178
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: qjm
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Attachments: HDFS-8178.000.patch
>
>
> HDFS-5919 fixes the issue for {{FileJournalManager}}. A similar fix is needed
> for QJM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
