[
https://issues.apache.org/jira/browse/HDFS-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912236#comment-16912236
]
Istvan Fajth commented on HDFS-8303:
------------------------------------
As this one is a dependency of HDFS-8178 I started to work on, I am assigning
this one as well to me. I am also attaching a patch based on the original
changeset that applies.
> QJM should purge old logs in the current directory through FJM
> --------------------------------------------------------------
>
> Key: HDFS-8303
> URL: https://issues.apache.org/jira/browse/HDFS-8303
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Priority: Major
> Labels: BB2015-05-TBR
> Attachments: HDFS-8303.0.patch, HDFS-8303.1.patch
>
>
> As the first step of the consolidation effort, QJM should call its FJM to
> purge the current directory.
> The current QJM logic of purging current dir is very similar to FJM purging
> logic.
> QJM:
> {code}
> private static final List<Pattern> CURRENT_DIR_PURGE_REGEXES =
> ImmutableList.of(
> Pattern.compile("edits_\\d+-(\\d+)"),
> Pattern.compile("edits_inprogress_(\\d+)(?:\\..*)?"));
> ...
> long txid = Long.parseLong(matcher.group(1));
> if (txid < minTxIdToKeep) {
> LOG.info("Purging no-longer needed file " + txid);
> if (!f.delete()) {
> ...
> {code}
> FJM:
> {code}
> private static final Pattern EDITS_REGEX = Pattern.compile(
> NameNodeFile.EDITS.getName() + "_(\\d+)-(\\d+)");
> private static final Pattern EDITS_INPROGRESS_REGEX = Pattern.compile(
> NameNodeFile.EDITS_INPROGRESS.getName() + "_(\\d+)");
> private static final Pattern EDITS_INPROGRESS_STALE_REGEX = Pattern.compile(
> NameNodeFile.EDITS_INPROGRESS.getName() + "_(\\d+).*(\\S+)");
> ...
> List<EditLogFile> editLogs = matchEditLogs(files, true);
> for (EditLogFile log : editLogs) {
> if (log.getFirstTxId() < minTxIdToKeep &&
> log.getLastTxId() < minTxIdToKeep) {
> purger.purgeLog(log);
> }
> }
> {code}
> I can see 2 differences:
> # Different regex in matching for empty/corrupt in-progress files. The FJM
> pattern makes more sense to me.
> # FJM verifies that both start and end txID of a finalized edit file to be
> old enough. This doesn't make sense because end txID is always larger than
> start txID
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
