[
https://issues.apache.org/jira/browse/HDFS-17963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104331#comment-18104331
]
Geqi Hu commented on HDFS-17963:
--------------------------------
Additional operational impact:
I have encountered this segment-count problem in multiple recovery incidents
with strict recovery-time objectives, not only in a single test environment.
When edit-log replay is projected to take several days, operators may face two
unacceptable choices: wait for the complete replay, or, only when the resulting
data loss is explicitly accepted, restore from the last usable fsimage and
discard the subsequent edit history.
The latter loses namespace changes recorded after the checkpoint and is not a
safe or generally recommended workaround. A validated offline compaction
command could provide a safer recovery option by preserving the complete
transaction stream while reducing the fixed per-segment processing overhead.
> Add an offline command to compact excessive finalized edit-log segments
> -----------------------------------------------------------------------
>
> Key: HDFS-17963
> URL: https://issues.apache.org/jira/browse/HDFS-17963
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, tools
> Reporter: Geqi Hu
> Priority: Major
>
> Healthy HDFS clusters with regularly completed checkpoints normally do not
> accumulate an excessive number of finalized edit-log segments. However,
> long-lived idle or test clusters may continue rolling edit logs while
> checkpointing is disabled, misconfigured, or failing.
> I have encountered this situation repeatedly while handling HDFS recovery
> incidents with strict recovery-time objectives.
> In one anonymized case from an HDFS-compatible distribution:
> * 113,340 finalized edit-log segments had accumulated;
> * their combined size was only 7,583,956 bytes;
> * 107,382 segments (94.74%) were 42-byte segments containing only
> \{{OP_START_LOG_SEGMENT}} and \{{OP_END_LOG_SEGMENT}};
> * only 5,958 segment files contained non-marker operations; and
> * the required replay range was txid 806689 through 1045052.
> Two NameNodes replayed the sequence at nearly identical rates of
> approximately 704 and 705 txids/hour. At txid 827760, only about 8.8% had
> been processed, with an estimated 308 hours (12.8 days) remaining.
> This suggests that repeated segment opening, initialization, validation, and
> closing dominated the recovery time rather than edit-log byte size or
> business-operation replay.
> Deleting the marker-only files is not safe because their transaction IDs are
> part of the continuous history and valid operations are interspersed among
> them. Fixing checkpointing prevents future accumulation but does not reduce
> an existing backlog when the NameNode cannot complete startup.
> I propose adding an explicitly offline and experimental command to rewrite a
> continuous sequence of finalized edit-log segments into fewer output
> segments. The command should:
> * validate filename ranges and decoded transaction IDs;
> * reject gaps, overlaps, duplicate ranges, corruption, and incompatible
> layout versions;
> * preserve transaction IDs, operation order, and operation contents;
> * write only to a separate output directory; and
> * re-read and validate all generated output before reporting success.
> It would not process in-progress segments, modify input files, access a
> running NameNode or JournalNode, replace live metadata, renumber
> transactions, or manage service startup and rollback.
> I implemented an independent prototype. For the case above, it rewrote the
> 113,340 segments into one output file, produced identical source/output
> operation-stream SHA-256 digests, and the NameNode successfully completed
> startup with the validated result in a controlled recovery test.
> HDFS-2982 addressed repeated directory scans during startup. This proposal
> addresses the fixed per-file overhead of processing an already accumulated
> set of tens or hundreds of thousands of finalized segments.
> Would offline compaction be an acceptable recovery mechanism for this
> scenario, and would OfflineEditsViewer be the appropriate integration point?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
