[
https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu updated HDFS-15887:
--------------------------------
Description:
In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and
when a checkpoint occurs, it will compete with TailEdits for lock
(FSNamesystem#cpLock).
Usually, it takes a long time to execute checkpoint, which will cause the size
of the generated edit log file to be relatively large.
For example, here is an actual effect:
The StandbyCheckpointer log is triggered as follows : edit_files.jpg
2021-03-11 09:18:42,513 [769071096]-INFO [Standby State
Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint
because there have been 5142154 txns since the last checkpoint, which exceeds
the configured threshold 1000000
was:
When checkpoint action and roll edit log action occur at the same time, the
generated edit file will be larger, which is unfavorable for the tail edit file.
For example, here is an actual effect:
The StandbyCheckpointer log is triggered as follows : edit_files.jpg
2021-03-11 09:18:42,513 [769071096]-INFO [Standby State
Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint
because there have been 5142154 txns since the last checkpoint, which exceeds
the configured threshold 1000000
> Make LogRoll and TailEdits execute in parallel
> ----------------------------------------------
>
> Key: HDFS-15887
> URL: https://issues.apache.org/jira/browse/HDFS-15887
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
> Attachments: edit_files.jpg
>
>
> In the EditLogTailer class, LogRoll and TailEdits are executed in a thread,
> and when a checkpoint occurs, it will compete with TailEdits for lock
> (FSNamesystem#cpLock).
> Usually, it takes a long time to execute checkpoint, which will cause the
> size of the generated edit log file to be relatively large.
> For example, here is an actual effect:
> The StandbyCheckpointer log is triggered as follows : edit_files.jpg
> 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State
> Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering
> checkpoint because there have been 5142154 txns since the last checkpoint,
> which exceeds the configured threshold 1000000
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
