[
https://issues.apache.org/jira/browse/HDFS-15887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangHua Zhu updated HDFS-15887:
--------------------------------
Description:
In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and
when a checkpoint occurs, it will compete with TailEdits for lock
(FSNamesystem#cpLock).
Usually, it takes a long time to execute checkpoint, which will cause the size
of the generated edit log file to be relatively large.
For example, here is an actual effect:
The StandbyCheckpointer log is triggered as follows : edit_files.jpg
2021-03-11 09:18:42,513 [769071096]-INFO [Standby State
Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint
because there have been 5142154 txns since the last checkpoint, which exceeds
the configured threshold 1000000
When loading an edit log with a large amount of data, the processing time will
be longer. We should make the edit log size as even as possible, which is good
for the operation of the system.
was:
In the EditLogTailer class, LogRoll and TailEdits are executed in a thread, and
when a checkpoint occurs, it will compete with TailEdits for lock
(FSNamesystem#cpLock).
Usually, it takes a long time to execute checkpoint, which will cause the size
of the generated edit log file to be relatively large.
For example, here is an actual effect:
The StandbyCheckpointer log is triggered as follows : edit_files.jpg
2021-03-11 09:18:42,513 [769071096]-INFO [Standby State
Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering checkpoint
because there have been 5142154 txns since the last checkpoint, which exceeds
the configured threshold 1000000
> Make LogRoll and TailEdits execute in parallel
> ----------------------------------------------
>
> Key: HDFS-15887
> URL: https://issues.apache.org/jira/browse/HDFS-15887
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
> Attachments: edit_files.jpg
>
>
> In the EditLogTailer class, LogRoll and TailEdits are executed in a thread,
> and when a checkpoint occurs, it will compete with TailEdits for lock
> (FSNamesystem#cpLock).
> Usually, it takes a long time to execute checkpoint, which will cause the
> size of the generated edit log file to be relatively large.
> For example, here is an actual effect:
> The StandbyCheckpointer log is triggered as follows : edit_files.jpg
> 2021-03-11 09:18:42,513 [769071096]-INFO [Standby State
> Checkpointer:StandbyCheckpointer$CheckpointerThread@335]-Triggering
> checkpoint because there have been 5142154 txns since the last checkpoint,
> which exceeds the configured threshold 1000000
> When loading an edit log with a large amount of data, the processing time
> will be longer. We should make the edit log size as even as possible, which
> is good for the operation of the system.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
