[
https://issues.apache.org/jira/browse/HBASE-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704092#comment-13704092
]
Varun Sharma commented on HBASE-8599:
-------------------------------------
Attached a v3 without the logging...
Do we want to do the 0.94 backport in this JIRA ?
> HLogs in ZK are not cleaned up when replication lag is minimal
> --------------------------------------------------------------
>
> Key: HBASE-8599
> URL: https://issues.apache.org/jira/browse/HBASE-8599
> Project: HBase
> Issue Type: Improvement
> Components: Replication
> Affects Versions: 0.98.0, 0.94.7
> Reporter: Varun Sharma
> Assignee: Varun Sharma
> Fix For: 0.98.0, 0.94.10
>
> Attachments: 8599-0.94.patch, 8599-trunk.patch, 8599-trunk-v2.patch,
> 8599-trunk-v3.patch
>
>
> On a cluster with very low replication lag (as measured by ageOfLastShippedOp
> on source), we found HLogs accumulating and not being cleaned up as new
> WAL(s) are rolled.
> Each time, we call logPositionAndCleanOldLogs() to clean older logs whenever
> the current WAL is not being written to any more - as suggested by
> currentWALBeingWrittenTo being false. However, when lags are small, we may
> hit the following block first and continue onto the next WAL without clearing
> the old WAL(s)...
> ReplicationSource::run() {
> if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo =
> false)) {
> // If we are here, then we advance to the next WAL without any
> cleaning
> // and close existing WAL
> continue;
> }
> // Ship some edits and call logPositionAndCleanOldLogs
> }
> If we hit readAllEntriesToReplicateOrNextFile(false) only once - then older
> logs are not cleaned out and persist in the zookeeper node since we simply
> call "continue" and skip the subsequent logPositionAndCleanOldLogs call - if
> its called more than once, we do end up clearing the old logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
