[ https://issues.apache.org/jira/browse/ZOOKEEPER-1413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thawan Kooburat updated ZOOKEEPER-1413: --------------------------------------- Attachment: ZOOKEEPER-1413.patch Fix bug > Use on-disk transaction log for learner sync up > ----------------------------------------------- > > Key: ZOOKEEPER-1413 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1413 > Project: ZooKeeper > Issue Type: Improvement > Components: server > Affects Versions: 3.4.3 > Reporter: Thawan Kooburat > Assignee: Thawan Kooburat > Priority: Minor > Labels: performance > Fix For: 3.5.0 > > Attachments: ZOOKEEPER-1413.patch, ZOOKEEPER-1413.patch, > ZOOKEEPER-1413.patch, ZOOKEEPER-1413.patch, ZOOKEEPER-1413.patch, > ZOOKEEPER-1413.patch > > > Motivation: > The learner syncs up with leader by retrieving committed log from the leader. > Currently, the leader only keeps 500 entries of recently committed log in > memory. If the learner falls behind more than 500 updates, the leader will > send the entire snapshot to the learner. > With the size of the snapshot for some of our Zookeeper deployments (~10G), > it is prohibitively expensive to send the entire snapshot over network. > Additionally, our Zookeeper may serve more than 4K updates per seconds. As a > result, a network hiccups for less than a second will cause the learner to > use snapshot transfer. > Design: > Instead of looking only at committed log in memory, the leader will also look > at transaction log on disk. The amount of transaction log kept on disk is > configurable and the current default is 100k. This will allow Zookeeper to > tolerate longer temporal network failure before initiating the snapshot > transfer. > Implementation: > We plan to add interface to the persistence layer will can be use to retrieve > proposals from on-disk transaction log. These proposals can then be used to > send to the learner using existing protocol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira