On Tue, Mar 22, 2016 at 7:53 AM, ibrahim El-sanosi <[email protected]> wrote: > Thank you Patrick, > > > >>That's correct. ZK is an in-memory store. As such all znodes > >>representing the current state are held in memory. > >> > >>> 2. If the leader crashes or lost a quorum, recovery mechanism applies > >>> latest snapshots. In this scenario, it is possible to eliminate the old > ZK > >>> states and could save memory space. > >>> > >> > >>This won't affect the in-memory state. It will be the same as prior to > >>the recovery. > > > > Observation (2), I assume the worst scenario when all ZK servers want down > and start recovery phase (No ZK states persent in memory at the moment). > Does ZK servers reapply all snapshot in memory or just takes the latest one? >
The most recent valid snapshot is read and any transactions written to the logs subsequent to that snapshot are replayed. Patrick > > > Ibrahim > > On Tue, Mar 22, 2016 at 2:43 PM, Patrick Hunt <[email protected]> wrote: > >> On Fri, Mar 18, 2016 at 10:10 AM, ibrahim El-sanosi >> <[email protected]> wrote: >> > Ok, Thank you for answering the question. >> > >> > So, what I understand from this thread is as following: >> > >> > 1. When the crash does occur (Leader does not crash and majority of >> > servers is always up), the state from Time T0 (ZK start of ZK) and until >> > time Tn (ZK state of now) will be in memory (even the size of ZK state >> > reaches to the memory size). >> > >> >> That's correct. ZK is an in-memory store. As such all znodes >> representing the current state are held in memory. >> >> > 2. If the leader crashes or lost a quorum, recovery mechanism applies >> > latest snapshots. In this scenario, it is possible to eliminate the old >> ZK >> > states and could save memory space. >> > >> >> This won't affect the in-memory state. It will be the same as prior to >> the recovery. >> >> Patrick >> >> > >> > Is that right? >> > >> > >> > >> > Regards, >> > >> > Ibrahim >> > >> > On Fri, Mar 18, 2016 at 4:49 PM, Flavio Junqueira <[email protected]> >> wrote: >> > >> >> It is currently as Jordan says, although it is not entirely unreasonable >> >> to think of memory as cache and keep in memory only the current working >> >> set, possibly using say an SSD to store the remaining part of the state >> >> that you don't want to keep in memory. One key issue here is that if you >> >> have a pretty large state (that you don't want to keep all in memory), >> then >> >> you'll end up increasing worst-case recovery time. >> >> >> >> We certainly do not have any kind of expiration mechanism for znodes >> other >> >> than the ephemeral flag, which is associated to sessions. >> >> >> >> -Flavio >> >> >> >> > On 18 Mar 2016, at 16:38, ibrahim El-sanosi <[email protected] >> > >> >> wrote: >> >> > >> >> > Thank you for replaying. >> >> > >> >> > >> >> > Take this example, Zookeeper started on 1/1/2015, and assume ZK >> servers >> >> are >> >> > never crashed. Also, the ZK are very busy, reciveing continusly write >> >> > requests from clients and accordingly snapshots are generated >> overtime. >> >> On >> >> > 1/3/2015, what Znodes will be in memory data tree? do all Znodes >> still >> >> > store in memory (from 1/1/2015 to 1/3/2015) as there is no crashed >> occur. >> >> > Or Dsnapshhot is taken the data tree is >> >> > >> >> > >> >> > Note that I am not asking about data in log or snapshoot. I am asking >> >> > about the current data in memory. >> >> > >> >> > On Fri, Mar 18, 2016 at 4:12 PM, Flavio Junqueira <[email protected]> >> >> wrote: >> >> > >> >> >> Hi Ibrahim, >> >> >> >> >> >> Are you asking about how we compact old logs? We do it by taking >> >> snapshots >> >> >> so that upon recovery, we only load the latest snapshot and replay >> the >> >> txn >> >> >> log from the snapshot tag. The snapshot tag is the last zxid >> committed >> >> when >> >> >> we start producing the snapshot. >> >> >> >> >> >> We don't actually delete anything, though, unless you intentionally >> turn >> >> >> on auto purge: >> >> >> >> >> >> >> >> >> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering >> >> >> < >> >> >> >> >> >> https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering >> >> >>> >> >> >> >> >> >> -Flavio >> >> >> >> >> >>> On 18 Mar 2016, at 16:05, ibrahim El-sanosi < >> [email protected]> >> >> >> wrote: >> >> >>> >> >> >>> Hi all, >> >> >>> >> >> >>> Assume the Zookeeper have been running for about one year (from >> >> 1/1/2015 >> >> >>> until now), how does ZooKeeper deal with old delivered write >> requests >> >> (To >> >> >>> optimize a memory used) (say from 1/1/2015 to 03/04/2015). I am >> >> assuming >> >> >>> the old delivered request are no longer used. >> >> >>> >> >> >>> On the other words, what strategy ZK does to find and delete unused >> >> >> Znodes? >> >> >>> >> >> >>> Ibrahim >> >> >> >> >> >> >> >> >> >> >>
