Thank you Patrick,
Ibrahim On Wed, Mar 23, 2016 at 4:44 PM, Patrick Hunt <[email protected]> wrote: > On Tue, Mar 22, 2016 at 7:53 AM, ibrahim El-sanosi > <[email protected]> wrote: > > Thank you Patrick, > > > > > > > >>That's correct. ZK is an in-memory store. As such all znodes > > > >>representing the current state are held in memory. > > > >> > > > >>> 2. If the leader crashes or lost a quorum, recovery mechanism applies > > > >>> latest snapshots. In this scenario, it is possible to eliminate the old > > ZK > > > >>> states and could save memory space. > > > >>> > > > >> > > > >>This won't affect the in-memory state. It will be the same as prior to > > > >>the recovery. > > > > > > > > Observation (2), I assume the worst scenario when all ZK servers want > down > > and start recovery phase (No ZK states persent in memory at the moment). > > Does ZK servers reapply all snapshot in memory or just takes the latest > one? > > > > The most recent valid snapshot is read and any transactions written to > the logs subsequent to that snapshot are replayed. > > Patrick > > > > > > > Ibrahim > > > > On Tue, Mar 22, 2016 at 2:43 PM, Patrick Hunt <[email protected]> wrote: > > > >> On Fri, Mar 18, 2016 at 10:10 AM, ibrahim El-sanosi > >> <[email protected]> wrote: > >> > Ok, Thank you for answering the question. > >> > > >> > So, what I understand from this thread is as following: > >> > > >> > 1. When the crash does occur (Leader does not crash and majority of > >> > servers is always up), the state from Time T0 (ZK start of ZK) and > until > >> > time Tn (ZK state of now) will be in memory (even the size of ZK state > >> > reaches to the memory size). > >> > > >> > >> That's correct. ZK is an in-memory store. As such all znodes > >> representing the current state are held in memory. > >> > >> > 2. If the leader crashes or lost a quorum, recovery mechanism applies > >> > latest snapshots. In this scenario, it is possible to eliminate the > old > >> ZK > >> > states and could save memory space. > >> > > >> > >> This won't affect the in-memory state. It will be the same as prior to > >> the recovery. > >> > >> Patrick > >> > >> > > >> > Is that right? > >> > > >> > > >> > > >> > Regards, > >> > > >> > Ibrahim > >> > > >> > On Fri, Mar 18, 2016 at 4:49 PM, Flavio Junqueira <[email protected]> > >> wrote: > >> > > >> >> It is currently as Jordan says, although it is not entirely > unreasonable > >> >> to think of memory as cache and keep in memory only the current > working > >> >> set, possibly using say an SSD to store the remaining part of the > state > >> >> that you don't want to keep in memory. One key issue here is that if > you > >> >> have a pretty large state (that you don't want to keep all in > memory), > >> then > >> >> you'll end up increasing worst-case recovery time. > >> >> > >> >> We certainly do not have any kind of expiration mechanism for znodes > >> other > >> >> than the ephemeral flag, which is associated to sessions. > >> >> > >> >> -Flavio > >> >> > >> >> > On 18 Mar 2016, at 16:38, ibrahim El-sanosi < > [email protected] > >> > > >> >> wrote: > >> >> > > >> >> > Thank you for replaying. > >> >> > > >> >> > > >> >> > Take this example, Zookeeper started on 1/1/2015, and assume ZK > >> servers > >> >> are > >> >> > never crashed. Also, the ZK are very busy, reciveing continusly > write > >> >> > requests from clients and accordingly snapshots are generated > >> overtime. > >> >> On > >> >> > 1/3/2015, what Znodes will be in memory data tree? do all Znodes > >> still > >> >> > store in memory (from 1/1/2015 to 1/3/2015) as there is no crashed > >> occur. > >> >> > Or Dsnapshhot is taken the data tree is > >> >> > > >> >> > > >> >> > Note that I am not asking about data in log or snapshoot. I am > asking > >> >> > about the current data in memory. > >> >> > > >> >> > On Fri, Mar 18, 2016 at 4:12 PM, Flavio Junqueira <[email protected]> > >> >> wrote: > >> >> > > >> >> >> Hi Ibrahim, > >> >> >> > >> >> >> Are you asking about how we compact old logs? We do it by taking > >> >> snapshots > >> >> >> so that upon recovery, we only load the latest snapshot and replay > >> the > >> >> txn > >> >> >> log from the snapshot tag. The snapshot tag is the last zxid > >> committed > >> >> when > >> >> >> we start producing the snapshot. > >> >> >> > >> >> >> We don't actually delete anything, though, unless you > intentionally > >> turn > >> >> >> on auto purge: > >> >> >> > >> >> >> > >> >> > >> > https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering > >> >> >> < > >> >> >> > >> >> > >> > https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_administering > >> >> >>> > >> >> >> > >> >> >> -Flavio > >> >> >> > >> >> >>> On 18 Mar 2016, at 16:05, ibrahim El-sanosi < > >> [email protected]> > >> >> >> wrote: > >> >> >>> > >> >> >>> Hi all, > >> >> >>> > >> >> >>> Assume the Zookeeper have been running for about one year (from > >> >> 1/1/2015 > >> >> >>> until now), how does ZooKeeper deal with old delivered write > >> requests > >> >> (To > >> >> >>> optimize a memory used) (say from 1/1/2015 to 03/04/2015). I am > >> >> assuming > >> >> >>> the old delivered request are no longer used. > >> >> >>> > >> >> >>> On the other words, what strategy ZK does to find and delete > unused > >> >> >> Znodes? > >> >> >>> > >> >> >>> Ibrahim > >> >> >> > >> >> >> > >> >> > >> >> > >> >
