Hi Dejan,
For a typical ZK application, granularity of hours is more than enough, since
it is supposed to be an infrequent background task. In your case, it sounds
like durability isn't an important property because if it is you shouldn't be
getting rid of disk data this fast. I'm also wondering about the amount of data
you're generating. What's the size of your snapshots and txn logs?
-Flavio
On Thursday, April 16, 2015 10:26 AM, Dejan Markic
<[email protected]> wrote:
Hello Flavio!
Would that mean, that zkCleanup.sh would not be needed?
PurgeInterval is minimum 1 hour? Why is it so high?
Thanks!
Kind regards,
Dejan Markic
-----Original Message-----
From: Flavio Junqueira [mailto:[email protected]]
Sent: Thursday, April 16, 2015 11:15 AM
To: [email protected]
Subject: Re: Transaction logs and snapshots
Hi Dejan,
Check if the autopurge feature solves your problem:
http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html#sc_advancedConfiguration
-Flavio
On Thursday, April 16, 2015 9:17 AM, Dejan Markic
<[email protected]> wrote:
Hello all!
We are running 3 ZK servers in ensemble, and ZK is processing a lot of commands
per seconds. There are probably around 300 nodes created/checked/set/get per
second.
Since we have only information about live sessions we handle in ZK, we don't
need any data persistency - eg: we can stop all nodes, clean all transaction
logs/snapshots, and start them up again, without any issues.
Since we have a lot of requests/changes, we have moved dataDir onto ramdisk, so
we have no problems with disk IOPS, etc.
Is there a way, to minimze the usage of snapshots/logs so ramdisk would not get
filled up? It happens that transaction logs/snapshots grow so large, that we
run out of space on ramdisk.
We issue >/usr/share/zookeeper/bin/zkCleanup.sh -n 3< every 2 minutes, so this
should cleanup the dataDir quite often. Why is >count number of snapshots/logs
to keep< limited to 3 and not below?
I assume, in my setup, I don't even need snapshots/logs to be stored after they
are not actively needed?
So my basic questions are:
- can I somehow get rid of snapshot/logs sooner, more often ... ?
- when is snapshot created? Can it be created sooner, so it would be smaller?
- Is it possible to get rid of snapshot/logs all together?
Thank you for all your inputs and kind regards, Dejan Markic
