Re: zxid integer overflow
I'm not aware of sustained 1k/sec, Ben might know how long the 20k/sec test runs for (and for how long that rate is sustained). You'd definitely want to tune the GC, GC related pauses would be the biggest obstacle for this (assuming you are using a dedicated log device for the transaction logs). Patrick On Tue, Oct 19, 2010 at 3:14 PM, Sandy Pratt prat...@adobe.com wrote: Follow up question: does anyone have a production cluster that handles a similar sustained rate of changes? -Original Message- From: Benjamin Reed [mailto:br...@yahoo-inc.com] Sent: Tuesday, October 19, 2010 2:53 PM To: zookeeper-user@hadoop.apache.org Subject: Re: zxid integer overflow we should put in a test for that. it is certainly a plausible scenario. in theory it will just flow into the next epoch and everything will be fine, but we should try it and see. ben On 10/19/2010 11:33 AM, Sandy Pratt wrote: Just as a thought experiment, I was pondering the following: ZK stamps each change to its managed state with a zxid ( http://hadoop.apache.org/zookeeper/docs/r3.2.1/zookeeperInternals.html). That ID consists of a 64 bit number in which the upper 32 bits are the epoch, which changes when the leader does, and the bottom 32 bits are a counter, which is incremented by the leader with every change. If 1000 changes are made to ZK state each second (which is 1/20th of the peak rate advertised), then the counter portion will roll over in 2^32 / (86400 * 1000) = 49 days. Now, assuming that my math is correct, is this an actual concern? For example, if I'm using ZK to provide locking for a key value store that handles transactions at about that rate, am I setting myself up for failure? Thanks, Sandy
zxid integer overflow
Just as a thought experiment, I was pondering the following: ZK stamps each change to its managed state with a zxid (http://hadoop.apache.org/zookeeper/docs/r3.2.1/zookeeperInternals.html). That ID consists of a 64 bit number in which the upper 32 bits are the epoch, which changes when the leader does, and the bottom 32 bits are a counter, which is incremented by the leader with every change. If 1000 changes are made to ZK state each second (which is 1/20th of the peak rate advertised), then the counter portion will roll over in 2^32 / (86400 * 1000) = 49 days. Now, assuming that my math is correct, is this an actual concern? For example, if I'm using ZK to provide locking for a key value store that handles transactions at about that rate, am I setting myself up for failure? Thanks, Sandy
Re: zxid integer overflow
we should put in a test for that. it is certainly a plausible scenario. in theory it will just flow into the next epoch and everything will be fine, but we should try it and see. ben On 10/19/2010 11:33 AM, Sandy Pratt wrote: Just as a thought experiment, I was pondering the following: ZK stamps each change to its managed state with a zxid (http://hadoop.apache.org/zookeeper/docs/r3.2.1/zookeeperInternals.html). That ID consists of a 64 bit number in which the upper 32 bits are the epoch, which changes when the leader does, and the bottom 32 bits are a counter, which is incremented by the leader with every change. If 1000 changes are made to ZK state each second (which is 1/20th of the peak rate advertised), then the counter portion will roll over in 2^32 / (86400 * 1000) = 49 days. Now, assuming that my math is correct, is this an actual concern? For example, if I'm using ZK to provide locking for a key value store that handles transactions at about that rate, am I setting myself up for failure? Thanks, Sandy
RE: zxid integer overflow
Follow up question: does anyone have a production cluster that handles a similar sustained rate of changes? -Original Message- From: Benjamin Reed [mailto:br...@yahoo-inc.com] Sent: Tuesday, October 19, 2010 2:53 PM To: zookeeper-user@hadoop.apache.org Subject: Re: zxid integer overflow we should put in a test for that. it is certainly a plausible scenario. in theory it will just flow into the next epoch and everything will be fine, but we should try it and see. ben On 10/19/2010 11:33 AM, Sandy Pratt wrote: Just as a thought experiment, I was pondering the following: ZK stamps each change to its managed state with a zxid (http://hadoop.apache.org/zookeeper/docs/r3.2.1/zookeeperInternals.html). That ID consists of a 64 bit number in which the upper 32 bits are the epoch, which changes when the leader does, and the bottom 32 bits are a counter, which is incremented by the leader with every change. If 1000 changes are made to ZK state each second (which is 1/20th of the peak rate advertised), then the counter portion will roll over in 2^32 / (86400 * 1000) = 49 days. Now, assuming that my math is correct, is this an actual concern? For example, if I'm using ZK to provide locking for a key value store that handles transactions at about that rate, am I setting myself up for failure? Thanks, Sandy
