Hi, I am afraid I don’t understand your question.
Do you not understand the difference between synchronous and asynchronous communication? Look: Synchronous… I’m not going to do anything until I hear from you or I time out and resend my request. Think of having a phone conversation. You say something and then wait for a response. Asynchronous… I’m going to send a bit of information and then go on and do something else and not wait for a response. Think of writing a post-it note and leaving on the fridge for your wife to find. Or leaving a voice mail message that you’re heading out to the pub for a quick drink and you will be late to dinner. ;-) Ok… I realize I’m stating the obvious… but that really should explain what you are seeing. The message is sent and then ZK goes on doing something else… and the response is somewhere in the queue to be processed at a later time. What’s wrong with that? Your own results show that the more activity ZK is doing, the longer the delay in receiving the ACK from the response. -Mike On Oct 23, 2014, at 7:21 PM, Ibrahim El-sanosi (PGR) <[email protected]> wrote: > Hi Rakesh, > > First of all, the zookeeper ensemble consists of five Zookeeper servers. Also > I have another 10 clients machines used to send write requests to Zookeeper. > The benchmark code creates 5 threads (equal to number of Zookeeper server) , > each thread associates with one Zookeeper server. So, in this case, each > zookeeper server will receive a set of write requests. The benchmark code > runs for 30 seconds. > > Async tests: > > * Number of clients > In fact, I have different test, each test has different number of clients. > For example, the bellow shows the latency corresponds to different number of > clients: > Five clients: Latency min/avg/max: 235/366/515 > Ten clients: Latency min/avg/max: 252/368/505 > > * Number of threads > As explained above, each client creates 5 threads and each thread connects to > one Zookeeper server. For instance, test using 5 clients’ machines, each > Zookeeper server receives five threads. > > * data size storing in each znode > The data size store in znode is 100 bytes > > Also, it would be good to monitor : > > 1) JVM stats(one way is through JMX) like heap, gc activities. This is to see > if latency spike corresponds to gc activity or not. > > If you mean by JVM stats the four word stat command, then the latency result > showed above is generated using this command. If you mean something else then > I have to read about and tell you late on. > > 2) Since you are doubting fsync, I think $ iostat would be helpful to see > disk statistics. For example, $ iostat -d -x 2 10 and collects the disk > latency. > > Yes, the batch size that I use in SyncrequestProcessor class is 1000 > requests. I think this is preferable size. Also, I will try to use iostat. > > 3) CPU usage through top or sar unix commands. I didn't use sar , but I could > see it gives more details like percent of CPU running idle with a process > waiting for block I/O etc. > > Yes, I will use the top command to gathering the resource utilization. > However, I don’t think top or sar will answer my question. Because I am > thinking there is different between Asynchroned and Synchronized mode for > measuring the latency. > > Thank you for your attention > > I look forward to hearing from you > > > Ibrahim > > -----Original Message----- > From: Rakesh Radhakrishnan [mailto:[email protected]] > Sent: Thursday, October 23, 2014 03:58 م > To: [email protected] > Subject: Re: Latency in asynchronous mode > > Hi Ibrahim, > > In async tests, could you give the details like: > > * number of clients > * number of threads > * data size storing in each znode > > Also, it would be good to monitor : > > 1) JVM stats(one way is through JMX) like heap, gc activities. This is to see > if latency spike corresponds to gc activity or not. > > 2) Since you are doubting fsync, I think $ iostat would be helpful to see > disk statistics. For example, $ iostat -d -x 2 10 and collects the disk > latency. > > 3) CPU usage through top or sar unix commands. I didn't use sar , but I could > see it gives more details like percent of CPU running idle with a process > waiting for block I/O etc. > > > -Rakesh > > > On Thu, Oct 23, 2014 at 6:44 PM, Alexander Shraer <[email protected]> wrote: > >> Maybe due to queueing at the leader in asynchronous mode - if in your >> experiment you have one client in sync mode the leader has just one op >> in the queue at a time On Oct 23, 2014 1:57 PM, "Ibrahim" >> <[email protected]> wrote: >> >>> Hi folks, >>> >>> I am testing ZooKeeper latency in Asynchronous mode. I am sending >>> update >>> (write) requests to Zookeeper cluster that consists of 5 physical >>> Zookeeper. >>> >>> So, when I run the stat command I get high latency like: >>> Latency min/avg/max: 7/339/392 >>> Latency min/avg/max: 1/371/627 >>> Latency min/avg/max: 1/371/627 >>> Latency min/avg/max: 1/364/674 >>> I guess such high latency correspond to fsync (batch requests). But >>> I >> wish >>> if someone could help me and explain this behaviour. >>> >>> However, testing Zookeeper using Synchronous mode, it gives me >>> reasonable result like: >>> Latency min/avg/max: 6/24/55 >>> Latency min/avg/max: 7/22/61 >>> Latency min/avg/max: 7/30/65 >>> >>> Note that the latency measures in milliseconds. >>> >>> I look forward to hearing from you. >>> >>> Ibrahim >>> >>> >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> >> http://zookeeper-user.578899.n2.nabble.com/Latency-in-asynchronous-mod >> e-tp7580446.html >>> Sent from the zookeeper-user mailing list archive at Nabble.com. >>> >>
