Re: permanent ZSESSIONMOVED

2010-03-18 Thread Ted Dunning
Thanks to you for figuring out that it was a problem (and for having a new and strange environment to test ZK in!) On Thu, Mar 18, 2010 at 1:36 PM, Łukasz Osipiuk wrote: > That is great. Thank you very much for your work!

Re: permanent ZSESSIONMOVED

2010-03-18 Thread Łukasz Osipiuk
Hi! On Thu, Mar 18, 2010 at 21:23, Patrick Hunt wrote: > We've identified the problem and are working on a fix. Full details here: > > https://issues.apache.org/jira/browse/ZOOKEEPER-710 > That is great. Thank you very much for your work! All the best, Łukasz > Patrick > > Łukasz Osipiuk wr

Re: permanent ZSESSIONMOVED

2010-03-18 Thread Patrick Hunt
We've identified the problem and are working on a fix. Full details here: https://issues.apache.org/jira/browse/ZOOKEEPER-710 Patrick Łukasz Osipiuk wrote: Hello Mahadev, 2010/3/18 Mahadev Konar : Hi Lukasz, I looked at the logs and havent been able to determine how the session move can hap

Re: permanent ZSESSIONMOVED

2010-03-18 Thread Łukasz Osipiuk
Hello Mahadev, 2010/3/18 Mahadev Konar : > Hi Lukasz, >  I looked at the logs and havent been able to determine how the session > move can happen. > > >  You should go ahead and open a jira for this! Please attach the logs to > that jira. Mark it for 3.3.0 until we can say that it isnt a problem a

Re: permanent ZSESSIONMOVED

2010-03-18 Thread Michael Bauland
Hi Patrick, thanks for your detailed answer. >> I'm not sure, but maybe my input could help, too. As I mentioned >> earlier, I also run the three zookeeper servers not in a local >> environment but across two sites in different countries (soon, in >> production, it'll be three sites). >> I'm acc

Re: permanent ZSESSIONMOVED

2010-03-17 Thread Patrick Hunt
JIRA is great because we can move the discussion off the list and track explicitly along with possible fixes. Also if you create the jira you'll get updates on any changes to the record (you could 'watch' the JIRA instead, that's fine too). Can you include the session id in your client logs? T

Re: permanent ZSESSIONMOVED

2010-03-17 Thread Mahadev Konar
Hi Lukasz, I looked at the logs and havent been able to determine how the session move can happen. You should go ahead and open a jira for this! Please attach the logs to that jira. Mark it for 3.3.0 until we can say that it isnt a problem as of now. Also, please go ahead and attach non fi

Re: permanent ZSESSIONMOVED

2010-03-17 Thread Patrick Hunt
Thanks for this Michael, a bit of clarification/advice inline below: Michael Bauland wrote: Hi, This is very different from most uses of ZK. Normally Zookeeper is used on a private network with almost no packet loss. Your high rate of loss may be tickling a protocol bug that other people jus

Re: permanent ZSESSIONMOVED

2010-03-17 Thread Łukasz Osipiuk
Ok. I analysed some logs from time when problem last occurred. brokensession.txt - logs concerning session which had a problem leader.txt - logs concerning leader election (i grepped for Leader in zookeeper.log) Some observations below - Network glitch which resulted in problem occurred at about

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Ted Dunning
Hmm... this inspires me to have a thought as well. Łukasz, there isn't any fancy network stuff going on here is there? No NATing or fancy load balancing or reassignment of IP addresses of servers, right? On Tue, Mar 16, 2010 at 4:51 PM, Patrick Hunt wrote: > It will be good to see the logs, ho

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Patrick Hunt
It will be good to see the logs, however I had one additional thought. The leader (the zk leader) is the one checking for session MOVED. It keeps track of which server the session is currently attached to and will throw the moved exception if the session proposes a request through a server oth

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Patrick Hunt
Agree. Patrick Ted Dunning wrote: This is very different from most uses of ZK. Normally Zookeeper is used on a private network with almost no packet loss. Your high rate of loss may be tickling a protocol bug that other people just never see. On Tue, Mar 16, 2010 at 12:24 PM, Łukasz Osipiuk

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Ted Dunning
This is very different from most uses of ZK. Normally Zookeeper is used on a private network with almost no packet loss. Your high rate of loss may be tickling a protocol bug that other people just never see. On Tue, Mar 16, 2010 at 12:24 PM, Łukasz Osipiuk wrote: > We get reconnects > quite

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Patrick Hunt
Yes, if you search "back" (older entries) in the server log you will be able to see who the leader is, it will say something like "LEADING" or "FOLLOWING", but this may change over time (which is why you need to search "back" as I mention) if leadership within the ZK cluster changes (say due to

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Patrick Hunt
Yea, that's great. (no complaints on my end, just wasn't sure what you meant, wanted to make sure I was clear.). Can you identify some sort of pattern? We're discussing on our end, but this is the first time we've heard of such an issue (outside 3.2.0 bug) and we're looking for a direction to

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Łukasz Osipiuk
On Tue, Mar 16, 2010 at 20:05, Patrick Hunt wrote: > We'll probably need the ZK server/client logs to hunt this down. Can you > tell if the MOVED happens in some particular scenario, say you are connected > to a follower and move to a leader, or perhaps you are connected to server > A, get disconn

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Łukasz Osipiuk
On Tue, Mar 16, 2010 at 20:05, Patrick Hunt wrote: > We'll probably need the ZK server/client logs to hunt this down. Can you > tell if the MOVED happens in some particular scenario, say you are connected > to a follower and move to a leader, or perhaps you are connected to server > A, get disconn

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Patrick Hunt
We'll probably need the ZK server/client logs to hunt this down. Can you tell if the MOVED happens in some particular scenario, say you are connected to a follower and move to a leader, or perhaps you are connected to server A, get disconnected and reconnected to server A? is there some pa

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Łukasz Osipiuk
On Tue, Mar 16, 2010 at 19:22, Patrick Hunt wrote: > Łukasz Osipiuk wrote: >> >> On Tue, Mar 16, 2010 at 17:18, Patrick Hunt wrote: >>> >>> Can you verify that you are using 3.2.2 on all servers? You can do this >>> by >>> running the "stat" command against each of your servers and look at the >>

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Patrick Hunt
Łukasz Osipiuk wrote: On Tue, Mar 16, 2010 at 17:18, Patrick Hunt wrote: Can you verify that you are using 3.2.2 on all servers? You can do this by running the "stat" command against each of your servers and look at the very top of the output (we include the version of zk server there). http://

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Łukasz Osipiuk
On Tue, Mar 16, 2010 at 17:18, Patrick Hunt wrote: > Can you verify that you are using 3.2.2 on all servers? You can do this by > running the "stat" command against each of your servers and look at the very > top of the output (we include the version of zk server there). > http://bit.ly/dglVld Th

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Patrick Hunt
Can you verify that you are using 3.2.2 on all servers? You can do this by running the "stat" command against each of your servers and look at the very top of the output (we include the version of zk server there). http://bit.ly/dglVld Are you using synchronous or async operations in your clien

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Łukasz Osipiuk
not really - it happens occasionally - every few days :( I believe it is somewhat connected with our network environment which suffers from some packet loss which leads to connection timeouts. I can switch on some more logging if you can lead me which categories are worth to enable DEBUG for them.

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Benjamin Reed
weird, this does sound like a bug. do you have a reliable way of reproducing the problem? thanx ben On 03/16/2010 08:27 AM, Łukasz Osipiuk wrote: nope. I always pass 0 as clientid. Łukasz On Tue, Mar 16, 2010 at 16:20, Benjamin Reed wrote: do you ever use zookeeper_init() with the cli

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Łukasz Osipiuk
nope. I always pass 0 as clientid. Łukasz On Tue, Mar 16, 2010 at 16:20, Benjamin Reed wrote: > do you ever use zookeeper_init() with the clientid field set to something > other than null? > > ben > > On 03/16/2010 07:43 AM, Łukasz Osipiuk wrote: >> >> Hi everyone! >> >> I am writing to this gr

Re: permanent ZSESSIONMOVED

2010-03-16 Thread Benjamin Reed
do you ever use zookeeper_init() with the clientid field set to something other than null? ben On 03/16/2010 07:43 AM, Łukasz Osipiuk wrote: Hi everyone! I am writing to this group because recently we are getting some strange errors with our production zookeeper setup. From time to time we

permanent ZSESSIONMOVED

2010-03-16 Thread Łukasz Osipiuk
Hi everyone! I am writing to this group because recently we are getting some strange errors with our production zookeeper setup. >From time to time we are observing that our client application (C++ based) disconnects from zookeeper (session state is changed to 1) and reconnects (state changed to