Re: 3.4.6: Unable to read additional data from client sessionid 0xABC, likely client has closed socket
Mate, Thanks for the reply! We identified several thousand excess znodes lingering in the cluster and after deleting them we see no difference in the error rate. We are exclusively using java clients in our code, no C or python. We do indeed have plans to update our ensembles this year, thanks! -- John On Monday, January 18, 2021, 12:23:47 AM PST, Szalay-Bekő Máté wrote: resending my answer (it was sent originally to jlindw...@yahoo.com.invalid - I wonder why) On Mon, Jan 18, 2021 at 9:12 AM Szalay-Bekő Máté wrote: > Hi John, > > > Could an excessive number/size of znodes be a factor? > I don't think this would be a likely case... more like a client error I > think. > > The stack-trace suggests that the client closed the TCP session > unexpectedly. Maybe the clients are missing the zookeeper.close() calls? > > Also I wonder if it is caused by > https://issues.apache.org/jira/browse/ZOOKEEPER-1105 - are you using C > (or python) client? > (there was this bug, that the C client didn't wait for the proper session > close to finish) > > > Btw: please consider upgrading your cluster, 3.4 is end-of-life now and it > is not supported by the community anymore. > > Cheers, > Mate > > On Thu, Jan 14, 2021 at 7:55 PM John Lindwall > wrote: > >> We're seeing thousands of these a day in our zookeeper logs (zookeeper >> 3.4.6): >> WARN [NIOServerCxn.Factory:X.X.X.X/X.X.X.X:Y:NIOServerCnxn@357] - >> caught end of stream exceptionEndOfStreamException: Unable to read >> additional data from client sessionid 0xABC, likely client has closed socket >> at >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) >> at >> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) >> at java.lang.Thread.run(Thread.java:745) >> >> Any ideas of the cause of this? Could an excessive number/size of znodes >> be a factor? We're not seeing any obvious client-side issues. We're not >> sure but we believe that these were not happening earlier. >> >> -- John > >
Re: 3.4.6: Unable to read additional data from client sessionid 0xABC, likely client has closed socket
resending my answer (it was sent originally to jlindw...@yahoo.com.invalid - I wonder why) On Mon, Jan 18, 2021 at 9:12 AM Szalay-Bekő Máté wrote: > Hi John, > > > Could an excessive number/size of znodes be a factor? > I don't think this would be a likely case... more like a client error I > think. > > The stack-trace suggests that the client closed the TCP session > unexpectedly. Maybe the clients are missing the zookeeper.close() calls? > > Also I wonder if it is caused by > https://issues.apache.org/jira/browse/ZOOKEEPER-1105 - are you using C > (or python) client? > (there was this bug, that the C client didn't wait for the proper session > close to finish) > > > Btw: please consider upgrading your cluster, 3.4 is end-of-life now and it > is not supported by the community anymore. > > Cheers, > Mate > > On Thu, Jan 14, 2021 at 7:55 PM John Lindwall > wrote: > >> We're seeing thousands of these a day in our zookeeper logs (zookeeper >> 3.4.6): >> WARN [NIOServerCxn.Factory:X.X.X.X/X.X.X.X:Y:NIOServerCnxn@357] - >> caught end of stream exceptionEndOfStreamException: Unable to read >> additional data from client sessionid 0xABC, likely client has closed socket >> at >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) >> at >> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) >> at java.lang.Thread.run(Thread.java:745) >> >> Any ideas of the cause of this? Could an excessive number/size of znodes >> be a factor? We're not seeing any obvious client-side issues. We're not >> sure but we believe that these were not happening earlier. >> >> -- John > >
Re: 3.4.6: Unable to read additional data from client sessionid 0xABC, likely client has closed socket
Hi John, > Could an excessive number/size of znodes be a factor? I don't think this would be a likely case... more like a client error I think. The stack-trace suggests that the client closed the TCP session unexpectedly. Maybe the clients are missing the zookeeper.close() calls? Also I wonder if it is caused by https://issues.apache.org/jira/browse/ZOOKEEPER-1105 - are you using C (or python) client? (there was this bug, that the C client didn't wait for the proper session close to finish) Btw: please consider upgrading your cluster, 3.4 is end-of-life now and it is not supported by the community anymore. Cheers, Mate On Thu, Jan 14, 2021 at 7:55 PM John Lindwall wrote: > We're seeing thousands of these a day in our zookeeper logs (zookeeper > 3.4.6): > WARN [NIOServerCxn.Factory:X.X.X.X/X.X.X.X:Y:NIOServerCnxn@357] - caught > end of stream exceptionEndOfStreamException: Unable to read additional data > from client sessionid 0xABC, likely client has closed socket > at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > at > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) > at java.lang.Thread.run(Thread.java:745) > > Any ideas of the cause of this? Could an excessive number/size of znodes > be a factor? We're not seeing any obvious client-side issues. We're not > sure but we believe that these were not happening earlier. > > -- John
3.4.6: Unable to read additional data from client sessionid 0xABC, likely client has closed socket
We're seeing thousands of these a day in our zookeeper logs (zookeeper 3.4.6): WARN [NIOServerCxn.Factory:X.X.X.X/X.X.X.X:Y:NIOServerCnxn@357] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0xABC, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) Any ideas of the cause of this? Could an excessive number/size of znodes be a factor? We're not seeing any obvious client-side issues. We're not sure but we believe that these were not happening earlier. -- John