Re: Expiring session... timeout of 600000ms exceeded
Am not sure, if anyone responded to this or not. Are the clients getting session expired or getting Connectionloss? In any case, zookeeper client has its own thread to updated the server with active connection status. Did you take a look at the GC activity at your client? Thanks mahadev On 9/21/10 8:24 AM, Tim Robertson timrobertson...@gmail.com wrote: Hi all, I am seeing a lot of my clients being kicked out after the 10 minute negotiated timeout is exceeded. My clients are each a JVM (around 100 running on a machine) which are doing web crawling of specific endpoints and handling the response XML - so they do wait around for 3-4 minutes on HTTP timeouts, but certainly not 10 mins. I am just prototyping right now on a 2xquad core mac pro with 12GB memory, and the 100 child processes only get -Xmx64m and I don't see my machine exhausted. Do my clients need to do anything in order to initiate keep alive heart beats or should this be automatic (I thought the ticktime would dictate this)? # my conf is: tickTime=2000 dataDir=/Volumes/Data/zookeeper clientPort=2181 maxClientCnxns=1 minSessionTimeout=4000 maxSessionTimeout=80 Thanks for any pointers to this newbie, Tim
Expiring session... timeout of 600000ms exceeded
Hi all, I am seeing a lot of my clients being kicked out after the 10 minute negotiated timeout is exceeded. My clients are each a JVM (around 100 running on a machine) which are doing web crawling of specific endpoints and handling the response XML - so they do wait around for 3-4 minutes on HTTP timeouts, but certainly not 10 mins. I am just prototyping right now on a 2xquad core mac pro with 12GB memory, and the 100 child processes only get -Xmx64m and I don't see my machine exhausted. Do my clients need to do anything in order to initiate keep alive heart beats or should this be automatic (I thought the ticktime would dictate this)? # my conf is: tickTime=2000 dataDir=/Volumes/Data/zookeeper clientPort=2181 maxClientCnxns=1 minSessionTimeout=4000 maxSessionTimeout=80 Thanks for any pointers to this newbie, Tim
Re: Expiring session... timeout of 600000ms exceeded
Thanks Ted, To answer your last question first, no you don't have to do anything explicit to keep the ZK connection alive. It is maintained by a dedicated thread. You do have to keep your java program responsive and ZK problems like this almost always indicate that you have a problem with your program checking out for extended periods of time. My strong guess is that you have something evil happening with your java process that is actually causing this delay. Since you have tiny memory, it probably isn't GC. Since you have a bunch of processes, swap and process wakeup delays seem plausible. What is the load average on your box? CPU spikes when responses come in, but mostly it's IO wait on the endpoints (timeout of 3 minutes). I suspect HTTP client 4 is dropping into a retry mechanism though, but have not investigated this yet. On the topic of your application, why you are using processes instead of threads? With threads, you can get your memory overhead down to 10's of kilobytes as opposed to 10's of megabytes. I am just prototyping scaling out many processes and potentially across multiple machines. Our live crawler runs in a single JVM, but some of these crawlers take 4-6 weeks, so long running processes block others, so I was looking at alternatives - our live crawler also uses DOM based XML parsing so hitting memory limits - SAX would address this. Also we want to be able to deploy patches to the crawlers without interrupting those long running jobs if possible. Also, why not use something like Bixo so you don't have to prototype a threaded crawler? It is not a web crawler but more of a custom web service client that issues queries for pages of data. A second query is assembled based on the response of the first. These are Biodiversity domain specific protocols DiGIR, TAPIR and BioCASe which are closer to SOAP based requests / response. I'll look at Bixo. Thanks again, Tim On Tue, Sep 21, 2010 at 8:24 AM, Tim Robertson timrobertson...@gmail.comwrote: Hi all, I am seeing a lot of my clients being kicked out after the 10 minute negotiated timeout is exceeded. My clients are each a JVM (around 100 running on a machine) which are doing web crawling of specific endpoints and handling the response XML - so they do wait around for 3-4 minutes on HTTP timeouts, but certainly not 10 mins. I am just prototyping right now on a 2xquad core mac pro with 12GB memory, and the 100 child processes only get -Xmx64m and I don't see my machine exhausted. Do my clients need to do anything in order to initiate keep alive heart beats or should this be automatic (I thought the ticktime would dictate this)? # my conf is: tickTime=2000 dataDir=/Volumes/Data/zookeeper clientPort=2181 maxClientCnxns=1 minSessionTimeout=4000 maxSessionTimeout=80 Thanks for any pointers to this newbie, Tim
Re: Expiring session... timeout of 600000ms exceeded
Generally best practices for crawlers is that no process runs more than an hour or five. All crawler processes update a central state store with their progress, but they exit when they reach a time limit knowing that somebody else will take up the work where they leave off. This avoids a multitude of ills. On Tue, Sep 21, 2010 at 11:53 AM, Tim Robertson timrobertson...@gmail.comwrote: On the topic of your application, why you are using processes instead of threads? With threads, you can get your memory overhead down to 10's of kilobytes as opposed to 10's of megabytes. I am just prototyping scaling out many processes and potentially across multiple machines. Our live crawler runs in a single JVM, but some of these crawlers take 4-6 weeks, so long running processes block others, so I was looking at alternatives - our live crawler also uses DOM based XML parsing so hitting memory limits - SAX would address this. Also we want to be able to deploy patches to the crawlers without interrupting those long running jobs if possible.
