I have a largish index with a lot of churn, and inserts that come in large bursts. My server is a multiprocessor with plenty of memory, so I can multi-thread and stuff in about 1.6 million records per hour, going full speed. I use a dozen or so threads to post curl inserts, and monitor the responses.

Using jetty, there is ~10% failure rate with no server response code received. Switching to tomcat reduces the error rate to around 2%. (which makes me like tomcat a lot, even though I'm a dog person...). I suspect I'm overrunning the capacity of the servlet container. Tweaking parameters in Jetty improved performance, and I can tune Tomcat. But then I'll just be overrunning a tuned system, at a slightly faster rate.

My work around is to keep track of which inserts fail, but I suspect there's a better approach. Any suggestions how I can balance maximum insert speed with a low error rate? Thanks!

         -Jim

Reply via email to