Your code has no precautions against overwhelming the cluster. 20 worker threads that are not coordinated is a challenge.
I recommend the BulkProcessor class at https://github .com/elasticsearch/elasticsearch/blob/master/src /main/java/org/elasticsearch/action/bulk/BulkProcessor.java SYN flood message is not related to ES. If you have opened your port to public internet access, take care! Don't do it, risk of DOS attack is too high. Jörg On Fri, Mar 28, 2014 at 5:53 PM, <[email protected]> wrote: > You could be right: I can't test right now but this is my code: > > (there may be 20 workerThreads) > As you can see, as each thread submits work, the thread will do a > client.prepareBulk() ... is that sufficient clear out the documents? > > workerThread() { > Client client = getMyGlobalTransportClient(); > BulkRequestBuilder bulkRequest = client.prepareBulk(); > for (...) { > bulkRequest.add(...) > if (bulkRequest.numberOfActions() >= chunksize) { > BulkResponse bulkResponse = bulkRequest.execute().actionGet(); > if (bulkResponse.hasFailures()) { > ... > } else { > ... > } > bulkRequest = client.prepareBulk(); > } > > etc > > > > > On Friday, March 28, 2014 2:44:07 PM UTC, InquiringMind wrote: > >> When I use the Java TransportClient and the BulkRequest builder, my >> throughput is like a scalded cat racing a bolt of greased lightning, with >> the cat way ahead! >> >> "the Java API" does not say how you are using it. Since I cannot see your >> code, I cannot comment on where your mistake is located. >> >> But I have noticed that a small number of folks use BulkRequestBuilder >> but keep adding documents to it. Since BulkRequestBuilder is additive, >> their first bulk load batch contains N documents, their second contains 2N >> (the first batch plus the second batch), their third batch contains 3N, and >> so on until they crash the JVM with OOM errors. >> >> So if this is your mistake, then simply create a new BulkRequestBuilder >> for each batch of documents to submit, and let the previous >> BulkRequestBuilder get garbage collected, and your Java build will run >> lightning fast and never run into memory or thread issues. >> >> If not, the problem is still in your Java code and not in ES. I have been >> working with ES at the Java API level for over a year now. I cannot recall >> any issue that I've had that was not my own fault (a few breaking changes >> during release upgrades have given me some problems, but none that I >> couldn't solve). ES has been remarkably rock solid, and for something as >> elemental as bulk loading, it's the Rock of Gibraltar. >> >> Hope this helps. >> >> Brian >> > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/bf4f5daf-8d4a-4574-bfd2-809b37972ace%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/bf4f5daf-8d4a-4574-bfd2-809b37972ace%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGWt5C1rZ9SJ7jbOauZRPn4L91s7fMPM%2B68cKe9_BJa%2BQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
