You could be right: I can't test right now but this is my code:
(there may be 20 workerThreads)
As you can see, as each thread submits work, the thread will do a
client.prepareBulk() ... is that sufficient clear out the documents?
workerThread() {
Client client = getMyGlobalTransportClient();
BulkRequestBuilder bulkRequest = client.prepareBulk();
for (...) {
bulkRequest.add(...)
if (bulkRequest.numberOfActions() >= chunksize) {
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
if (bulkResponse.hasFailures()) {
...
} else {
...
}
bulkRequest = client.prepareBulk();
}
etc
On Friday, March 28, 2014 2:44:07 PM UTC, InquiringMind wrote:
> When I use the Java TransportClient and the BulkRequest builder, my
> throughput is like a scalded cat racing a bolt of greased lightning, with
> the cat way ahead!
>
> "the Java API" does not say how you are using it. Since I cannot see your
> code, I cannot comment on where your mistake is located.
>
> But I have noticed that a small number of folks use BulkRequestBuilder but
> keep adding documents to it. Since BulkRequestBuilder is additive, their
> first bulk load batch contains N documents, their second contains 2N (the
> first batch plus the second batch), their third batch contains 3N, and so
> on until they crash the JVM with OOM errors.
>
> So if this is your mistake, then simply create a new BulkRequestBuilder
> for each batch of documents to submit, and let the previous
> BulkRequestBuilder get garbage collected, and your Java build will run
> lightning fast and never run into memory or thread issues.
>
> If not, the problem is still in your Java code and not in ES. I have been
> working with ES at the Java API level for over a year now. I cannot recall
> any issue that I've had that was not my own fault (a few breaking changes
> during release upgrades have given me some problems, but none that I
> couldn't solve). ES has been remarkably rock solid, and for something as
> elemental as bulk loading, it's the Rock of Gibraltar.
>
> Hope this helps.
>
> Brian
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bf4f5daf-8d4a-4574-bfd2-809b37972ace%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
