Oh, thanks for the pointer. Have you tried the blocking close in case of bulk failures? It looks like it could hang forever...
Personally I would prefer a separate waitForResponses() method on the semaphore without synchronization, but that is a matter of taste. Jörg On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston < [email protected]> wrote: > There is an open issue to add a blocking close method to BulkProcessor > > https://github.com/elasticsearch/elasticsearch/pull/4180 > > Matt > > > On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote: > >> You must flush the BulkProcessor and wait until your code has received >> all responses from outstanding requests. The close() method in >> BulkProcessor performs a flush - and recently an explicit flush() is also >> implemented - >> >> https://github.com/elasticsearch/elasticsearch/issues/5570 >> >> but BulkProcessor does not wait for all responses. If you close your >> client prematurely, you will see "no node available" or other things. >> >> Since a "flush" is also executed on a periodical interval, you can either >> simply wait in your code for an estimated time so all the bulk requests >> should be processed (which assumes the client is up for a long time after >> BulkProcessor close), or you can write a BulkProcessor wrapper that is >> maintaining a counter on the listener calls of "beforeBulk"/"afterBulk" >> listener methods so you know when all responses have been received. >> >> I wrote an improved BulkProcessor which provides a "waitForResponses" >> method: >> >> https://github.com/jprante/elasticsearch-support/blob/ >> master/src/main/java/org/xbib/elasticsearch/action/ingest/ >> IngestProcessor.java >> >> Jörg >> >> >> On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 <[email protected]> wrote: >> >>> Ok, so how do I close the client? I have something like this? >>> >>> main(){ >>> >>> MyIndexer indexer(); >>> >>> for (doc=0; doc <n; doc ++) >>> indexer.pushDocumentToBulk(doc); >>> >>> indexer.Shutdown(); >>> >>> } >>> >>> class MyIndexer(){ >>> >>> //create client >>> >>> // MyBulkProcesor myBulk; >>> >>> //PushDocumentToBulk(Doc) >>> >>> void ShutDown(){ >>> myBulk.Close(); >>> //client.close() ; //This creates No node available exception here, >>> my guess bulk is still processing ? >>> } >>> >>> } >>> >>> class MyBulkProcessor(){ >>> >>> BlulkProcessor bulk; >>> >>> @BeforeBulk ... >>> @AfterBulk .... >>> >>> PushDocumentToBulk() { ..push doc ..} >>> void Close(){ bulk.close(); } >>> } >>> >>> >>> >>> On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote: >>> >>>> No, BulkProcessor does not close the client. >>>> >>>> Jörg >>>> >>>> >>>> On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 <[email protected]>wrote: >>>> >>>>> bulkProcessor.close(); >>>>> >>>>> Does this also close the transportClient? When I follow up the above >>>>> line with client.close(), I get no available exception. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40goo >>>>> glegroups.com<https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752% >>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvQQjwBRuSU7Ke3iUp6%3D%2BQBX-LznqKrYf3yF59Rm3Dvw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
