Hi, yes, I've tested it with bulk failures - it seems to work well. Internally, BulkProcessor releases semaphores for bulk failures and other exceptions that can be thrown by the client, like NoNodeAvailableException, so it shouldn't ever block forever.
Matt On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote: > > Oh, thanks for the pointer. > > Have you tried the blocking close in case of bulk failures? It looks like > it could hang forever... > > Personally I would prefer a separate waitForResponses() method on the > semaphore without synchronization, but that is a matter of taste. > > Jörg > > > On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston < > [email protected] <javascript:>> wrote: > >> There is an open issue to add a blocking close method to BulkProcessor >> >> https://github.com/elasticsearch/elasticsearch/pull/4180 >> >> Matt >> >> >> On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote: >> >>> You must flush the BulkProcessor and wait until your code has received >>> all responses from outstanding requests. The close() method in >>> BulkProcessor performs a flush - and recently an explicit flush() is also >>> implemented - >>> >>> https://github.com/elasticsearch/elasticsearch/issues/5570 >>> >>> but BulkProcessor does not wait for all responses. If you close your >>> client prematurely, you will see "no node available" or other things. >>> >>> Since a "flush" is also executed on a periodical interval, you can >>> either simply wait in your code for an estimated time so all the bulk >>> requests should be processed (which assumes the client is up for a long >>> time after BulkProcessor close), or you can write a BulkProcessor wrapper >>> that is maintaining a counter on the listener calls of >>> "beforeBulk"/"afterBulk" listener methods so you know when all responses >>> have been received. >>> >>> I wrote an improved BulkProcessor which provides a "waitForResponses" >>> method: >>> >>> https://github.com/jprante/elasticsearch-support/blob/ >>> master/src/main/java/org/xbib/elasticsearch/action/ingest/ >>> IngestProcessor.java >>> >>> Jörg >>> >>> >>> On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 <[email protected]>wrote: >>> >>>> Ok, so how do I close the client? I have something like this? >>>> >>>> main(){ >>>> >>>> MyIndexer indexer(); >>>> >>>> for (doc=0; doc <n; doc ++) >>>> indexer.pushDocumentToBulk(doc); >>>> >>>> indexer.Shutdown(); >>>> >>>> } >>>> >>>> class MyIndexer(){ >>>> >>>> //create client >>>> >>>> // MyBulkProcesor myBulk; >>>> >>>> //PushDocumentToBulk(Doc) >>>> >>>> void ShutDown(){ >>>> myBulk.Close(); >>>> //client.close() ; //This creates No node available exception here, >>>> my guess bulk is still processing ? >>>> } >>>> >>>> } >>>> >>>> class MyBulkProcessor(){ >>>> >>>> BlulkProcessor bulk; >>>> >>>> @BeforeBulk ... >>>> @AfterBulk .... >>>> >>>> PushDocumentToBulk() { ..push doc ..} >>>> void Close(){ bulk.close(); } >>>> } >>>> >>>> >>>> >>>> On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote: >>>> >>>>> No, BulkProcessor does not close the client. >>>>> >>>>> Jörg >>>>> >>>>> >>>>> On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 <[email protected]>wrote: >>>>> >>>>>> bulkProcessor.close(); >>>>>> >>>>>> Does this also close the transportClient? When I follow up the above >>>>>> line with client.close(), I get no available exception. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40goo >>>>>> glegroups.com<https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752% >>>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b637cef4-35a3-4d3a-92d9-0334189cd112%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
