Matt, I see the open issue, but where is the code for this?
On Wednesday, April 23, 2014 10:21:13 AM UTC-4, Matt Preston wrote: > > Hi, > > yes, I've tested it with bulk failures - it seems to work well. > Internally, BulkProcessor releases semaphores for bulk failures and other > exceptions that can be thrown by the client, like NoNodeAvailableException, > so it shouldn't ever block forever. > > Matt > > On Wednesday, 23 April 2014 13:47:20 UTC+1, Jörg Prante wrote: >> >> Oh, thanks for the pointer. >> >> Have you tried the blocking close in case of bulk failures? It looks like >> it could hang forever... >> >> Personally I would prefer a separate waitForResponses() method on the >> semaphore without synchronization, but that is a matter of taste. >> >> Jörg >> >> >> On Wed, Apr 23, 2014 at 12:58 PM, Matt Preston < >> [email protected]> wrote: >> >>> There is an open issue to add a blocking close method to BulkProcessor >>> >>> https://github.com/elasticsearch/elasticsearch/pull/4180 >>> >>> Matt >>> >>> >>> On Wednesday, 23 April 2014 10:51:50 UTC+1, Jörg Prante wrote: >>> >>>> You must flush the BulkProcessor and wait until your code has received >>>> all responses from outstanding requests. The close() method in >>>> BulkProcessor performs a flush - and recently an explicit flush() is also >>>> implemented - >>>> >>>> https://github.com/elasticsearch/elasticsearch/issues/5570 >>>> >>>> but BulkProcessor does not wait for all responses. If you close your >>>> client prematurely, you will see "no node available" or other things. >>>> >>>> Since a "flush" is also executed on a periodical interval, you can >>>> either simply wait in your code for an estimated time so all the bulk >>>> requests should be processed (which assumes the client is up for a long >>>> time after BulkProcessor close), or you can write a BulkProcessor wrapper >>>> that is maintaining a counter on the listener calls of >>>> "beforeBulk"/"afterBulk" listener methods so you know when all responses >>>> have been received. >>>> >>>> I wrote an improved BulkProcessor which provides a "waitForResponses" >>>> method: >>>> >>>> https://github.com/jprante/elasticsearch-support/blob/ >>>> master/src/main/java/org/xbib/elasticsearch/action/ingest/ >>>> IngestProcessor.java >>>> >>>> Jörg >>>> >>>> >>>> On Wed, Apr 23, 2014 at 12:01 AM, IronMan2014 <[email protected]>wrote: >>>> >>>>> Ok, so how do I close the client? I have something like this? >>>>> >>>>> main(){ >>>>> >>>>> MyIndexer indexer(); >>>>> >>>>> for (doc=0; doc <n; doc ++) >>>>> indexer.pushDocumentToBulk(doc); >>>>> >>>>> indexer.Shutdown(); >>>>> >>>>> } >>>>> >>>>> class MyIndexer(){ >>>>> >>>>> //create client >>>>> >>>>> // MyBulkProcesor myBulk; >>>>> >>>>> //PushDocumentToBulk(Doc) >>>>> >>>>> void ShutDown(){ >>>>> myBulk.Close(); >>>>> //client.close() ; //This creates No node available exception here, >>>>> my guess bulk is still processing ? >>>>> } >>>>> >>>>> } >>>>> >>>>> class MyBulkProcessor(){ >>>>> >>>>> BlulkProcessor bulk; >>>>> >>>>> @BeforeBulk ... >>>>> @AfterBulk .... >>>>> >>>>> PushDocumentToBulk() { ..push doc ..} >>>>> void Close(){ bulk.close(); } >>>>> } >>>>> >>>>> >>>>> >>>>> On Tuesday, April 22, 2014 5:42:23 PM UTC-4, Jörg Prante wrote: >>>>> >>>>>> No, BulkProcessor does not close the client. >>>>>> >>>>>> Jörg >>>>>> >>>>>> >>>>>> On Tue, Apr 22, 2014 at 11:03 PM, IronMan2014 <[email protected]>wrote: >>>>>> >>>>>>> bulkProcessor.close(); >>>>>>> >>>>>>> Does this also close the transportClient? When I follow up the above >>>>>>> line with client.close(), I get no available exception. >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/a923766f-1f7 >>>>>>> 0-4e78-8b0f-f443e16a2cd7%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/a923766f-1f70-4e78-8b0f-f443e16a2cd7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752% >>>>> 40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/018765f1-b97f-4fbb-a2ae-794a0283d752%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/6b471e65-fcc0-4e85-ba59-eadc94ec3ea5%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c331a55c-69e4-4538-b63f-aee8fc3f9375%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
