I was not able to see any errors in the logs. I think it is not the case of one bulk request being completely rejected or errored out. I see bulk requests being completed in the logs, but it seems only some of the documents are getting indexed during one bulk request. I can see the count increasing only upto around 150 during one bulk request of 200 documents.
It seems that if we decrease the number of shards of the index to 50, there is no data loss. But increasing it to 100 or 500 shards is making the bulk insertion to have data loss during insertion. On Thursday, March 27, 2014 6:12:41 PM UTC+5:30, Binh Ly wrote: > > By any chance, did you see any errors in the logs? It might be possible > that some of the bulk calls were rejected. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5e805ecf-ba21-4023-97b0-fbbc3a84411d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
