Not that I know of. But there is a known but very rare bug (fixed in 0.90.8) which can cause data loss upon a node restart: https://github.com/elasticsearch/elasticsearch/issues/4502
Maybe you run into that? On Sun, Jun 22, 2014 at 10:18 PM, Rohit Jaiswal <[email protected]> wrote: > Yes, it did when we restarted the node while trying to reproduce this > problem. We also were able to access the data using the Scan search api > after restarting the node. > > However we have seen quite a few of the bulk update errors in our 20-node > production cluster and have suffered data loss on other aliases (The alias > filter being the user-id) as well. We think the data loss is because of > this bulk update error. > > Is there a chance of losing data on shards when enough of these bulk > updates happen concurrently on multiple aliases (users)? > > Thanks > > > On Sun, Jun 22, 2014 at 1:10 PM, Boaz Leskes <[email protected]> wrote: > >> If you restart the node it's on, it doesn't come back? >> >> >> On Sun, Jun 22, 2014 at 10:01 PM, Rohit Jaiswal <[email protected]> >> wrote: >> >>> Hi Boaz, >>> Thanks for replying. After we get this error, the cluster >>> health changes to Yellow with a replica shard in Unassigned state. Is there >>> a specific way to recover that shard? We dont want to lose other data on >>> that shard. >>> >>> Thanks, >>> Rohit >>> >>> >>> On Sun, Jun 22, 2014 at 12:50 PM, Boaz Leskes <[email protected]> >>> wrote: >>> >>>> Hi Rohit, >>>> >>>> This issue means update fails anyway, but it breaks the entire request. >>>> You should indeed set the retry_on_conflict option to make the update >>>> request succeed. PS - you should really upgrade - a lot has happened and >>>> was fixed since 0.90.2 ... >>>> >>>> Cheers, >>>> Boaz >>>> >>>> >>>> On Monday, June 16, 2014 10:26:06 PM UTC+2, Rohit Jaiswal wrote: >>>>> >>>>> Hi Boaz, >>>>> We are using 0.90.2 and run into this issue. As i >>>>> understand, one option is to upgrade to 0.90.3. If we continue using >>>>> 0.90.2 >>>>> and use (increase) retry_on_conflict, we will not see the problem? Please >>>>> clarify. >>>>> >>>>> Thanks, >>>>> Rohit >>>>> On Wednesday, August 7, 2013 9:39:56 AM UTC-7, Boaz Leskes wrote: >>>>> >>>>>> HI Eric, >>>>>> >>>>>> OK. Based on the gist you sent, i tracked down a problem at fixed it: >>>>>> https://github.com/elasticsearch/elasticsearch/issues/3448 . >>>>>> Thanks!! The fix is part of 0.90.3, so I'd recommend upgrading. This is a >>>>>> secondary problem which occurs when two requests try to update the same >>>>>> document at exactly the same time. One of them succeeds and the other >>>>>> fails >>>>>> with a version conflict (that error was masked by the error you were >>>>>> seeing). You can use (or increase) the retry_on_conflict parameter to >>>>>> make >>>>>> the failing request try again. >>>>>> >>>>>> I'm still curious about your reporting of loosing replicas. Can you >>>>>> elaborate more about what happens? Do you see anything in the logs? >>>>>> >>>>>> Cheers, >>>>>> Boaz >>>>>> >>>>>> On Tuesday, August 6, 2013 5:09:26 AM UTC+2, Eric Sites wrote: >>>>>>> >>>>>>> Boaz, >>>>>>> >>>>>>> Sorry but I no longer have those logs, I upgraded to 0.90.2 from >>>>>>> 0.90.0 and wiped the logs when I did. >>>>>>> I did the upgrade to use the _bulk api for my update. >>>>>>> >>>>>>> Basically the "lang", "js" was not the issue. >>>>>>> >>>>>>> I was using different scripts with the same set of params and an >>>>>>> upcert. The fix was to use a different param name for different scripts, >>>>>>> about 10 unique scripts in total. >>>>>>> >>>>>>> I was losing replicated shards about every 10,000 to 30,000 updates, >>>>>>> never the primary shard. >>>>>>> >>>>>>> I have 185 million + large json documents, with 100 shards in 1 >>>>>>> index with 1 replication, so 200 shards total over 6 servers. Each >>>>>>> shard is >>>>>>> about 10.4 GB in size. >>>>>>> About 2 TB of data, 1 TB primary, 1 TB replicated. >>>>>>> >>>>>>> Cheers, >>>>>>> Eric Sites >>>>>>> >>>>>>> From: Boaz Leskes <[email protected]> >>>>>>> Reply-To: <[email protected]> >>>>>>> Date: Monday, August 5, 2013 5:38 PM >>>>>>> To: <[email protected]> >>>>>>> Subject: Re: 0.90.2 _update or _bulk update causing >>>>>>> NullPointerException in logs and I start losing shards >>>>>>> >>>>>>> Hi Eric, >>>>>>> >>>>>>> Glad to hear you solved it. It would be great if you can share the >>>>>>> failed logs from the _update (non bulk call). A failed script shouldn't >>>>>>> cause shards to drop so I would like to research it some more. >>>>>>> >>>>>>> Cheers, >>>>>>> Boaz >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 5, 2013 at 6:40 PM, Eric Sites <[email protected]> wrote: >>>>>>> >>>>>>>> Boaz, >>>>>>>> >>>>>>>> I found and fixed the problem. >>>>>>>> >>>>>>>> I added the "lang", "js" to the update json, that was not needed >>>>>>>> before in es 0.90.0. >>>>>>>> I also changed the name of new_tracking to match the name of the >>>>>>>> action in the params section. >>>>>>>> So for example the script now looks like this: >>>>>>>> >>>>>>>> if (ctx._source['tracking'] != null) { >>>>>>>> if (ctx._source.tracking['some_action'] != null) { >>>>>>>> ctx._source.tracking.some_action += param1; >>>>>>>> } else { >>>>>>>> ctx._source.tracking['some_action'] = 1; >>>>>>>> } >>>>>>>> } else { >>>>>>>> ctx._source.tracking = new_some_action; >>>>>>>> } >>>>>>>> >>>>>>>> "params" : { "param1" : 1, "new_some_action" : { "some_action" : 1 >>>>>>>> } } >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Eric Sites >>>>>>>> >>>>>>>> From: Boaz Leskes <[email protected]> >>>>>>>> Reply-To: <[email protected]> >>>>>>>> Date: Monday, August 5, 2013 10:35 AM >>>>>>>> To: <[email protected]> >>>>>>>> Subject: Re: 0.90.2 _update or _bulk update causing >>>>>>>> NullPointerException in logs and I start losing shards >>>>>>>> >>>>>>>> Hi Eric, >>>>>>>> >>>>>>>> This is interesting. The log stack trace from the gist comes from >>>>>>>> the bulk calls. Can you also post one from a failed _update? Cross >>>>>>>> checking >>>>>>>> them might help pin pointing the issue. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Boaz >>>>>>>> >>>>>>>> On Monday, August 5, 2013 1:34:16 AM UTC+2, [email protected] wrote: >>>>>>>>> >>>>>>>>> I am getting java.lang.NullPointerException exception in my >>>>>>>>> ElasticSearch cluster logs when I am doing a _bulk update or just an >>>>>>>>> _update. >>>>>>>>> I am sending a lot of data to my clusters. After I get this error >>>>>>>>> I lose a shard and it has to be recreated. >>>>>>>>> >>>>>>>>> version 0.90.2 >>>>>>>>> >>>>>>>>> gist: https://gist.github.com/EricSites/6152468 >>>>>>>>> >>>>>>>>> I get this using the _bulk api or just normal _update api. >>>>>>>>> >>>>>>>>> My update script is a little complicated. >>>>>>>>> I am adding a tracking object to my document if it does not >>>>>>>>> exists. There should only be one of these and it should not be an >>>>>>>>> array of >>>>>>>>> these. >>>>>>>>> If the object does exists, I am trying to add a new field to the >>>>>>>>> tracking object to keep track on counts. >>>>>>>>> So if the field does not exists I create it, else just += to it. >>>>>>>>> >>>>>>>>> if (ctx._source['tracking'] != null) { >>>>>>>>> if (ctx._source.tracking['some_action'] != null) { >>>>>>>>> ctx._source.tracking.some_action += param1; >>>>>>>>> } else { >>>>>>>>> ctx._source.tracking['some_action'] = 1; >>>>>>>>> } >>>>>>>>> } else { >>>>>>>>> ctx._source.tracking = new_tracking; >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> Here is my mapping for this: >>>>>>>>> { >>>>>>>>> "sample" : { >>>>>>>>> "index_options" : "docs", >>>>>>>>> "properties" : { >>>>>>>>> "tracking" : { >>>>>>>>> "type" : "object", >>>>>>>>> "dynamic" : true >>>>>>>>> } >>>>>>>>> } >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "elasticsearch" group. >>>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/ >>>>>>>> topic/elasticsearch/yk7HvjqCgOg/unsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/groups/opt_out. >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "elasticsearch" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/a22ffbaa-af7e-4d15-ac5a-e1dcd5b76976%40googlegroups.com >>>> <https://groups.google.com/d/msgid/elasticsearch/a22ffbaa-af7e-4d15-ac5a-e1dcd5b76976%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAP_rV8GKheAXK%3Dq%2BG2vdyfgRBURuk4_udO8XFLNCTmDV3EnWiA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAP_rV8GKheAXK%3Dq%2BG2vdyfgRBURuk4_udO8XFLNCTmDV3EnWiA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "elasticsearch" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAKzwz0oDUnYonpURtCVis-9UxS0FRiRMvLW1wZZybo2gOZboTA%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CAKzwz0oDUnYonpURtCVis-9UxS0FRiRMvLW1wZZybo2gOZboTA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAP_rV8GexzoN8Nrf3GBaCrrXrVdKjUzzrkqw%3DYLwTW9YwEst5A%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAP_rV8GexzoN8Nrf3GBaCrrXrVdKjUzzrkqw%3DYLwTW9YwEst5A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKzwz0qcZw2SR0Bt6GU06-FEp%2BL%2BRyAin3oCnWhpefGGVH99Zg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
