Hi Boaz,
Thanks for replying. After we get this error, the cluster
health changes to Yellow with a replica shard in Unassigned state. Is there
a specific way to recover that shard? We dont want to lose other data on
that shard.
Thanks,
Rohit
On Sun, Jun 22, 2014 at 12:50 PM, Boaz Leskes <[email protected]> wrote:
> Hi Rohit,
>
> This issue means update fails anyway, but it breaks the entire request.
> You should indeed set the retry_on_conflict option to make the update
> request succeed. PS - you should really upgrade - a lot has happened and
> was fixed since 0.90.2 ...
>
> Cheers,
> Boaz
>
>
> On Monday, June 16, 2014 10:26:06 PM UTC+2, Rohit Jaiswal wrote:
>>
>> Hi Boaz,
>> We are using 0.90.2 and run into this issue. As i
>> understand, one option is to upgrade to 0.90.3. If we continue using 0.90.2
>> and use (increase) retry_on_conflict, we will not see the problem? Please
>> clarify.
>>
>> Thanks,
>> Rohit
>> On Wednesday, August 7, 2013 9:39:56 AM UTC-7, Boaz Leskes wrote:
>>
>>> HI Eric,
>>>
>>> OK. Based on the gist you sent, i tracked down a problem at fixed it:
>>> https://github.com/elasticsearch/elasticsearch/issues/3448 . Thanks!!
>>> The fix is part of 0.90.3, so I'd recommend upgrading. This is a secondary
>>> problem which occurs when two requests try to update the same document at
>>> exactly the same time. One of them succeeds and the other fails with a
>>> version conflict (that error was masked by the error you were seeing). You
>>> can use (or increase) the retry_on_conflict parameter to make the failing
>>> request try again.
>>>
>>> I'm still curious about your reporting of loosing replicas. Can you
>>> elaborate more about what happens? Do you see anything in the logs?
>>>
>>> Cheers,
>>> Boaz
>>>
>>> On Tuesday, August 6, 2013 5:09:26 AM UTC+2, Eric Sites wrote:
>>>>
>>>> Boaz,
>>>>
>>>> Sorry but I no longer have those logs, I upgraded to 0.90.2 from 0.90.0
>>>> and wiped the logs when I did.
>>>> I did the upgrade to use the _bulk api for my update.
>>>>
>>>> Basically the "lang", "js" was not the issue.
>>>>
>>>> I was using different scripts with the same set of params and an
>>>> upcert. The fix was to use a different param name for different scripts,
>>>> about 10 unique scripts in total.
>>>>
>>>> I was losing replicated shards about every 10,000 to 30,000 updates,
>>>> never the primary shard.
>>>>
>>>> I have 185 million + large json documents, with 100 shards in 1 index
>>>> with 1 replication, so 200 shards total over 6 servers. Each shard is about
>>>> 10.4 GB in size.
>>>> About 2 TB of data, 1 TB primary, 1 TB replicated.
>>>>
>>>> Cheers,
>>>> Eric Sites
>>>>
>>>> From: Boaz Leskes <[email protected]>
>>>> Reply-To: <[email protected]>
>>>> Date: Monday, August 5, 2013 5:38 PM
>>>> To: <[email protected]>
>>>> Subject: Re: 0.90.2 _update or _bulk update causing
>>>> NullPointerException in logs and I start losing shards
>>>>
>>>> Hi Eric,
>>>>
>>>> Glad to hear you solved it. It would be great if you can share the
>>>> failed logs from the _update (non bulk call). A failed script shouldn't
>>>> cause shards to drop so I would like to research it some more.
>>>>
>>>> Cheers,
>>>> Boaz
>>>>
>>>>
>>>> On Mon, Aug 5, 2013 at 6:40 PM, Eric Sites <[email protected]> wrote:
>>>>
>>>>> Boaz,
>>>>>
>>>>> I found and fixed the problem.
>>>>>
>>>>> I added the "lang", "js" to the update json, that was not needed
>>>>> before in es 0.90.0.
>>>>> I also changed the name of new_tracking to match the name of the
>>>>> action in the params section.
>>>>> So for example the script now looks like this:
>>>>>
>>>>> if (ctx._source['tracking'] != null) {
>>>>> if (ctx._source.tracking['some_action'] != null) {
>>>>> ctx._source.tracking.some_action += param1;
>>>>> } else {
>>>>> ctx._source.tracking['some_action'] = 1;
>>>>> }
>>>>> } else {
>>>>> ctx._source.tracking = new_some_action;
>>>>> }
>>>>>
>>>>> "params" : { "param1" : 1, "new_some_action" : { "some_action" : 1 } }
>>>>>
>>>>> Cheers,
>>>>> Eric Sites
>>>>>
>>>>> From: Boaz Leskes <[email protected]>
>>>>> Reply-To: <[email protected]>
>>>>> Date: Monday, August 5, 2013 10:35 AM
>>>>> To: <[email protected]>
>>>>> Subject: Re: 0.90.2 _update or _bulk update causing
>>>>> NullPointerException in logs and I start losing shards
>>>>>
>>>>> Hi Eric,
>>>>>
>>>>> This is interesting. The log stack trace from the gist comes from the
>>>>> bulk calls. Can you also post one from a failed _update? Cross checking
>>>>> them might help pin pointing the issue.
>>>>>
>>>>> Cheers,
>>>>> Boaz
>>>>>
>>>>> On Monday, August 5, 2013 1:34:16 AM UTC+2, [email protected] wrote:
>>>>>>
>>>>>> I am getting java.lang.NullPointerException exception in my
>>>>>> ElasticSearch cluster logs when I am doing a _bulk update or just an
>>>>>> _update.
>>>>>> I am sending a lot of data to my clusters. After I get this error I
>>>>>> lose a shard and it has to be recreated.
>>>>>>
>>>>>> version 0.90.2
>>>>>>
>>>>>> gist: https://gist.github.com/EricSites/6152468
>>>>>>
>>>>>> I get this using the _bulk api or just normal _update api.
>>>>>>
>>>>>> My update script is a little complicated.
>>>>>> I am adding a tracking object to my document if it does not exists.
>>>>>> There should only be one of these and it should not be an array of these.
>>>>>> If the object does exists, I am trying to add a new field to the
>>>>>> tracking object to keep track on counts.
>>>>>> So if the field does not exists I create it, else just += to it.
>>>>>>
>>>>>> if (ctx._source['tracking'] != null) {
>>>>>> if (ctx._source.tracking['some_action'] != null) {
>>>>>> ctx._source.tracking.some_action += param1;
>>>>>> } else {
>>>>>> ctx._source.tracking['some_action'] = 1;
>>>>>> }
>>>>>> } else {
>>>>>> ctx._source.tracking = new_tracking;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Here is my mapping for this:
>>>>>> {
>>>>>> "sample" : {
>>>>>> "index_options" : "docs",
>>>>>> "properties" : {
>>>>>> "tracking" : {
>>>>>> "type" : "object",
>>>>>> "dynamic" : true
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elasticsearch" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "elasticsearch" group.
>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/
>>>>> topic/elasticsearch/yk7HvjqCgOg/unsubscribe.
>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>> [email protected].
>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>>
>>>>
>>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/a22ffbaa-af7e-4d15-ac5a-e1dcd5b76976%40googlegroups.com
> <https://groups.google.com/d/msgid/elasticsearch/a22ffbaa-af7e-4d15-ac5a-e1dcd5b76976%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAP_rV8GKheAXK%3Dq%2BG2vdyfgRBURuk4_udO8XFLNCTmDV3EnWiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
