Hi Boaz,
We are using 0.90.2 and run into this issue. As i
understand, one option is to upgrade to 0.90.3. If we continue using 0.90.2
and use (increase) retry_on_conflict, we will not see the problem? Please
clarify.
Thanks,
Rohit
On Wednesday, August 7, 2013 9:39:56 AM UTC-7, Boaz Leskes wrote:
> HI Eric,
>
> OK. Based on the gist you sent, i tracked down a problem at fixed it:
> https://github.com/elasticsearch/elasticsearch/issues/3448 . Thanks!! The
> fix is part of 0.90.3, so I'd recommend upgrading. This is a secondary
> problem which occurs when two requests try to update the same document at
> exactly the same time. One of them succeeds and the other fails with a
> version conflict (that error was masked by the error you were seeing). You
> can use (or increase) the retry_on_conflict parameter to make the failing
> request try again.
>
> I'm still curious about your reporting of loosing replicas. Can you
> elaborate more about what happens? Do you see anything in the logs?
>
> Cheers,
> Boaz
>
> On Tuesday, August 6, 2013 5:09:26 AM UTC+2, Eric Sites wrote:
>>
>> Boaz,
>>
>> Sorry but I no longer have those logs, I upgraded to 0.90.2 from 0.90.0
>> and wiped the logs when I did.
>> I did the upgrade to use the _bulk api for my update.
>>
>> Basically the "lang", "js" was not the issue.
>>
>> I was using different scripts with the same set of params and an upcert.
>> The fix was to use a different param name for different scripts, about 10
>> unique scripts in total.
>>
>> I was losing replicated shards about every 10,000 to 30,000 updates,
>> never the primary shard.
>>
>> I have 185 million + large json documents, with 100 shards in 1 index
>> with 1 replication, so 200 shards total over 6 servers. Each shard is about
>> 10.4 GB in size.
>> About 2 TB of data, 1 TB primary, 1 TB replicated.
>>
>> Cheers,
>> Eric Sites
>>
>> From: Boaz Leskes <[email protected]>
>> Reply-To: <[email protected]>
>> Date: Monday, August 5, 2013 5:38 PM
>> To: <[email protected]>
>> Subject: Re: 0.90.2 _update or _bulk update causing NullPointerException
>> in logs and I start losing shards
>>
>> Hi Eric,
>>
>> Glad to hear you solved it. It would be great if you can share the failed
>> logs from the _update (non bulk call). A failed script shouldn't cause
>> shards to drop so I would like to research it some more.
>>
>> Cheers,
>> Boaz
>>
>>
>> On Mon, Aug 5, 2013 at 6:40 PM, Eric Sites <[email protected]> wrote:
>>
>>> Boaz,
>>>
>>> I found and fixed the problem.
>>>
>>> I added the "lang", "js" to the update json, that was not needed before
>>> in es 0.90.0.
>>> I also changed the name of new_tracking to match the name of the action
>>> in the params section.
>>> So for example the script now looks like this:
>>>
>>> if (ctx._source['tracking'] != null) {
>>> if (ctx._source.tracking['some_action'] != null) {
>>> ctx._source.tracking.some_action += param1;
>>> } else {
>>> ctx._source.tracking['some_action'] = 1;
>>> }
>>> } else {
>>> ctx._source.tracking = new_some_action;
>>> }
>>>
>>> "params" : { "param1" : 1, "new_some_action" : { "some_action" : 1 } }
>>>
>>> Cheers,
>>> Eric Sites
>>>
>>> From: Boaz Leskes <[email protected]>
>>> Reply-To: <[email protected]>
>>> Date: Monday, August 5, 2013 10:35 AM
>>> To: <[email protected]>
>>> Subject: Re: 0.90.2 _update or _bulk update causing
>>> NullPointerException in logs and I start losing shards
>>>
>>> Hi Eric,
>>>
>>> This is interesting. The log stack trace from the gist comes from the
>>> bulk calls. Can you also post one from a failed _update? Cross checking
>>> them might help pin pointing the issue.
>>>
>>> Cheers,
>>> Boaz
>>>
>>> On Monday, August 5, 2013 1:34:16 AM UTC+2, [email protected] wrote:
>>>>
>>>> I am getting java.lang.NullPointerException exception in my
>>>> ElasticSearch cluster logs when I am doing a _bulk update or just an
>>>> _update.
>>>> I am sending a lot of data to my clusters. After I get this error I
>>>> lose a shard and it has to be recreated.
>>>>
>>>> version 0.90.2
>>>>
>>>> gist: https://gist.github.com/EricSites/6152468
>>>>
>>>> I get this using the _bulk api or just normal _update api.
>>>>
>>>> My update script is a little complicated.
>>>> I am adding a tracking object to my document if it does not exists.
>>>> There should only be one of these and it should not be an array of these.
>>>> If the object does exists, I am trying to add a new field to the
>>>> tracking object to keep track on counts.
>>>> So if the field does not exists I create it, else just += to it.
>>>>
>>>> if (ctx._source['tracking'] != null) {
>>>> if (ctx._source.tracking['some_action'] != null) {
>>>> ctx._source.tracking.some_action += param1;
>>>> } else {
>>>> ctx._source.tracking['some_action'] = 1;
>>>> }
>>>> } else {
>>>> ctx._source.tracking = new_tracking;
>>>> }
>>>>
>>>>
>>>> Here is my mapping for this:
>>>> {
>>>> "sample" : {
>>>> "index_options" : "docs",
>>>> "properties" : {
>>>> "tracking" : {
>>>> "type" : "object",
>>>> "dynamic" : true
>>>> }
>>>> }
>>>> }
>>>> }
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "elasticsearch" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/elasticsearch/yk7HvjqCgOg/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> [email protected].
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ecf3d1cf-8b21-44cb-8b17-76f818805a7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
