More info: I've tested it with ES versions 1.4.2, 1.4.3, and 1.4.4, and the
reproduction script demonstrates the issue on all of them. Sometimes it
happens right away, other times you have to wait for a few
minutes/iterations, but each time the mapping is retrieved is after the
status of the cluster is green.
Here's my test environment: Windows 7 Enterprise, default configuration ES
with two nodes. Java version: jre1.8.0_31.
On Tuesday, March 3, 2015 at 10:34:49 AM UTC-5, Andrew Pak wrote:
>
> Thanks for your reply Jörg! However, even after I wait for green status, I
> still get the issue, here's the output of the updated script that does a
> synchronous call to get the health of the cluster with the
> 'wait_for_status=green' parameter.
>
> /bin/python es_dynamic_mapping_test.py
> Delete index /documents: {"acknowledged":true}
> Create index /documents: {"acknowledged":true}
> Create mapping /documents/pdf: {"acknowledged":true}
> Create new /documents/pdf document:
> {"_index":"documents","_type":"pdf","_id":"AUvgNoIq70doA8_gZ7fd","_version":1,"created":true}
> Refresh index /documents {"_shards":{"total":1,"successful":1,"failed":0}}
> Wait for green status:
> {"cluster_name":"helixdev","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shar
>
> ds":1,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}
> Retrieved fields: [u'title_str', u'title_str.tags_str']
> ########################################################################
> #
> # Field 'new_field_str' was not found in the mapping.
> #
> ########################################################################
>
> https://gist.github.com/elasticsearcher/2dff7ee44e11c6a39bfb
>
> Thanks again!
> Andrew
>
>
> On Monday, March 2, 2015 at 5:04:05 PM UTC-5, Jörg Prante wrote:
>>
>> You have to wait for green health of the index before retrieving the new
>> mapping.
>>
>>
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html
>>
>> While the mapping changes, cluster has some work to do to propagate the
>> new field to all nodes. This takes some time. It is not a race condition.
>>
>> Jörg
>>
>> On Mon, Mar 2, 2015 at 10:44 PM, Andrew Pak <[email protected]> wrote:
>>
>>> Hello!
>>>
>>> Here's my issue:
>>>
>>> 1. I create an index with a mapping that has a dynamic template.
>>> 2. I create a new document that contains a new field that will match a
>>> dynamic template in the mapping and that will be added to the mapping. The
>>> PUT succeeds with 201 Created.
>>> 3. I then retrieve the mapping with a GET to check that the new field
>>> has been correctly added. The GET succeeds with 200 OK.
>>> 4. The returned mapping doesn't always contain the newly created field.
>>> It sometimes takes a small delay (< 1s) for the updated mapping to be
>>> returned.
>>>
>>> Seems like a race condition, so I'm wondering if this is expected
>>> behaviour and if there is a setting or parameter that can be set to
>>> guarantee that the mapping gets refreshed atomically, just like you can
>>> force-refresh an index. This is not currently a show-stopping issue, I'd
>>> just like to have a definitive answer.
>>>
>>> The reason I need the mapping to reflect the changes immediately is that
>>> I have unit tests that depend on field information retrieved from the
>>> mapping at run-time and they sporadically fail because the returned mapping
>>> and the expected mapping don't always match up.
>>>
>>> Here's a curl recreation script:
>>> https://gist.github.com/elasticsearcher/7f116867603a44575ce1
>>>
>>> However, it seems that curl is slow enough that the small delay needed
>>> for the mapping to refresh passes in-between curl calls and the script
>>> doesn't actually reproduce the issue. I hence created an equivalent Python
>>> script that retries the whole process until the issue is demonstrated:
>>> https://gist.github.com/elasticsearcher/2dff7ee44e11c6a39bfb
>>>
>>> Here's a sample output from the script (contains a few iterations of the
>>> entire recreation cycle):
>>>
>>> > /bin/python es_mapping_race_condition.py
>>> Delete index /documents: {"acknowledged":true}
>>> Create index /documents: {"acknowledged":true}
>>> Create mapping /documents/pdf: {"acknowledged":true}
>>> Create new /documents/pdf document:
>>> {"_index":"documents","_type":"pdf","_id":"AUvcWwZgoOHS3xjDkoFH","_version":1,"created":true}
>>> Refresh index /documents
>>> {"_shards":{"total":10,"successful":5,"failed":0}}
>>> Retrieved fields: [u'new_field_str', u'new_field_str.tags_str',
>>> u'title_str', u'title_str.tags_str']
>>> Delete index /documents: {"acknowledged":true}
>>> Create index /documents: {"acknowledged":true}
>>> Create mapping /documents/pdf: {"acknowledged":true}
>>> Create new /documents/pdf document:
>>> {"_index":"documents","_type":"pdf","_id":"AUvcWwsCoOHS3xjDkoFI","_version":1,"created":true}
>>> Refresh index /documents
>>> {"_shards":{"total":10,"successful":5,"failed":0}}
>>> Retrieved fields: [u'new_field_str', u'new_field_str.tags_str',
>>> u'title_str', u'title_str.tags_str']
>>> Delete index /documents: {"acknowledged":true}
>>> Create index /documents: {"acknowledged":true}
>>> Create mapping /documents/pdf: {"acknowledged":true}
>>> Create new /documents/pdf document:
>>> {"_index":"documents","_type":"pdf","_id":"AUvcWxDioOHS3xjDkoFJ","_version":1,"created":true}
>>> Refresh index /documents
>>> {"_shards":{"total":10,"successful":5,"failed":0}}
>>> Retrieved fields: [u'title_str', u'title_str.tags_str']
>>> ########################################################################
>>> #
>>> # Field 'new_field_str' was not found in the mapping.
>>> #
>>> ########################################################################
>>>
>>> Thanks in advance for any help!
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/50af4889-5bfa-41e2-8c1a-acffebe7d6fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
