No, when things are running everything is ok, indexes break during restart/powerdown 17-03-2014 13:11, "Clinton Gormley" <[email protected]> napisał(a):
> Are you sure you didn't run out of disk space or file handles at some > stage, or have an OOM exception? > > > On 16 March 2014 16:37, bizzorama <[email protected]> wrote: > >> Hi, >> >> it turned out that it was not a problem of ES version (we tested on both >> 0.90.10 and 0.90.9) but just a ES bug ... >> after restarting pc or even just the service indices got broken ... we >> found out that this was the case of missing mappings. >> We observed that broken indices had their mappings corrupted (only some >> default fields were observed). >> You can check this by calling: http:\\es_address:9200\indexName\_mapping >> >> Our mappings were dynamic (not set manually - just figured out by ES when >> the records were incoming). >> >> The solution was to add a static mapping file like the one described here: >> >> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-conf-mappings.html(we >> added the default one). >> >> I just copied mappings from a healty index, made some changes, turned it >> to a mapping file and copied to the ES server. >> >> Now everything works just fine. >> >> Regards, >> Karol >> >> >> W dniu niedziela, 16 marca 2014 14:54:00 UTC+1 użytkownik Mac Jouz >> napisał: >> >>> >>> Hi Bizzorama, >>> >>> I had a similar problem with the same configuration than you gave. >>> ES ran since the 11th of February and was fed every day at 6:00 AM by 2 >>> LS. >>> Everything worked well (kibana reports were correct and no data loss) >>> until >>> I restarted yesterday ES :-( >>> Among 30 index (1 per day), 4 were unusable and data within kibana report >>> for the related period were unavailable (same org.elasticsearch.search. >>> facet.FacetPhaseExecutionException: Facet[0]: (key) field [@timestamp] >>> not found) >>> >>> Do you confirm when you downgraded ES to 0.90.9 that you retrieved your >>> data >>> (i.e you was able to show your data in kibana reports) ? >>> >>> I will try to downgrade ES version as you suggested and will let you know >>> more >>> >>> Thanks for your answer >>> >>> >>> >>> Sorry for the delay. >>> >>> Looks like you were right, after downgrading ES to 0.90.9 i couldn't >>> reproduce the issue in such manner. >>> >>> Unfortunately, I found some other problems, and one looks like a blocker >>> .... >>> >>> After whole ES cluster powerdown, ES just started replaying 'no mapping >>> for ... <name of field>' for each request. >>> >>> W dniu czwartek, 20 lutego 2014 16:42:20 UTC+1 użytkownik Binh Ly >>> napisał: >>>> >>>> Your error logs seem to indicate some kind of version mismatch. Is it >>>> possible for you to test LS 1.3.2 against ES 0.90.9 and take a sample of >>>> raw logs from those 3 days and test them through to see if those 3 days >>>> work in Kibana? The reason I ask is because LS 1.3.2 (specifically the >>>> elasticsearch output) was built using the binaries from ES 0.90.9. >>>> >>>> Thanks. >>>> >>> >>> Le mardi 11 février 2014 13:18:01 UTC+1, bizzorama a écrit : >>>> >>>> Hi, >>>> >>>> I've noticed a very disturbing ElasticSearch behaviour ... >>>> my environment is: >>>> >>>> 1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch >>>> (0.90.10) + kibana >>>> >>>> which process about 7 000 000 records per day, >>>> everything worked fine on our test environment, untill we run some >>>> tests for a longer period (about 15 days). >>>> >>>> After that time, kibana was unable to show any data. >>>> I did some investigation and it looks like some of the indexes (for 3 >>>> days to be exact) seem to be corrupted. >>>> Now every query from kibana, using those corrupted indexes - failes. >>>> >>>> Errors read from elasticsearch logs: >>>> >>>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: >>>> Facet[terms]: failed to find mapping for Name* ... a couple of other >>>> columns* >>>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet >>>> [0]: (key) field [@timestamp] not found >>>> >>>> ... generaly all queries end with those errors >>>> >>>> When elasticsearch is started we find something like this: >>>> >>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>> Message not fully read (request) for [243445] and action >>>> [cluster/nodeIndexCreated], resetting >>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>> Message not fully read (request) for [249943] and action >>>> [cluster/nodeIndexCreated], resetting >>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>> Message not fully read (request) for [246740] and action >>>> [cluster/nodeIndexCreated], resetting >>>> >>>> And a little observations: >>>> >>>> 1. When using elasticsearch-head plugin, when querying records >>>> 'manually', i can see only elasticsearch columns (_index, _type, _id, >>>> _score). >>>> But when I 'randomly' select columns and overview their raw json >>>> they look ok. >>>> >>>> 2, When I tried to process same data again - everything is ok >>>> >>>> Is it possible that some corrupted data found its way to elasticsearch >>>> and now whole index is broken ? >>>> Can this be fixed ? reindexed or sth ? >>>> This data is very importand and can't be lost ... >>>> >>>> Best Regards, >>>> Karol >>>> >>>> >>> Le mardi 11 février 2014 13:18:01 UTC+1, bizzorama a écrit : >>>> >>>> Hi, >>>> >>>> I've noticed a very disturbing ElasticSearch behaviour ... >>>> my environment is: >>>> >>>> 1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch >>>> (0.90.10) + kibana >>>> >>>> which process about 7 000 000 records per day, >>>> everything worked fine on our test environment, untill we run some >>>> tests for a longer period (about 15 days). >>>> >>>> After that time, kibana was unable to show any data. >>>> I did some investigation and it looks like some of the indexes (for 3 >>>> days to be exact) seem to be corrupted. >>>> Now every query from kibana, using those corrupted indexes - failes. >>>> >>>> Errors read from elasticsearch logs: >>>> >>>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: >>>> Facet[terms]: failed to find mapping for Name* ... a couple of other >>>> columns* >>>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet >>>> [0]: (key) field [@timestamp] not found >>>> >>>> ... generaly all queries end with those errors >>>> >>>> When elasticsearch is started we find something like this: >>>> >>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>> Message not fully read (request) for [243445] and action >>>> [cluster/nodeIndexCreated], resetting >>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>> Message not fully read (request) for [249943] and action >>>> [cluster/nodeIndexCreated], resetting >>>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>>> Message not fully read (request) for [246740] and action >>>> [cluster/nodeIndexCreated], resetting >>>> >>>> And a little observations: >>>> >>>> 1. When using elasticsearch-head plugin, when querying records >>>> 'manually', i can see only elasticsearch columns (_index, _type, _id, >>>> _score). >>>> But when I 'randomly' select columns and overview their raw json >>>> they look ok. >>>> >>>> 2, When I tried to process same data again - everything is ok >>>> >>>> Is it possible that some corrupted data found its way to elasticsearch >>>> and now whole index is broken ? >>>> Can this be fixed ? reindexed or sth ? >>>> This data is very importand and can't be lost ... >>>> >>>> Best Regards, >>>> Karol >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/2f25e8f8-7c62-4474-a7a4-ee64a433eeca%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/2f25e8f8-7c62-4474-a7a4-ee64a433eeca%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/7ZwB6SNFkDc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAPt3XKSE6%3DX-QjwJYzs0Aq7FXOHbE6Vu8Am-5BGvznmAsbFikw%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAPt3XKSE6%3DX-QjwJYzs0Aq7FXOHbE6Vu8Am-5BGvznmAsbFikw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAMSRqaaudS6_xZGiJojLe5Ccr7BpEcJOzuY_39ffZ3m9ry_htQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
