Are you sure you didn't run out of disk space or file handles at some stage, or have an OOM exception?
On 16 March 2014 16:37, bizzorama <[email protected]> wrote: > Hi, > > it turned out that it was not a problem of ES version (we tested on both > 0.90.10 and 0.90.9) but just a ES bug ... > after restarting pc or even just the service indices got broken ... we > found out that this was the case of missing mappings. > We observed that broken indices had their mappings corrupted (only some > default fields were observed). > You can check this by calling: http:\\es_address:9200\indexName\_mapping > > Our mappings were dynamic (not set manually - just figured out by ES when > the records were incoming). > > The solution was to add a static mapping file like the one described here: > > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-conf-mappings.html(we > added the default one). > > I just copied mappings from a healty index, made some changes, turned it > to a mapping file and copied to the ES server. > > Now everything works just fine. > > Regards, > Karol > > > W dniu niedziela, 16 marca 2014 14:54:00 UTC+1 użytkownik Mac Jouz napisał: > >> >> Hi Bizzorama, >> >> I had a similar problem with the same configuration than you gave. >> ES ran since the 11th of February and was fed every day at 6:00 AM by 2 >> LS. >> Everything worked well (kibana reports were correct and no data loss) >> until >> I restarted yesterday ES :-( >> Among 30 index (1 per day), 4 were unusable and data within kibana report >> for the related period were unavailable (same org.elasticsearch.search. >> facet.FacetPhaseExecutionException: Facet[0]: (key) field [@timestamp] >> not found) >> >> Do you confirm when you downgraded ES to 0.90.9 that you retrieved your >> data >> (i.e you was able to show your data in kibana reports) ? >> >> I will try to downgrade ES version as you suggested and will let you know >> more >> >> Thanks for your answer >> >> >> >> Sorry for the delay. >> >> Looks like you were right, after downgrading ES to 0.90.9 i couldn't >> reproduce the issue in such manner. >> >> Unfortunately, I found some other problems, and one looks like a blocker >> .... >> >> After whole ES cluster powerdown, ES just started replaying 'no mapping >> for ... <name of field>' for each request. >> >> W dniu czwartek, 20 lutego 2014 16:42:20 UTC+1 użytkownik Binh Ly napisał: >>> >>> Your error logs seem to indicate some kind of version mismatch. Is it >>> possible for you to test LS 1.3.2 against ES 0.90.9 and take a sample of >>> raw logs from those 3 days and test them through to see if those 3 days >>> work in Kibana? The reason I ask is because LS 1.3.2 (specifically the >>> elasticsearch output) was built using the binaries from ES 0.90.9. >>> >>> Thanks. >>> >> >> Le mardi 11 février 2014 13:18:01 UTC+1, bizzorama a écrit : >>> >>> Hi, >>> >>> I've noticed a very disturbing ElasticSearch behaviour ... >>> my environment is: >>> >>> 1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch >>> (0.90.10) + kibana >>> >>> which process about 7 000 000 records per day, >>> everything worked fine on our test environment, untill we run some tests >>> for a longer period (about 15 days). >>> >>> After that time, kibana was unable to show any data. >>> I did some investigation and it looks like some of the indexes (for 3 >>> days to be exact) seem to be corrupted. >>> Now every query from kibana, using those corrupted indexes - failes. >>> >>> Errors read from elasticsearch logs: >>> >>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: >>> Facet[terms]: failed to find mapping for Name* ... a couple of other >>> columns* >>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet >>> [0]: (key) field [@timestamp] not found >>> >>> ... generaly all queries end with those errors >>> >>> When elasticsearch is started we find something like this: >>> >>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>> Message not fully read (request) for [243445] and action >>> [cluster/nodeIndexCreated], resetting >>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>> Message not fully read (request) for [249943] and action >>> [cluster/nodeIndexCreated], resetting >>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>> Message not fully read (request) for [246740] and action >>> [cluster/nodeIndexCreated], resetting >>> >>> And a little observations: >>> >>> 1. When using elasticsearch-head plugin, when querying records >>> 'manually', i can see only elasticsearch columns (_index, _type, _id, >>> _score). >>> But when I 'randomly' select columns and overview their raw json >>> they look ok. >>> >>> 2, When I tried to process same data again - everything is ok >>> >>> Is it possible that some corrupted data found its way to elasticsearch >>> and now whole index is broken ? >>> Can this be fixed ? reindexed or sth ? >>> This data is very importand and can't be lost ... >>> >>> Best Regards, >>> Karol >>> >>> >> Le mardi 11 février 2014 13:18:01 UTC+1, bizzorama a écrit : >>> >>> Hi, >>> >>> I've noticed a very disturbing ElasticSearch behaviour ... >>> my environment is: >>> >>> 1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch >>> (0.90.10) + kibana >>> >>> which process about 7 000 000 records per day, >>> everything worked fine on our test environment, untill we run some tests >>> for a longer period (about 15 days). >>> >>> After that time, kibana was unable to show any data. >>> I did some investigation and it looks like some of the indexes (for 3 >>> days to be exact) seem to be corrupted. >>> Now every query from kibana, using those corrupted indexes - failes. >>> >>> Errors read from elasticsearch logs: >>> >>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: >>> Facet[terms]: failed to find mapping for Name* ... a couple of other >>> columns* >>> - org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet >>> [0]: (key) field [@timestamp] not found >>> >>> ... generaly all queries end with those errors >>> >>> When elasticsearch is started we find something like this: >>> >>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>> Message not fully read (request) for [243445] and action >>> [cluster/nodeIndexCreated], resetting >>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>> Message not fully read (request) for [249943] and action >>> [cluster/nodeIndexCreated], resetting >>> [2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] >>> Message not fully read (request) for [246740] and action >>> [cluster/nodeIndexCreated], resetting >>> >>> And a little observations: >>> >>> 1. When using elasticsearch-head plugin, when querying records >>> 'manually', i can see only elasticsearch columns (_index, _type, _id, >>> _score). >>> But when I 'randomly' select columns and overview their raw json >>> they look ok. >>> >>> 2, When I tried to process same data again - everything is ok >>> >>> Is it possible that some corrupted data found its way to elasticsearch >>> and now whole index is broken ? >>> Can this be fixed ? reindexed or sth ? >>> This data is very importand and can't be lost ... >>> >>> Best Regards, >>> Karol >>> >>> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/2f25e8f8-7c62-4474-a7a4-ee64a433eeca%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/2f25e8f8-7c62-4474-a7a4-ee64a433eeca%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKSE6%3DX-QjwJYzs0Aq7FXOHbE6Vu8Am-5BGvznmAsbFikw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
