Hi,
I've noticed a very disturbing ElasticSearch behaviour ...
my environment is:
1 logstash (1.3.2) (+ redis to store some data) + 1 elasticsearch (0.90.10)
+ kibana
which process about 7 000 000 records per day,
everything worked fine on our test environment, untill we run some tests
for a longer period (about 15 days).
After that time, kibana was unable to show any data.
I did some investigation and it looks like some of the indexes (for 3 days
to be exact) seem to be corrupted.
Now every query from kibana, using those corrupted indexes - failes.
Errors read from elasticsearch logs:
- org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet[terms]: failed to find mapping for Name* ... a couple of other
columns*
- org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet [0]:
(key) field [@timestamp] not found
... generaly all queries end with those errors
When elasticsearch is started we find something like this:
[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] Message
not fully read (request) for [243445] and action
[cluster/nodeIndexCreated], resetting
[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] Message
not fully read (request) for [249943] and action
[cluster/nodeIndexCreated], resetting
[2014-02-07 15:02:08,147][WARN ][transport.netty ] [Name] Message
not fully read (request) for [246740] and action
[cluster/nodeIndexCreated], resetting
And a little observations:
1. When using elasticsearch-head plugin, when querying records 'manually',
i can see only elasticsearch columns (_index, _type, _id, _score).
But when I 'randomly' select columns and overview their raw json they
look ok.
2, When I tried to process same data again - everything is ok
Is it possible that some corrupted data found its way to elasticsearch and
now whole index is broken ?
Can this be fixed ? reindexed or sth ?
This data is very importand and can't be lost ...
Best Regards,
Karol
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/426a8411-bf03-42d3-9b77-1e45dace98ff%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
