The elasticsearch log file was not showing any activity during the time when this was happening , but when I restarted the elasticsearch service - I got some warnings (see attached log file) , but then seem to proceed normally and has not complained any more . Note: My process is still indexing new documents.
On Monday, October 20, 2014 3:52:47 PM UTC+2, Pieter Agenbag wrote:
>
> Good afternoon.
>
> The subject might be a little misleading as to the true nature of my
> problem - which I'll try to explain here in as much detail as possible.
>
> First of all , I am rather new to Elasticsearch.
>
> Secondly , this problem has happened more than once (after dumping all
> indexes and starting over).
>
> Ok ,here goes:
>
> I have a single elasticsearch node running on a dedicated Dell PowerEdge
> R720 with dual 6 core cpus and 96GB ram. (of which 32GB is assigned to the
> HEAP)
> The machine is connected back to back with a 10GB fiber to another Dell
> R620.
>
> My processing happens on the R620 which then uses the bulk api to index
> (currently testing at 8000 documents per second) into the ES on the R720.
> The documents are all of the same type and indexed into daily "partitions"
> .. events-2014-10-18 , events-2014-10-19 , events-2014-10-20 etc.
> I then have a Kibana dashboard on top of that.
>
> All of this works perfectly for several days and then seemingly stops. I
> noticed that my Kibana dashboard (defaulting to the last hours data)
> stopped plotting. Further investigation showed that my application is still
> processing and indexing documents,
> but a search on todays index , with descending timestamp ordering shows
> the last document as 3:59 this morning.
>
> POST /events-2014-10-20/event/_search
> {
> "size":20
> ,"from": 0
> ,"query":
> {
> "filtered":
> {
> "query":
> {
> "match_all": {}
> }
> }
> }
> ,"sort":
> [
> {
> "timestamp":
> {
> "order": "desc"
> }
> }
> ]
> }
>
> .... result....
> {
> "took": 22836,
> "timed_out": false,
> "_shards": {
> "total": 5,
> "successful": 5,
> "failed": 0
> },
> "hits": {
> "total": 385153635,
> "max_score": null,
> "hits": [
> {
> "_index": "events-2014-10-20",
> "_type": "event",
> "_id": "HYOy53Q3TM-TwBz6SLRB5w",
> "_score": null,
> "_source": {
> "timestamp": "2014-10-20 03:59:14",
> .. etc
>
> Now, when I do a bulk index , the API tells me that the documents were
> created and specifies the autogenerated ids.
> So , taking one of those IDs and querying it directly , I DO get the
> document back.
> In the correct index , with the correct timestamp at 10:12 (from earlier
> when I was investigating)
>
> GET /events-2014-10-20/event/kY6MaTgCThizVGgb3iSGrw
>
> {
> "_index": "events-2014-10-20",
> "_type": "event",
> "_id": "kY6MaTgCThizVGgb3iSGrw",
> "_version": 1,
> "found": true,
> "_source": {
> "timestamp": "2014-10-20 10:12:35",
>
> ------------------------
> Some system stats :
> ------------------------
> The machine's cpu spikes every now and then (as I issue bulk indexes) ,
> but drops down to idle again afterwards.
> There's plenty left on the heap and the disks is only 33% used.
>
>
> curl -XGET 'localhost:9200/_cat/health?v'
> epoch timestamp cluster status node.total node.data shards pri
> relo init unassign
> 1413812685 15:44:45 elasticsearch yellow 1 1 85 85
> 0 0 15
>
> curl -XGET 'localhost:9200/_cat/count?v'
> epoch timestamp count
> 1413812740 15:45:40 4171812520
>
> curl -XGET '10.15.3.19:9200/_cat/indices/events*?v'
> health index pri rep docs.count docs.deleted store.size
> pri.store.size
> green events-2014-10-19 5 0 702422696 0 230.5gb
> 230.5gb
> green events-2014-10-17 5 0 702341395 0 230.4gb
> 230.4gb
> green events-2014-10-18 5 0 702352975 0 230.3gb
> 230.3gb
> green events-2014-10-15 5 0 703229636 0 230.8gb
> 230.8gb
> green events-2014-10-16 5 0 701547992 0 230.2gb
> 230.2gb
> green events-2014-10-14 5 0 252394923 0 83.1gb
> 83.1gb
> yellow events-2014-10-20 5 1 407331075 0 127.9gb
> 127.9gb
>
> curl -XGET 'localhost:9200/_cat/pending_tasks?v'
> insertOrder timeInQueue priority source
>
> curl -XGET '10.15.3.19:9200/_cat/shards/events-2014-10-20?v'
> index shard prirep state docs store ip node
> events-2014-10-20 4 p STARTED 81862132 27.8gb 10.15.3.19 X-Ray
> events-2014-10-20 4 r UNASSIGNED
> events-2014-10-20 0 p STARTED 81869999 25.4gb 10.15.3.19 X-Ray
> events-2014-10-20 0 r UNASSIGNED
> events-2014-10-20 3 p STARTED 81885822 25.5gb 10.15.3.19 X-Ray
> events-2014-10-20 3 r UNASSIGNED
> events-2014-10-20 1 p STARTED 81868103 25.5gb 10.15.3.19 X-Ray
> events-2014-10-20 1 r UNASSIGNED
> events-2014-10-20 2 p STARTED 81871297 26.5gb 10.15.3.19 X-Ray
> events-2014-10-20 2 r UNASSIGNED
>
> I can obviously not put this system into production and my deadline is
> fast approaching - so , any help will be greatly appreciated.
>
> Regards.
> Pieter
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e5ce48f5-8b1d-4a64-a0b7-ef7d6f45dc2f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
eslog_2014_10_20.log
Description: Binary data
