Hi Eran,
Which version of Elasticsearch are you using?
Are you assigning your own document IDs or letting Elasticsearch assign
them automatically?
Best regards,
Christian
On Friday, April 24, 2015 at 7:49:56 AM UTC+1, Eran wrote:
>
> Hello,
>
> I've created an index I use for logging.
>
> This means there are mostly writes, and some searches once in a while.
> In the phase of the first loading, I'm using several clients to
> concurrently index documents using the bulk API.
>
> At first, indexing takes 200 ms for a bulk of 5000 documents.
> As time goes by, the indexing time increases, and gets to 1000-4500 ms.
>
> I am using an EC2 c3.8xl machine with 32 cores, and 60 GB of memory, with
> an IO provisioned volume set to 7000 IOPS.
>
> Looking at the metrics, I see that the CPU and memory are fine, the write
> IOPS are at 300, but the read IOPS have slowly gone up and got to 7000.
>
> How come I'm only indexing, but most of the IOPS are read?
>
> I am attaching some screen captures from the BigDesk plugin, that show the
> two states of the index, ater about 20% of the graphs is the point in time
> where I stopped the clients, so you can see the load drop of.
>
> My settings are:
>
> threadpool.bulk.type: fixed
> threadpool.bulk.size: 32 # availableProcessors
> threadpool.bulk.queue_size: 1000
>
> # Indices settings
> indices.memory.index_buffer_size: 50%
>
>
> 376,1 97%
> indices.cache.filter.expire: 6h
>
> bootstrap.mlockall: true
>
>
> and I've change the index settings to:
>
>
> {"index":{"refresh_interval":"60m","translog":{"flush_threshold_size":"1gb","flush_threshold_ops":"50000"}}}
> I also tried "refresh_interval":"-1"
>
>
> Please let me know what else I need to provide if needed (settings, logs,
> metrics)
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1ee7a991-d6d5-4240-be92-e73db63cccf5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
