if it can help :
_nodes/hot_threads?pretty=1
::: [Sepulchre][tcr0m37nRAS9appE1pvWLA][sd-70975][inet[/x.x.x.x:9300]]
20.6% (102.7ms out of 500ms) cpu usage by thread
'elasticsearch[Sepulchre][search][T#78]'
8/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.LinkedTransferQueue.awaitMatch(
LinkedTransferQueue.java:735)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.
java:644)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.
java:1137)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(
SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.
java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 15 elements
org.apache.lucene.search.TopFieldCollector.populateResults(
TopFieldCollector.java:1203)
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.
java:156)
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.
java:93)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:582)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:275)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
20.2% (101ms out of 500ms) cpu usage by thread
'elasticsearch[Sepulchre][search][T#4]'
10/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.LinkedTransferQueue.awaitMatch(
LinkedTransferQueue.java:735)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.
java:644)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.
java:1137)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(
SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.
java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
19.3% (96.4ms out of 500ms) cpu usage by thread
'elasticsearch[Sepulchre][search][T#42]'
9/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.LinkedTransferQueue.awaitMatch(
LinkedTransferQueue.java:735)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.
java:644)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.
java:1137)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(
SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.
java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
unique snapshot
org.apache.lucene.search.
FieldValueHitQueue$MultiComparatorsFieldValueHitQueue.lessThan(
FieldValueHitQueue.java:122)
org.apache.lucene.search.
FieldValueHitQueue$MultiComparatorsFieldValueHitQueue.lessThan(
FieldValueHitQueue.java:99)
org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:241)
org.apache.lucene.util.PriorityQueue.pop(PriorityQueue.java:177)
org.apache.lucene.search.TopFieldCollector.populateResults(
TopFieldCollector.java:1203)
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.
java:156)
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.
java:93)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:582)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:275)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
_nodes/stats?pretty=1
{
"cluster_name" : "elasticsearch",
"nodes" : {
"tcr0m37nRAS9appE1pvWLA" : {
"timestamp" : 1422495166302,
"name" : "Sepulchre",
"transport_address" : "inet[/x.x.x.x:9300]",
"host" : "sd-70975",
"ip" : [ "inet[/x.x.x.x:9300]", "NONE" ],
"indices" : {
"docs" : {
"count" : 936269,
"deleted" : 545
},
"store" : {
"size_in_bytes" : 856723820,
"throttle_time_in_millis" : 36
},
"indexing" : {
"index_total" : 49,
"index_time_in_millis" : 1058,
"index_current" : 0,
"delete_total" : 184,
"delete_time_in_millis" : 84,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time_in_millis" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 16,
"query_total" : 11557772,
"query_time_in_millis" : 91259801,
"query_current" : 9,
"fetch_total" : 10139269,
"fetch_time_in_millis" : 37413865,
"fetch_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 5,
"total_time_in_millis" : 5835,
"total_docs" : 7258,
"total_size_in_bytes" : 12852963
},
"refresh" : {
"total" : 287,
"total_time_in_millis" : 5430
},
"flush" : {
"total" : 45,
"total_time_in_millis" : 1016
},
"warmer" : {
"current" : 0,
"total" : 445,
"total_time_in_millis" : 308
},
"filter_cache" : {
"memory_size_in_bytes" : 204470568,
"evictions" : 0
},
"id_cache" : {
"memory_size_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 6131132,
"evictions" : 0
},
"percolate" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0,
"memory_size_in_bytes" : -1,
"memory_size" : "-1b",
"queries" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 679,
"memory_in_bytes" : 41166270,
"index_writer_memory_in_bytes" : 0,
"index_writer_max_memory_in_bytes" : 3168274004,
"version_map_memory_in_bytes" : 3268,
"fixed_bit_set_memory_in_bytes" : 0
},
"translog" : {
"operations" : 122,
"size_in_bytes" : 17
},
"suggest" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0
},
"query_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0,
"hit_count" : 0,
"miss_count" : 0
}
},
"os" : {
"timestamp" : 1422495166542,
"uptime_in_millis" : 10821,
"load_average" : [ 49.66, 47.58, 46.68 ],
"cpu" : {
"sys" : 3,
"user" : 83,
"idle" : 12,
"usage" : 86,
"stolen" : 0
},
"mem" : {
"free_in_bytes" : 221818286080,
"used_in_bytes" : 48629456896,
"free_percent" : 84,
"used_percent" : 15,
"actual_free_in_bytes" : 228507299840,
"actual_used_in_bytes" : 41940443136
},
"swap" : {
"used_in_bytes" : 0,
"free_in_bytes" : 10238291968
}
},
"process" : {
"timestamp" : 1422495166543,
"open_file_descriptors" : 2162,
"cpu" : {
"percent" : 982,
"sys_in_millis" : 3025760,
"user_in_millis" : 103162570,
"total_in_millis" : 106188330
},
"mem" : {
"resident_in_bytes" : 7449333760,
"share_in_bytes" : 75313152,
"total_virtual_in_bytes" : 53864665088
}
},
"jvm" : {
"timestamp" : 1422495166630,
"uptime_in_millis" : 10807489,
"mem" : {
"heap_used_in_bytes" : 4758146816,
"heap_used_percent" : 15,
"heap_committed_in_bytes" : 30894260224,
"heap_max_in_bytes" : 30894260224,
"non_heap_used_in_bytes" : 54495664,
"non_heap_committed_in_bytes" : 54984704,
"pools" : {
"young" : {
"used_in_bytes" : 536440,
"max_in_bytes" : 1954217984,
"peak_used_in_bytes" : 1954217984,
"peak_max_in_bytes" : 1954217984
},
"survivor" : {
"used_in_bytes" : 28468776,
"max_in_bytes" : 244252672,
"peak_used_in_bytes" : 151595664,
"peak_max_in_bytes" : 244252672
},
"old" : {
"used_in_bytes" : 4729141600,
"max_in_bytes" : 28695789568,
"peak_used_in_bytes" : 4729141600,
"peak_max_in_bytes" : 28695789568
}
}
},
"threads" : {
"count" : 346,
"peak_count" : 362
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 16969,
"collection_time_in_millis" : 1384852
},
"old" : {
"collection_count" : 0,
"collection_time_in_millis" : 0
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 1673,
"used_in_bytes" : 70117388,
"total_capacity_in_bytes" : 70117388
},
"mapped" : {
"count" : 314,
"used_in_bytes" : 154987920,
"total_capacity_in_bytes" : 154987920
}
}
},
"thread_pool" : {
"generic" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 4,
"completed" : 1443
},
"index" : {
"threads" : 32,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 32,
"completed" : 233
},
"bench" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"get" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"merge" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 5,
"completed" : 116
},
"suggest" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bulk" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"optimize" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"warmer" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 2,
"completed" : 208
},
"flush" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 5,
"completed" : 113
},
"search" : {
"threads" : 96,
"queue" : 1,
"active" : 9,
"rejected" : 0,
"largest" : 96,
"completed" : 24008797
},
"listener" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"percolate" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"management" : {
"threads" : 5,
"queue" : 0,
"active" : 1,
"rejected" : 0,
"largest" : 5,
"completed" : 63903
},
"refresh" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 6,
"completed" : 112
}
},
"network" : {
"tcp" : {
"active_opens" : 2312428,
"passive_opens" : 2484460,
"curr_estab" : 582,
"in_segs" : 209057765,
"out_segs" : 216923023,
"retrans_segs" : 49717,
"estab_resets" : 1144,
"attempt_fails" : 135,
"in_errs" : 34,
"out_rsts" : 2459
}
},
"fs" : {
"timestamp" : 1422495166632,
"total" : {
"total_in_bytes" : 302674501632,
"free_in_bytes" : 264467443712,
"available_in_bytes" : 249068793856,
"disk_reads" : 351417,
"disk_writes" : 205904,
"disk_io_op" : 557321,
"disk_read_size_in_bytes" : 3433067520,
"disk_write_size_in_bytes" : 3127025664,
"disk_io_size_in_bytes" : 6560093184,
"disk_queue" : "0",
"disk_service_time" : "0.1"
},
"data" : [ {
"path" : "/var/lib/elasticsearch/elasticsearch/nodes/0",
"mount" : "/",
"dev" : "/dev/sda2",
"total_in_bytes" : 302674501632,
"free_in_bytes" : 264467443712,
"available_in_bytes" : 249068793856,
"disk_reads" : 351417,
"disk_writes" : 205904,
"disk_io_op" : 557321,
"disk_read_size_in_bytes" : 3433067520,
"disk_write_size_in_bytes" : 3127025664,
"disk_io_size_in_bytes" : 6560093184,
"disk_queue" : "0",
"disk_service_time" : "0.1"
} ]
},
"transport" : {
"server_open" : 13,
"rx_count" : 6,
"rx_size_in_bytes" : 1380,
"tx_count" : 6,
"tx_size_in_bytes" : 1380
},
"http" : {
"current_open" : 11,
"total_opened" : 2311818
},
"breakers" : {
"request" : {
"limit_size_in_bytes" : 12357704089,
"limit_size" : "11.5gb",
"estimated_size_in_bytes" : 16440,
"estimated_size" : "16kb",
"overhead" : 1.0,
"tripped" : 0
},
"fielddata" : {
"limit_size_in_bytes" : 18536556134,
"limit_size" : "17.2gb",
"estimated_size_in_bytes" : 6131132,
"estimated_size" : "5.8mb",
"overhead" : 1.03,
"tripped" : 0
},
"parent" : {
"limit_size_in_bytes" : 21625982156,
"limit_size" : "20.1gb",
"estimated_size_in_bytes" : 6147572,
"estimated_size" : "5.8mb",
"overhead" : 1.0,
"tripped" : 0
}
}
}
}
}
On Wednesday, January 28, 2015 at 11:41:16 PM UTC+1, Oto Iashvili wrote:
>
> Hi,
>
> thanks a lot for answer,
> Ive tried several value for heap, between 26 and 32, but I didnt see any
> difference.
>
> I remove G1 and put back default parameter. But still same pb.
> I was saying around 500%, but it is just average, i goes sometimes up to
> 2000%
>
> I was also thinking to take several server, but right now it is not
> possible.
>
> I was using a smaller server just before, with 96go of ram and I was
> working better. I trie to put same parameters as before, but it is not much
> slower.
>
> On Wednesday, January 28, 2015 at 2:36:29 PM UTC+1, Jilles van Gurp wrote:
>>
>> How much heap are you giving to ES? With this many requests, if your
>> setup is not falling over it is probably not garbage collect related
>> because that would result in very noticable delays/unavailability of es.
>> 32GB should be a good value given how much memory you have. Also, you
>> probably want to use doc_values in your mapping so that you can utilize the
>> os file cache and move some of the memory pressure on the heap. You seem to
>> have plenty of ram, so your entire dataset should easily fit in RAM.
>>
>> Also, don't use G1 for elasticsearch. There are known issues with that
>> particular garbage collector in combination with lucene. CMS is the best
>> option for ES.
>>
>> 500% of 20 cores doesn't sound that bad; you'd max them out at 4000%.
>> Still, it would be nice to know what it is doing. In any case, you might
>> want to try out marvel to find out where your setup is bottlenecked. Also,
>> you might want to consider scaling horizontally instead of vertically. Many
>> smaller servers can be nicer than one big one.
>>
>> On Wednesday, January 28, 2015 at 2:09:22 PM UTC+1, Oto Iashvili wrote:
>>>
>>> Hi
>>>
>>> I have a website for classified. For this I'm using elasticsearch,
>>> postgres and rails on a same ubuntu 14.04 dedicated server, with 256go of
>>> RAM and 20 cores, 40 threads .
>>>
>>> I have 10 indexes on elasticsearch, each have default numbers of shardes
>>> (5). They have between 1000 and 400 000 classifieds dependings on which
>>> index.
>>> approximatly 5000 requests per minute, 2/3 making an elasticsearch
>>> request.
>>>
>>> according to htop, jvm is using around 500% of CPU
>>> I try different options, I reduce number of shardes per index, I also
>>> try to change JAVA_OPTS of followed
>>>
>>> #JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
>>> #JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"
>>>
>>> #JAVA_OPTS="$JAVA_OPTS -XX:CMSInitiatingOccupancyFraction=75"
>>> #JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
>>>
>>> JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
>>>
>>> but it doesnt seems to change anything.
>>>
>>> so to questions :
>>> - when you change any setting on elasticsearch, and then restart,
>>> should the improvement (if any) be visible immediatly or can it arrive a
>>> bit later thanks to cache or any thing else ?
>>> - can any one help me to find good configuration for JVM / elasticsearch
>>> so it will not take that many ressources
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d1b4c785-d6d0-4677-a319-1dc9ea00c6da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
