SSDs help, though there is likely some other issue here so it's probably not worth looking at, at this time.
Have you checked hot threads or the slow query log? Can you provide more specs on your hardware? What java version are you running? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 11 June 2014 08:41, <[email protected]> wrote: > The Heap Size is being reduced to 30GB to ensure that's not the bottleneck. > > The servers currently run SAS Drives. Though SSDs are usually preferred > for Elasticsearch, can this cause such disparities in > performance? ElasticHQ reports very high Refresh Rates, Search-Fetch and > Search-Query rates. > > On Tuesday, June 10, 2014 3:17:49 PM UTC-7, Mark Walkom wrote: >> >> You will likely see an increase by distributing it to one shard per >> machine, but that's hard to quantify without actually doing it. >> >> Also, you may be doing yourself a disservice with such a large heap size >> as Nik mentioned. Over 32GB, Java pointers are not compressed and you do >> lose a bit of performance due to this. >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: [email protected] >> web: www.campaignmonitor.com >> >> >> On 11 June 2014 07:20, <[email protected]> wrote: >> >>> Thanks for the clarification. The servers aren't under any (read) load >>> yet. There is constant update of data in the background - Roughly about 60 >>> Index Writes per second. The refresh interval is set to 60s. Can this be a >>> performance bottleneck? >>> >>> We can add in more nodes to bring it up to 10 Nodes - 5 Shards with 1 >>> Replica. But I doubt if that will reduce the Empty Search Query to 50ms. >>> Are there any other profiling tools out there to debug the response time? >>> >>> >>> On Tuesday, June 10, 2014 11:30:03 AM UTC-7, Nikolas Everett wrote: >>> >>>> Short answer: yes. >>>> Long answer: 500ms is a long time for the empty query. I see 2ms from >>>> elasticsearch and 23ms from time in development. In production I see maybe >>>> 54ms from elasticsearch and 70 from time across far far more shards and >>>> more data. When I do the same query across thousands of shards and a >>>> couple of TB of data I get ~250ms. Production is 16 servers with 96GB of >>>> ram and 30GB heaps. >>>> >>>> The analyzers really aren't going to hurt performance. >>>> >>>> I'd have a look at your servers themselves: what kind of load are they >>>> under? What is your indexing rate? that kind of thing. >>>> >>>> Also, 30GB is normally the sweet spot for heap sizes, making ~64GB of >>>> total ram the sweet spot for total ram. 110GB heap is pretty high and I'd >>>> expect for new generation (pause the world) garbage collection to take a >>>> while there. >>>> >>>> >>>> Nik >>>> >>>> >>>> On Tue, Jun 10, 2014 at 2:20 PM, <[email protected]> wrote: >>>> >>>>> I am currently running only 1 index with 5 shards. So the both of >>>>> those queries yield the same response time. My main question is to >>>>> understand if scaling out is an Option given the current replication >>>>> scheme. >>>>> >>>>> >>>>> <https://lh5.googleusercontent.com/-bz8iQd0KUaA/U5dMSGLNNFI/AAAAAAAAABg/tGJl0HOj4xo/s1600/Elasticsearch+Cluster.png> >>>>> >>>>> >>>>> On Tuesday, June 10, 2014 11:15:26 AM UTC-7, Nikolas Everett wrote: >>>>> >>>>>> I imagine that depends on lots of stuff. Are you doing >>>>>> elasticsearch:9200/_search or elasticsearch:9200/index/_search ? >>>>>> The former can take quite a while if you have lots index and lots of >>>>>> shards. If you can get away with not doing it, I would. The latter will >>>>>> only take a long time if you have tons of shards. It should otherwise be >>>>>> pretty quick. >>>>>> >>>>>> >>>>>> On Tue, Jun 10, 2014 at 2:10 PM, <[email protected]> wrote: >>>>>> >>>>>>> We currently run our Elasticsearch (*v1.0.2*) cluster on *3 Nodes* >>>>>>> with *5 Shards and 1 Replication* Scheme. The total index size is >>>>>>> about 70GB (~140GB with replication). >>>>>>> >>>>>>> The Empty Search (/_search) query takes 500-600 ms to respond. Will >>>>>>> adding in more Nodes help in this case? The Servers are have 252gb >>>>>>> of RAM and 110gb for Heap. >>>>>>> >>>>>>> The Index uses the following analyzers - standard, lowercase, stop, >>>>>>> porter_stem. Will this degrade Query performance? >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "elasticsearch" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/elasticsearch/9941cfd1-d21 >>>>>>> 1-4706-aa45-6c545c66baff%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/elasticsearch/9941cfd1-d211-4706-aa45-6c545c66baff%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "elasticsearch" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/elasticsearch/d5291841-e791-40b3-93e6-d8fbe9921ac5%40goo >>>>> glegroups.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/d5291841-e791-40b3-93e6-d8fbe9921ac5%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/ecb24d7b-404e-482c-b70e-9b90d33fd18d% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/ecb24d7b-404e-482c-b70e-9b90d33fd18d%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/c6cf345b-762e-4093-80fe-b7c535da77b9%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/c6cf345b-762e-4093-80fe-b7c535da77b9%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aytE%3Dm1CnLLmagMpFJPOLxN2sytLKN%2B7ZW344fYHHbRA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
