HI David, thanks v much - I'll certainly try this - it'll be quick and easy to test.
Derek On Wed, Sep 21, 2022 at 3:21 PM David Hastings <hastings.recurs...@gmail.com> wrote: > | I think because the instance has 64Gbytes of RAM and I've > divided this up into 32Gbytes to the JVM > > reduce it to 30 or 31 for the JVM, 32 ->64 is not a good idea. it triggers > a pointer flip to 64bit > > https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/ > > > > On Wed, Sep 21, 2022 at 9:40 AM Derek C <de...@hssl.ie> wrote: > > > Hi Deepak, > > > > IOPs is something I have looked at and adjusted over the last few weeks. > > I'm now using a "GP3" disk type set to 3,000 IOPS. > > (The other week I tested IOPS/data throughput with fio and it's > delivering > > as promised). > > > > Funny though - I think because the instance has 64Gbytes of RAM and I've > > divided this up into 32Gbytes to the JVM and 32Gbytes to the O/S, and > > my collection size is 20Gbytes I suspect the O/S has cached the entire > > index because when I was hitting SOLR with JMeter I > > wasn't really seeing the disk reads in iotop (I could see the disk writes > > in iotop - the SOLR log I think). > > > > I suspect the O/S cache because if I say, cat all the SOLR logs to > > /dev/null I see the disk reads in iotop but then if I repeat the cat > > command it'll be > > instant and I won't see any disk reads. Also I didn't see the disk reads > > in the AWS storage "Read throughput (Ops/s)" chart which matches > > me not seeing the disk reads in iotop. > > > > So that made me think the problem isn't around a disk I/O bottleneck. > > > > Derek > > > > On Wed, Sep 21, 2022 at 2:27 PM Deepak Goel <deic...@gmail.com> wrote: > > > > > what is the iops allocated to your aws instance? > > > > > > On Wed, 21 Sep 2022, 17:36 Derek C, <de...@hssl.ie> wrote: > > > > > > > Hi all, > > > > > > > > I'm trying to identify a SOLR query performance issue (or rather how > to > > > > size [AWS EC2] hardware to support many users). > > > > > > > > Right now I'm testing on a single SOLR node (with distrib=false to > keep > > > the > > > > query to the one node). > > > > > > > > I'm using Apache JMeter to test (with many different queries in a > CSV) > > > and > > > > I'm using prometheus-exporter / prometheus / Grafana to view the > status > > > of > > > > the SOLR node/replica). I'm also looking at top for CPU usage and > > iotop > > > > for disk IO on the VM/instance. > > > > > > > > Right now the VM/instance is a x2gd.xlarge which is Graviton based > > with 4 > > > > [v]CPUs and 64Gbytes of memory (32Gbytes to SOLR and 32Gbytes to > O/S). > > > > > > > > My collection is 20Gbytes in size with about 2.5 million documents. > > > > > > > > Interestingly when I run a test I don't see any disk I/O reads at > all - > > > so > > > > I think [debian] linux has cached the index in RAM (great I think). > > > > > > > > What I don't understand is that if I run a lot of parallel/rolling > > > queries > > > > (probably 1,500) with JMeter what I see [in top] is the CPU *not* > > hitting > > > > 400% (4 cores) and, from time to time, it's at 0% which is really > weird > > > - I > > > > would expect the CPU to be the the bottleneck (and the thing to be > > scaled > > > > up for more performance). Meanwhile queries are taking over 10 > seconds > > > to > > > > respond. > > > > > > > > I'm not really sure where to look (in the Grafana SOLR dashboard?) to > > > look > > > > further and try to find out more. > > > > > > > > If anyone has any suggestions it's be great > > > > > > > > thanks, > > > > > > > > Derek > > > > > > > > -- > > > > Derek Conniffe > > > > Harvey Software Systems Ltd T/A HSSL > > > > Skype: dconnrt > > > > Email: de...@hssl.ie > > > > > > > > > > > > *Disclaimer:* This email and any files transmitted with it are > > > confidential > > > > and intended solely for the use of the individual or entity to whom > > they > > > > are addressed. If you have received this email in error please delete > > it > > > > (if you are not the intended recipient you are notified that > > disclosing, > > > > copying, distributing or taking any action in reliance on the > contents > > of > > > > this information is strictly prohibited). > > > > *Warning*: Although HSSL have taken reasonable precautions to ensure > no > > > > viruses are present in this email, HSSL cannot accept responsibility > > for > > > > any loss or damage arising from the use of this email or attachments. > > > > P For the Environment, please only print this email if necessary. > > > > > > > > > > > > > -- > > -- > > Derek Conniffe > > Harvey Software Systems Ltd T/A HSSL > > Telephone (IRL): 086 856 3823 > > Telephone (US): (650) 443 8285 > > Skype: dconnrt > > Email: de...@hssl.ie > > > > > > *Disclaimer:* This email and any files transmitted with it are > confidential > > and intended solely for the use of the individual or entity to whom they > > are addressed. If you have received this email in error please delete it > > (if you are not the intended recipient you are notified that disclosing, > > copying, distributing or taking any action in reliance on the contents of > > this information is strictly prohibited). > > *Warning*: Although HSSL have taken reasonable precautions to ensure no > > viruses are present in this email, HSSL cannot accept responsibility for > > any loss or damage arising from the use of this email or attachments. > > P For the Environment, please only print this email if necessary. > > > -- -- Derek Conniffe Harvey Software Systems Ltd T/A HSSL Telephone (IRL): 086 856 3823 Telephone (US): (650) 443 8285 Skype: dconnrt Email: de...@hssl.ie *Disclaimer:* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please delete it (if you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited). *Warning*: Although HSSL have taken reasonable precautions to ensure no viruses are present in this email, HSSL cannot accept responsibility for any loss or damage arising from the use of this email or attachments. P For the Environment, please only print this email if necessary.
