| I think because the instance has 64Gbytes of RAM and I've divided this up into 32Gbytes to the JVM
reduce it to 30 or 31 for the JVM, 32 ->64 is not a good idea. it triggers a pointer flip to 64bit https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/ On Wed, Sep 21, 2022 at 9:40 AM Derek C <de...@hssl.ie> wrote: > Hi Deepak, > > IOPs is something I have looked at and adjusted over the last few weeks. > I'm now using a "GP3" disk type set to 3,000 IOPS. > (The other week I tested IOPS/data throughput with fio and it's delivering > as promised). > > Funny though - I think because the instance has 64Gbytes of RAM and I've > divided this up into 32Gbytes to the JVM and 32Gbytes to the O/S, and > my collection size is 20Gbytes I suspect the O/S has cached the entire > index because when I was hitting SOLR with JMeter I > wasn't really seeing the disk reads in iotop (I could see the disk writes > in iotop - the SOLR log I think). > > I suspect the O/S cache because if I say, cat all the SOLR logs to > /dev/null I see the disk reads in iotop but then if I repeat the cat > command it'll be > instant and I won't see any disk reads. Also I didn't see the disk reads > in the AWS storage "Read throughput (Ops/s)" chart which matches > me not seeing the disk reads in iotop. > > So that made me think the problem isn't around a disk I/O bottleneck. > > Derek > > On Wed, Sep 21, 2022 at 2:27 PM Deepak Goel <deic...@gmail.com> wrote: > > > what is the iops allocated to your aws instance? > > > > On Wed, 21 Sep 2022, 17:36 Derek C, <de...@hssl.ie> wrote: > > > > > Hi all, > > > > > > I'm trying to identify a SOLR query performance issue (or rather how to > > > size [AWS EC2] hardware to support many users). > > > > > > Right now I'm testing on a single SOLR node (with distrib=false to keep > > the > > > query to the one node). > > > > > > I'm using Apache JMeter to test (with many different queries in a CSV) > > and > > > I'm using prometheus-exporter / prometheus / Grafana to view the status > > of > > > the SOLR node/replica). I'm also looking at top for CPU usage and > iotop > > > for disk IO on the VM/instance. > > > > > > Right now the VM/instance is a x2gd.xlarge which is Graviton based > with 4 > > > [v]CPUs and 64Gbytes of memory (32Gbytes to SOLR and 32Gbytes to O/S). > > > > > > My collection is 20Gbytes in size with about 2.5 million documents. > > > > > > Interestingly when I run a test I don't see any disk I/O reads at all - > > so > > > I think [debian] linux has cached the index in RAM (great I think). > > > > > > What I don't understand is that if I run a lot of parallel/rolling > > queries > > > (probably 1,500) with JMeter what I see [in top] is the CPU *not* > hitting > > > 400% (4 cores) and, from time to time, it's at 0% which is really weird > > - I > > > would expect the CPU to be the the bottleneck (and the thing to be > scaled > > > up for more performance). Meanwhile queries are taking over 10 seconds > > to > > > respond. > > > > > > I'm not really sure where to look (in the Grafana SOLR dashboard?) to > > look > > > further and try to find out more. > > > > > > If anyone has any suggestions it's be great > > > > > > thanks, > > > > > > Derek > > > > > > -- > > > Derek Conniffe > > > Harvey Software Systems Ltd T/A HSSL > > > Skype: dconnrt > > > Email: de...@hssl.ie > > > > > > > > > *Disclaimer:* This email and any files transmitted with it are > > confidential > > > and intended solely for the use of the individual or entity to whom > they > > > are addressed. If you have received this email in error please delete > it > > > (if you are not the intended recipient you are notified that > disclosing, > > > copying, distributing or taking any action in reliance on the contents > of > > > this information is strictly prohibited). > > > *Warning*: Although HSSL have taken reasonable precautions to ensure no > > > viruses are present in this email, HSSL cannot accept responsibility > for > > > any loss or damage arising from the use of this email or attachments. > > > P For the Environment, please only print this email if necessary. > > > > > > > > -- > -- > Derek Conniffe > Harvey Software Systems Ltd T/A HSSL > Telephone (IRL): 086 856 3823 > Telephone (US): (650) 443 8285 > Skype: dconnrt > Email: de...@hssl.ie > > > *Disclaimer:* This email and any files transmitted with it are confidential > and intended solely for the use of the individual or entity to whom they > are addressed. If you have received this email in error please delete it > (if you are not the intended recipient you are notified that disclosing, > copying, distributing or taking any action in reliance on the contents of > this information is strictly prohibited). > *Warning*: Although HSSL have taken reasonable precautions to ensure no > viruses are present in this email, HSSL cannot accept responsibility for > any loss or damage arising from the use of this email or attachments. > P For the Environment, please only print this email if necessary. >
