Do you have a list of what uses each queue? I don't know the exact differences between each statistic. The documentation is non-existent as to the meaning of each. I do know that a count in either is not good though.
What this means is that you need more threads in those queues to handle the workload. Try increasing the value by 5 at a time until you reach a number that results in no blocked counters. I would also suggest setting the minimum threads for each queue to 1. There is really no reason that I can see to have a value greater than 1 for the min, except in the case of the escalation queue. ARServer will create new threads as needed if the min is set to 1. I did not see the times in the spreadsheet. What was the collection interval? Do you have any reports of timeouts during the times the blocked item counter increased? On Tue, Apr 3, 2012 at 12:28 PM, Goodall, Andrew C <ago...@jcp.com> wrote: > ** ** ** ** ** > > Thanks Axton – I’ve got support issues open this, and call outs to BMC R&D > and PS.**** > > ** ** > > I appreciate your help.**** > > ** ** > > Sorry to trouble you…**** > > ** ** > > On the Server Statistics…I’ve attached the output from our 7.6.04 sandbox > (VM Windows 2003 x64 Quad CPU 2.2 GHz Quad Core – 12GB Ram)**** > > ** ** > > This is what is config’d in ar.cfg**** > > ** ** > > Private-RPC-Socket: 390601 1 1 **** > > Private-RPC-Socket: 390603 1 6 **** > > Private-RPC-Socket: 390620 12 12 (FAST) - set by total # of CPU’s > (4) x3**** > > Private-RPC-Socket: 390621 5 16 **** > > Private-RPC-Socket: 390626 5 12 **** > > Private-RPC-Socket: 390632 2 4 **** > > Private-RPC-Socket: 390633 2 2 **** > > Private-RPC-Socket: 390635 20 20 (LIST) - set by total # of CPU’s > (4) x5**** > > Private-RPC-Socket: 390680 2 2 **** > > Private-RPC-Socket: 390681 4 8 **** > > Private-RPC-Socket: 390685 16 32 **** > > Private-RPC-Socket: 390698 10 10**** > > ** ** > > On the server statistics I’m confused on the difference between (“Queue > Items Blocked Count” and “Num Queue Items Blocked Count”)**** > > **** > > I’m seeing high numbers in “Queue Items Blocked Count” but zero in “Num > Queue Items Blocked Count”**** > > ** ** > > The threads with high values in “Queue Items Blocked Count” are:**** > > 390602 – FTS**** > > 390620 - FAST**** > > 390635 - LIST**** > > 390680 – PRIVATE (Approval-RPC-Socket)**** > > ** ** > > ** ** > > Regards,**** > > **** > > *Andrew Goodall***** > > Software Engineer 2 | Development Services | jcpenney . www.jcp.com > <http://www.jcp.com/>| > **** > ------------------------------ > > *From:* Action Request System discussion list(ARSList) [mailto:** > arslist@ARSLIST.ORG**] *On Behalf Of *Axton > *Sent:* Monday, April 02, 2012 11:50 AM > > *To:* **arslist@ARSLIST.ORG** > *Subject:* Re: fast and list threads - by cores or cpus? - documentation > conflicts. > **** > > ** ** > > ** On the pdf from BMC, don't assume that everyone that creates a pdf > knows what they are talking about. Always ask questions. How did they > come to that number? I don't see any text that justifies that setting. I > could pick it apart, but you should ask the questions.**** > > ** ** > > Axton Grams**** > > On Mon, Apr 2, 2012 at 11:45 AM, Axton <axton.gr...@gmail.com> wrote:**** > > Disruptive? Not unless you go crazy with it.**** > > If you want to know the thread utilization for a given queue (e.g., Fast, > List, etc.) you will need the individual queues. It's going to be hard to > tune a queue if you don't know the numbers for that queue.**** > > If you are only tuning the thread counts, a 1 hour interval should be more > than sufficient.**** > > The server stats data is recorded in a form name Server Statistics.**** > > ** ** > > Axton Grams**** > > ** ** > > On Mon, Apr 2, 2012 at 11:29 AM, Goodall, Andrew C <ago...@jcp.com> wrote: > **** > > ** **** > > Thanks so much for all this input Axton.**** > > Great info.**** > > **** > > Question regarding turning on the Server statistics recording…**** > > **** > > 1. Is it disruptive ? wondering whether I can turn on in prod for a > short time to get a true load picture during peak time.**** > > 2. Do I need cumulative and individual queue mode or just cumulative > **** > > 3. What’s a good recording interval?**** > > 4. I assume it saves the data to a .log in BMC > Software\ARSystem\Arserver\Db location**** > > **** > > This all started because we’ve been having performance issues on 7.5 and > we’re upgrading to 7.6.04 with all new hardware (Windows x64 Quad CPU / > Quad core 2.6 GHz – 12GB Ram VMs)**** > > We use full ITSM suite (mainly incident, problem, change, SRM, RKM)**** > > Between 300 – 400 concurrent users**** > > **** > > In recent doc from Performance and Scalability White Paper from BMC - > http://documents.bmc.com/supportu/documents/97/70/249770/249770.pdf**** > > They use a 2 CPU Quad Core Windows x64 (page 10/77) 8 total cores and set > threads (page 48/77) List = 24 and Fast = 40**** > > **** > > **** > > Regards,**** > > **** > > *Andrew Goodall***** > > Software Engineer 2 | Development Services | jcpenney . www.jcp.com > <http://www.jcp.com/>| > **** > > *From:* Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG] *On Behalf Of *Axton > *Sent:* Monday, April 02, 2012 11:12 AM > *To:* arslist@ARSLIST.ORG > *Subject:* Re: fast and list threads - by cores or cpus? - documentation > conflicts.**** > > **** > > ** To add to that, the goals of setting the queue thread settings should > only include:**** > > - you don't want the min thread count too high**** > > - you don't want the max thread count too low**** > > - you don't want the max thread count too high > > Getting the mix right will ensure that there are sufficient threads to > handle the workload. Having too many threads is bad and will result in > excessive overhead within the kernel and the database (each thread creates > a connection to the database server). Having too few threads (without the > ability to increase the ceiling) is bad and will result in performance > problems or timeouts.**** > > **** > > There are no options to adjust how **Remedy** dispatches work to the > threads available or to alter how **Remedy** breaks up the work when > handling. It either has sufficient threads to do what it needs to do, or > it does not. Giving the process more threads than it will use will not > increase the performance of the application, it will just add work during > cache operations, create more db connections, and add overhead to the OS in > managing all those threads.**** > > **** > > Probably equally important to getting the thread configuration right, is > moving certain subsystems to separate queues. Things like email, approval, > reconciliation, etc. should be moved to separate queues. By doing this, > you can leave the fast and list queues available for user requests. This > makes managing the threads (by you) much easier because you don't have many > subsystems competing for a limited pool of threads. Instead you have known > subsystems using private queues that are all theirs and you have queues > available to end users that are all theirs.**** > > **** > > Axton Grams**** > > On Mon, Apr 2, 2012 at 10:57 AM, Axton <axton.gr...@gmail.com> wrote:**** > > Can 2 cores execute 2 threads of execution in parallel? If so, why would > you not want to leverage that capability? I would not suggest putting your > min threads to the max value expected. Start low, say 1. Creating more > threads than are needed adds overhead, esp. during cache operations. ** > Remedy** will create a new thread when all threads are busy and another > request is in the queue. Set the max to the target of what you think you > might need. With this approach, **Remedy** will allocate what it needs.** > ** > > **** > > There is no formula that will magically tell you the ideal number of > threads that you need for your system. Two people can write an application > where both applications provide a comparable set of capabilities. > Depending on how the workflow is written, different thread requirements > will be needed.**** > > **** > > Thread to parallel execution ratio is not the best way to measure things. > An execution path, while only capable of handling one instruction at a > time, can effectively handle more than one thread of execution by > performing a context switch. While context switching does cost some > overhead, it may in fact enable the server to handle a larger concurrency > or provide a higher throughput of operations. That's not to say that in > your case, 4 threads per execution path, is the best scenario, but it might > be.**** > > **** > > The server statistics, coupled with OS statistics, are your best indicator > of whether or not you have allocated the appropriate number of threads. > It's a game of trial and error until you find the right fit for your > environment. To start, set the min threads to 1 for all queues (except > escalation, if you use escalation pools) and the max to some nominal > number, like 10. If you start to see queu blocked counts in the server > statistics, increase the max for that queue. If you do not see queue > blocked counts, you have enough threads to handle the load the system is > experiencing. Check the thread count for each queue after a period of > time. If the number of threads never reaches the max threads allowed for > that queue, your max is too high. Once you figure where the thread count > hovers, set the minimum to match that number. Continue to monitor for > queue blocked items, and increase the max as needed.**** > > **** > > Axton Grams**** > > **** > > On Mon, Apr 2, 2012 at 10:32 AM, LJ LongWing <lj.longw...@gmail.com> > wrote:**** > > ** **** > > Andrew,**** > > I think you should go with 16. I know that a quad core is not the same > thing as 4 CPU’s….BUT…it is for most practical purposes…I believe the terms > core and cpu are used loosely to mean the same thing these days when > referring to CPU power….it’s my understanding that to make a quad core cpu, > they shrunk the technology of a single core to be able to fit 4 dies on the > same chip….so while you are more constrained on bus because you are all on > one chip…your processing power should be better than a single core on a > single cpu….**** > > **** > > I know I’m going to get a lot of flack for my loose and probably improper > usage of terms….but that’s roughly how I think of it J**** > > **** > > *From:* Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG] *On Behalf Of *Goodall, Andrew C > *Sent:* Thursday, March 22, 2012 5:57 PM > *To:* arslist@ARSLIST.ORG > *Subject:* fast and list threads - by cores or cpus? - documentation > conflicts.**** > > **** > > ** **** > > I’m confused – the doc seems to conflict and BMC architects have signed > off on our ar.cfg settings in the past in which they were set by CPU and > not by Cores.**** > > **** > > My previous understanding was that these values should be set by the > number of CPUs and not Cores, in fact BMC architects have signed off on > these settings many times – and if you look at “BMC **Remedy** AR System > Server 7.6 – Performance Tuning” page 33 – it states CPUs**** > > **** > > However, in the recent publication for performance and scalability > “Performance and Scalability White Paper” released this month – see page 4 > and 48 – this was set at the number of total cores.**** > > **** > > So – we have a Quad CPU 2.2 GHz Quad Core AMD with 12GB ram Windows box.* > *** > > **** > > So should I use x4 or x16 = a huge difference in the number of threads, > which could substantially impact performance.**** > > **** > > **** > > See:**** > > **** > > “Performance and Scalability White Paper” page 4 and 48 - > http://documents.bmc.com/supportu/documents/97/70/249770/249770.pdf**** > > **** > > “BMC ******Remedy**** **AR**** System Server 7.6 – Performance Tuning” > page 33 - > http://documents.bmc.com/supportu/documents/90/37/199037/199037.pdf**** > > **** > > **** > > Regards,**** > > **** > > *Andrew Goodall***** > > Software Engineer 2 | Development Services | jcpenney . www.jcp.com > <http://www.jcp.com/> > **** > > **** > > > The information transmitted is intended only for the person or entity to > which it is addressed and > may contain confidential and/or privileged material. If the reader of this > message is not the intended > recipient, you are hereby notified that your access is unauthorized, and > any review, dissemination, > distribution or copying of this message including any attachments is > strictly prohibited. If you are not > the intended recipient, please contact the sender and delete the material > from any computer.**** > > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ **** > > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_**** > > **** > > **** > > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ **** > > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_**** > > ** ** > > ** ** > > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ **** > _attend WWRUG12 www.wwrug.com ARSlist: "Where the Answers Are"_ > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: "Where the Answers Are"
