LJ, How much Remedy interaction is _required_ during the processing of a single request? What I'm wondering is if some of the processing could be off-loaded into an external process that would then take care of the majority of the processing and then only hit Remedy when it needs to create or retrieve records from Remedy. If much of the processing you're currently doing in Remedy is just pseudo-processing in order to trigger this or that (e.g., push a field here to trigger this workflow there, etc.), then moving that type of processing into an external, say Java, process could potentially reduce the load on your Remedy server speeding things up in the end. For example, if you have a web service that the external system is calling, and then Remedy is calling other web services, etc., to process the request, you could write a web service in Java that the external system would call, that could then also call the other web services and create/get entries in Remedy when necessary. Doing something like that in an external process could also make it easier to implement some kind of caching scheme. You could even host it as a separate web application on the same mid-tier server so that you don't need to set up a another web server to host it.
Lyle -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of LJ Longwing Sent: Tuesday, January 19, 2010 8:47 AM To: [email protected] Subject: Re: Fast/List Concurrent settings? Doug, As usual, in a word, eloquent. I had already suggested a private queue, but the problem I see with it, and this is a minor one, is that if I'm not mistaken, if I want to use a private queue on an existing mid-tier server, just for these 'unruly users', then I need to specify two servers on the same mid-tier which has caused me issues in the past...so my thought was to further isolate the users and setup a mid-tier server specifically for this function and set it to that same private queue, this would remove the burst load from my main load balanced midtier cluster and then isolate them onto their own RPC queue so they didn't interfere with the normal users....does that sound best? I was trying to figure out how to turn down the amount of SQL calls in general that Remedy was doing but had forgotten about the history option....that is a great option that I could certainly use here and will likely provide several seconds of savings which will be a huge percentage. You are correct that I don't have control over the burst behavior of the external system, which is truly unfortunate...I'm simply trying to find a high water mark and attempting to mitigate the amount of failures they will receive because of timeout. We have seen bursts of 500 calls within a 20 min window, then nothing for an hour....frustrating, but it's what we deal with :). Thanks again for your time in not only reading the list but regularly posting detailed answers. -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of Mueller, Doug Sent: Monday, January 18, 2010 11:32 AM To: [email protected] Subject: Re: Fast/List Concurrent settings? Now, we are back to the original topic of blasts of concurrent operations. First, is it really the only way things work that there is no work and then a blast of xxx simultaneous and then no work? Could the other system have a more steady stream of work rather than the periodic blasts of work? Even if there was some differentiation, it would balance the load -- of both systems. Now, this all depends on the other system being able to spread things in some way. If it can, this would be a big step to any system for dealing with volume. If not, then we have to look at other things on the AR System side. Assuming you cannot spread the load.... First, your note didn't indicate you were loosing operations or they were being rejected. That is because of the dispatcher model and all the operations are being accepted and put on the list for processing, but they are not able to all be processed simultaneously. So, to start with, we are not loosing operations, we just have a tuning issue to deal with. With the queue model of the system, the first thing I would recommend is to configure a private queue for the use of this system. I assume your system is not only for the use of this one automated processor. So, the first thing I would do is to define a private queue just for that automated system to isolate the load of that system from other users and especially from interactive users. You don't want to affect other users of the system when the "flood" hits from this automated system. Using a private queue will isolate the load -- just like in the stadium example where you have a special entrance for the "unrully crowd". Then, we can look at the number of threads that are appropriate for this private queue. That can be looked at independently of the number of threads for other queues in the system for other purposes. In theory, there is no reason you cannot have 10s or even 100s of threads in a queue. It is just a number to us. We will start up that many if needed. What you need to be aware of is that each thread will open a database connnection so you need to have a database allowing that many connnections AND that each thread will take some amount of memory. With the 7.5 release on UNIX and Linux supporting a 64-bit address space, the memory can be grown more if needed (still 32-bit on windows but 64-bit is coming). You have to make the call about how much memory you have, and performance of swapping, and overall overhead of processes on the system for the number of threads you want to configure. You also have to worry about system configuration of per process memory and file descriptors (open connections count as a file descriptor) and other such things that you may encounter if configuring very large numbers of threads. BUT, there is no inherent restriction in the AR System about the number of threads you could configure if you wanted to. You already have seen some of the issues that high simultaneous operations on a single table can do. There are some settings that can help with that, for example Next ID blocks -- limits contention on the next ID database column No status history -- If you don't need it for your table, you can set this option and eliminate the create/management of the status history table and entries (definitely available in 7.5; maybe in 7.1 since I cannot remember the release this option was added for) If your logic has workflow that does pushes to other tables that perform create operations there, you should really look at these settings on those forms as well because you are indirectly issuing creates against them too. There are options to control where your DB indexes are stored vs. the data to put indexes and data on different disc areas -- which helps throughput on heavy scale create/modify operations. You should investigate operations your workflow is performing to make sure that they are tuned as well as possible and that there are no inefficient searches or steps to the process so that the total processing time for each create can be minimized. If you don't have to do something during the interactive processing of the create, don't. Don't do it at all if it is not needed or perform the processing later if it is not necessary for interactive response to the user. Hopefully, this gives some ideas to look at. It is not a definitive answer to your inquiry, but hopefully some useful thoughts that lead you toward the best answer for your situation. And, maybe some ideas for others to consider. Hopefully, this note has been useful in terms of providing some information about how the system functions -- and maybe some reasoning behind why choices were made. I hope it also provided some ideas for dealing with high simultaneous operations situations. Doug Mueller _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are" NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
