Thanks Doug Axton
On Thu, Jun 19, 2014 at 12:08 PM, Mueller, Doug <doug_muel...@bmc.com> wrote: > ** > > Axton, > > > > I know a bit about the system and am sometimes even correct in my > answers!!!! > > > > For this case….. > > > > I can definitively state that the AR System server cache of metadata is > SHARED across threads. There is by > > default 1 and only 1 copy of the definition cache for a server regardless > of the number of threads. > > > > If you make an Administrative change to any definition (include groups by > the way), the system will make a > > copy of the cache and the Administrative thread will use that copy and all > other threads will use the original. > > When an interval has passed since the last administrative change > (configurable, but generally around 30 > > seconds), the "new" cache copy with all the changes made is made active > and as each thread that is running > > completes its currently processing API call, it will switch from the cache > copy it is using to the new "active" > > cache. When the last thread switches, the old cache copy is deleted and > you are back to 1 cache copy. > > > > If you have a really active system and have lots of threads and you make > admin changes with pauses between > > them, you can sometimes have 3 or 4 copies of the cache in memory at once > all running through the threads > > finishing and then switching to the most current but that will be for > seconds/minutes depending on the > > length of the longest running API call connected to an older cache copy. > > > > Steady state and except during change, there is 1 and only 1 copy of the > cache that is shared by all threads > > (in a thread safe manner which is easy as it is always only READ by > threads and never written). > > > > Now, there is a mode – Developer-cache-mode – that you can configure a > system in that we recommend is > > used on DEVELOPMENT machines but not on production that will have 1 and > only 1 copy EVER of the cache. > > There is additional locking in place so that when there are Admin updates, > other threads are blocked > > temporarily to let that update happen. It is good on development > environments where there are lots of > > changes and you want them reflected immediately. > > > > > > Anyway, way more than the question asked. > > > > > > To be clear, the memory cache of definitions is several hundred MB. There > is no way there could be a copy > > per thread (10 threads times 300 MB = blowout and that is just 10 > threads). And there is NOT. > > > > I hope this helps clarify things. > > > > Doug Mueller > > > > *From:* Action Request System discussion list(ARSList) [mailto: > arslist@ARSLIST.ORG] *On Behalf Of *Axton > *Sent:* Wednesday, June 18, 2014 9:14 PM > > *To:* arslist@ARSLIST.ORG > *Subject:* Re: Threads and queues > > > > ** > > Are you sure about 1 copy of the cache in memory? In theory, in a multi > threaded application there should, ideally, be one copy of the cache that > is accessed in a thread safe way, but I am almost certain that is not the > case with Remedy. It would be great if someone who knows for certain wants > to clarify. > > On Jun 18, 2014 3:29 PM, "Misi Mladoniczky" <m...@rrr.se> wrote: > > Hi Axton, > > What do you mean by cache data? > > The threads do NOT have an individual copy of the Cache (definitions of > your > system including FLTR, ESCL, etc). They share a single copy of this cache. > If > they did, it would require an enormous amount of memory to have multiple > threads. > > The following is slightly off topic I guess... > > In ar.cfg/conf you can set Cache-Mode which affects things when you change > an > object through DevStudio or change/add a group in the Group-form. > > Cache-Mode: 0 (default) > => threads continue to work with the active cache, and a new cache is built > from scratch. When the re-caching has been completed, all threads > immediately > switch to the new cache. The change will not be noticed by end users at > all if > you have abundant memory on your system. The disadvantage is that the > memory > consumption of cache data doubles for a while, and everything is reread > from > the database which takes some effort to do. > > Cache-Mode: 1 (Development) > => all threads are frozen while the system performs an in-place update of > the > current cache. This is a fairly fast operation, but it should only take a > split-second for a single change. > > Best Regards - Misi, RRR AB, http://www.rrr.se (ARSList MVP 2011) > > Ask the Remedy Licensing Experts (Best R.O.I. Award at WWRUG10/11/12/13): > * RRR|License - Not enough Remedy licenses? Save money by optimizing. > * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs. > Find these products, and many free tools and utilities, at http://rrr.se. > > > Nice analogy Doug. I have historically separated out various subsystems > to > > separate queues (e.g., approval server, AIE, email engine, midtier, > etc.). > > I did this to provide a means to prevent one subsystem from impacting > > other subsystems. This has happened to me in the past where some > subsystem > > ate up more threads than the allowed max. Using separate queues allows > the > > server to remain stable by preventing one bad actor from impacting other > > things. > > > > Also worth mentioning is that each thread has a private copy of cache > data. > > The more threads you have the longer cache related operations take to > > complete and more memory is required to manage all the extra copies of > > cache. > > > > I have practiced setting the min threads to 1 for all queues (with the > > exception of the escalation queue) and set the max to some reasonable > > value, based on what uses it and historical needs. The server then > manages > > the thread count based on the actual demand. Remedy creates new threads > > when some operation in a queue is blocked (i.e., no free threads) up to > the > > max threads. > > > > Separate queues also have the advantage of the separate Server Statistics > > counters. These counters can be valuable when analyzing performance, > > utilization, and demand by various external components. > > > > Separating things adds to overall system stability by keeping things > > contained. > > > > With Remedy, once a thread is instantiated by arserverd it will never be > > destroyed. The only way to destroy a thread is to stop the server. > > > > > > On Wed, Jun 18, 2014 at 11:30 AM, Mueller, Doug <doug_muel...@bmc.com> > > wrote: > > > >> James, > >> > >> Let me try a different approach to describing things. > >> > >> First, some definitions.... > >> > >> A Thread is an independent path of execution capability. It is > equivalent > >> to a > >> thread in a Java application in your question. With the AR System > >> environment, > >> each thread includes a DB connection and the ability to perform > operations > >> to the > >> DB in response to API calls. The outside world (the API) cannot address > >> threads > >> directly but is directed to a thread by a controller. > >> > >> A Queue is a pool of one or more threads that provide the externally > >> identified > >> access points into the system. A given access identifies a queue to > >> interact with > >> and that queue is a controller to pass work onto the threads within its > >> control. > >> > >> > >> So, let's use a bank as an analogy. You are going to your local branch > to > >> see a > >> teller. There are three tellers working -- so there are 3 THREADS of > >> execution. > >> As most banks are set up, you get access to the threads (the tellers) by > >> getting in > >> the line for a teller -- so there is 1 QUEUE where you go -- and you are > >> in line > >> with other customers to be assigned to a teller -- thread -- when it is > >> your turn > >> for processing. > >> > >> Now, let's take it further. Instead of a teller to perform a > transaction, > >> you > >> actually want to access your safe deposit box. Well, there is a > DIFFERENT > >> QUEUE > >> you get in to get access and there may be 2 safe deposit box review > rooms. > >> So, you > >> are in a different queue with two threads for performing safe deposit > box > >> operations. > >> > >> One step further, you instead want to perform an Administrative > operation > >> like > >> setting up an account or closing an account. You don't go to the teller > >> queue or > >> to the safe deposit box queue, but to another queue for these types of > >> operations. > >> Say in your bank you had 1 person who helped with these operations so > you > >> have > >> a queue with 1 thread. > >> > >> Now, depending on what operation you want to perform, you access the > >> appropriate > >> queue and then get directed to a thread to perform the work. > >> > >> Note that you could be in the Administrative queue waiting and there > could > >> be no > >> one in the teller queue and there could be tellers just standing there > >> (available > >> threads) but there are no threads available in your queue so you are > still > >> waiting > >> for your threads to become available. > >> > >> Queues allow bucketing of operations and threads perform the work. > >> > >> Now, if you had the bottleneck, you may want to increase the threads on > the > >> Administrative queue above to handle higher load but you do not want to > >> block the > >> teller threads with long Administrative operations as you may be > getting a > >> flood of > >> customers for standard transactions coming in just a minute. You get to > >> control > >> the flow and the load and how much resource you dedicate to different > >> operations. > >> > >> > >> This is what threads and queues are all about. > >> > >> > >> Kind of a simple example and it obviously can get more complex with > >> multiple > >> branches and you pick a branch and then a queue within that branch and > >> many more > >> types of queues and then server groups and load balancers and several > >> other layers > >> show up. > >> > >> But, hopefully, this gives you a way to understand the idea of threads > and > >> queues > >> and what they are and how they interact. > >> > >> Doug Mueller > >> > >> -----Original Message----- > >> From: Action Request System discussion list(ARSList) [mailto: > >> arslist@ARSLIST.ORG] On Behalf Of Misi Mladoniczky > >> Sent: Tuesday, June 17, 2014 12:32 PM > >> To: arslist@ARSLIST.ORG > >> Subject: Re: Threads and queues > >> > >> Hi, > >> > >> The threads we are talking about are threads in a single process that > >> performs in parallel the way you are describing it. > >> > >> The threads themselves perform more or less identical tasks. I a minimal > >> system with a single thread (the Admin thread) the system will perform > all > >> the tasks within that single thread. > >> > >> The queues are queues only in a bogged down system where the AR > >> Server/Database threads are all in use at the same time where user calls > >> will be queued until a thread is free to service them. This is why they > are > >> called queues. > >> > >> The Admin/Fast/List/Escalation/Private queues are just a way to > designate > >> threads to certain types of operations. > >> > >> Each thread can service API-calls, which typically triggers a database > >> search or a bunch of filter operations followerd by a write to the > database. > >> > >> Each thread keeps an active database connection open to the database at > >> all times. > >> > >> Best Regards - Misi, RRR AB, http://www.rrr.se (ARSList MVP > 2011) > >> > >> Ask the Remedy Licensing Experts (Best R.O.I. Award at > WWRUG10/11/12/13): > >> * RRR|License - Not enough Remedy licenses? Save money by optimizing. > >> * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy > logs. > >> Find these products, and many free tools and utilities, at > http://rrr.se. > >> > >> > In Java a thread means a part of the program that executes independent > >> > of other parts. Is that same in remedy? Can you give me example of > >> > execution in remedy? > >> > > >> > A queue means lining up of jobs/threads like a pages in a printer. > >> > > >> > Feel free to correct me. > >> > > >> > > >> > >> Hi, > >> > >> There is a legacy here which might make it a little obscure. > >> > >> But nowadays the AR Server uses threads (similar to Java) to compute > >> things parallel, and make parallel connections/calls to the database. > >> > >> Each active queue has one ore more threads, and listen to different RPC > >> numbers, but except for Private queues you will only use RPC# 390601 > (if I > >> remember right) and the transactions will be routed to the right queue. > >> > >> In a normal system you have these queues (and each can have multiple > >> threads): > >> - Fast (handles most single record operations like Submit and Modify of > a > >> ticket) > >> - List (handles most multiple record operations like a Search or getting > >> the list of fields for a form) > >> - Admin (the core thread that also handles all admin operations from > >> DevStudio) > >> - Escalation (performs the escalation searches and any resulting filter > >> operations on updated records) > >> - Private (private threads can be configured if you for example have an > >> API integration that should be separated from the normal queues. Either > to > >> give priority access or to prevent it from bogging down the other queues > >> servicing normal users) > >> > >> Best Regards - Misi, RRR AB, http://www.rrr.se (ARSList MVP > 2011) > >> > >> > >> > _______________________________________________________________________________ > >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the > >> Answers Are, and have been for 20 years" > >> > >> > >> > _______________________________________________________________________________ > >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > >> "Where the Answers Are, and have been for 20 years" > >> > > > > > _______________________________________________________________________________ > > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > > "Where the Answers Are, and have been for 20 years" > > > > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > "Where the Answers Are, and have been for 20 years" > > _ARSlist: "Where the Answers Are" and have been for 20 years_ > _ARSlist: "Where the Answers Are" and have been for 20 years_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years"
