You really need to be careful with development cache mode on production servers for the following reason;
With this option set the server has only one copy of the cache to work with. This may provide some advantages in terms of memory usage but consider the following.. A user starts a long running query, or a long running escalation kicks off. In both of these cases the thread performing the work will place a flag against the cache to say that it is in use - effectively a read lock. Now you make an admin change which requires exclusive use of the one cache that you have. This exclusive lock cannot be obtained because the cache is in use so it is queued up and your admin thread is out of action until it gets the lock. Not only that but any subsequent user API call that tries to get a read lock on the cache will fail because there is an outstanding exclusive lock request pending. All of your users are locked out until the orginal activity completes and then the pending admin change is done. Mark -----Original Message----- From: Action Request System discussion list(ARSList) [mailto:[email protected]] On Behalf Of Misi Mladoniczky Sent: 10 December 2009 11:59 To: [email protected] Subject: Re: Development Cache Mode revisited Hi, Just to clarify. This more or less proves that it is a BAD idea to have Cache-Mode set to Development on any server in the Server-Group but the Admin-server. The Admin-server seems to just patch in the modification directly in memory. The other servers allways need to recache everything when a change has occured. If Dev-Cache-Mode is set on a server that rereads the whole cache, the users need to wait until the operation is completed. Best Regards - Misi, RRR AB, http://www.rrr.se Products from RRR Scandinavia: * RRR|License - Not enough Remedy licenses? Save money by optimizing. * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs. * RRR|Translator - Manage and automate your language translations. Find these products, and many free tools and utilities, at http://rrr.se. > Hi, > > We have now gone into production, and I have a couple of things I want to > share about this... > > The two servers in the server-group are (now) 7.1.0 patch 008. > > We have ITSM7 instaleld. > > We can call the servers USER and ESCL, to indicate their major use. > > The final way we did it was to have the following setup, it is shown last > in the last table below: > USER, Cache-Mode=Dev, Admin > ESCL, Cache-Mode=0 > > These are results from various settings when an admin-change was applied. > > Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem > USER Dev - 420 sec 420 sec 620Mb 620Mb > ESCL Dev Admin 1 sec 1 sec 620Mb 620Mb > (not good, as users got locked out) > > Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem > USER Dev Admin 1 sec 1 sec 620Mb 620Mb > ESCL Normal - 550 sec 0 sec 1200Mb 620Mb > (requires more memory on the ESCL-server) > > Note that if you import multiple things, as opposed to applying a small > change, the 1-second lockout may be considerably higher. > > Best Regards - Misi, RRR AB, http://rrr.se > >> Yep. If anyone is interested, p171 7.5 Config. >> >> When object definition changes are made to the administrative server in >> a >> server >> group, other members of the group can be notified either by a signal or >> by >> a >> database posting. With signaling, other servers are notified of changes >> immediately, and there is no delay in resynchronization or updating >> definitions. >> The database method reduces server activity when object definition >> changes >> are >> communicated between servers and is an effective way to reduce the >> number >> of >> cache reloads when a series of changes are made, but there is a delay >> before other >> members of the server group pick up the changes. >> >> By default, AR System uses a combination of these methods for different >> types of >> updates in a server group. Server definition changes such as changes to >> forms, >> active links, filters, and escalations, as well as user group changes, >> are >> handled by >> database posting. All other changes, such as Alert registration, DSO >> activity, and >> so on, are handled by signaling. However, you can configure the server >> to >> use >> signaling for all changes including server object changes by following >> the >> procedure in this section. >> >> The configuration file setting Server-Group-Signal-Option tells the >> server >> whether to use arsignald for all signals, rather than using a >> combination >> of >> signaling and database posting. If set to true (T), server object >> changes >> are >> communicated by arsignald instead of database posting. Use this option >> if >> you >> don?t want any delay in communicating server object changes to other >> servers in >> the server group. >> >> >> Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores >> N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) >> 703-7763 | e-mail: [email protected] >> >> >> >> From: >> Lyle Taylor <[email protected]> >> To: >> [email protected] >> Date: >> 11/20/2009 03:13 PM >> Subject: >> Re: Development Cache Mode revisited >> Sent by: >> "Action Request System discussion list(ARSList)" <[email protected]> >> >> >> >> ** >> Concerning the other servers in the server group, the behavior is >> discussed in one of the AR server guides, but I?m not sure which one >> (it?s >> been a while since I read it). It probably wouldn?t be too hard to find >> if needed. >> >> In a nutshell, though, each of the servers in the server group >> periodically checks to see if the schema has been updated (or if new >> groups have been added, etc.), and builds a new copy of the cache from >> the >> database when they see that it has changed. If I recall correctly, the >> polling frequency may be able to be configured in ar.conf, and it >> doesn?t >> immediately update once it notices a change. Rather, it will wait until >> something like two successive polling cycles have passed without >> additional changes occurring before rebuilding its local cache. That >> way, >> it?s less likely to do an update in the middle of a set of changes. >> >> If Dev Cache Mode is turned off on the other servers in the group, and >> users are logged into those servers, users will continue to use the old >> copy of the cache until they log out and back in again. Note that this >> means that, depending on the frequency of changes (including adding new >> Remedy group or ITSM Support Groups), this can be a significant source >> of >> memory usage on the server. We have run out of memory and had the >> server >> crash in the past due to creating several new Support Groups throughout >> the day, and each one causing yet another copy of the cache to be held >> in >> memory for the people that logged in since the last time it was updated. >> This is even more of an issue if people don?t log out correctly (e.g., >> if >> they?re using the mid-tier and simply close their browser window without >> logging out), as the system will NOT log them out automatically, even >> though it will eventually release their write licenses. This causes old >> copies of the cache to be retained for longer periods than necessary. >> The >> old copies of the cache will be held in memory until everyone that was >> using that particular copy log out of Remedy; it will then release it, >> and >> you memory usage will drop. >> >> Like I said, that?s from memory and recent experience. I do recall >> reading it in one of the guides in the standard documentation, so you >> should be able to find the specifics if you want them. >> >> Lyle >> >> From: Action Request System discussion list(ARSList) [ >> mailto:[email protected]] On Behalf Of Tony Worthington >> Sent: Friday, November 20, 2009 12:28 PM >> To: [email protected] >> Subject: Re: Development Cache Mode revisited >> >> ** Be aware that (mysterious) things in ITSM7 trigger recaches -- not >> just >> definition imports or changes. I haven't quite figured out what -- but >> we >> have one or two a day -- no admin involved. They show as Remedy >> Application Service user. Some can be traced to group changes, which we >> now perform after-hours. Others are a mystery. >> >> I don't think there are db read differences when updating the cache - >> just >> the server handling of the in-memory copy. If you watch the size of the >> server when doing changes on vs. off you should see the differences. >> >> What happens with the cache in the other servers in the server group? >> >> I think this is dealt with in the server_cache, servgrp_board and >> servgrp_cache and related servgrp_* tables. No server group here so I >> can't say for sure. My thinking is the server watches for a signal to >> trigger a recache. That signal is is an API call or a change in the >> meta-data. Can anyone confirm how it really works? >> >> >> Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores >> N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) >> 703-7763 | e-mail: [email protected] >> >> >> From: >> Misi Mladoniczky <[email protected]> >> To: >> [email protected] >> Date: >> 11/20/2009 01:13 PM >> Subject: >> Re: Development Cache Mode revisited >> Sent by: >> "Action Request System discussion list(ARSList)" <[email protected]> >> >> >> >> >> >> Lyle and Tony, >> >> Thank you for the reply. >> >> In other words, there should be no impact whatsoever if you perform NO >> admin-changes. >> >> If you apply your changes when both users, email engine and escalations >> are taking it easy, it would probably be faster to use Developer Cache >> Mode even in production (Cache-Mode: 1). >> >> Two more questions: >> >> 1. I turned on SQL-logs to see what happened when disabling an >> escalation >> with both settings. Nothing much happened, and the server did not seem >> to >> reread much from the database repository. I had expected a bunch of >> SQL-selects from the repository. Maybe it just updated the memory cache >> without rereading the DB? >> >> 2. What happens with the cache in the other servers in the server group? >> Is the admin-change propagated to these in some magic way? >> >> Best Regards - Misi, RRR AB, http://rrr.se >> >>> I just worked through a longstanding issue that we have had in our >>> environment related to Dev Cache Mode. Due to extremely slow >> performance >>> when making admin changes on a 7.0 server, I had gotten into the habit >> of >>> keeping our admin server set with Dev Cache Mode turned on, which >>> significantly sped up migrations to production and such. However, in >> this >>> environment (7.1 p5 server), I was seeing exactly the opposite - admin >>> changes were very frequently timing out and migrations were painfully >> slow >>> (for example, a patch that took a few minutes to apply in dev and stage >>> took an hour and twenty minutes to apply in production). Thinking that >> it >>> should go faster since Dev Cache Mode was turned on, I looked >>> everywhere >> I >>> could think of to try and isolate why it was so slow. I got BMC >> involved >>> in my search twice. This last time, the support rep forwarded this >>> same >>> KB article to me and recommended that I leave Dev Cache Mode turned OFF >> in >>> order to improve performance with admin actions. I tried it, and it >>> worked. My migrations now go very quickly, and I haven't seen any >>> timeouts since. The only slowness is in the initial period where the >>> AR >>> server creates a copy of the cache in memory, but that's generally >> fairly >>> quick, since it's just doing an in memory copy rather than trying to >> pull >>> it all from the database again (like I suspect it was doing on the 7.0 >>> server). >>> >>> In short, in our environment, we see significantly better performance >>> making admin changes in production with Dev Cache Mode turned off. The >>> support rep told me that since AR Server 7.1, their recommendation is >>> to >>> keep it turned off. >>> >>> That said, if I understand it correctly, your mileage may vary >>> depending >>> on what kind of activity you have going on on the server you're trying >> to >>> make changes on. As I understand it, when Dev Cache Mode is turned on, >>> since it doesn't make a copy of the cache for existing users, it has to >>> wait until user operations have ceased before it can make the change, >> then >>> it makes the change in between user operations. In our case, since >>> this >>> server also runs all our back-end processes (e-mail, escalations, >>> etc.), >>> and we apparently have some things going on that take a long time to >>> complete, the admin thread ended up having to wait a long time (more >> than >>> 10 minutes in some cases) before it could get a lock on the cache and >> make >>> the changes. This was what was causing the timeouts and slow >> performance >>> that we saw. Now, since it simply makes a relatively quick copy of the >>> cache for existing user operations (just system operations in our case, >>> but still, they're ongoing), the overall time to make the changes is >>> quick, because once the cache is copied, it can proceed with the >>> changes >>> without having to wait for a break between user operations. >>> >>> So, if you don't have much going on on the server you're trying to >>> make >>> admin changes on, it could be quicker to leave Dev Cache Mode turned >>> on, >>> because then you don't have to wait for the cache to get copied (I >>> think >>> it takes less than 30 seconds in our case to copy about 400MB of >>> cache). >>> However, if you have lots of stuff going on (e-mail, user access, >>> escalations, etc.), you may see better performance with it turned off. >>> >>> I hope this helps a bit. >>> >>> Lyle >>> >>> >>> From: Action Request System discussion list(ARSList) >> _Platinum Sponsor: [email protected] ARSlist: "Where the Answers >> Are"_ >> >> >> NOTICE: This email message is for the sole use of the intended >> recipient(s) and may contain confidential and privileged information. >> Any >> unauthorized review, use, disclosure or distribution is prohibited. If >> you >> are not the intended recipient, please contact the sender by reply email >> and destroy all copies of the original message. >> >> _Platinum Sponsor: [email protected] ARSlist: "Where the Answers >> Are"_ >> >> ********************************************************************** >> CONFIDENTIALITY NOTICE: >> This is a transmission from Kohl's Department Stores, Inc. >> and may contain information which is confidential and proprietary. >> If you are not the addressee, any disclosure, copying or distribution or >> use of the contents of this message is expressly prohibited. >> If you have received this transmission in error, please destroy it and >> notify us immediately at 262-703-7000. >> >> CAUTION: >> Internet and e-mail communications are Kohl's property and Kohl's >> reserves >> the right to retrieve and read any message created, sent and received. >> Kohl's reserves the right to monitor messages by authorized Kohl's >> Associates at any time >> without any further consent. >> >> _______________________________________________________________________________ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> Platinum Sponsor:[email protected] ARSlist: "Where the Answers >> Are" >> >> -- >> This message was scanned by ESVA and is believed to be clean. >> >> > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are" > > -- > This message was scanned by ESVA and is believed to be clean. > > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are" _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
