Hi,
We have taken all this into account.
1. The escalations are run on another server
2. Admin-changes will never be applied during production hours
Without Development-Cache-Mode set, the server frequently timed out when
doing admin changes. This seemed to be because of the buffer-copying when
the cache was duplicated.
Best Regards - Misi, RRR AB, http://www.rrr.se
Products from RRR Scandinavia:
* RRR|License - Not enough Remedy licenses? Save money by optimizing.
* RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs.
Find these products, and many free tools and utilities, at http://rrr.se.
> You really need to be careful with development cache mode on production
> servers for the following reason;
>
> With this option set the server has only one copy of the cache to work
> with. This may provide some advantages in terms of memory usage but
> consider the following..
>
> A user starts a long running query, or a long running escalation kicks
> off. In both of these cases the thread performing the work will place a
> flag against the cache to say that it is in use - effectively a read lock.
> Now you make an admin change which requires exclusive use of the one
> cache that you have. This exclusive lock cannot be obtained because the
> cache is in use so it is queued up and your admin thread is out of action
> until it gets the lock. Not only that but any subsequent user API call
> that tries to get a read lock on the cache will fail because there is an
> outstanding exclusive lock request pending. All of your users are locked
> out until the orginal activity completes and then the pending admin change
> is done.
>
> Mark
>
> -----Original Message-----
> From: Action Request System discussion list(ARSList)
> [mailto:[email protected]] On Behalf Of Misi Mladoniczky
> Sent: 10 December 2009 11:59
> To: [email protected]
> Subject: Re: Development Cache Mode revisited
>
> Hi,
>
> Just to clarify.
>
> This more or less proves that it is a BAD idea to have Cache-Mode set to
> Development on any server in the Server-Group but the Admin-server.
>
> The Admin-server seems to just patch in the modification directly in
> memory. The other servers allways need to recache everything when a change
> has occured.
>
> If Dev-Cache-Mode is set on a server that rereads the whole cache, the
> users need to wait until the operation is completed.
>
> Best Regards - Misi, RRR AB, http://www.rrr.se
>
> Products from RRR Scandinavia:
> * RRR|License - Not enough Remedy licenses? Save money by optimizing.
> * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs.
> * RRR|Translator - Manage and automate your language translations.
> Find these products, and many free tools and utilities, at http://rrr.se.
>
>> Hi,
>>
>> We have now gone into production, and I have a couple of things I want
>> to
>> share about this...
>>
>> The two servers in the server-group are (now) 7.1.0 patch 008.
>>
>> We have ITSM7 instaleld.
>>
>> We can call the servers USER and ESCL, to indicate their major use.
>>
>> The final way we did it was to have the following setup, it is shown
>> last
>> in the last table below:
>> USER, Cache-Mode=Dev, Admin
>> ESCL, Cache-Mode=0
>>
>> These are results from various settings when an admin-change was
>> applied.
>>
>> Server CacheMode AdminServer RecacheTime UserLockout RecacheMem
>> StableMem
>> USER Dev - 420 sec 420 sec 620Mb 620Mb
>> ESCL Dev Admin 1 sec 1 sec 620Mb 620Mb
>> (not good, as users got locked out)
>>
>> Server CacheMode AdminServer RecacheTime UserLockout RecacheMem
>> StableMem
>> USER Dev Admin 1 sec 1 sec 620Mb 620Mb
>> ESCL Normal - 550 sec 0 sec 1200Mb 620Mb
>> (requires more memory on the ESCL-server)
>>
>> Note that if you import multiple things, as opposed to applying a small
>> change, the 1-second lockout may be considerably higher.
>>
>> Best Regards - Misi, RRR AB, http://rrr.se
>>
>>> Yep. If anyone is interested, p171 7.5 Config.
>>>
>>> When object definition changes are made to the administrative server in
>>> a
>>> server
>>> group, other members of the group can be notified either by a signal or
>>> by
>>> a
>>> database posting. With signaling, other servers are notified of changes
>>> immediately, and there is no delay in resynchronization or updating
>>> definitions.
>>> The database method reduces server activity when object definition
>>> changes
>>> are
>>> communicated between servers and is an effective way to reduce the
>>> number
>>> of
>>> cache reloads when a series of changes are made, but there is a delay
>>> before other
>>> members of the server group pick up the changes.
>>>
>>> By default, AR System uses a combination of these methods for different
>>> types of
>>> updates in a server group. Server definition changes such as changes to
>>> forms,
>>> active links, filters, and escalations, as well as user group changes,
>>> are
>>> handled by
>>> database posting. All other changes, such as Alert registration, DSO
>>> activity, and
>>> so on, are handled by signaling. However, you can configure the server
>>> to
>>> use
>>> signaling for all changes including server object changes by following
>>> the
>>> procedure in this section.
>>>
>>> The configuration file setting Server-Group-Signal-Option tells the
>>> server
>>> whether to use arsignald for all signals, rather than using a
>>> combination
>>> of
>>> signaling and database posting. If set to true (T), server object
>>> changes
>>> are
>>> communicated by arsignald instead of database posting. Use this option
>>> if
>>> you
>>> don?t want any delay in communicating server object changes to other
>>> servers in
>>> the server group.
>>>
>>>
>>> Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores
>>> N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262)
>>> 703-7763 | e-mail: [email protected]
>>>
>>>
>>>
>>> From:
>>> Lyle Taylor <[email protected]>
>>> To:
>>> [email protected]
>>> Date:
>>> 11/20/2009 03:13 PM
>>> Subject:
>>> Re: Development Cache Mode revisited
>>> Sent by:
>>> "Action Request System discussion list(ARSList)" <[email protected]>
>>>
>>>
>>>
>>> **
>>> Concerning the other servers in the server group, the behavior is
>>> discussed in one of the AR server guides, but I?m not sure which one
>>> (it?s
>>> been a while since I read it). It probably wouldn?t be too hard to
>>> find
>>> if needed.
>>>
>>> In a nutshell, though, each of the servers in the server group
>>> periodically checks to see if the schema has been updated (or if new
>>> groups have been added, etc.), and builds a new copy of the cache from
>>> the
>>> database when they see that it has changed. If I recall correctly, the
>>> polling frequency may be able to be configured in ar.conf, and it
>>> doesn?t
>>> immediately update once it notices a change. Rather, it will wait
>>> until
>>> something like two successive polling cycles have passed without
>>> additional changes occurring before rebuilding its local cache. That
>>> way,
>>> it?s less likely to do an update in the middle of a set of changes.
>>>
>>> If Dev Cache Mode is turned off on the other servers in the group, and
>>> users are logged into those servers, users will continue to use the old
>>> copy of the cache until they log out and back in again. Note that this
>>> means that, depending on the frequency of changes (including adding new
>>> Remedy group or ITSM Support Groups), this can be a significant source
>>> of
>>> memory usage on the server. We have run out of memory and had the
>>> server
>>> crash in the past due to creating several new Support Groups throughout
>>> the day, and each one causing yet another copy of the cache to be held
>>> in
>>> memory for the people that logged in since the last time it was
>>> updated.
>>> This is even more of an issue if people don?t log out correctly (e.g.,
>>> if
>>> they?re using the mid-tier and simply close their browser window
>>> without
>>> logging out), as the system will NOT log them out automatically, even
>>> though it will eventually release their write licenses. This causes
>>> old
>>> copies of the cache to be retained for longer periods than necessary.
>>> The
>>> old copies of the cache will be held in memory until everyone that was
>>> using that particular copy log out of Remedy; it will then release it,
>>> and
>>> you memory usage will drop.
>>>
>>> Like I said, that?s from memory and recent experience. I do recall
>>> reading it in one of the guides in the standard documentation, so you
>>> should be able to find the specifics if you want them.
>>>
>>> Lyle
>>>
>>> From: Action Request System discussion list(ARSList) [
>>> mailto:[email protected]] On Behalf Of Tony Worthington
>>> Sent: Friday, November 20, 2009 12:28 PM
>>> To: [email protected]
>>> Subject: Re: Development Cache Mode revisited
>>>
>>> ** Be aware that (mysterious) things in ITSM7 trigger recaches -- not
>>> just
>>> definition imports or changes. I haven't quite figured out what -- but
>>> we
>>> have one or two a day -- no admin involved. They show as Remedy
>>> Application Service user. Some can be traced to group changes, which we
>>> now perform after-hours. Others are a mystery.
>>>
>>> I don't think there are db read differences when updating the cache -
>>> just
>>> the server handling of the in-memory copy. If you watch the size of
>>> the
>>> server when doing changes on vs. off you should see the differences.
>>>
>>> What happens with the cache in the other servers in the server group?
>>>
>>> I think this is dealt with in the server_cache, servgrp_board and
>>> servgrp_cache and related servgrp_* tables. No server group here so I
>>> can't say for sure. My thinking is the server watches for a signal to
>>> trigger a recache. That signal is is an API call or a change in the
>>> meta-data. Can anyone confirm how it really works?
>>>
>>>
>>> Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores
>>> N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262)
>>> 703-7763 | e-mail: [email protected]
>>>
>>>
>>> From:
>>> Misi Mladoniczky <[email protected]>
>>> To:
>>> [email protected]
>>> Date:
>>> 11/20/2009 01:13 PM
>>> Subject:
>>> Re: Development Cache Mode revisited
>>> Sent by:
>>> "Action Request System discussion list(ARSList)" <[email protected]>
>>>
>>>
>>>
>>>
>>>
>>> Lyle and Tony,
>>>
>>> Thank you for the reply.
>>>
>>> In other words, there should be no impact whatsoever if you perform NO
>>> admin-changes.
>>>
>>> If you apply your changes when both users, email engine and escalations
>>> are taking it easy, it would probably be faster to use Developer Cache
>>> Mode even in production (Cache-Mode: 1).
>>>
>>> Two more questions:
>>>
>>> 1. I turned on SQL-logs to see what happened when disabling an
>>> escalation
>>> with both settings. Nothing much happened, and the server did not seem
>>> to
>>> reread much from the database repository. I had expected a bunch of
>>> SQL-selects from the repository. Maybe it just updated the memory cache
>>> without rereading the DB?
>>>
>>> 2. What happens with the cache in the other servers in the server
>>> group?
>>> Is the admin-change propagated to these in some magic way?
>>>
>>> Best Regards - Misi, RRR AB, http://rrr.se
>>>
>>>> I just worked through a longstanding issue that we have had in our
>>>> environment related to Dev Cache Mode. Due to extremely slow
>>> performance
>>>> when making admin changes on a 7.0 server, I had gotten into the habit
>>> of
>>>> keeping our admin server set with Dev Cache Mode turned on, which
>>>> significantly sped up migrations to production and such. However, in
>>> this
>>>> environment (7.1 p5 server), I was seeing exactly the opposite - admin
>>>> changes were very frequently timing out and migrations were painfully
>>> slow
>>>> (for example, a patch that took a few minutes to apply in dev and
>>>> stage
>>>> took an hour and twenty minutes to apply in production). Thinking
>>>> that
>>> it
>>>> should go faster since Dev Cache Mode was turned on, I looked
>>>> everywhere
>>> I
>>>> could think of to try and isolate why it was so slow. I got BMC
>>> involved
>>>> in my search twice. This last time, the support rep forwarded this
>>>> same
>>>> KB article to me and recommended that I leave Dev Cache Mode turned
>>>> OFF
>>> in
>>>> order to improve performance with admin actions. I tried it, and it
>>>> worked. My migrations now go very quickly, and I haven't seen any
>>>> timeouts since. The only slowness is in the initial period where the
>>>> AR
>>>> server creates a copy of the cache in memory, but that's generally
>>> fairly
>>>> quick, since it's just doing an in memory copy rather than trying to
>>> pull
>>>> it all from the database again (like I suspect it was doing on the 7.0
>>>> server).
>>>>
>>>> In short, in our environment, we see significantly better performance
>>>> making admin changes in production with Dev Cache Mode turned off.
>>>> The
>>>> support rep told me that since AR Server 7.1, their recommendation is
>>>> to
>>>> keep it turned off.
>>>>
>>>> That said, if I understand it correctly, your mileage may vary
>>>> depending
>>>> on what kind of activity you have going on on the server you're trying
>>> to
>>>> make changes on. As I understand it, when Dev Cache Mode is turned
>>>> on,
>>>> since it doesn't make a copy of the cache for existing users, it has
>>>> to
>>>> wait until user operations have ceased before it can make the change,
>>> then
>>>> it makes the change in between user operations. In our case, since
>>>> this
>>>> server also runs all our back-end processes (e-mail, escalations,
>>>> etc.),
>>>> and we apparently have some things going on that take a long time to
>>>> complete, the admin thread ended up having to wait a long time (more
>>> than
>>>> 10 minutes in some cases) before it could get a lock on the cache and
>>> make
>>>> the changes. This was what was causing the timeouts and slow
>>> performance
>>>> that we saw. Now, since it simply makes a relatively quick copy of
>>>> the
>>>> cache for existing user operations (just system operations in our
>>>> case,
>>>> but still, they're ongoing), the overall time to make the changes is
>>>> quick, because once the cache is copied, it can proceed with the
>>>> changes
>>>> without having to wait for a break between user operations.
>>>>
>>>> So, if you don't have much going on on the server you're trying to
>>>> make
>>>> admin changes on, it could be quicker to leave Dev Cache Mode turned
>>>> on,
>>>> because then you don't have to wait for the cache to get copied (I
>>>> think
>>>> it takes less than 30 seconds in our case to copy about 400MB of
>>>> cache).
>>>> However, if you have lots of stuff going on (e-mail, user access,
>>>> escalations, etc.), you may see better performance with it turned off.
>>>>
>>>> I hope this helps a bit.
>>>>
>>>> Lyle
>>>>
>>>>
>>>> From: Action Request System discussion list(ARSList)
>>> _Platinum Sponsor: [email protected] ARSlist: "Where the Answers
>>> Are"_
>>>
>>>
>>> NOTICE: This email message is for the sole use of the intended
>>> recipient(s) and may contain confidential and privileged information.
>>> Any
>>> unauthorized review, use, disclosure or distribution is prohibited. If
>>> you
>>> are not the intended recipient, please contact the sender by reply
>>> email
>>> and destroy all copies of the original message.
>>>
>>> _Platinum Sponsor: [email protected] ARSlist: "Where the Answers
>>> Are"_
>>>
>>> **********************************************************************
>>> CONFIDENTIALITY NOTICE:
>>> This is a transmission from Kohl's Department Stores, Inc.
>>> and may contain information which is confidential and proprietary.
>>> If you are not the addressee, any disclosure, copying or distribution
>>> or
>>> use of the contents of this message is expressly prohibited.
>>> If you have received this transmission in error, please destroy it and
>>> notify us immediately at 262-703-7000.
>>>
>>> CAUTION:
>>> Internet and e-mail communications are Kohl's property and Kohl's
>>> reserves
>>> the right to retrieve and read any message created, sent and received.
>>> Kohl's reserves the right to monitor messages by authorized Kohl's
>>> Associates at any time
>>> without any further consent.
>>>
>>> _______________________________________________________________________________
>>> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
>>> Platinum Sponsor:[email protected] ARSlist: "Where the Answers
>>> Are"
>>>
>>> --
>>> This message was scanned by ESVA and is believed to be clean.
>>>
>>>
>>
>> _______________________________________________________________________________
>> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
>> Platinum Sponsor:[email protected] ARSlist: "Where the Answers
>> Are"
>>
>> --
>> This message was scanned by ESVA and is believed to be clean.
>>
>>
>
> _______________________________________________________________________________
> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
> Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
>
> _______________________________________________________________________________
> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
> Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
>
> --
> This message was scanned by ESVA and is believed to be clean.
>
>
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
