Hi,
Just to clarify.
This more or less proves that it is a BAD idea to have Cache-Mode set to
Development on any server in the Server-Group but the Admin-server.
The Admin-server seems to just patch in the modification directly in
memory. The other servers allways need to recache everything when a change
has occured.
If Dev-Cache-Mode is set on a server that rereads the whole cache, the
users need to wait until the operation is completed.
Best Regards - Misi, RRR AB, http://www.rrr.se
Products from RRR Scandinavia:
* RRR|License - Not enough Remedy licenses? Save money by optimizing.
* RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs.
* RRR|Translator - Manage and automate your language translations.
Find these products, and many free tools and utilities, at http://rrr.se.
> Hi,
>
> We have now gone into production, and I have a couple of things I want to
> share about this...
>
> The two servers in the server-group are (now) 7.1.0 patch 008.
>
> We have ITSM7 instaleld.
>
> We can call the servers USER and ESCL, to indicate their major use.
>
> The final way we did it was to have the following setup, it is shown last
> in the last table below:
> USER, Cache-Mode=Dev, Admin
> ESCL, Cache-Mode=0
>
> These are results from various settings when an admin-change was applied.
>
> Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem
> USER Dev - 420 sec 420 sec 620Mb 620Mb
> ESCL Dev Admin 1 sec 1 sec 620Mb 620Mb
> (not good, as users got locked out)
>
> Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem
> USER Dev Admin 1 sec 1 sec 620Mb 620Mb
> ESCL Normal - 550 sec 0 sec 1200Mb 620Mb
> (requires more memory on the ESCL-server)
>
> Note that if you import multiple things, as opposed to applying a small
> change, the 1-second lockout may be considerably higher.
>
> Best Regards - Misi, RRR AB, http://rrr.se
>
>> Yep. If anyone is interested, p171 7.5 Config.
>>
>> When object definition changes are made to the administrative server in
>> a
>> server
>> group, other members of the group can be notified either by a signal or
>> by
>> a
>> database posting. With signaling, other servers are notified of changes
>> immediately, and there is no delay in resynchronization or updating
>> definitions.
>> The database method reduces server activity when object definition
>> changes
>> are
>> communicated between servers and is an effective way to reduce the
>> number
>> of
>> cache reloads when a series of changes are made, but there is a delay
>> before other
>> members of the server group pick up the changes.
>>
>> By default, AR System uses a combination of these methods for different
>> types of
>> updates in a server group. Server definition changes such as changes to
>> forms,
>> active links, filters, and escalations, as well as user group changes,
>> are
>> handled by
>> database posting. All other changes, such as Alert registration, DSO
>> activity, and
>> so on, are handled by signaling. However, you can configure the server
>> to
>> use
>> signaling for all changes including server object changes by following
>> the
>> procedure in this section.
>>
>> The configuration file setting Server-Group-Signal-Option tells the
>> server
>> whether to use arsignald for all signals, rather than using a
>> combination
>> of
>> signaling and database posting. If set to true (T), server object
>> changes
>> are
>> communicated by arsignald instead of database posting. Use this option
>> if
>> you
>> don?t want any delay in communicating server object changes to other
>> servers in
>> the server group.
>>
>>
>> Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores
>> N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262)
>> 703-7763 | e-mail: [email protected]
>>
>>
>>
>> From:
>> Lyle Taylor <[email protected]>
>> To:
>> [email protected]
>> Date:
>> 11/20/2009 03:13 PM
>> Subject:
>> Re: Development Cache Mode revisited
>> Sent by:
>> "Action Request System discussion list(ARSList)" <[email protected]>
>>
>>
>>
>> **
>> Concerning the other servers in the server group, the behavior is
>> discussed in one of the AR server guides, but I?m not sure which one
>> (it?s
>> been a while since I read it). It probably wouldn?t be too hard to find
>> if needed.
>>
>> In a nutshell, though, each of the servers in the server group
>> periodically checks to see if the schema has been updated (or if new
>> groups have been added, etc.), and builds a new copy of the cache from
>> the
>> database when they see that it has changed. If I recall correctly, the
>> polling frequency may be able to be configured in ar.conf, and it
>> doesn?t
>> immediately update once it notices a change. Rather, it will wait until
>> something like two successive polling cycles have passed without
>> additional changes occurring before rebuilding its local cache. That
>> way,
>> it?s less likely to do an update in the middle of a set of changes.
>>
>> If Dev Cache Mode is turned off on the other servers in the group, and
>> users are logged into those servers, users will continue to use the old
>> copy of the cache until they log out and back in again. Note that this
>> means that, depending on the frequency of changes (including adding new
>> Remedy group or ITSM Support Groups), this can be a significant source
>> of
>> memory usage on the server. We have run out of memory and had the
>> server
>> crash in the past due to creating several new Support Groups throughout
>> the day, and each one causing yet another copy of the cache to be held
>> in
>> memory for the people that logged in since the last time it was updated.
>> This is even more of an issue if people don?t log out correctly (e.g.,
>> if
>> they?re using the mid-tier and simply close their browser window without
>> logging out), as the system will NOT log them out automatically, even
>> though it will eventually release their write licenses. This causes old
>> copies of the cache to be retained for longer periods than necessary.
>> The
>> old copies of the cache will be held in memory until everyone that was
>> using that particular copy log out of Remedy; it will then release it,
>> and
>> you memory usage will drop.
>>
>> Like I said, that?s from memory and recent experience. I do recall
>> reading it in one of the guides in the standard documentation, so you
>> should be able to find the specifics if you want them.
>>
>> Lyle
>>
>> From: Action Request System discussion list(ARSList) [
>> mailto:[email protected]] On Behalf Of Tony Worthington
>> Sent: Friday, November 20, 2009 12:28 PM
>> To: [email protected]
>> Subject: Re: Development Cache Mode revisited
>>
>> ** Be aware that (mysterious) things in ITSM7 trigger recaches -- not
>> just
>> definition imports or changes. I haven't quite figured out what -- but
>> we
>> have one or two a day -- no admin involved. They show as Remedy
>> Application Service user. Some can be traced to group changes, which we
>> now perform after-hours. Others are a mystery.
>>
>> I don't think there are db read differences when updating the cache -
>> just
>> the server handling of the in-memory copy. If you watch the size of the
>> server when doing changes on vs. off you should see the differences.
>>
>> What happens with the cache in the other servers in the server group?
>>
>> I think this is dealt with in the server_cache, servgrp_board and
>> servgrp_cache and related servgrp_* tables. No server group here so I
>> can't say for sure. My thinking is the server watches for a signal to
>> trigger a recache. That signal is is an API call or a change in the
>> meta-data. Can anyone confirm how it really works?
>>
>>
>> Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores
>> N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262)
>> 703-7763 | e-mail: [email protected]
>>
>>
>> From:
>> Misi Mladoniczky <[email protected]>
>> To:
>> [email protected]
>> Date:
>> 11/20/2009 01:13 PM
>> Subject:
>> Re: Development Cache Mode revisited
>> Sent by:
>> "Action Request System discussion list(ARSList)" <[email protected]>
>>
>>
>>
>>
>>
>> Lyle and Tony,
>>
>> Thank you for the reply.
>>
>> In other words, there should be no impact whatsoever if you perform NO
>> admin-changes.
>>
>> If you apply your changes when both users, email engine and escalations
>> are taking it easy, it would probably be faster to use Developer Cache
>> Mode even in production (Cache-Mode: 1).
>>
>> Two more questions:
>>
>> 1. I turned on SQL-logs to see what happened when disabling an
>> escalation
>> with both settings. Nothing much happened, and the server did not seem
>> to
>> reread much from the database repository. I had expected a bunch of
>> SQL-selects from the repository. Maybe it just updated the memory cache
>> without rereading the DB?
>>
>> 2. What happens with the cache in the other servers in the server group?
>> Is the admin-change propagated to these in some magic way?
>>
>> Best Regards - Misi, RRR AB, http://rrr.se
>>
>>> I just worked through a longstanding issue that we have had in our
>>> environment related to Dev Cache Mode. Due to extremely slow
>> performance
>>> when making admin changes on a 7.0 server, I had gotten into the habit
>> of
>>> keeping our admin server set with Dev Cache Mode turned on, which
>>> significantly sped up migrations to production and such. However, in
>> this
>>> environment (7.1 p5 server), I was seeing exactly the opposite - admin
>>> changes were very frequently timing out and migrations were painfully
>> slow
>>> (for example, a patch that took a few minutes to apply in dev and stage
>>> took an hour and twenty minutes to apply in production). Thinking that
>> it
>>> should go faster since Dev Cache Mode was turned on, I looked
>>> everywhere
>> I
>>> could think of to try and isolate why it was so slow. I got BMC
>> involved
>>> in my search twice. This last time, the support rep forwarded this
>>> same
>>> KB article to me and recommended that I leave Dev Cache Mode turned OFF
>> in
>>> order to improve performance with admin actions. I tried it, and it
>>> worked. My migrations now go very quickly, and I haven't seen any
>>> timeouts since. The only slowness is in the initial period where the
>>> AR
>>> server creates a copy of the cache in memory, but that's generally
>> fairly
>>> quick, since it's just doing an in memory copy rather than trying to
>> pull
>>> it all from the database again (like I suspect it was doing on the 7.0
>>> server).
>>>
>>> In short, in our environment, we see significantly better performance
>>> making admin changes in production with Dev Cache Mode turned off. The
>>> support rep told me that since AR Server 7.1, their recommendation is
>>> to
>>> keep it turned off.
>>>
>>> That said, if I understand it correctly, your mileage may vary
>>> depending
>>> on what kind of activity you have going on on the server you're trying
>> to
>>> make changes on. As I understand it, when Dev Cache Mode is turned on,
>>> since it doesn't make a copy of the cache for existing users, it has to
>>> wait until user operations have ceased before it can make the change,
>> then
>>> it makes the change in between user operations. In our case, since
>>> this
>>> server also runs all our back-end processes (e-mail, escalations,
>>> etc.),
>>> and we apparently have some things going on that take a long time to
>>> complete, the admin thread ended up having to wait a long time (more
>> than
>>> 10 minutes in some cases) before it could get a lock on the cache and
>> make
>>> the changes. This was what was causing the timeouts and slow
>> performance
>>> that we saw. Now, since it simply makes a relatively quick copy of the
>>> cache for existing user operations (just system operations in our case,
>>> but still, they're ongoing), the overall time to make the changes is
>>> quick, because once the cache is copied, it can proceed with the
>>> changes
>>> without having to wait for a break between user operations.
>>>
>>> So, if you don't have much going on on the server you're trying to
>>> make
>>> admin changes on, it could be quicker to leave Dev Cache Mode turned
>>> on,
>>> because then you don't have to wait for the cache to get copied (I
>>> think
>>> it takes less than 30 seconds in our case to copy about 400MB of
>>> cache).
>>> However, if you have lots of stuff going on (e-mail, user access,
>>> escalations, etc.), you may see better performance with it turned off.
>>>
>>> I hope this helps a bit.
>>>
>>> Lyle
>>>
>>>
>>> From: Action Request System discussion list(ARSList)
>> _Platinum Sponsor: [email protected] ARSlist: "Where the Answers
>> Are"_
>>
>>
>> NOTICE: This email message is for the sole use of the intended
>> recipient(s) and may contain confidential and privileged information.
>> Any
>> unauthorized review, use, disclosure or distribution is prohibited. If
>> you
>> are not the intended recipient, please contact the sender by reply email
>> and destroy all copies of the original message.
>>
>> _Platinum Sponsor: [email protected] ARSlist: "Where the Answers
>> Are"_
>>
>> **********************************************************************
>> CONFIDENTIALITY NOTICE:
>> This is a transmission from Kohl's Department Stores, Inc.
>> and may contain information which is confidential and proprietary.
>> If you are not the addressee, any disclosure, copying or distribution or
>> use of the contents of this message is expressly prohibited.
>> If you have received this transmission in error, please destroy it and
>> notify us immediately at 262-703-7000.
>>
>> CAUTION:
>> Internet and e-mail communications are Kohl's property and Kohl's
>> reserves
>> the right to retrieve and read any message created, sent and received.
>> Kohl's reserves the right to monitor messages by authorized Kohl's
>> Associates at any time
>> without any further consent.
>>
>> _______________________________________________________________________________
>> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
>> Platinum Sponsor:[email protected] ARSlist: "Where the Answers
>> Are"
>>
>> --
>> This message was scanned by ESVA and is believed to be clean.
>>
>>
>
> _______________________________________________________________________________
> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
> Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
>
> --
> This message was scanned by ESVA and is believed to be clean.
>
>
_______________________________________________________________________________
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
Platinum Sponsor:[email protected] ARSlist: "Where the Answers Are"
