Re: Development Cache Mode revisited
Hi, We have now gone into production, and I have a couple of things I want to share about this... The two servers in the server-group are (now) 7.1.0 patch 008. We have ITSM7 instaleld. We can call the servers USER and ESCL, to indicate their major use. The final way we did it was to have the following setup, it is shown last in the last table below: USER, Cache-Mode=Dev, Admin ESCL, Cache-Mode=0 These are results from various settings when an admin-change was applied. Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem USER Dev - 420 sec 420 sec 620Mb 620Mb ESCL Dev Admin 1 sec 1 sec 620Mb 620Mb (not good, as users got locked out) Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem USER Dev Admin 1 sec 1 sec 620Mb 620Mb ESCL Normal- 550 sec 0 sec 1200Mb 620Mb (requires more memory on the ESCL-server) Note that if you import multiple things, as opposed to applying a small change, the 1-second lockout may be considerably higher. Best Regards - Misi, RRR AB, http://rrr.se Yep. If anyone is interested, p171 7.5 Config. When object definition changes are made to the administrative server in a server group, other members of the group can be notified either by a signal or by a database posting. With signaling, other servers are notified of changes immediately, and there is no delay in resynchronization or updating definitions. The database method reduces server activity when object definition changes are communicated between servers and is an effective way to reduce the number of cache reloads when a series of changes are made, but there is a delay before other members of the server group pick up the changes. By default, AR System uses a combination of these methods for different types of updates in a server group. Server definition changes such as changes to forms, active links, filters, and escalations, as well as user group changes, are handled by database posting. All other changes, such as Alert registration, DSO activity, and so on, are handled by signaling. However, you can configure the server to use signaling for all changes including server object changes by following the procedure in this section. The configuration file setting Server-Group-Signal-Option tells the server whether to use arsignald for all signals, rather than using a combination of signaling and database posting. If set to true (T), server object changes are communicated by arsignald instead of database posting. Use this option if you don?t want any delay in communicating server object changes to other servers in the server group. Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) 703-7763 | e-mail: tony.worthing...@kohls.com From: Lyle Taylor tayl...@ldschurch.org To: arslist@ARSLIST.ORG Date: 11/20/2009 03:13 PM Subject: Re: Development Cache Mode revisited Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG ** Concerning the other servers in the server group, the behavior is discussed in one of the AR server guides, but I?m not sure which one (it?s been a while since I read it). It probably wouldn?t be too hard to find if needed. In a nutshell, though, each of the servers in the server group periodically checks to see if the schema has been updated (or if new groups have been added, etc.), and builds a new copy of the cache from the database when they see that it has changed. If I recall correctly, the polling frequency may be able to be configured in ar.conf, and it doesn?t immediately update once it notices a change. Rather, it will wait until something like two successive polling cycles have passed without additional changes occurring before rebuilding its local cache. That way, it?s less likely to do an update in the middle of a set of changes. If Dev Cache Mode is turned off on the other servers in the group, and users are logged into those servers, users will continue to use the old copy of the cache until they log out and back in again. Note that this means that, depending on the frequency of changes (including adding new Remedy group or ITSM Support Groups), this can be a significant source of memory usage on the server. We have run out of memory and had the server crash in the past due to creating several new Support Groups throughout the day, and each one causing yet another copy of the cache to be held in memory for the people that logged in since the last time it was updated. This is even more of an issue if people don?t log out correctly (e.g., if they?re using the mid-tier and simply close their browser window without logging out), as the system will NOT log them out automatically, even though it will eventually release their write
Re: Development Cache Mode revisited
Hi, Just to clarify. This more or less proves that it is a BAD idea to have Cache-Mode set to Development on any server in the Server-Group but the Admin-server. The Admin-server seems to just patch in the modification directly in memory. The other servers allways need to recache everything when a change has occured. If Dev-Cache-Mode is set on a server that rereads the whole cache, the users need to wait until the operation is completed. Best Regards - Misi, RRR AB, http://www.rrr.se Products from RRR Scandinavia: * RRR|License - Not enough Remedy licenses? Save money by optimizing. * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs. * RRR|Translator - Manage and automate your language translations. Find these products, and many free tools and utilities, at http://rrr.se. Hi, We have now gone into production, and I have a couple of things I want to share about this... The two servers in the server-group are (now) 7.1.0 patch 008. We have ITSM7 instaleld. We can call the servers USER and ESCL, to indicate their major use. The final way we did it was to have the following setup, it is shown last in the last table below: USER, Cache-Mode=Dev, Admin ESCL, Cache-Mode=0 These are results from various settings when an admin-change was applied. Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem USER Dev - 420 sec 420 sec 620Mb 620Mb ESCL Dev Admin 1 sec 1 sec 620Mb 620Mb (not good, as users got locked out) Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem USER Dev Admin 1 sec 1 sec 620Mb 620Mb ESCL Normal- 550 sec 0 sec 1200Mb 620Mb (requires more memory on the ESCL-server) Note that if you import multiple things, as opposed to applying a small change, the 1-second lockout may be considerably higher. Best Regards - Misi, RRR AB, http://rrr.se Yep. If anyone is interested, p171 7.5 Config. When object definition changes are made to the administrative server in a server group, other members of the group can be notified either by a signal or by a database posting. With signaling, other servers are notified of changes immediately, and there is no delay in resynchronization or updating definitions. The database method reduces server activity when object definition changes are communicated between servers and is an effective way to reduce the number of cache reloads when a series of changes are made, but there is a delay before other members of the server group pick up the changes. By default, AR System uses a combination of these methods for different types of updates in a server group. Server definition changes such as changes to forms, active links, filters, and escalations, as well as user group changes, are handled by database posting. All other changes, such as Alert registration, DSO activity, and so on, are handled by signaling. However, you can configure the server to use signaling for all changes including server object changes by following the procedure in this section. The configuration file setting Server-Group-Signal-Option tells the server whether to use arsignald for all signals, rather than using a combination of signaling and database posting. If set to true (T), server object changes are communicated by arsignald instead of database posting. Use this option if you don?t want any delay in communicating server object changes to other servers in the server group. Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) 703-7763 | e-mail: tony.worthing...@kohls.com From: Lyle Taylor tayl...@ldschurch.org To: arslist@ARSLIST.ORG Date: 11/20/2009 03:13 PM Subject: Re: Development Cache Mode revisited Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG ** Concerning the other servers in the server group, the behavior is discussed in one of the AR server guides, but I?m not sure which one (it?s been a while since I read it). It probably wouldn?t be too hard to find if needed. In a nutshell, though, each of the servers in the server group periodically checks to see if the schema has been updated (or if new groups have been added, etc.), and builds a new copy of the cache from the database when they see that it has changed. If I recall correctly, the polling frequency may be able to be configured in ar.conf, and it doesn?t immediately update once it notices a change. Rather, it will wait until something like two successive polling cycles have passed without additional changes occurring before rebuilding its local cache. That way, it?s less likely to do an update in the middle of a set of changes. If Dev Cache Mode is turned off on the other servers in the group
Re: Development Cache Mode revisited
You really need to be careful with development cache mode on production servers for the following reason; With this option set the server has only one copy of the cache to work with. This may provide some advantages in terms of memory usage but consider the following.. A user starts a long running query, or a long running escalation kicks off. In both of these cases the thread performing the work will place a flag against the cache to say that it is in use - effectively a read lock. Now you make an admin change which requires exclusive use of the one cache that you have. This exclusive lock cannot be obtained because the cache is in use so it is queued up and your admin thread is out of action until it gets the lock. Not only that but any subsequent user API call that tries to get a read lock on the cache will fail because there is an outstanding exclusive lock request pending. All of your users are locked out until the orginal activity completes and then the pending admin change is done. Mark -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Misi Mladoniczky Sent: 10 December 2009 11:59 To: arslist@ARSLIST.ORG Subject: Re: Development Cache Mode revisited Hi, Just to clarify. This more or less proves that it is a BAD idea to have Cache-Mode set to Development on any server in the Server-Group but the Admin-server. The Admin-server seems to just patch in the modification directly in memory. The other servers allways need to recache everything when a change has occured. If Dev-Cache-Mode is set on a server that rereads the whole cache, the users need to wait until the operation is completed. Best Regards - Misi, RRR AB, http://www.rrr.se Products from RRR Scandinavia: * RRR|License - Not enough Remedy licenses? Save money by optimizing. * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs. * RRR|Translator - Manage and automate your language translations. Find these products, and many free tools and utilities, at http://rrr.se. Hi, We have now gone into production, and I have a couple of things I want to share about this... The two servers in the server-group are (now) 7.1.0 patch 008. We have ITSM7 instaleld. We can call the servers USER and ESCL, to indicate their major use. The final way we did it was to have the following setup, it is shown last in the last table below: USER, Cache-Mode=Dev, Admin ESCL, Cache-Mode=0 These are results from various settings when an admin-change was applied. Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem USER Dev - 420 sec 420 sec 620Mb 620Mb ESCL Dev Admin 1 sec 1 sec 620Mb 620Mb (not good, as users got locked out) Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem USER Dev Admin 1 sec 1 sec 620Mb 620Mb ESCL Normal- 550 sec 0 sec 1200Mb 620Mb (requires more memory on the ESCL-server) Note that if you import multiple things, as opposed to applying a small change, the 1-second lockout may be considerably higher. Best Regards - Misi, RRR AB, http://rrr.se Yep. If anyone is interested, p171 7.5 Config. When object definition changes are made to the administrative server in a server group, other members of the group can be notified either by a signal or by a database posting. With signaling, other servers are notified of changes immediately, and there is no delay in resynchronization or updating definitions. The database method reduces server activity when object definition changes are communicated between servers and is an effective way to reduce the number of cache reloads when a series of changes are made, but there is a delay before other members of the server group pick up the changes. By default, AR System uses a combination of these methods for different types of updates in a server group. Server definition changes such as changes to forms, active links, filters, and escalations, as well as user group changes, are handled by database posting. All other changes, such as Alert registration, DSO activity, and so on, are handled by signaling. However, you can configure the server to use signaling for all changes including server object changes by following the procedure in this section. The configuration file setting Server-Group-Signal-Option tells the server whether to use arsignald for all signals, rather than using a combination of signaling and database posting. If set to true (T), server object changes are communicated by arsignald instead of database posting. Use this option if you don?t want any delay in communicating server object changes to other servers in the server group. Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores N56 W17000 Ridgewood Drive
Re: Development Cache Mode revisited
Hi, We have taken all this into account. 1. The escalations are run on another server 2. Admin-changes will never be applied during production hours Without Development-Cache-Mode set, the server frequently timed out when doing admin changes. This seemed to be because of the buffer-copying when the cache was duplicated. Best Regards - Misi, RRR AB, http://www.rrr.se Products from RRR Scandinavia: * RRR|License - Not enough Remedy licenses? Save money by optimizing. * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs. Find these products, and many free tools and utilities, at http://rrr.se. You really need to be careful with development cache mode on production servers for the following reason; With this option set the server has only one copy of the cache to work with. This may provide some advantages in terms of memory usage but consider the following.. A user starts a long running query, or a long running escalation kicks off. In both of these cases the thread performing the work will place a flag against the cache to say that it is in use - effectively a read lock. Now you make an admin change which requires exclusive use of the one cache that you have. This exclusive lock cannot be obtained because the cache is in use so it is queued up and your admin thread is out of action until it gets the lock. Not only that but any subsequent user API call that tries to get a read lock on the cache will fail because there is an outstanding exclusive lock request pending. All of your users are locked out until the orginal activity completes and then the pending admin change is done. Mark -Original Message- From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Misi Mladoniczky Sent: 10 December 2009 11:59 To: arslist@ARSLIST.ORG Subject: Re: Development Cache Mode revisited Hi, Just to clarify. This more or less proves that it is a BAD idea to have Cache-Mode set to Development on any server in the Server-Group but the Admin-server. The Admin-server seems to just patch in the modification directly in memory. The other servers allways need to recache everything when a change has occured. If Dev-Cache-Mode is set on a server that rereads the whole cache, the users need to wait until the operation is completed. Best Regards - Misi, RRR AB, http://www.rrr.se Products from RRR Scandinavia: * RRR|License - Not enough Remedy licenses? Save money by optimizing. * RRR|Log - Performance issues or elusive bugs? Analyze your Remedy logs. * RRR|Translator - Manage and automate your language translations. Find these products, and many free tools and utilities, at http://rrr.se. Hi, We have now gone into production, and I have a couple of things I want to share about this... The two servers in the server-group are (now) 7.1.0 patch 008. We have ITSM7 instaleld. We can call the servers USER and ESCL, to indicate their major use. The final way we did it was to have the following setup, it is shown last in the last table below: USER, Cache-Mode=Dev, Admin ESCL, Cache-Mode=0 These are results from various settings when an admin-change was applied. Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem USER Dev - 420 sec 420 sec 620Mb 620Mb ESCL Dev Admin 1 sec 1 sec 620Mb 620Mb (not good, as users got locked out) Server CacheMode AdminServer RecacheTime UserLockout RecacheMem StableMem USER Dev Admin 1 sec 1 sec 620Mb 620Mb ESCL Normal- 550 sec 0 sec 1200Mb 620Mb (requires more memory on the ESCL-server) Note that if you import multiple things, as opposed to applying a small change, the 1-second lockout may be considerably higher. Best Regards - Misi, RRR AB, http://rrr.se Yep. If anyone is interested, p171 7.5 Config. When object definition changes are made to the administrative server in a server group, other members of the group can be notified either by a signal or by a database posting. With signaling, other servers are notified of changes immediately, and there is no delay in resynchronization or updating definitions. The database method reduces server activity when object definition changes are communicated between servers and is an effective way to reduce the number of cache reloads when a series of changes are made, but there is a delay before other members of the server group pick up the changes. By default, AR System uses a combination of these methods for different types of updates in a server group. Server definition changes such as changes to forms, active links, filters, and escalations, as well as user group changes, are handled by database posting. All other changes, such as Alert registration, DSO activity, and so on, are handled by signaling. However, you
Re: Development Cache Mode revisited
Yep. If anyone is interested, p171 7.5 Config. When object definition changes are made to the administrative server in a server group, other members of the group can be notified either by a signal or by a database posting. With signaling, other servers are notified of changes immediately, and there is no delay in resynchronization or updating definitions. The database method reduces server activity when object definition changes are communicated between servers and is an effective way to reduce the number of cache reloads when a series of changes are made, but there is a delay before other members of the server group pick up the changes. By default, AR System uses a combination of these methods for different types of updates in a server group. Server definition changes such as changes to forms, active links, filters, and escalations, as well as user group changes, are handled by database posting. All other changes, such as Alert registration, DSO activity, and so on, are handled by signaling. However, you can configure the server to use signaling for all changes including server object changes by following the procedure in this section. The configuration file setting Server-Group-Signal-Option tells the server whether to use arsignald for all signals, rather than using a combination of signaling and database posting. If set to true (T), server object changes are communicated by arsignald instead of database posting. Use this option if you don?t want any delay in communicating server object changes to other servers in the server group. Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) 703-7763 | e-mail: tony.worthing...@kohls.com From: Lyle Taylor tayl...@ldschurch.org To: arslist@ARSLIST.ORG Date: 11/20/2009 03:13 PM Subject: Re: Development Cache Mode revisited Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG ** Concerning the other servers in the server group, the behavior is discussed in one of the AR server guides, but I?m not sure which one (it?s been a while since I read it). It probably wouldn?t be too hard to find if needed. In a nutshell, though, each of the servers in the server group periodically checks to see if the schema has been updated (or if new groups have been added, etc.), and builds a new copy of the cache from the database when they see that it has changed. If I recall correctly, the polling frequency may be able to be configured in ar.conf, and it doesn?t immediately update once it notices a change. Rather, it will wait until something like two successive polling cycles have passed without additional changes occurring before rebuilding its local cache. That way, it?s less likely to do an update in the middle of a set of changes. If Dev Cache Mode is turned off on the other servers in the group, and users are logged into those servers, users will continue to use the old copy of the cache until they log out and back in again. Note that this means that, depending on the frequency of changes (including adding new Remedy group or ITSM Support Groups), this can be a significant source of memory usage on the server. We have run out of memory and had the server crash in the past due to creating several new Support Groups throughout the day, and each one causing yet another copy of the cache to be held in memory for the people that logged in since the last time it was updated. This is even more of an issue if people don?t log out correctly (e.g., if they?re using the mid-tier and simply close their browser window without logging out), as the system will NOT log them out automatically, even though it will eventually release their write licenses. This causes old copies of the cache to be retained for longer periods than necessary. The old copies of the cache will be held in memory until everyone that was using that particular copy log out of Remedy; it will then release it, and you memory usage will drop. Like I said, that?s from memory and recent experience. I do recall reading it in one of the guides in the standard documentation, so you should be able to find the specifics if you want them. Lyle From: Action Request System discussion list(ARSList) [ mailto:arsl...@arslist.org] On Behalf Of Tony Worthington Sent: Friday, November 20, 2009 12:28 PM To: arslist@ARSLIST.ORG Subject: Re: Development Cache Mode revisited ** Be aware that (mysterious) things in ITSM7 trigger recaches -- not just definition imports or changes. I haven't quite figured out what -- but we have one or two a day -- no admin involved. They show as Remedy Application Service user. Some can be traced to group changes, which we now perform after-hours. Others are a mystery. I don't think there are db read differences when updating the cache - just the server handling of the in-memory copy. If you watch the size
Development Cache Mode revisited
Hi all, If you have Development Cache Mode ON for a production environment. What would be the impact, if: - No admin-changes were done - And No changes to Groups of the type View/Change were done The client runs ARServer 7.1.0 patch 4, ITSM 7.0, Windows and Oracle. it is a server group with 2 servers. The reason I ask, is that if they have Cache-Mode turned OFF (0), the production server seems to time out when they actually do an admin change. For example enabling an escalation. They want to do occasional admin-changes during off-hours when activity is very low. I tried to catch the actual recaching in my AR Server SQL-log, but could not find any such activity. Maybe it is not logged for some reason??? Best Regards - Misi, RRR AB, http://rrr.se ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
Re: Development Cache Mode revisited
Misi - We have the same issue. We run with dev cache mode off in production and are required to bounce the server (extending the outage by at least 10+10 minutes) to flip in and out of dev cache mode. I have also found that making changes with dev cache off is risky, and has caused serious issues with timeouts halfway through object changes. The re-cache events can be captured in the thread logs, with the additional server config file setting of: Copy-Cache-Logging: T This is the first initial build... THRD /* Sun Nov 15 2009 02:36:12.1180 */ InitServerCache Begin THRD /* Sun Nov 15 2009 02:43:38.1260 */ InitServerCache End: rpcCallProc=0 tid=2708 And subsequent recaches.. THRD /* Sun Nov 15 2009 10:28:47.2310 */ CopyCache Begin: rpcCallProc=10002 user=Remedy Application Service tid=2708 rpcId=0 THRD /* Sun Nov 15 2009 10:30:30.8860 */ CopyCache End THRD /* Sun Nov 15 2009 10:38:01.5740 */ FreeServerCache: rpcCallProc=94 user=tibco tid=532 rpcId=286130 I am excited for the pre-load settings of 7.5 -- and am hoping it will alleviate the time-out issues by allocating more threads and specifying No for at init only -- thereby allowing small admin changes, while leaving the server in non-dev cache mode. Tony Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) 703-7763 | e-mail: tony.worthing...@kohls.com From: Misi Mladoniczky m...@rrr.se To: arslist@ARSLIST.ORG Date: 11/20/2009 08:49 AM Subject: Development Cache Mode revisited Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG Hi all, If you have Development Cache Mode ON for a production environment. What would be the impact, if: - No admin-changes were done - And No changes to Groups of the type View/Change were done The client runs ARServer 7.1.0 patch 4, ITSM 7.0, Windows and Oracle. it is a server group with 2 servers. The reason I ask, is that if they have Cache-Mode turned OFF (0), the production server seems to time out when they actually do an admin change. For example enabling an escalation. They want to do occasional admin-changes during off-hours when activity is very low. I tried to catch the actual recaching in my AR Server SQL-log, but could not find any such activity. Maybe it is not logged for some reason??? Best Regards - Misi, RRR AB, http://rrr.se ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are ** CONFIDENTIALITY NOTICE: This is a transmission from Kohl's Department Stores, Inc. and may contain information which is confidential and proprietary. If you are not the addressee, any disclosure, copying or distribution or use of the contents of this message is expressly prohibited. If you have received this transmission in error, please destroy it and notify us immediately at 262-703-7000. CAUTION: Internet and e-mail communications are Kohl's property and Kohl's reserves the right to retrieve and read any message created, sent and received. Kohl's reserves the right to monitor messages by authorized Kohl's Associates at any time without any further consent. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: Where the Answers Are
Re: Development Cache Mode revisited
users. In a production environment the opposite is true. So, the key thing to understand when using this parameter is that we lock the object when a change is applied (when we issue a Set, Delete, or Create API call), not when it is opened. This means that any number of admin tools may have an object open; if one of them changes the object and a second one makes another change, the first is overwritten. That behavior is no different than without the change, since there is only one admin thread and all admin operations are serialized. This should improve performance immensely when saving a form because you are only dealing with the current form and not all objects on the server being cached. Is there any risk to having Development Cache Mode enabled in production? Yes. An issue was discovered when development cache mode is enabled that can cause AR Server to appear hung. This is thoroughly described in KM-00025743. This is NOT a common issue. Normally, there would be no risk other than the described behavior of AR Server when Admin changes occur. -- Here's 25743 There is an important side-effect that can cause AR Server to appear hung and users receive timeout errors, like ARERR 93. This occurs under specific circumstances, but specifically when the Development Cache Mode is selected in the Admin Tool under File-Server Information on the Configuration tab (see KB 24487 for information on Development Cache Mode). Here is the specific course of events that leads to the problem: An escalation starts running and puts a Read Lock on the Server Cache tables (in memory). An Admin Operation (like a form or workflow save) starts but cannot put an Exclusive lock on the cache until the Escalation is done with its lock. The escalation continues to run. Users and utilities try to perform ARServer work (APIs). But they need a read lock on the cache and are forced to wait. The escalation keeps running. Users start getting timeouts (ARERR 93, etc.) and the Admin Tool times out. The escalation keeps running. Finally the escalation completes and the Admin Operation can start. Admin Operation completes and users work can start. Since the users already timed out, they've likely given up and moved into trouble shooting mode, unaware of whats happening with AR Server. Normally, escalations run pretty quick, but in some cases, this isn't true. For instance, lets say an escalation fires on a form with 200,000 records in it, updating each one. Each record touched causes Filters to fire which do Push Fields actions to several other forms for example, resulting in each of the 200,000 records to take roughly 18 seconds to complete. This escalation would take a very long time to complete and all the while, AR Server would appear as though its hung because of the Development Cache mode being enabled. When Development Cache mode is enabled, it will attempt to put an exclusive lock on the server cache tables in memory. When disabled, a separate memory footprint is created so there is no locking conflict with the cache and therefore this problem would not occur. This behavior is not considered a bug, but rather a circumstantial issue caused by the design of the product. Currently there is no workaround to this issue, but being aware of it means that there are options. For instance, normally Development Cache Mode is only enabled on a development server. If this is the case, then usually you can disable escalations or temporarily disable development cache mode in development (if the escalation must run for testing, etc.). You could also alter the escalation(s) so it doesn't touch so many records or doesn't cause filters to fire, etc. In other words, speed up the escalation processing time so any locks are very quick, or disable the escalation (or all escalations), or disable development cache mode. Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) 703-7763 | e-mail: tony.worthing...@kohls.com From: Misi Mladoniczky m...@rrr.se To: arslist@ARSLIST.ORG Date: 11/20/2009 08:49 AM Subject: Development Cache Mode revisited Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG Hi all, If you have Development Cache Mode ON for a production environment. What would be the impact, if: - No admin-changes were done - And No changes to Groups of the type View/Change were done The client runs ARServer 7.1.0 patch 4, ITSM 7.0, Windows and Oracle. it is a server group with 2 servers. The reason I ask, is that if they have Cache-Mode turned OFF (0), the production server seems to time out when they actually do an admin change. For example enabling an escalation. They want to do occasional admin-changes during off-hours when activity is very low. I tried to catch the actual recaching in my AR Server SQL-log, but could not find any such activity. Maybe
Re: Development Cache Mode revisited
I just worked through a longstanding issue that we have had in our environment related to Dev Cache Mode. Due to extremely slow performance when making admin changes on a 7.0 server, I had gotten into the habit of keeping our admin server set with Dev Cache Mode turned on, which significantly sped up migrations to production and such. However, in this environment (7.1 p5 server), I was seeing exactly the opposite - admin changes were very frequently timing out and migrations were painfully slow (for example, a patch that took a few minutes to apply in dev and stage took an hour and twenty minutes to apply in production). Thinking that it should go faster since Dev Cache Mode was turned on, I looked everywhere I could think of to try and isolate why it was so slow. I got BMC involved in my search twice. This last time, the support rep forwarded this same KB article to me and recommended that I leave Dev Cache Mode turned OFF in order to improve performance with admin actions. I tried it, and it worked. My migrations now go very quickly, and I haven't seen any timeouts since. The only slowness is in the initial period where the AR server creates a copy of the cache in memory, but that's generally fairly quick, since it's just doing an in memory copy rather than trying to pull it all from the database again (like I suspect it was doing on the 7.0 server). In short, in our environment, we see significantly better performance making admin changes in production with Dev Cache Mode turned off. The support rep told me that since AR Server 7.1, their recommendation is to keep it turned off. That said, if I understand it correctly, your mileage may vary depending on what kind of activity you have going on on the server you're trying to make changes on. As I understand it, when Dev Cache Mode is turned on, since it doesn't make a copy of the cache for existing users, it has to wait until user operations have ceased before it can make the change, then it makes the change in between user operations. In our case, since this server also runs all our back-end processes (e-mail, escalations, etc.), and we apparently have some things going on that take a long time to complete, the admin thread ended up having to wait a long time (more than 10 minutes in some cases) before it could get a lock on the cache and make the changes. This was what was causing the timeouts and slow performance that we saw. Now, since it simply makes a relatively quick copy of the cache for existing user operations (just system operations in our case, but still, they're ongoing), the overall time to make the changes is quick, because once the cache is copied, it can proceed with the changes without having to wait for a break between user operations. So, if you don't have much going on on the server you're trying to make admin changes on, it could be quicker to leave Dev Cache Mode turned on, because then you don't have to wait for the cache to get copied (I think it takes less than 30 seconds in our case to copy about 400MB of cache). However, if you have lots of stuff going on (e-mail, user access, escalations, etc.), you may see better performance with it turned off. I hope this helps a bit. Lyle From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Tony Worthington Sent: Friday, November 20, 2009 9:36 AM To: arslist@ARSLIST.ORG Subject: Re: Development Cache Mode revisited ** I missed your question... sorry. - No admin-changes were done - And No changes to Groups of the type View/Change were done This is from BMC KB KM-00024487. It doesn't answer your ? directly, but is full of good information. What may cause poor performance in the Admin Tool? When poor performance is seen in the Admin Tool when saving objects, it may be a result of how AR Server handles information it has read into memory. At server startup, AR Server reads information from the database into memory, for example all field properties are read into memory, all users and groups and active links, etc. When this information that has been read into memory changes, there are two different ways for AR Server to handle the event. It can update the existing memory footprint so that users all see the change that was made. It can create a separate memory footprint so existing users still utilize the existing memory footprint, while new users logging in would utilize the new memory footprint. Either method will cause a performance hit, but the big difference between the two ways of handling the event is where the performance impact is seen. Note: How noticeable the performance hit is depends on several factors, but mostly the amount of data that needs to be read into memory. This is usually a direct result of the number of objects (forms and workflow) that exist. The more objects, the greater the performance impact. Is there a way to improve performance or control
Re: Development Cache Mode revisited
Lyle and Tony, Thank you for the reply. In other words, there should be no impact whatsoever if you perform NO admin-changes. If you apply your changes when both users, email engine and escalations are taking it easy, it would probably be faster to use Developer Cache Mode even in production (Cache-Mode: 1). Two more questions: 1. I turned on SQL-logs to see what happened when disabling an escalation with both settings. Nothing much happened, and the server did not seem to reread much from the database repository. I had expected a bunch of SQL-selects from the repository. Maybe it just updated the memory cache without rereading the DB? 2. What happens with the cache in the other servers in the server group? Is the admin-change propagated to these in some magic way? Best Regards - Misi, RRR AB, http://rrr.se I just worked through a longstanding issue that we have had in our environment related to Dev Cache Mode. Due to extremely slow performance when making admin changes on a 7.0 server, I had gotten into the habit of keeping our admin server set with Dev Cache Mode turned on, which significantly sped up migrations to production and such. However, in this environment (7.1 p5 server), I was seeing exactly the opposite - admin changes were very frequently timing out and migrations were painfully slow (for example, a patch that took a few minutes to apply in dev and stage took an hour and twenty minutes to apply in production). Thinking that it should go faster since Dev Cache Mode was turned on, I looked everywhere I could think of to try and isolate why it was so slow. I got BMC involved in my search twice. This last time, the support rep forwarded this same KB article to me and recommended that I leave Dev Cache Mode turned OFF in order to improve performance with admin actions. I tried it, and it worked. My migrations now go very quickly, and I haven't seen any timeouts since. The only slowness is in the initial period where the AR server creates a copy of the cache in memory, but that's generally fairly quick, since it's just doing an in memory copy rather than trying to pull it all from the database again (like I suspect it was doing on the 7.0 server). In short, in our environment, we see significantly better performance making admin changes in production with Dev Cache Mode turned off. The support rep told me that since AR Server 7.1, their recommendation is to keep it turned off. That said, if I understand it correctly, your mileage may vary depending on what kind of activity you have going on on the server you're trying to make changes on. As I understand it, when Dev Cache Mode is turned on, since it doesn't make a copy of the cache for existing users, it has to wait until user operations have ceased before it can make the change, then it makes the change in between user operations. In our case, since this server also runs all our back-end processes (e-mail, escalations, etc.), and we apparently have some things going on that take a long time to complete, the admin thread ended up having to wait a long time (more than 10 minutes in some cases) before it could get a lock on the cache and make the changes. This was what was causing the timeouts and slow performance that we saw. Now, since it simply makes a relatively quick copy of the cache for existing user operations (just system operations in our case, but still, they're ongoing), the overall time to make the changes is quick, because once the cache is copied, it can proceed with the changes without having to wait for a break between user operations. So, if you don't have much going on on the server you're trying to make admin changes on, it could be quicker to leave Dev Cache Mode turned on, because then you don't have to wait for the cache to get copied (I think it takes less than 30 seconds in our case to copy about 400MB of cache). However, if you have lots of stuff going on (e-mail, user access, escalations, etc.), you may see better performance with it turned off. I hope this helps a bit. Lyle From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Tony Worthington Sent: Friday, November 20, 2009 9:36 AM To: arslist@ARSLIST.ORG Subject: Re: Development Cache Mode revisited ** I missed your question... sorry. - No admin-changes were done - And No changes to Groups of the type View/Change were done This is from BMC KB KM-00024487. It doesn't answer your ? directly, but is full of good information. What may cause poor performance in the Admin Tool? When poor performance is seen in the Admin Tool when saving objects, it may be a result of how AR Server handles information it has read into memory. At server startup, AR Server reads information from the database into memory, for example all field properties are read into memory, all users and groups and active links, etc. When this information
Re: Development Cache Mode revisited
Be aware that (mysterious) things in ITSM7 trigger recaches -- not just definition imports or changes. I haven't quite figured out what -- but we have one or two a day -- no admin involved. They show as Remedy Application Service user. Some can be traced to group changes, which we now perform after-hours. Others are a mystery. I don't think there are db read differences when updating the cache - just the server handling of the in-memory copy. If you watch the size of the server when doing changes on vs. off you should see the differences. What happens with the cache in the other servers in the server group? I think this is dealt with in the server_cache, servgrp_board and servgrp_cache and related servgrp_* tables. No server group here so I can't say for sure. My thinking is the server watches for a signal to trigger a recache. That signal is is an API call or a change in the meta-data. Can anyone confirm how it really works? Tony Worthington | Sr. Technical Analyst | Kohl?s Department Stores N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) 703-7763 | e-mail: tony.worthing...@kohls.com From: Misi Mladoniczky m...@rrr.se To: arslist@ARSLIST.ORG Date: 11/20/2009 01:13 PM Subject: Re: Development Cache Mode revisited Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG Lyle and Tony, Thank you for the reply. In other words, there should be no impact whatsoever if you perform NO admin-changes. If you apply your changes when both users, email engine and escalations are taking it easy, it would probably be faster to use Developer Cache Mode even in production (Cache-Mode: 1). Two more questions: 1. I turned on SQL-logs to see what happened when disabling an escalation with both settings. Nothing much happened, and the server did not seem to reread much from the database repository. I had expected a bunch of SQL-selects from the repository. Maybe it just updated the memory cache without rereading the DB? 2. What happens with the cache in the other servers in the server group? Is the admin-change propagated to these in some magic way? Best Regards - Misi, RRR AB, http://rrr.se I just worked through a longstanding issue that we have had in our environment related to Dev Cache Mode. Due to extremely slow performance when making admin changes on a 7.0 server, I had gotten into the habit of keeping our admin server set with Dev Cache Mode turned on, which significantly sped up migrations to production and such. However, in this environment (7.1 p5 server), I was seeing exactly the opposite - admin changes were very frequently timing out and migrations were painfully slow (for example, a patch that took a few minutes to apply in dev and stage took an hour and twenty minutes to apply in production). Thinking that it should go faster since Dev Cache Mode was turned on, I looked everywhere I could think of to try and isolate why it was so slow. I got BMC involved in my search twice. This last time, the support rep forwarded this same KB article to me and recommended that I leave Dev Cache Mode turned OFF in order to improve performance with admin actions. I tried it, and it worked. My migrations now go very quickly, and I haven't seen any timeouts since. The only slowness is in the initial period where the AR server creates a copy of the cache in memory, but that's generally fairly quick, since it's just doing an in memory copy rather than trying to pull it all from the database again (like I suspect it was doing on the 7.0 server). In short, in our environment, we see significantly better performance making admin changes in production with Dev Cache Mode turned off. The support rep told me that since AR Server 7.1, their recommendation is to keep it turned off. That said, if I understand it correctly, your mileage may vary depending on what kind of activity you have going on on the server you're trying to make changes on. As I understand it, when Dev Cache Mode is turned on, since it doesn't make a copy of the cache for existing users, it has to wait until user operations have ceased before it can make the change, then it makes the change in between user operations. In our case, since this server also runs all our back-end processes (e-mail, escalations, etc.), and we apparently have some things going on that take a long time to complete, the admin thread ended up having to wait a long time (more than 10 minutes in some cases) before it could get a lock on the cache and make the changes. This was what was causing the timeouts and slow performance that we saw. Now, since it simply makes a relatively quick copy of the cache for existing user operations (just system operations in our case, but still, they're ongoing), the overall time to make the changes is quick, because once the cache is copied, it can proceed with the changes without having to wait for a break
Re: Development Cache Mode revisited
Concerning the other servers in the server group, the behavior is discussed in one of the AR server guides, but I'm not sure which one (it's been a while since I read it). It probably wouldn't be too hard to find if needed. In a nutshell, though, each of the servers in the server group periodically checks to see if the schema has been updated (or if new groups have been added, etc.), and builds a new copy of the cache from the database when they see that it has changed. If I recall correctly, the polling frequency may be able to be configured in ar.conf, and it doesn't immediately update once it notices a change. Rather, it will wait until something like two successive polling cycles have passed without additional changes occurring before rebuilding its local cache. That way, it's less likely to do an update in the middle of a set of changes. If Dev Cache Mode is turned off on the other servers in the group, and users are logged into those servers, users will continue to use the old copy of the cache until they log out and back in again. Note that this means that, depending on the frequency of changes (including adding new Remedy group or ITSM Support Groups), this can be a significant source of memory usage on the server. We have run out of memory and had the server crash in the past due to creating several new Support Groups throughout the day, and each one causing yet another copy of the cache to be held in memory for the people that logged in since the last time it was updated. This is even more of an issue if people don't log out correctly (e.g., if they're using the mid-tier and simply close their browser window without logging out), as the system will NOT log them out automatically, even though it will eventually release their write licenses. This causes old copies of the cache to be retained for longer periods than necessary. The old copies of the cache will be held in memory until everyone that was using that particular copy log out of Remedy; it will then release it, and you memory usage will drop. Like I said, that's from memory and recent experience. I do recall reading it in one of the guides in the standard documentation, so you should be able to find the specifics if you want them. Lyle From: Action Request System discussion list(ARSList) [mailto:arsl...@arslist.org] On Behalf Of Tony Worthington Sent: Friday, November 20, 2009 12:28 PM To: arslist@ARSLIST.ORG Subject: Re: Development Cache Mode revisited ** Be aware that (mysterious) things in ITSM7 trigger recaches -- not just definition imports or changes. I haven't quite figured out what -- but we have one or two a day -- no admin involved. They show as Remedy Application Service user. Some can be traced to group changes, which we now perform after-hours. Others are a mystery. I don't think there are db read differences when updating the cache - just the server handling of the in-memory copy. If you watch the size of the server when doing changes on vs. off you should see the differences. What happens with the cache in the other servers in the server group? I think this is dealt with in the server_cache, servgrp_board and servgrp_cache and related servgrp_* tables. No server group here so I can't say for sure. My thinking is the server watches for a signal to trigger a recache. That signal is is an API call or a change in the meta-data. Can anyone confirm how it really works? Tony Worthington | Sr. Technical Analyst | Kohl's Department Stores N56 W17000 Ridgewood Drive | Menomonee Falls, WI 53051 | office: (262) 703-7763 | e-mail: tony.worthing...@kohls.com From: Misi Mladoniczky m...@rrr.se To: arslist@ARSLIST.ORG Date: 11/20/2009 01:13 PM Subject: Re: Development Cache Mode revisited Sent by: Action Request System discussion list(ARSList) arslist@ARSLIST.ORG Lyle and Tony, Thank you for the reply. In other words, there should be no impact whatsoever if you perform NO admin-changes. If you apply your changes when both users, email engine and escalations are taking it easy, it would probably be faster to use Developer Cache Mode even in production (Cache-Mode: 1). Two more questions: 1. I turned on SQL-logs to see what happened when disabling an escalation with both settings. Nothing much happened, and the server did not seem to reread much from the database repository. I had expected a bunch of SQL-selects from the repository. Maybe it just updated the memory cache without rereading the DB? 2. What happens with the cache in the other servers in the server group? Is the admin-change propagated to these in some magic way? Best Regards - Misi, RRR AB, http://rrr.sehttp://rrr.se/ I just worked through a longstanding issue that we have had in our environment related to Dev Cache Mode. Due to extremely slow performance when making admin changes on a 7.0 server, I had gotten into the habit of keeping our
