Re: Group emails slow

[Doug Blair]

** Greg, I’ve seen tis exact symptom before. Support is on the right track. There are two escalations that process records in this form. One turns Notifications addressed to a group into a bunch of Notifications addressed to individual people, and the other processes the individual messages taking into account the user’s preference, performing keyword substitution and sending the message to AR System Email Messages. The problem you’re seeing happens when a large number of notifications are to be processed and the amount of time taken by the first escalation, plus the amount of time for other escalations running in the same pool, takes longer than the escalation interval. In that condition the first escalation is skipped or delayed until the next time for it to be called comes around. This delays the expansion of the group messages while individual ones continue tone processed every minute. For example, if it takes 5 minutes to process the group notifications and another 7 minutes to process all the individual messages, it is 12 minutes before the group escalation could be considered again. There are other escalations running in the same pool (or if you don’t use pools, all on the same escalation thread). So the group escalation is delayed. As I recall the group escalation runs every 10 minutes and the individual one every minute.  if you turn on escalation logging you can see the order in which the escalations fire, and that one has to run all the way through before the next one starts, and that ALL the escalations due to run at a specific time have to be processed before the system gets the list again. To deal with this we did three things.  First we added several pools for escalations, and moved the one that processes individual messages to its own pool. This means nothing gets in its way.  The second was to move the group escalation also to its own pool, that way nothing delays it either. The third thing was to add multiple threads for 390603. This means that in cases where both of these escalations are due to run at the same time they can in fact be processed in parallel. In this configuration that big escalation that closes old incidents or removed processed email messages will not tie up an escalation thread that needs to process outgoing messages. I suspect you’ll find something that does a large scale operation on incidents or people records or some such. And… of course, anything that your DBA tells you about bad queries looking up people’s email address or their user preferences for notifications should be looked at, but changes the the huber of users with notification preferences, as well as the number of pending group notifications, are both going to be fairly small numbers. Feel free to contact me off list if you need some more detail. Doug Blair On Feb 24, 2014, at 10:03 AM, Greg Donalson <cdonal...@slb.com> wrote: Schlumberger-Public ________________________________________ Hi all, I am wondering if anyone else has experienced the issues that I am seeing.  Group emails will fly out of the system with no issue for about 4 days, then on that 5th day, they start lagging and taking longer and longer to leave the system.  The form that I am looking at is the NTE:SYS-NT Process Control form.  So, I know that there are escalations around this form depending on if it is an individual email or a group email.  The individual ones will continue to go out at a fast pace, it is just the group emails that take longer.  I understand that it has to loop through a table with all of the people that are assigned to that group, but like I said in the first 4 days after a restart of the Remedy service, they go out at a fast pace.  Just to give you an idea, I have seen the group emails get 20-30 minutes behind.  I am thinking this is more of an escalation issue as if I restart just the email service, it does not help any.  It does not matter if I restart the Remedy service on a Monday at 8am or a Thursday at 2pm, within 7 days, the group emails will be slow in going out.  Also during this time, we will see the CPU of this server go up to around 30% or above and stay there, where in the first 4 days it is between 10-20%.  I have relayed all of this to BMC and they have tons of logs (we have been working for 6 month on this 1 issue).  They have asked me to increase escalation thread, move the group escalations around, and move the escalations from one server to another.  None of this has helped at all.  I have asked for BMC to escalate the issue and the same person keeps calling me and says they are working with the escalated person - starting to have doubts about that. Here is our environment: ARS - 7.6.04 SP3 ITSM (Incident, Change, and Asset) - 7.6.04 SP2 Database - Oracle 11g - 64bit Application server - 2 Linux servers that are VMs that are in a server group MidTier - 3 servers - 7.6.04 SP4 with the March 4, 2013 Hotfix _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years" Doug -- Doug Blair d...@blairing.com +1 224-558-5462 1208 East Fremont Street Arlington Heights, Illinois 60004 ITILv3 _ARSlist: "Where the Answers Are" and have been for 20 years_