William, Personally, I like the part about it being broken when you have multiple servers working and that does jive with the idea that it only ran when you had the admin server up. It seems odd, but your logic works in my eyes. I think maybe creating an escalation to delete all the AE-ASSIGN tickets would be a good idea. Or a filter that automatically addressed them as they were created (most likely deleting them).
It sucks that it happened and I am certain management is NOT amused. Good luck on getting everything fixed and resolved. Brian Goralczyk On Tue, Apr 1, 2014 at 8:43 AM, William Rentfrow <[email protected] > wrote: > ** > > Backstory: Linux 7.6.04 sp 4 with remote Oracle DB. Not really relevant > to this issue... > > > > The other night we were adding servers to our server group. So we had all > of the servers down and had started installing the new servers. We got > through 2 ARS installs. Then we licensed those and installed CMDB - during > the CMDB install we had to go add a record for the AIS:GlobalPreferences > for each server, so we started one of the other servers up, added the data, > and moved on. > > > > After we got done with those installs we stopped (we were in an outage > window and just about out of time). Then we started up our admin server > and finally the other 5 ARS servers. > > > > The next morning I come in to find that ALL of of the existing active > tasks have been re-assigned by the Remedy Application Service. In most > cases they have all been assigned to the same person in each group. All of > them are last modified by "Remedy Application Service" and the task audit > shows a clear assignment to the individual by the RAS. It didn't matter > if the tasks were already assigned to a person - they all got assigned to > one person in each group. > > > > Management was not impressed. > > > > At this point - to the best of my knowledge - we have no applications > using the Assignment Engine. As far as I know that is the only thing that > can change Assignments in a bulk way. > > > > So I re-check the configuration of everything - and find that Task > Management is set to have Round Robin assignment. That doesn't fit what > happened - if anything it was using capactiy management. > > > > The audit logs compared to the arerror logs that show shutdowns, etc, > indicate this only happened during our outage window. And it happened > around the time we brought up the admin server. Also, it definitely > happened at the end when only the admin server was up. So the admin server > had to be the one to do this. > > > > It has not happened since. I have noticed that the records in Application > Pending for AE-ASSIGN are just constantly growing. > > > > So here's my working theory: Please poke holes in it. I can't find any > other explanation. > > > > 1. The assignment engine is not in use and has been broken for a > long time - we just didn't notice. > > 2. Application Pending - which we manually clean out from time to > time- had a bunch of AE-ASSIGN records in it > > 3. For some unknown reason the AE started to work when we bounced > the admin server during the installs and processed the AE-ASSIGN records > > 4. When AE-ASSIGN records are processed they do not check to see if > there is an assignee because the workflow that creates the record did that. > > 5. Bringing up the other servers for some reason broke it again, > and it stopped. > > > > We've turned on AE logging but so far there's nothing in it - this sort of > confirms my SWAG above. > > > > Ideas? I really have no direct proof it was the AE - but there are no > escalations affecting assignment and I don't know of any other processes > that do. > > > > William Rentfrow > > [email protected] > > Office: 715-204-3061 > > Cell: 715-398-5056 > > > _ARSlist: "Where the Answers Are" and have been for 20 years_ _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org "Where the Answers Are, and have been for 20 years"
