Re: Remedy System comes to a standstill/unresonsive
The changes we made were based on recommendations from support. He really didn't give me any indication of why specific changes were being made except for one. Probably just best practice settings. The only setting he explained was the Cache-mode. Ron -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Jamie Sent: Saturday, April 21, 2012 6:04 AM To: arslist@ARSLIST.ORG Subject: Re: Remedy System comes to a standstill/unresonsive Thank you all for all of your responses. Sorry, I have left you a little dry, but was OOO for a while for vacation. We have looked into our private threads and thread counts and will soon be: Creating private queue for recon process Seperating RPC loop back and approval process onto their own private queue (currently sharing the same thread) We are also working on improving some table scans we've seen crop up that are increasing the IO on our database server. Also, we're taking steps with our DBAs to move our redo logs and turn off an un-needed piece of oracle audit logging that should help with database IO Some answers for the posts: Jesus - we currently have our get list set to 5000 on user servers and 15000 on admin operations server. The 5000 is for a business purpose, but from what we heard from BMC the midtier heap can't really handle any more than 2,000 so we're going to work back with customers to see if we can decrease this. We also have unqualified searches turned off. Thanks for your help Ron - thanks for your infomration on your issue. This looks similar to our issue, I'm curious to know what processes were using the threads you increased. Thanks! Mark - thanks for the information. I took a look at our ar.conf file and see that setting on 2 out of 4 of our app servers (not sure why it is not on all 4), but will look at applying it to the other 2. LJ - thanks, i'll take a look at the other posts Phil - Thanks for the information, we have the AR Email Messages form setup to purge all entries after 1 week. Susan - thanks for your input on the rrr log. I know that Misi and team have some great products out there, just haven't had the time to research and look into them, but sounds to have great benefit. Again - thanks for everyone who has helped! ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
Thank you all for all of your responses. Sorry, I have left you a little dry, but was OOO for a while for vacation. We have looked into our private threads and thread counts and will soon be: Creating private queue for recon process Seperating RPC loop back and approval process onto their own private queue (currently sharing the same thread) We are also working on improving some table scans we've seen crop up that are increasing the IO on our database server. Also, we're taking steps with our DBAs to move our redo logs and turn off an un-needed piece of oracle audit logging that should help with database IO Some answers for the posts: Jesus - we currently have our get list set to 5000 on user servers and 15000 on admin operations server. The 5000 is for a business purpose, but from what we heard from BMC the midtier heap can't really handle any more than 2,000 so we're going to work back with customers to see if we can decrease this. We also have unqualified searches turned off. Thanks for your help Ron - thanks for your infomration on your issue. This looks similar to our issue, I'm curious to know what processes were using the threads you increased. Thanks! Mark - thanks for the information. I took a look at our ar.conf file and see that setting on 2 out of 4 of our app servers (not sure why it is not on all 4), but will look at applying it to the other 2. LJ - thanks, i'll take a look at the other posts Phil - Thanks for the information, we have the AR Email Messages form setup to purge all entries after 1 week. Susan - thanks for your input on the rrr log. I know that Misi and team have some great products out there, just haven't had the time to research and look into them, but sounds to have great benefit. Again - thanks for everyone who has helped! ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
I've seen similar symptoms on solaris before, it was resolved by setting RPC-Non-Blocking-IO: T in the ar.conf. I think you might need to ensure a couple of patches are installed as well. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
Jamie, I think the most likely of the solutions, are those presented by Ron regarding thread counts. Recently there have been some good discussions on the list regarding server statistics and how to determine if you need more threads, I would recommend checking them out -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Jamie Sent: Wednesday, April 11, 2012 11:20 AM To: arslist@ARSLIST.ORG Subject: Remedy System comes to a standstill/unresonsive We've been experiencing intermittent issues where all of a sudden our Remedy system comes to a standstill and is unresponsive to the end user for 10-15mins. The end users are experiencing this in both the client and the web. The strange thing is that there are no errors outputted to the arerror.log file and the end users are not receiving any errors such as timeouts. We have automated montioring setup to ping the IP Address every 5 secs and have not received any alerts. We have montiored the databse and haven't found really any long running queries during the time of the issues that we think is contributing to the issues. We've now turned on the normal app logging of API/SQL/Filter/Esc and opened up a ticket with BMC Support, but looking for some Out of the Box thinking from the group on where to maybe look or if anyone has had similar issues. ARS/Midtier 7.5 Patch 6 ITSM 7.5.01 Patch 1 Atrium Core 7.5 Patch 2 SLM 7.5.00 Kinetic Survey Kinetic Request 4 Sun Solaris AR Servers in a server group behind loadbalancer Oracle Database 4 Windows VM Midtier servers behind load balancer Integrations with AR servers: OIM BladeLogic/BAO IBRSD with ProactiveNet, Patrol Control M TMART Yep, we've been drinking the BMC blue kool-aid for the most part :) ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
Check if you are retaining your incoming/outgoing emails in the AR System Eamil Messages form. I have noticed that same server issue when a lot of emails are stored in the form. Phil On April 12, 2012 at 10:26 AM LJ LongWing lj.longw...@gmail.com wrote: Jamie, I think the most likely of the solutions, are those presented by Ron regarding thread counts. Recently there have been some good discussions on the list regarding server statistics and how to determine if you need more threads, I would recommend checking them out -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Jamie Sent: Wednesday, April 11, 2012 11:20 AM To: arslist@ARSLIST.ORG Subject: Remedy System comes to a standstill/unresonsive We've been experiencing intermittent issues where all of a sudden our Remedy system comes to a standstill and is unresponsive to the end user for 10-15mins. The end users are experiencing this in both the client and the web. The strange thing is that there are no errors outputted to the arerror.log file and the end users are not receiving any errors such as timeouts. We have automated montioring setup to ping the IP Address every 5 secs and have not received any alerts. We have montiored the databse and haven't found really any long running queries during the time of the issues that we think is contributing to the issues. We've now turned on the normal app logging of API/SQL/Filter/Esc and opened up a ticket with BMC Support, but looking for some Out of the Box thinking from the group on where to maybe look or if anyone has had similar issues. ARS/Midtier 7.5 Patch 6 ITSM 7.5.01 Patch 1 Atrium Core 7.5 Patch 2 SLM 7.5.00 Kinetic Survey Kinetic Request 4 Sun Solaris AR Servers in a server group behind loadbalancer Oracle Database 4 Windows VM Midtier servers behind load balancer Integrations with AR servers: OIM BladeLogic/BAO IBRSD with ProactiveNet, Patrol Control M TMART Yep, we've been drinking the BMC blue kool-aid for the most part :) ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
We were having some intermittent response issues and of course the finger always points to the application or database. We're also on Sun Solaris but ARS v7.5, Oracle 10g. I've started using rrr|Log and love, love, love that program (www.rrr.se). I can quickly see any sql statements taking longer than I'd expect and work to correct them. Although when things take 2/100's of a second I think they're in pretty good shape. But I was able to spot people doing full table scan queries and coach them (polite terminology) in how to do them properly, and the subsequently make sure they're doing it that way. It also shows thread information and lapse times. You can use the evaluation portion but for us was about 2 minutes worth of log file. It was a one minute demo to the CTO to get approval for full version. I do believe I saw a twinkle in his eye and his mouth involuntary dropped open. Good job Misi !!! Plus now I get to point to the network guys ... lol !!! Susan On Thu, Apr 12, 2012 at 10:01 AM, arsl...@remedy.us.com arsl...@remedy.us.com wrote: ** ** Check if you are retaining your incoming/outgoing emails in the AR System Eamil Messages form. I have noticed that same server issue when a lot of emails are stored in the form. Phil On April 12, 2012 at 10:26 AM LJ LongWing lj.longw...@gmail.com wrote: Jamie, I think the most likely of the solutions, are those presented by Ron regarding thread counts. Recently there have been some good discussions on the list regarding server statistics and how to determine if you need more threads, I would recommend checking them out -Original Message- From: Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] On Behalf Of Jamie Sent: Wednesday, April 11, 2012 11:20 AM To: arslist@ARSLIST.ORG Subject: Remedy System comes to a standstill/unresonsive We've been experiencing intermittent issues where all of a sudden our Remedy system comes to a standstill and is unresponsive to the end user for 10-15mins. The end users are experiencing this in both the client and the web. The strange thing is that there are no errors outputted to the arerror.log file and the end users are not receiving any errors such as timeouts. We have automated montioring setup to ping the IP Address every 5 secs and have not received any alerts. We have montiored the databse and haven't found really any long running queries during the time of the issues that we think is contributing to the issues. We've now turned on the normal app logging of API/SQL/Filter/Esc and opened up a ticket with BMC Support, but looking for some Out of the Box thinking from the group on where to maybe look or if anyone has had similar issues. ARS/Midtier 7.5 Patch 6 ITSM 7.5.01 Patch 1 Atrium Core 7.5 Patch 2 SLM 7.5.00 Kinetic Survey Kinetic Request 4 Sun Solaris AR Servers in a server group behind loadbalancer Oracle Database 4 Windows VM Midtier servers behind load balancer Integrations with AR servers: OIM BladeLogic/BAO IBRSD with ProactiveNet, Patrol Control M TMART Yep, we've been drinking the BMC blue kool-aid for the most part :) ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are _attend WWRUG12 www.wwrug.com ARSlist: Where the Answers Are_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Remedy System comes to a standstill/unresonsive
We've been experiencing intermittent issues where all of a sudden our Remedy system comes to a standstill and is unresponsive to the end user for 10-15mins. The end users are experiencing this in both the client and the web. The strange thing is that there are no errors outputted to the arerror.log file and the end users are not receiving any errors such as timeouts. We have automated montioring setup to ping the IP Address every 5 secs and have not received any alerts. We have montiored the databse and haven't found really any long running queries during the time of the issues that we think is contributing to the issues. We've now turned on the normal app logging of API/SQL/Filter/Esc and opened up a ticket with BMC Support, but looking for some Out of the Box thinking from the group on where to maybe look or if anyone has had similar issues. ARS/Midtier 7.5 Patch 6 ITSM 7.5.01 Patch 1 Atrium Core 7.5 Patch 2 SLM 7.5.00 Kinetic Survey Kinetic Request 4 Sun Solaris AR Servers in a server group behind loadbalancer Oracle Database 4 Windows VM Midtier servers behind load balancer Integrations with AR servers: OIM BladeLogic/BAO IBRSD with ProactiveNet, Patrol Control M TMART Yep, we've been drinking the BMC blue kool-aid for the most part :) ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
Jaime, Install the desktop client on the app server. When you see the issue, RDP to the app server and then start the desktop client. If you still see an issue this rules out the network. Also try connecting to the Oracle instance from your desktop. This will give an indication if it's an issue with the connection to the DB instance. Dave -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Jamie Sent: Wednesday, April 11, 2012 1:20 PM To: arslist@ARSLIST.ORG Subject: Remedy System comes to a standstill/unresonsive We've been experiencing intermittent issues where all of a sudden our Remedy system comes to a standstill and is unresponsive to the end user for 10-15mins. The end users are experiencing this in both the client and the web. The strange thing is that there are no errors outputted to the arerror.log file and the end users are not receiving any errors such as timeouts. We have automated montioring setup to ping the IP Address every 5 secs and have not received any alerts. We have montiored the databse and haven't found really any long running queries during the time of the issues that we think is contributing to the issues. We've now turned on the normal app logging of API/SQL/Filter/Esc and opened up a ticket with BMC Support, but looking for some Out of the Box thinking from the group on where to maybe look or if anyone has had similar issues. ARS/Midtier 7.5 Patch 6 ITSM 7.5.01 Patch 1 Atrium Core 7.5 Patch 2 SLM 7.5.00 Kinetic Survey Kinetic Request 4 Sun Solaris AR Servers in a server group behind loadbalancer Oracle Database 4 Windows VM Midtier servers behind load balancer Integrations with AR servers: OIM BladeLogic/BAO IBRSD with ProactiveNet, Patrol Control M TMART Yep, we've been drinking the BMC blue kool-aid for the most part :) ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
Try to look at your getlist setting in AR System Administration\server information\config\Max entries for Getlist. If you have it set to 0, that could be your issue. I have it set to 1. Also, make sure that Allow Unqualified Searches is unchecked. Those two things taken together cause unresponsiveness. Are you having any issues with the mid-tier locking up? Jesus Ortega Senior II, Implementation Engineer LyondellBasell Industries Office: 713 309-4914 Cell: 281 546-0735 -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Jamie Sent: Wednesday, April 11, 2012 12:20 PM To: arslist@ARSLIST.ORG Subject: Remedy System comes to a standstill/unresonsive We've been experiencing intermittent issues where all of a sudden our Remedy system comes to a standstill and is unresponsive to the end user for 10-15mins. The end users are experiencing this in both the client and the web. The strange thing is that there are no errors outputted to the arerror.log file and the end users are not receiving any errors such as timeouts. We have automated montioring setup to ping the IP Address every 5 secs and have not received any alerts. We have montiored the databse and haven't found really any long running queries during the time of the issues that we think is contributing to the issues. We've now turned on the normal app logging of API/SQL/Filter/Esc and opened up a ticket with BMC Support, but looking for some Out of the Box thinking from the group on where to maybe look or if anyone has had similar issues. ARS/Midtier 7.5 Patch 6 ITSM 7.5.01 Patch 1 Atrium Core 7.5 Patch 2 SLM 7.5.00 Kinetic Survey Kinetic Request 4 Sun Solaris AR Servers in a server group behind loadbalancer Oracle Database 4 Windows VM Midtier servers behind load balancer Integrations with AR servers: OIM BladeLogic/BAO IBRSD with ProactiveNet, Patrol Control M TMART Yep, we've been drinking the BMC blue kool-aid for the most part :) ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are Information contained in this email is subject to the disclaimer found by clicking on the following link: http://www.lyondellbasell.com/Footer/Disclaimer/ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
Are you performing any of these activities at the time these delays seem to happen? : 1) Modifying or making changes to any of the large data tables from the Admin or the Development Studio? 2) Creating new 'permission groups' (View or Change groups) 3) Having a short interval for a Mid-Tier recache? 4) Expensive Escalations running on intervals? 5) Constantly creating new or modifying CMDB classes? 6) AIE engine polling activity? These are things I can think of at the top of my head which can lock up users.. Joe -Original Message- From: Jamie Sent: Wednesday, April 11, 2012 1:19 PM Newsgroups: public.remedy.arsystem.general To: arslist@ARSLIST.ORG Subject: Remedy System comes to a standstill/unresonsive We've been experiencing intermittent issues where all of a sudden our Remedy system comes to a standstill and is unresponsive to the end user for 10-15mins. The end users are experiencing this in both the client and the web. The strange thing is that there are no errors outputted to the arerror.log file and the end users are not receiving any errors such as timeouts. We have automated montioring setup to ping the IP Address every 5 secs and have not received any alerts. We have montiored the databse and haven't found really any long running queries during the time of the issues that we think is contributing to the issues. We've now turned on the normal app logging of API/SQL/Filter/Esc and opened up a ticket with BMC Support, but looking for some Out of the Box thinking from the group on where to maybe look or if anyone has had similar issues. ARS/Midtier 7.5 Patch 6 ITSM 7.5.01 Patch 1 Atrium Core 7.5 Patch 2 SLM 7.5.00 Kinetic Survey Kinetic Request 4 Sun Solaris AR Servers in a server group behind loadbalancer Oracle Database 4 Windows VM Midtier servers behind load balancer Integrations with AR servers: OIM BladeLogic/BAO IBRSD with ProactiveNet, Patrol Control M TMART Yep, we've been drinking the BMC blue kool-aid for the most part :) ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
All good questions Joe and thanks for contributing to our issue. I should have also included, that we are pretty much 95% out of the box and do not do very much Remedy workflow changes unless it's a patch or hotfix. Which non have been applied recently. Below are answers to your questions: 1) No changes are being done to data tables or anything in dev studio 2) No groups have been created, modified, or renamed 3) We have Definition Check inteval turned off 4) I have reviewed the escalation log during the time, and no significant log running esclations that would cause any issues. We also have escalations running on it's on queue 5) No CMDB classes have been created outside of the out of box classes along time ago 6) We are not using AIE for anything, is there a way I can check this to make sure nothing is being tripped? ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
We had a similar issue start cropping up in December with more in January and February. We're on 7.5 and as much OOTB as possible. It started by being unresponsive for ~15-20min and then got longer. We ended up needing to restart the AR server service to get the system back which worked fine but was extremely annoying. Support helped us work through the problem and though we still don't know the root cause of why it started happening, the resolution was some updates to the ar.cfg. This is what we changed: Here is the diff from before to after Private-RPC-Socket: 390620 2 6 Private-RPC-Socket: 390620 4 10 Private-RPC-Socket: 390635 2 3 Private-RPC-Socket: 390635 4 10 # This line was added Private-RPC-Socket: 390626 2 3 Cache-Mode: 1 Cache-Mode: 0 Hope it's helpful. -Original Message- From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Jamie Sent: Wednesday, April 11, 2012 11:28 AM To: arslist@ARSLIST.ORG Subject: Re: Remedy System comes to a standstill/unresonsive All good questions Joe and thanks for contributing to our issue. I should have also included, that we are pretty much 95% out of the box and do not do very much Remedy workflow changes unless it's a patch or hotfix. Which non have been applied recently. Below are answers to your questions: 1) No changes are being done to data tables or anything in dev studio 2) No groups have been created, modified, or renamed 3) We have Definition Check inteval turned off 4) I have reviewed the escalation log during the time, and no significant log running esclations that would cause any issues. We also have escalations running on it's on queue 5) No CMDB classes have been created outside of the out of box classes along time ago 6) We are not using AIE for anything, is there a way I can check this to make sure nothing is being tripped? ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
Re: Remedy System comes to a standstill/unresonsive
Jamie, What do your SQL/API (combination of these into a single file) logs show? Any obvious latencies? Joe -Original Message- From: Jamie Sent: Wednesday, April 11, 2012 2:27 PM Newsgroups: public.remedy.arsystem.general To: arslist@ARSLIST.ORG Subject: Re: Remedy System comes to a standstill/unresonsive All good questions Joe and thanks for contributing to our issue. I should have also included, that we are pretty much 95% out of the box and do not do very much Remedy workflow changes unless it's a patch or hotfix. Which non have been applied recently. Below are answers to your questions: 1) No changes are being done to data tables or anything in dev studio 2) No groups have been created, modified, or renamed 3) We have Definition Check inteval turned off 4) I have reviewed the escalation log during the time, and no significant log running esclations that would cause any issues. We also have escalations running on it's on queue 5) No CMDB classes have been created outside of the out of box classes along time ago 6) We are not using AIE for anything, is there a way I can check this to make sure nothing is being tripped? ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org attend wwrug12 www.wwrug12.com ARSList: Where the Answers Are
