ADV: MonitorRemedy from Panacea WAS Open question to BMC on troubleshooting without the WUT
Hi Larry and others,
Troubleshooting without the WUT in a Server Group environment is one of the
use-cases the MonitorRemedy product is designed for. Managing load balanced
server group environments can be a complicated and a time consuming process
Typically, Remedy specialists have to trawl around multiple Mid-Tier and AR
Servers in order to ascertain which server or servers are the problematic ones.
Diagnosing issues involves logging in to each server and looking at resource
usage and multiple logs files.
A unique user search feature of MonitorRemedy allows you to identify which
users are connected to which Mid-tier and AR Servers – allowing specialists to
quickly home in on problematic Mid-tier or AR Servers.
The MonitorRemedy product also provides a specialist graphical view of your
application's health focused on your BMC Remedy AR System and Application
installations.
A graphical warning system lets you quickly identify which Application
function, AR System, Mid-tier or Load Balancer components of your application
are either down or trending towards a potential issue. Drill down features for
example allow you to see:
· If key application components are working (e.g. approvals, email)
· Trends on resource usage , standard query and submit times.
· Log files of all your servers
· Real time ARSystem and Application licence usage.
Setup time for monitoring tools can be time consuming. MonitorRemedy is a
dynamic web application easily deployed to an Apache Tomcat server. The
configuration interface allows you to define the additional applications to be
monitored in minutes, allowing rapid product deployment.
BMC Engage 2014 pricing applies until 17th November 2014 and is £5k GBP per
environment monitored.
You can learn more online at http://www.pws-europe.com/products/MonitorRemedy
Thanks,
Deepak Thohan
cid:image001.gif@01C7FECB.9C4B3860
deepak.tho...@pws-europe.com
_
Telephone: +44 1784 497 047 | Mobile: +44 7875 009519
Fax: +44 1784 497 048 | Skype: deepak.thohan
http://www.pws-europe.com/products/MonitorRemedy
http://www.pws-europe.com/products/ShareRemedy
http://www.pws-europe.com/products/Workflow%20Studio
ShareRemedy | Leverage your investment in Remedy and SharePoint
AnalyseRemedy | The unique Migration, Analysis and Maintenance Tool
http://www.pws-europe.com/products/MobileRemedy MonitorRemedy | BMC Remedy
Environment Monitoring Solution
MobileRemedy| BMC Remedy On Your Mobile
On Wed, Oct 22, 2014 at 5:17 PM, Larry Robinson n...@ncsu.edu wrote:
**
Hi Folks,
John Baker was kind enough to let me know that the script I posted had some
potential security vulnerabilities. He provided the following alternative:
%@ page import=java.net.* %
htmlhead/headbodyp
%
String myhostname= null;
try {
myhostname= InetAddress.getLocalHost().getHostName();
} catch (UnknownHostException e) {
}
if (myhostname!=null) { %
This application server is running on hostname %= myhostname %.
% } %
/p/body/html
Works great. Thanks John!
Larry
On Thu, Oct 16, 2014 at 11:44 PM, Larry Robinson n...@ncsu.edu wrote:
Hi William,
I too receive problem reports from our Mid-tier users and struggled to
determine which server they were connected to. I asked one of my JSP experts to
write us a program that could run from the browser that would emit the name and
IP of the Tomcat server that was serving the users session. We use LVS as our
load balancer, which is IP based so once you are locked onto a server, you stay
there.
I put this JSP program in the Tomcat webapps/ROOT directory and ask the user to
invoke it as:
https://myservicename/server_id.jsp
and have them tell me which server they are locked onto.
Here is the code:
server_id.jsp:
%@ page import=java.util.*,java.io.* %
%
String cmd = hostname;
String outstr = ;
try {
Runtime rt = Runtime.getRuntime();
Process p = rt.exec(cmd);
try {
InputStreamReader ise = new InputStreamReader(p.getErrorStream());
BufferedReader bre = new BufferedReader(ise);
InputStreamReader iso = new InputStreamReader(p.getInputStream());
BufferedReader bro = new BufferedReader(iso);
String line=null;
while ( (line = bre.readLine()) != null ) {
outstr += line;
}
while ( (line = bro.readLine()) != null ) {
outstr += line;
}
}
catch (IOException ioe) {
ioe.printStackTrace();
}
int exitVal = p.waitFor();
}
catch (Throwable t) {
t.printStackTrace();
}
%
br
Hostname: nbsp;nbsp;nbsp; strong%=outstr%/strongbr
Your IP:
nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;strong%=request.getRemoteAddr()%/strongbr
Service IP: nbsp;nbsp;nbsp;strong%=request.getLocalAddr()%/strongbr
Hope this is helpful.
Larry
On Thu, Oct 16, 2014 at 1:02 PM, William Rentfrow wrentf...@stratacominc.com
wrote:
**
I sent this to our premiere
Re: Open question to BMC on troubleshooting without the WUT
Hi Folks,
John Baker was kind enough to let me know that the script I posted had some
potential security vulnerabilities. He provided the following alternative:
%@ page import=java.net.* %
htmlhead/headbodyp
%
String myhostname= null;
try {
myhostname= InetAddress.getLocalHost().getHostName();
} catch (UnknownHostException e) {
}
if (myhostname!=null) { %
This application server is running on hostname %= myhostname %.
% } %
/p/body/html
Works great. Thanks John!
Larry
On Thu, Oct 16, 2014 at 11:44 PM, Larry Robinson n...@ncsu.edu wrote:
Hi William,
I too receive problem reports from our Mid-tier users and struggled to
determine which server they were connected to. I asked one of my JSP
experts to write us a program that could run from the browser that would
emit the name and IP of the Tomcat server that was serving the users
session. We use LVS as our load balancer, which is IP based so once you are
locked onto a server, you stay there.
I put this JSP program in the Tomcat webapps/ROOT directory and ask the
user to invoke it as:
https://myservicename/s*erver_id.jsp*
and have them tell me which server they are locked onto.
Here is the code:
*server_id.jsp*:
%@ page import=java.util.*,java.io.* %
%
String cmd = hostname;
String outstr = ;
try {
Runtime rt = Runtime.getRuntime();
Process p = rt.exec(cmd);
try {
InputStreamReader ise = new InputStreamReader(p.getErrorStream());
BufferedReader bre = new BufferedReader(ise);
InputStreamReader iso = new InputStreamReader(p.getInputStream());
BufferedReader bro = new BufferedReader(iso);
String line=null;
while ( (line = bre.readLine()) != null ) {
outstr += line;
}
while ( (line = bro.readLine()) != null ) {
outstr += line;
}
}
catch (IOException ioe) {
ioe.printStackTrace();
}
int exitVal = p.waitFor();
}
catch (Throwable t) {
t.printStackTrace();
}
%
br
Hostname: nbsp;nbsp;nbsp; strong%=outstr%/strongbr
Your IP:
nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;strong%=request.getRemoteAddr()%/strongbr
Service IP:
nbsp;nbsp;nbsp;strong%=request.getLocalAddr()%/strongbr
Hope this is helpful.
Larry
On Thu, Oct 16, 2014 at 1:02 PM, William Rentfrow
wrentf...@stratacominc.com wrote:
**
I sent this to our premiere support person and manager, but I'd be
interested to see what others have to say about this too.
Original message below:
Hi –
This came up on our call and I wanted to write it out.
BMC has stated that the Windows User Tool (WUT) is going to be
discontinued (in fact, it already was in 7.6x). What I need to know is
what is BMC’s recommendation for diagnosing problems with the AR Servers in
a server group?
Currently our users will report an issue like this: “Remedy is
slow/locked up/whatever”. Routinely we get no more information than this.
Right now our troubleshooting is to first diagnose which server(s) is
having problems. The fast way to do this is to login to every server with
the user tool. We usually know within a few seconds if one of the AR
servers is locked up, because we will not be able to log in to it. Then we
can bounce it and get service restored.
If they are responsive we then move on the Mid-tier servers, etc.
With a large load-balanced environment there is no way to * QUICKLY* do
this without the WUT. I could login with Developer Studio, but that
doesn’t use the same threads on the server as the WUT does. We have seen
instances where users are locked up and admins can log in with Dev studio
(and vice versa). Same goes for migrator and the import tool.
Support suggesting checking the AR Error log, but there are two problems
with that – first, many lock-up scenarios do not results in errors in the
arerror.log file. There are numerous other logs to check on every server
as well (CMDB, Email, AIE, etc). Checking every log file on every server
is time consuming and not 100% guaranteed to show us which server is locked
up.
The second problem with support’s suggestion is the sheer time it would
take to login to each server. We are on Linux, so we need to connect via
SSH using putty. We do that by first connecting to a gateway server. Then
we ssh to the actual AR server (direct access is not allowed). Finally, we
sudo to the user Remedy is running as. That means each time we connect
it’s 3X we login. If we multiply that by the 10 servers in our server
group it would take at least an hour just to triage the problem.
I can do the same thing with the WUT in seconds.
So here is the question: What is the proper way to QUICKLY triage which
server is having problems without using the WUT or Dev
Studio/Migrator/Import?
William Rentfrow
_ARSlist: Where the Answers Are and have been for 20 years_
___
UNSUBSCRIBE or access ARSlist
Open question to BMC on troubleshooting without the WUT
I've not tested this:
%@ page import=java.net.* %
htmlhead/headbodyp
%
String myhostname= null;
try {
myhostname= InetAddress.getLocalHost().getHostName();
} catch (UnknownHostException e) {
}
if (myhostname!=null) { %
This application server is running on hostname %= myhostname %.
% } %
/p/body/html
John
--
SSO Plugin for the BMC product set
http://www.javasystemsolutions.com/jss/ssoplugin
___
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
I'd suggest using the driver or JavaDriver utility that is included with the server. You can script this to login and issue any API call - you also get the option to set the RPC queue so you could create a test that logs and tests several different queues in case you're just using up all of one thread type. Mark From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: 16 October 2014 18:02 To: arslist@ARSLIST.ORG Subject: Open question to BMC on troubleshooting without the WUT ** I sent this to our premiere support person and manager, but I'd be interested to see what others have to say about this too. Original message below: Hi - This came up on our call and I wanted to write it out. BMC has stated that the Windows User Tool (WUT) is going to be discontinued (in fact, it already was in 7.6x). What I need to know is what is BMC's recommendation for diagnosing problems with the AR Servers in a server group? Currently our users will report an issue like this: Remedy is slow/locked up/whatever. Routinely we get no more information than this. Right now our troubleshooting is to first diagnose which server(s) is having problems. The fast way to do this is to login to every server with the user tool. We usually know within a few seconds if one of the AR servers is locked up, because we will not be able to log in to it. Then we can bounce it and get service restored. If they are responsive we then move on the Mid-tier servers, etc. With a large load-balanced environment there is no way to QUICKLY do this without the WUT. I could login with Developer Studio, but that doesn't use the same threads on the server as the WUT does. We have seen instances where users are locked up and admins can log in with Dev studio (and vice versa). Same goes for migrator and the import tool. Support suggesting checking the AR Error log, but there are two problems with that - first, many lock-up scenarios do not results in errors in the arerror.log file. There are numerous other logs to check on every server as well (CMDB, Email, AIE, etc). Checking every log file on every server is time consuming and not 100% guaranteed to show us which server is locked up. The second problem with support's suggestion is the sheer time it would take to login to each server. We are on Linux, so we need to connect via SSH using putty. We do that by first connecting to a gateway server. Then we ssh to the actual AR server (direct access is not allowed). Finally, we sudo to the user Remedy is running as. That means each time we connect it's 3X we login. If we multiply that by the 10 servers in our server group it would take at least an hour just to triage the problem. I can do the same thing with the WUT in seconds. So here is the question: What is the proper way to QUICKLY triage which server is having problems without using the WUT or Dev Studio/Migrator/Import? William Rentfrow _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Thanks John... I'll certainly pass that on. Larry On Fri, Oct 17, 2014 at 1:26 AM, John Baker jba...@javasystemsolutions.com wrote: Larry I wouldn't suggest using that JSP :) It is running a native application (hostname) to get the hostname that is readily available from a Java API call. Running native applications isn't going to do the performance of your Mid Tier any good, and anyone with a copy of wget can almost certainly kill your Mid Tier pretty quickly. The Java InetAddress API is what you should be using for this task. John ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Hi John, As it turns out, I no longer have access to the author of the program I posted. Could elaborate on what the vulnerability is and how to implement the same functionality in a more secure manner? Thanks for your insights. Larry On Fri, Oct 17, 2014 at 1:26 AM, John Baker jba...@javasystemsolutions.com wrote: Larry I wouldn't suggest using that JSP :) It is running a native application (hostname) to get the hostname that is readily available from a Java API call. Running native applications isn't going to do the performance of your Mid Tier any good, and anyone with a copy of wget can almost certainly kill your Mid Tier pretty quickly. The Java InetAddress API is what you should be using for this task. John ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Open question to BMC on troubleshooting without the WUT
I sent this to our premiere support person and manager, but I'd be interested to see what others have to say about this too. Original message below: Hi - This came up on our call and I wanted to write it out. BMC has stated that the Windows User Tool (WUT) is going to be discontinued (in fact, it already was in 7.6x). What I need to know is what is BMC's recommendation for diagnosing problems with the AR Servers in a server group? Currently our users will report an issue like this: Remedy is slow/locked up/whatever. Routinely we get no more information than this. Right now our troubleshooting is to first diagnose which server(s) is having problems. The fast way to do this is to login to every server with the user tool. We usually know within a few seconds if one of the AR servers is locked up, because we will not be able to log in to it. Then we can bounce it and get service restored. If they are responsive we then move on the Mid-tier servers, etc. With a large load-balanced environment there is no way to QUICKLY do this without the WUT. I could login with Developer Studio, but that doesn't use the same threads on the server as the WUT does. We have seen instances where users are locked up and admins can log in with Dev studio (and vice versa). Same goes for migrator and the import tool. Support suggesting checking the AR Error log, but there are two problems with that - first, many lock-up scenarios do not results in errors in the arerror.log file. There are numerous other logs to check on every server as well (CMDB, Email, AIE, etc). Checking every log file on every server is time consuming and not 100% guaranteed to show us which server is locked up. The second problem with support's suggestion is the sheer time it would take to login to each server. We are on Linux, so we need to connect via SSH using putty. We do that by first connecting to a gateway server. Then we ssh to the actual AR server (direct access is not allowed). Finally, we sudo to the user Remedy is running as. That means each time we connect it's 3X we login. If we multiply that by the 10 servers in our server group it would take at least an hour just to triage the problem. I can do the same thing with the WUT in seconds. So here is the question: What is the proper way to QUICKLY triage which server is having problems without using the WUT or Dev Studio/Migrator/Import? William Rentfrow ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Has anyone tried doing what William is asking for with BMC's BPPM suite? On paper that looks like the way to go, but I wonder if the LOE to set it up is worth the cost. From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Thursday, October 16, 2014 10:02 AM To: arslist@ARSLIST.ORG Subject: Open question to BMC on troubleshooting without the WUT ** I sent this to our premiere support person and manager, but I'd be interested to see what others have to say about this too. Original message below: Hi - This came up on our call and I wanted to write it out. BMC has stated that the Windows User Tool (WUT) is going to be discontinued (in fact, it already was in 7.6x). What I need to know is what is BMC's recommendation for diagnosing problems with the AR Servers in a server group? Currently our users will report an issue like this: Remedy is slow/locked up/whatever. Routinely we get no more information than this. Right now our troubleshooting is to first diagnose which server(s) is having problems. The fast way to do this is to login to every server with the user tool. We usually know within a few seconds if one of the AR servers is locked up, because we will not be able to log in to it. Then we can bounce it and get service restored. If they are responsive we then move on the Mid-tier servers, etc. With a large load-balanced environment there is no way to QUICKLY do this without the WUT. I could login with Developer Studio, but that doesn't use the same threads on the server as the WUT does. We have seen instances where users are locked up and admins can log in with Dev studio (and vice versa). Same goes for migrator and the import tool. Support suggesting checking the AR Error log, but there are two problems with that - first, many lock-up scenarios do not results in errors in the arerror.log file. There are numerous other logs to check on every server as well (CMDB, Email, AIE, etc). Checking every log file on every server is time consuming and not 100% guaranteed to show us which server is locked up. The second problem with support's suggestion is the sheer time it would take to login to each server. We are on Linux, so we need to connect via SSH using putty. We do that by first connecting to a gateway server. Then we ssh to the actual AR server (direct access is not allowed). Finally, we sudo to the user Remedy is running as. That means each time we connect it's 3X we login. If we multiply that by the 10 servers in our server group it would take at least an hour just to triage the problem. I can do the same thing with the WUT in seconds. So here is the question: What is the proper way to QUICKLY triage which server is having problems without using the WUT or Dev Studio/Migrator/Import? William Rentfrow _ARSlist: Where the Answers Are and have been for 20 years_ Please consider the environment before printing this email and any attachments. This e-mail and any attachments are intended only for the individual or company to which it is addressed and may contain information which is privileged, confidential and prohibited from disclosure or unauthorized use under applicable law. If you are not the intended recipient of this e-mail, you are hereby notified that any use, dissemination, or copying of this e-mail or the information contained in this e-mail is strictly prohibited by the sender. If you have received this transmission in error, please return the material received to the sender and delete all copies from your system. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Write a JSP that connects via the Java API to each machine directly. Done. Get lunch. -John On Thu, Oct 16, 2014 at 1:02 PM, William Rentfrow wrentf...@stratacominc.com wrote: ** I sent this to our premiere support person and manager, but I'd be interested to see what others have to say about this too. Original message below: Hi – This came up on our call and I wanted to write it out. BMC has stated that the Windows User Tool (WUT) is going to be discontinued (in fact, it already was in 7.6x). What I need to know is what is BMC’s recommendation for diagnosing problems with the AR Servers in a server group? Currently our users will report an issue like this: “Remedy is slow/locked up/whatever”. Routinely we get no more information than this. Right now our troubleshooting is to first diagnose which server(s) is having problems. The fast way to do this is to login to every server with the user tool. We usually know within a few seconds if one of the AR servers is locked up, because we will not be able to log in to it. Then we can bounce it and get service restored. If they are responsive we then move on the Mid-tier servers, etc. With a large load-balanced environment there is no way to * QUICKLY* do this without the WUT. I could login with Developer Studio, but that doesn’t use the same threads on the server as the WUT does. We have seen instances where users are locked up and admins can log in with Dev studio (and vice versa). Same goes for migrator and the import tool. Support suggesting checking the AR Error log, but there are two problems with that – first, many lock-up scenarios do not results in errors in the arerror.log file. There are numerous other logs to check on every server as well (CMDB, Email, AIE, etc). Checking every log file on every server is time consuming and not 100% guaranteed to show us which server is locked up. The second problem with support’s suggestion is the sheer time it would take to login to each server. We are on Linux, so we need to connect via SSH using putty. We do that by first connecting to a gateway server. Then we ssh to the actual AR server (direct access is not allowed). Finally, we sudo to the user Remedy is running as. That means each time we connect it’s 3X we login. If we multiply that by the 10 servers in our server group it would take at least an hour just to triage the problem. I can do the same thing with the WUT in seconds. So here is the question: What is the proper way to QUICKLY triage which server is having problems without using the WUT or Dev Studio/Migrator/Import? William Rentfrow _ARSlist: Where the Answers Are and have been for 20 years_ -- *John Sundberg* Kinetic Data, Inc. Your Business. Your Process. 651-556-0930 I john.sundb...@kineticdata.com www.kineticdata.com I community.kineticdata.com ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Even though support for the WUT was discontinued after 7.6.4 you can still use it to log into the servers (unless you set the Minimum API version value). Fred From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Thursday, October 16, 2014 12:02 PM To: arslist@ARSLIST.ORG Subject: Open question to BMC on troubleshooting without the WUT ** I sent this to our premiere support person and manager, but I'd be interested to see what others have to say about this too. Original message below: Hi - This came up on our call and I wanted to write it out. BMC has stated that the Windows User Tool (WUT) is going to be discontinued (in fact, it already was in 7.6x). What I need to know is what is BMC's recommendation for diagnosing problems with the AR Servers in a server group? Currently our users will report an issue like this: Remedy is slow/locked up/whatever. Routinely we get no more information than this. Right now our troubleshooting is to first diagnose which server(s) is having problems. The fast way to do this is to login to every server with the user tool. We usually know within a few seconds if one of the AR servers is locked up, because we will not be able to log in to it. Then we can bounce it and get service restored. If they are responsive we then move on the Mid-tier servers, etc. With a large load-balanced environment there is no way to QUICKLY do this without the WUT. I could login with Developer Studio, but that doesn't use the same threads on the server as the WUT does. We have seen instances where users are locked up and admins can log in with Dev studio (and vice versa). Same goes for migrator and the import tool. Support suggesting checking the AR Error log, but there are two problems with that - first, many lock-up scenarios do not results in errors in the arerror.log file. There are numerous other logs to check on every server as well (CMDB, Email, AIE, etc). Checking every log file on every server is time consuming and not 100% guaranteed to show us which server is locked up. The second problem with support's suggestion is the sheer time it would take to login to each server. We are on Linux, so we need to connect via SSH using putty. We do that by first connecting to a gateway server. Then we ssh to the actual AR server (direct access is not allowed). Finally, we sudo to the user Remedy is running as. That means each time we connect it's 3X we login. If we multiply that by the 10 servers in our server group it would take at least an hour just to triage the problem. I can do the same thing with the WUT in seconds. So here is the question: What is the proper way to QUICKLY triage which server is having problems without using the WUT or Dev Studio/Migrator/Import? William Rentfrow ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
John - You're right - a custom solution is pretty easy, but it kind of proves my point: there's no easy way to do it with the BMC tools. William Rentfrow From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of John Sundberg Sent: Thursday, October 16, 2014 12:20 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** Write a JSP that connects via the Java API to each machine directly. Done. Get lunch. -John On Thu, Oct 16, 2014 at 1:02 PM, William Rentfrow wrentf...@stratacominc.commailto:wrentf...@stratacominc.com wrote: ** I sent this to our premiere support person and manager, but I'd be interested to see what others have to say about this too. Original message below: Hi – This came up on our call and I wanted to write it out. BMC has stated that the Windows User Tool (WUT) is going to be discontinued (in fact, it already was in 7.6x). What I need to know is what is BMC’s recommendation for diagnosing problems with the AR Servers in a server group? Currently our users will report an issue like this: “Remedy is slow/locked up/whatever”. Routinely we get no more information than this. Right now our troubleshooting is to first diagnose which server(s) is having problems. The fast way to do this is to login to every server with the user tool. We usually know within a few seconds if one of the AR servers is locked up, because we will not be able to log in to it. Then we can bounce it and get service restored. If they are responsive we then move on the Mid-tier servers, etc. With a large load-balanced environment there is no way to QUICKLY do this without the WUT. I could login with Developer Studio, but that doesn’t use the same threads on the server as the WUT does. We have seen instances where users are locked up and admins can log in with Dev studio (and vice versa). Same goes for migrator and the import tool. Support suggesting checking the AR Error log, but there are two problems with that – first, many lock-up scenarios do not results in errors in the arerror.log file. There are numerous other logs to check on every server as well (CMDB, Email, AIE, etc). Checking every log file on every server is time consuming and not 100% guaranteed to show us which server is locked up. The second problem with support’s suggestion is the sheer time it would take to login to each server. We are on Linux, so we need to connect via SSH using putty. We do that by first connecting to a gateway server. Then we ssh to the actual AR server (direct access is not allowed). Finally, we sudo to the user Remedy is running as. That means each time we connect it’s 3X we login. If we multiply that by the 10 servers in our server group it would take at least an hour just to triage the problem. I can do the same thing with the WUT in seconds. So here is the question: What is the proper way to QUICKLY triage which server is having problems without using the WUT or Dev Studio/Migrator/Import? William Rentfrow _ARSlist: Where the Answers Are and have been for 20 years_ -- John Sundberg Kinetic Data, Inc. Your Business. Your Process. 651-556-0930 I john.sundb...@kineticdata.commailto:john.sundb...@kineticdata.com www.kineticdata.comhttp://www.kineticdata.com/ I community.kineticdata.comhttp://community.kineticdata.com/ _ARSlist: Where the Answers Are and have been for 20 years_ No virus found in this message. Checked by AVG - www.avg.comhttp://www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner -- The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Arner, Todd Sent: Thursday, October 16, 2014 12:56 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.orgmailto:no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. No virus found in this message. Checked by AVG - www.avg.comhttp://www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
How about issuing one of the command line utilities to each of the servers such as maybe the arsignal -g that reloads the def cache. If a server is not responding you would get a error 90. I do not remember all the switches to arsignal offhand but there should be one that has less of an impact to a server than -g if you are worried about the performance impact it may have to servers that are responding well. Or you could try to arcreateentry into a test form. Joe _ From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Thursday, October 16, 2014 2:00 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Arner, Todd Sent: Thursday, October 16, 2014 12:56 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner _ The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. _ No virus found in this message. Checked by AVG - www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Good ideas, Joe! Perhaps the arserver -v (or whatever it is now) that returns the version would fit the bill - quick return, confirms that the server is up, and almost zero performance impact. Rick Cook On Thu, Oct 16, 2014 at 11:19 AM, Joe D'Souza jdso...@shyle.net wrote: ** How about issuing one of the command line utilities to each of the servers such as maybe the arsignal –g that reloads the def cache. If a server is not responding you would get a error 90. I do not remember all the switches to arsignal offhand but there should be one that has less of an impact to a server than –g if you are worried about the performance impact it may have to servers that are responding well. Or you could try to arcreateentry into a test form. Joe -- *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *William Rentfrow *Sent:* Thursday, October 16, 2014 2:00 PM *To:* arslist@ARSLIST.ORG *Subject:* Re: Open question to BMC on troubleshooting without the WUT Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Arner, Todd *Sent:* Thursday, October 16, 2014 12:56 PM *To:* arslist@ARSLIST.ORG *Subject:* Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner -- The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. -- No virus found in this message. Checked by AVG - www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
I really just wish that when users do become hung, in the load balanced web/server group environment, and it is one of the servers which is unresponsive - that there would be some way that the server group would recognize it and restart the armonitor processes... OR at least place something in the log files which directs you to WHY the server locked up!! On Thu, Oct 16, 2014 at 2:21 PM, Rick Cook remedyr...@gmail.com wrote: ** Good ideas, Joe! Perhaps the arserver -v (or whatever it is now) that returns the version would fit the bill - quick return, confirms that the server is up, and almost zero performance impact. Rick Cook On Thu, Oct 16, 2014 at 11:19 AM, Joe D'Souza jdso...@shyle.net wrote: ** How about issuing one of the command line utilities to each of the servers such as maybe the arsignal –g that reloads the def cache. If a server is not responding you would get a error 90. I do not remember all the switches to arsignal offhand but there should be one that has less of an impact to a server than –g if you are worried about the performance impact it may have to servers that are responding well. Or you could try to arcreateentry into a test form. Joe -- *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *William Rentfrow *Sent:* Thursday, October 16, 2014 2:00 PM *To:* arslist@ARSLIST.ORG *Subject:* Re: Open question to BMC on troubleshooting without the WUT Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Arner, Todd *Sent:* Thursday, October 16, 2014 12:56 PM *To:* arslist@ARSLIST.ORG *Subject:* Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner -- The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. -- No virus found in this message. Checked by AVG - www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Thanks Rick, Almost forgot about the -v option. Yes that would be far quicker than even the user tool. That would be the least of impact. Joe _ From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Rick Cook Sent: Thursday, October 16, 2014 2:22 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** Good ideas, Joe! Perhaps the arserver -v (or whatever it is now) that returns the version would fit the bill - quick return, confirms that the server is up, and almost zero performance impact. Rick Cook On Thu, Oct 16, 2014 at 11:19 AM, Joe D'Souza jdso...@shyle.net wrote: ** How about issuing one of the command line utilities to each of the servers such as maybe the arsignal -g that reloads the def cache. If a server is not responding you would get a error 90. I do not remember all the switches to arsignal offhand but there should be one that has less of an impact to a server than -g if you are worried about the performance impact it may have to servers that are responding well. Or you could try to arcreateentry into a test form. Joe _ From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Thursday, October 16, 2014 2:00 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Arner, Todd Sent: Thursday, October 16, 2014 12:56 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner _ The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. _ No virus found in this message. Checked by AVG - www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
If you are on UNIX or LINUX you might be able to design a cron job that runs every few minutes to do that using the output of arsignal -v. Joe _ From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Rob Dudley Sent: Thursday, October 16, 2014 2:31 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** I really just wish that when users do become hung, in the load balanced web/server group environment, and it is one of the servers which is unresponsive - that there would be some way that the server group would recognize it and restart the armonitor processes... OR at least place something in the log files which directs you to WHY the server locked up!! On Thu, Oct 16, 2014 at 2:21 PM, Rick Cook remedyr...@gmail.com wrote: ** Good ideas, Joe! Perhaps the arserver -v (or whatever it is now) that returns the version would fit the bill - quick return, confirms that the server is up, and almost zero performance impact. Rick Cook On Thu, Oct 16, 2014 at 11:19 AM, Joe D'Souza jdso...@shyle.net wrote: ** How about issuing one of the command line utilities to each of the servers such as maybe the arsignal -g that reloads the def cache. If a server is not responding you would get a error 90. I do not remember all the switches to arsignal offhand but there should be one that has less of an impact to a server than -g if you are worried about the performance impact it may have to servers that are responding well. Or you could try to arcreateentry into a test form. Joe _ From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Thursday, October 16, 2014 2:00 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Arner, Todd Sent: Thursday, October 16, 2014 12:56 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner _ The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. _ No virus found in this message. Checked by AVG - www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
I think the issue is that you are not testing the same process that the users are utilizing. That would increase the chance of false positives. That can be more frustrating than taking the time to do it manually. The best answer that I can see is to write a java program that can run and monitor the servers and return (or react to) a problem. But that is not the original request. It was something created and supported by BMC. I could write something fairly quickly, but that isn't the solution he was looking for. Brian Goralczyk Brian Goralczyk Phone 574-643-1144 Email bgoralc...@gmail.com On Thu, Oct 16, 2014 at 2:52 PM, Joe D'Souza jdso...@shyle.net wrote: ** If you are on UNIX or LINUX you might be able to design a cron job that runs every few minutes to do that using the output of arsignal –v. Joe -- *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Rob Dudley *Sent:* Thursday, October 16, 2014 2:31 PM *To:* arslist@ARSLIST.ORG *Subject:* Re: Open question to BMC on troubleshooting without the WUT ** I really just wish that when users do become hung, in the load balanced web/server group environment, and it is one of the servers which is unresponsive - that there would be some way that the server group would recognize it and restart the armonitor processes... OR at least place something in the log files which directs you to WHY the server locked up!! On Thu, Oct 16, 2014 at 2:21 PM, Rick Cook remedyr...@gmail.com wrote: ** Good ideas, Joe! Perhaps the arserver -v (or whatever it is now) that returns the version would fit the bill - quick return, confirms that the server is up, and almost zero performance impact. Rick Cook On Thu, Oct 16, 2014 at 11:19 AM, Joe D'Souza jdso...@shyle.net wrote: ** How about issuing one of the command line utilities to each of the servers such as maybe the arsignal –g that reloads the def cache. If a server is not responding you would get a error 90. I do not remember all the switches to arsignal offhand but there should be one that has less of an impact to a server than –g if you are worried about the performance impact it may have to servers that are responding well. Or you could try to arcreateentry into a test form. Joe -- *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *William Rentfrow *Sent:* Thursday, October 16, 2014 2:00 PM *To:* arslist@ARSLIST.ORG *Subject:* Re: Open question to BMC on troubleshooting without the WUT Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 *From:* Action Request System discussion list(ARSList) [mailto: arslist@ARSLIST.ORG] *On Behalf Of *Arner, Todd *Sent:* Thursday, October 16, 2014 12:56 PM *To:* arslist@ARSLIST.ORG *Subject:* Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner -- The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. -- No virus found in this message. Checked by AVG - www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Moot point either way - cron jobs are not allowed on the servers for various reasons. There are other ways of doing it though similar to cron. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Brian Goralczyk Sent: Thursday, October 16, 2014 2:08 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** I think the issue is that you are not testing the same process that the users are utilizing. That would increase the chance of false positives. That can be more frustrating than taking the time to do it manually. The best answer that I can see is to write a java program that can run and monitor the servers and return (or react to) a problem. But that is not the original request. It was something created and supported by BMC. I could write something fairly quickly, but that isn't the solution he was looking for. Brian Goralczyk Brian Goralczyk Phone 574-643-1144 Email bgoralc...@gmail.commailto:bgoralc...@gmail.com On Thu, Oct 16, 2014 at 2:52 PM, Joe D'Souza jdso...@shyle.netmailto:jdso...@shyle.net wrote: ** If you are on UNIX or LINUX you might be able to design a cron job that runs every few minutes to do that using the output of arsignal –v. Joe From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORGmailto:arslist@ARSLIST.ORG] On Behalf Of Rob Dudley Sent: Thursday, October 16, 2014 2:31 PM To: arslist@ARSLIST.ORGmailto:arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** I really just wish that when users do become hung, in the load balanced web/server group environment, and it is one of the servers which is unresponsive - that there would be some way that the server group would recognize it and restart the armonitor processes... OR at least place something in the log files which directs you to WHY the server locked up!! On Thu, Oct 16, 2014 at 2:21 PM, Rick Cook remedyr...@gmail.commailto:remedyr...@gmail.com wrote: ** Good ideas, Joe! Perhaps the arserver -v (or whatever it is now) that returns the version would fit the bill - quick return, confirms that the server is up, and almost zero performance impact. Rick Cook On Thu, Oct 16, 2014 at 11:19 AM, Joe D'Souza jdso...@shyle.netmailto:jdso...@shyle.net wrote: ** How about issuing one of the command line utilities to each of the servers such as maybe the arsignal –g that reloads the def cache. If a server is not responding you would get a error 90. I do not remember all the switches to arsignal offhand but there should be one that has less of an impact to a server than –g if you are worried about the performance impact it may have to servers that are responding well. Or you could try to arcreateentry into a test form. Joe From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORGmailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Thursday, October 16, 2014 2:00 PM To: arslist@ARSLIST.ORGmailto:arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.commailto:wrentf...@stratacominc.com Office: 715-204-3061tel:715-204-3061 or 701-232-5697x25tel:701-232-5697x25 Cell: 715-498-5056tel:715-498-5056 From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORGmailto:arslist@ARSLIST.ORG] On Behalf Of Arner, Todd Sent: Thursday, October 16, 2014 12:56 PM To: arslist@ARSLIST.ORGmailto:arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.orgmailto:no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. No virus found in this message. Checked by AVG - www.avg.comhttp://www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the
Re: Open question to BMC on troubleshooting without the WUT
We do this with a Unix/Linux shell script every minute. Keep a simple log in a plain text file. It notifies via text message if everything isn't perfect. Check the log to see which server didn't respond. We actually test several paths - the status page of the midtiers gives you an end-to-end status and verifies that at least one combination server and web server works. That gives an availability metric. You can also use the api driver program from the command line to test different servers in a server group and even check each fast and list thread pool. #!/bin/ksh, it's a wonderful thing :-) Doug -- Doug Blair +1 224-558-5462 Sent from my iPhone6+ Auto-corrected typos, misspellings and non-sequiturs are gratefully attributed to Steve Jobs :-) On Oct 16, 2014, at 2:52 PM, Joe D'Souza jdso...@shyle.net wrote: ** If you are on UNIX or LINUX you might be able to design a cron job that runs every few minutes to do that using the output of arsignal –v. Joe From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Rob Dudley Sent: Thursday, October 16, 2014 2:31 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** I really just wish that when users do become hung, in the load balanced web/server group environment, and it is one of the servers which is unresponsive - that there would be some way that the server group would recognize it and restart the armonitor processes... OR at least place something in the log files which directs you to WHY the server locked up!! On Thu, Oct 16, 2014 at 2:21 PM, Rick Cook remedyr...@gmail.com wrote: ** Good ideas, Joe! Perhaps the arserver -v (or whatever it is now) that returns the version would fit the bill - quick return, confirms that the server is up, and almost zero performance impact. Rick Cook On Thu, Oct 16, 2014 at 11:19 AM, Joe D'Souza jdso...@shyle.net wrote: ** How about issuing one of the command line utilities to each of the servers such as maybe the arsignal –g that reloads the def cache. If a server is not responding you would get a error 90. I do not remember all the switches to arsignal offhand but there should be one that has less of an impact to a server than –g if you are worried about the performance impact it may have to servers that are responding well. Or you could try to arcreateentry into a test form. Joe From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of William Rentfrow Sent: Thursday, October 16, 2014 2:00 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT Yes and no. I've seen instances where users are locked up but the admin threads work (and vice versa). So it's not the same test really as connecting as a user. William Rentfrow wrentf...@stratacominc.com Office: 715-204-3061 or 701-232-5697x25 Cell: 715-498-5056 From: Action Request System discussion list(ARSList) [mailto:arslist@ARSLIST.ORG] On Behalf Of Arner, Todd Sent: Thursday, October 16, 2014 12:56 PM To: arslist@ARSLIST.ORG Subject: Re: Open question to BMC on troubleshooting without the WUT ** Could you confirm the server is available by opening the server up in Dev Studio? Just a thought. Todd Arner The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be protected under state or federal law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please forward the communication to no...@glhec.org immediately and destroy or delete the original message and any copy of it from your computer system. If you have any questions concerning this message, please contact the sender. No virus found in this message. Checked by AVG - www.avg.com Version: 2014.0.4765 / Virus Database: 4040/8371 - Release Date: 10/11/14 _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ _ARSlist: Where the Answers Are and have been for 20 years_ ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Open question to BMC on troubleshooting without the WUT
The problem is not so much the lack of WUT, with its confused Windows 3.1 UI, but the lack of quality tools within Mid Tier to troubleshoot problems. Everything involves using a workflow console, but John hit the nail on the head, there needs to be a quality user interface that isn't remotely tied to workflow that performs the kind of troubleshooting people get from WUT. And it needs search tools that are in WUT, such as typing a string and searching for a form. The current MId Tier config interface is very poor. There's nothing on it that makes you feel like its a quality product. Its unloved and in need of attention. And it really isn't a lot of effort. ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
We have a midtier hosted under a tomcat not part of the load balancer. So users are not routed to this midtier. We added all individual arsystem nodes to this midtier. That way when you login using this midtier or search for forms you see links for each arsystem node. You can open forms and do queries or submit entries to a specific node that you selected(instead of loadbalancer determining it). Since this midtier is really for admin purpose, you do not need to allocate huge cpu or ram. or you can use or share one of the legacy or less used system to host this midtier. https://lh6.googleusercontent.com/-5TEXDNkfgWw/VEBnX3c2RpI/Le0/TxKwXwlJ8aQ/s1600/pic1.bmp https://lh3.googleusercontent.com/-Opc8aLbf5EI/VEBnapTIr8I/Le8/8umkZYEwXf0/s1600/pic2.bmp ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
Re: Open question to BMC on troubleshooting without the WUT
Hi William,
I too receive problem reports from our Mid-tier users and struggled to
determine which server they were connected to. I asked one of my JSP
experts to write us a program that could run from the browser that would
emit the name and IP of the Tomcat server that was serving the users
session. We use LVS as our load balancer, which is IP based so once you are
locked onto a server, you stay there.
I put this JSP program in the Tomcat webapps/ROOT directory and ask the
user to invoke it as:
https://myservicename/s*erver_id.jsp*
and have them tell me which server they are locked onto.
Here is the code:
*server_id.jsp*:
%@ page import=java.util.*,java.io.* %
%
String cmd = hostname;
String outstr = ;
try {
Runtime rt = Runtime.getRuntime();
Process p = rt.exec(cmd);
try {
InputStreamReader ise = new InputStreamReader(p.getErrorStream());
BufferedReader bre = new BufferedReader(ise);
InputStreamReader iso = new InputStreamReader(p.getInputStream());
BufferedReader bro = new BufferedReader(iso);
String line=null;
while ( (line = bre.readLine()) != null ) {
outstr += line;
}
while ( (line = bro.readLine()) != null ) {
outstr += line;
}
}
catch (IOException ioe) {
ioe.printStackTrace();
}
int exitVal = p.waitFor();
}
catch (Throwable t) {
t.printStackTrace();
}
%
br
Hostname: nbsp;nbsp;nbsp; strong%=outstr%/strongbr
Your IP:
nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;nbsp;strong%=request.getRemoteAddr()%/strongbr
Service IP:
nbsp;nbsp;nbsp;strong%=request.getLocalAddr()%/strongbr
Hope this is helpful.
Larry
On Thu, Oct 16, 2014 at 1:02 PM, William Rentfrow
wrentf...@stratacominc.com wrote:
**
I sent this to our premiere support person and manager, but I'd be
interested to see what others have to say about this too.
Original message below:
Hi –
This came up on our call and I wanted to write it out.
BMC has stated that the Windows User Tool (WUT) is going to be
discontinued (in fact, it already was in 7.6x). What I need to know is
what is BMC’s recommendation for diagnosing problems with the AR Servers in
a server group?
Currently our users will report an issue like this: “Remedy is slow/locked
up/whatever”. Routinely we get no more information than this.
Right now our troubleshooting is to first diagnose which server(s) is
having problems. The fast way to do this is to login to every server with
the user tool. We usually know within a few seconds if one of the AR
servers is locked up, because we will not be able to log in to it. Then we
can bounce it and get service restored.
If they are responsive we then move on the Mid-tier servers, etc.
With a large load-balanced environment there is no way to * QUICKLY* do
this without the WUT. I could login with Developer Studio, but that
doesn’t use the same threads on the server as the WUT does. We have seen
instances where users are locked up and admins can log in with Dev studio
(and vice versa). Same goes for migrator and the import tool.
Support suggesting checking the AR Error log, but there are two problems
with that – first, many lock-up scenarios do not results in errors in the
arerror.log file. There are numerous other logs to check on every server
as well (CMDB, Email, AIE, etc). Checking every log file on every server
is time consuming and not 100% guaranteed to show us which server is locked
up.
The second problem with support’s suggestion is the sheer time it would
take to login to each server. We are on Linux, so we need to connect via
SSH using putty. We do that by first connecting to a gateway server. Then
we ssh to the actual AR server (direct access is not allowed). Finally, we
sudo to the user Remedy is running as. That means each time we connect
it’s 3X we login. If we multiply that by the 10 servers in our server
group it would take at least an hour just to triage the problem.
I can do the same thing with the WUT in seconds.
So here is the question: What is the proper way to QUICKLY triage which
server is having problems without using the WUT or Dev
Studio/Migrator/Import?
William Rentfrow
_ARSlist: Where the Answers Are and have been for 20 years_
___
UNSUBSCRIBE or access ARSlist Archives at www.arslist.org
Where the Answers Are, and have been for 20 years
Open question to BMC on troubleshooting without the WUT
Larry I wouldn't suggest using that JSP :) It is running a native application (hostname) to get the hostname that is readily available from a Java API call. Running native applications isn't going to do the performance of your Mid Tier any good, and anyone with a copy of wget can almost certainly kill your Mid Tier pretty quickly. The Java InetAddress API is what you should be using for this task. John ___ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Where the Answers Are, and have been for 20 years
