RE: [discuss] CloudStack logging
Hi Pierre-Luc, I have a pull request in to fix the rotating of the catalina.out logs - I'll check what it's status is. And yes, being able to view VR, CPVM and SSVM logs from mgmt. server would be awesome. Some of our clients have 1000s of VRs so, some real thought needs to go into it... Kind regards, Paul Angus Regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Pierre-Luc Dion [mailto:pd...@cloudops.com] Sent: 15 April 2016 13:57 To: dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging Something that could be interesting also is to remove VR and systemVMs logs from the cloudstack-management.log and have them into a separate files. I'm still not sure of those logs are traps only when there is problems, if it's the case it make sense to keep them in cloudstack-management.log. Anyhow, it might be interesting to collect VR and systemVMs logs from the management servers. The log structure is also pretty stable so far which make integration with ELK fairly easy which is good! What about the catalina.out? is there any way to reduce is size? I'll update the JIRA issue with problematic troubleshooting logs that are not clear to interpret... Thanks Paul for taking that one ! PL On Fri, Apr 15, 2016 at 4:44 AM, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > Paul, > > I think that asking for input is a good approach. Those that have > problem with a certain log message or - sequence are the best to say > what should be different. > > Whether we should make hundreds of PRs or one?.. neither. I would say > change anything from a big class to a small package, or get the flow > for one API call and change anything in that. That would be cross > package and may require classes in the management -, service -, > orchestration and resource layers to be revisited for several flows. > > > On Fri, Apr 15, 2016 at 10:06 AM, Paul Angus > <paul.an...@shapeblue.com> > wrote: > > > Great, I'll pick up the issue that Wido has already created > > https://issues.apache.org/jira/browse/CLOUDSTACK-8645 > > > > Does anyone have an suggestions wrt to structure of pull requests? > > I'm inclinded to ask the community for copies of logs - particularly > > ones that people have found difficult to read so that I or others > > can work through them looking for anamolies - it's very messy from a > > thoroughness point of view - but it would return the most useful > > changes first. It > also > > puts the error msg in a context to make the categorisation more acurate. > > > > > > > > Kind regards, > > > > Paul Angus > > > > Regards, > > > > Paul Angus > > > > paul.an...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > -Original Message- > > From: Wido den Hollander [mailto:w...@widodh.nl] > > Sent: 14 April 2016 15:04 > > To: Paul Angus <paul.an...@shapeblue.com>; dev@cloudstack.apache.org > > Subject: Re: [discuss] CloudStack logging > > > > > > > Op 14 april 2016 om 14:49 schreef Paul Angus > > > <paul.an...@shapeblue.com > >: > > > > > > > > > Hi All > > > > > > I think that we can all agree that CloudStack logs are very > > > difficult to read, especially for operational staff. > > > > > > I believe that the primary reason for this is that a large amount > > > of what should be INFO level events are categorised as DEBUG. The > > > logging therefore must be set at DEBUG for the logs to capture > > > these events. By defaulting the logging to DEBUG we include a lot > > > of detail which is counter-productive when using the logs to > > > diagnose operational > > issues. > > > > > > > Yes! > > > > > I think the situation can be greatly improved by reviewing the > > > categorisation of logged events and where necessary re-categorise > > > then such that INFO, WARN and ERROR logging would be sufficient > > > for an > > 'operational' admin. > > > > > > I think a phased approach where the default logging level remains > > > at DEBUG while re-categorisation occurs would mean that nothing > > > materially changes until we are happy with the categorisations. > > > Then we can default the logging level to INFO. > > > > > > > I created a issue for this a while ago: > > https://issues.apache.org/jira/browse/CLOUDSTACK-8645 > > > > Short: Yes. Just change this where needed. > > > > Wido > > > > > > > > > > > Thoughts everyone? > > > > > > > > > > > > > > > Kind regards, > > > > > > Paul Angus > > > > > > > > > Regards, > > > > > > Paul Angus > > > > > > paul.an...@shapeblue.com > > > www.shapeblue.com > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > -- > Daan >
Re: [discuss] CloudStack logging
Something that could be interesting also is to remove VR and systemVMs logs from the cloudstack-management.log and have them into a separate files. I'm still not sure of those logs are traps only when there is problems, if it's the case it make sense to keep them in cloudstack-management.log. Anyhow, it might be interesting to collect VR and systemVMs logs from the management servers. The log structure is also pretty stable so far which make integration with ELK fairly easy which is good! What about the catalina.out? is there any way to reduce is size? I'll update the JIRA issue with problematic troubleshooting logs that are not clear to interpret... Thanks Paul for taking that one ! PL On Fri, Apr 15, 2016 at 4:44 AM, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > Paul, > > I think that asking for input is a good approach. Those that have problem > with a certain log message or - sequence are the best to say what should be > different. > > Whether we should make hundreds of PRs or one?.. neither. I would say > change anything from a big class to a small package, or get the flow for > one API call and change anything in that. That would be cross package and > may require classes in the management -, service -, orchestration and > resource layers to be revisited for several flows. > > > On Fri, Apr 15, 2016 at 10:06 AM, Paul Angus <paul.an...@shapeblue.com> > wrote: > > > Great, I'll pick up the issue that Wido has already created > > https://issues.apache.org/jira/browse/CLOUDSTACK-8645 > > > > Does anyone have an suggestions wrt to structure of pull requests? > > I'm inclinded to ask the community for copies of logs - particularly ones > > that people have found difficult to read so that I or others can work > > through them looking for anamolies - it's very messy from a thoroughness > > point of view - but it would return the most useful changes first. It > also > > puts the error msg in a context to make the categorisation more acurate. > > > > > > > > Kind regards, > > > > Paul Angus > > > > Regards, > > > > Paul Angus > > > > paul.an...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > -----Original Message- > > From: Wido den Hollander [mailto:w...@widodh.nl] > > Sent: 14 April 2016 15:04 > > To: Paul Angus <paul.an...@shapeblue.com>; dev@cloudstack.apache.org > > Subject: Re: [discuss] CloudStack logging > > > > > > > Op 14 april 2016 om 14:49 schreef Paul Angus <paul.an...@shapeblue.com > >: > > > > > > > > > Hi All > > > > > > I think that we can all agree that CloudStack logs are very difficult > > > to read, especially for operational staff. > > > > > > I believe that the primary reason for this is that a large amount of > > > what should be INFO level events are categorised as DEBUG. The > > > logging therefore must be set at DEBUG for the logs to capture these > > > events. By defaulting the logging to DEBUG we include a lot of detail > > > which is counter-productive when using the logs to diagnose operational > > issues. > > > > > > > Yes! > > > > > I think the situation can be greatly improved by reviewing the > > > categorisation of logged events and where necessary re-categorise then > > > such that INFO, WARN and ERROR logging would be sufficient for an > > 'operational' admin. > > > > > > I think a phased approach where the default logging level remains at > > > DEBUG while re-categorisation occurs would mean that nothing > > > materially changes until we are happy with the categorisations. Then > > > we can default the logging level to INFO. > > > > > > > I created a issue for this a while ago: > > https://issues.apache.org/jira/browse/CLOUDSTACK-8645 > > > > Short: Yes. Just change this where needed. > > > > Wido > > > > > > > > > > > Thoughts everyone? > > > > > > > > > > > > > > > Kind regards, > > > > > > Paul Angus > > > > > > > > > Regards, > > > > > > Paul Angus > > > > > > paul.an...@shapeblue.com > > > www.shapeblue.com > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > -- > Daan >
Re: [discuss] CloudStack logging
Paul, I think that asking for input is a good approach. Those that have problem with a certain log message or - sequence are the best to say what should be different. Whether we should make hundreds of PRs or one?.. neither. I would say change anything from a big class to a small package, or get the flow for one API call and change anything in that. That would be cross package and may require classes in the management -, service -, orchestration and resource layers to be revisited for several flows. On Fri, Apr 15, 2016 at 10:06 AM, Paul Angus <paul.an...@shapeblue.com> wrote: > Great, I'll pick up the issue that Wido has already created > https://issues.apache.org/jira/browse/CLOUDSTACK-8645 > > Does anyone have an suggestions wrt to structure of pull requests? > I'm inclinded to ask the community for copies of logs - particularly ones > that people have found difficult to read so that I or others can work > through them looking for anamolies - it's very messy from a thoroughness > point of view - but it would return the most useful changes first. It also > puts the error msg in a context to make the categorisation more acurate. > > > > Kind regards, > > Paul Angus > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > -Original Message- > From: Wido den Hollander [mailto:w...@widodh.nl] > Sent: 14 April 2016 15:04 > To: Paul Angus <paul.an...@shapeblue.com>; dev@cloudstack.apache.org > Subject: Re: [discuss] CloudStack logging > > > > Op 14 april 2016 om 14:49 schreef Paul Angus <paul.an...@shapeblue.com>: > > > > > > Hi All > > > > I think that we can all agree that CloudStack logs are very difficult > > to read, especially for operational staff. > > > > I believe that the primary reason for this is that a large amount of > > what should be INFO level events are categorised as DEBUG. The > > logging therefore must be set at DEBUG for the logs to capture these > > events. By defaulting the logging to DEBUG we include a lot of detail > > which is counter-productive when using the logs to diagnose operational > issues. > > > > Yes! > > > I think the situation can be greatly improved by reviewing the > > categorisation of logged events and where necessary re-categorise then > > such that INFO, WARN and ERROR logging would be sufficient for an > 'operational' admin. > > > > I think a phased approach where the default logging level remains at > > DEBUG while re-categorisation occurs would mean that nothing > > materially changes until we are happy with the categorisations. Then > > we can default the logging level to INFO. > > > > I created a issue for this a while ago: > https://issues.apache.org/jira/browse/CLOUDSTACK-8645 > > Short: Yes. Just change this where needed. > > Wido > > > > > > > Thoughts everyone? > > > > > > > > > > Kind regards, > > > > Paul Angus > > > > > > Regards, > > > > Paul Angus > > > > paul.an...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > -- Daan
RE: [discuss] CloudStack logging
Great, I'll pick up the issue that Wido has already created https://issues.apache.org/jira/browse/CLOUDSTACK-8645 Does anyone have an suggestions wrt to structure of pull requests? I'm inclinded to ask the community for copies of logs - particularly ones that people have found difficult to read so that I or others can work through them looking for anamolies - it's very messy from a thoroughness point of view - but it would return the most useful changes first. It also puts the error msg in a context to make the categorisation more acurate. Kind regards, Paul Angus Regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Wido den Hollander [mailto:w...@widodh.nl] Sent: 14 April 2016 15:04 To: Paul Angus <paul.an...@shapeblue.com>; dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging > Op 14 april 2016 om 14:49 schreef Paul Angus <paul.an...@shapeblue.com>: > > > Hi All > > I think that we can all agree that CloudStack logs are very difficult > to read, especially for operational staff. > > I believe that the primary reason for this is that a large amount of > what should be INFO level events are categorised as DEBUG. The > logging therefore must be set at DEBUG for the logs to capture these > events. By defaulting the logging to DEBUG we include a lot of detail > which is counter-productive when using the logs to diagnose operational > issues. > Yes! > I think the situation can be greatly improved by reviewing the > categorisation of logged events and where necessary re-categorise then > such that INFO, WARN and ERROR logging would be sufficient for an > 'operational' admin. > > I think a phased approach where the default logging level remains at > DEBUG while re-categorisation occurs would mean that nothing > materially changes until we are happy with the categorisations. Then > we can default the logging level to INFO. > I created a issue for this a while ago: https://issues.apache.org/jira/browse/CLOUDSTACK-8645 Short: Yes. Just change this where needed. Wido > > > Thoughts everyone? > > > > > Kind regards, > > Paul Angus > > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
RE: [discuss] CloudStack logging
The file in a running installation you want is: /etc/cloudstack/management/ log4j-cloud.xml It is generally dynamic - you don't have to restart mgmt. server, it just takes a few seconds to kick in. If you're looking at the source code, various components use logging - SystemVMs, KVM agents, CloudStack mgmt. servers. So there'll be multiple occurrences As Koushik says, I think ./client/tomcatconf/log4j-cloud.xml.in is the one for the mgmt. server. Kind regards, Paul Angus Regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Chaz PC [mailto:dreeems4e...@hotmail.com] Sent: 14 April 2016 16:01 To: dev@cloudstack.apache.org Subject: RE: [discuss] CloudStack logging I am confused about the configuration file there two files one is named Log4j-cloud.xml The other is Log4j-cloud.xml.in What is the difference between these two files ? I hope someone can help me clear this confusion Original message From: Koushik Das <koushik@accelerite.com> Date: 14/04/2016 6:49 PM (GMT+04:00) To: dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging Based on the namespace log level can be set. It needs to be configured in the log4j. Check client/tomcatconf/log4j-cloud.xml.in in the source code and you will get an idea. -Koushik From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will Stevens <wstev...@cloudops.com> Sent: Thursday, April 14, 2016 7:29 PM To: dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging Do you have to recompile in order to turn off the logging for a specific package or class? If yes, that is a show stopper for almost everyone... If it only requires a management server restart, that is more realistic. *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Thu, Apr 14, 2016 at 9:48 AM, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > On Thu, Apr 14, 2016 at 3:41 PM, Will Stevens > <williamstev...@gmail.com> > wrote: > > > Is there an easy way to do that Daan, or is it a tedious task you > > just > have > > to power through? > > > It is hard work, mostly tedious, sometimes hilarious and most > definitely > > devops! It certainly wouldn't classify as development. > > > > > -- > Daan > DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
RE: [discuss] CloudStack logging
I am confused about the configuration file there two files one is named Log4j-cloud.xml The other is Log4j-cloud.xml.in What is the difference between these two files ? I hope someone can help me clear this confusion Original message From: Koushik Das <koushik@accelerite.com> Date: 14/04/2016 6:49 PM (GMT+04:00) To: dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging Based on the namespace log level can be set. It needs to be configured in the log4j. Check client/tomcatconf/log4j-cloud.xml.in in the source code and you will get an idea. -Koushik From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will Stevens <wstev...@cloudops.com> Sent: Thursday, April 14, 2016 7:29 PM To: dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging Do you have to recompile in order to turn off the logging for a specific package or class? If yes, that is a show stopper for almost everyone... If it only requires a management server restart, that is more realistic. *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Thu, Apr 14, 2016 at 9:48 AM, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > On Thu, Apr 14, 2016 at 3:41 PM, Will Stevens <williamstev...@gmail.com> > wrote: > > > Is there an easy way to do that Daan, or is it a tedious task you just > have > > to power through? > > > It is hard work, mostly tedious, sometimes hilarious and most definitely > > devops! It certainly wouldn't classify as development. > > > > > -- > Daan > DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: [discuss] CloudStack logging
Based on the namespace log level can be set. It needs to be configured in the log4j. Check client/tomcatconf/log4j-cloud.xml.in in the source code and you will get an idea. -Koushik From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will Stevens <wstev...@cloudops.com> Sent: Thursday, April 14, 2016 7:29 PM To: dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging Do you have to recompile in order to turn off the logging for a specific package or class? If yes, that is a show stopper for almost everyone... If it only requires a management server restart, that is more realistic. *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Thu, Apr 14, 2016 at 9:48 AM, Daan Hoogland <daan.hoogl...@gmail.com> wrote: > On Thu, Apr 14, 2016 at 3:41 PM, Will Stevens <williamstev...@gmail.com> > wrote: > > > Is there an easy way to do that Daan, or is it a tedious task you just > have > > to power through? > > > It is hard work, mostly tedious, sometimes hilarious and most definitely > > devops! It certainly wouldn't classify as development. > > > > > -- > Daan > DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: [discuss] CloudStack logging
Really agree with this one. With a toolchain like ELK, it isn't as bad because you can filter out the things you don't want, but generally speaking, the log levels need to be reevaluated. I would also add that many log statements would benefit from additional context, but that is a whole different story. Another example: INFO [com.cloud.agent.transport.Request] (StatsCollector-3:ctx-de473a88) not building log message for '[{}]', _cmds.length == 1 It even says "not building log message". On Thu, Apr 14, 2016 at 10:18 AM, Simon Weller <swel...@ena.com> wrote: > I'm all for this as well. > > Training our systems folks on how to work through the logs is a pain. > Focusing in on the most important items is more useful and also makes it > easier to automate log fetching and categorization. > > - Si > > > From: Paul Angus <paul.an...@shapeblue.com> > Sent: Thursday, April 14, 2016 9:06 AM > To: Wido den Hollander; dev@cloudstack.apache.org > Subject: RE: [discuss] CloudStack logging > > Grabbing a couple of examples - these should be debug - I can't 'do' > anything with this information. > > INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-4119d5bc) Begin cleanup expired async-jobs > INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > (AsyncJobMgr-Heartbeat-1:ctx-4119d5bc) End cleanup expired async-jobs > INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-2:ctx-3209f65c) Scan > hung worker VM to recycle > INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-113:ctx-56d9f9f2) > Scan hung worker VM to recycle > > Whereas this should be WARN. I wouldn't want to lose this by switching off > DEBUG > > DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-91226be7) > Detected management node left, id:2, nodeIP:10.2.0.6 > DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-0bccd078) > Detected management node left, id:2, nodeIP:10.2.0.6 > > > > Kind regards, > > Paul Angus > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > -Original Message- > From: Wido den Hollander [mailto:w...@widodh.nl] > Sent: 14 April 2016 15:04 > To: Paul Angus <paul.an...@shapeblue.com>; dev@cloudstack.apache.org > Subject: Re: [discuss] CloudStack logging > > > > Op 14 april 2016 om 14:49 schreef Paul Angus <paul.an...@shapeblue.com>: > > > > > > Hi All > > > > I think that we can all agree that CloudStack logs are very difficult > > to read, especially for operational staff. > > > > I believe that the primary reason for this is that a large amount of > > what should be INFO level events are categorised as DEBUG. The > > logging therefore must be set at DEBUG for the logs to capture these > > events. By defaulting the logging to DEBUG we include a lot of detail > > which is counter-productive when using the logs to diagnose operational > issues. > > > > Yes! > > > I think the situation can be greatly improved by reviewing the > > categorisation of logged events and where necessary re-categorise then > > such that INFO, WARN and ERROR logging would be sufficient for an > 'operational' admin. > > > > I think a phased approach where the default logging level remains at > > DEBUG while re-categorisation occurs would mean that nothing > > materially changes until we are happy with the categorisations. Then > > we can default the logging level to INFO. > > > > I created a issue for this a while ago: > https://issues.apache.org/jira/browse/CLOUDSTACK-8645 > > Short: Yes. Just change this where needed. > > Wido > > > > > > > Thoughts everyone? > > > > > > > > > > Kind regards, > > > > Paul Angus > > > > > > Regards, > > > > Paul Angus > > > > paul.an...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >
Re: [discuss] CloudStack logging
I'm all for this as well. Training our systems folks on how to work through the logs is a pain. Focusing in on the most important items is more useful and also makes it easier to automate log fetching and categorization. - Si From: Paul Angus <paul.an...@shapeblue.com> Sent: Thursday, April 14, 2016 9:06 AM To: Wido den Hollander; dev@cloudstack.apache.org Subject: RE: [discuss] CloudStack logging Grabbing a couple of examples - these should be debug - I can't 'do' anything with this information. INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4119d5bc) Begin cleanup expired async-jobs INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4119d5bc) End cleanup expired async-jobs INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-2:ctx-3209f65c) Scan hung worker VM to recycle INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-113:ctx-56d9f9f2) Scan hung worker VM to recycle Whereas this should be WARN. I wouldn't want to lose this by switching off DEBUG DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-91226be7) Detected management node left, id:2, nodeIP:10.2.0.6 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-0bccd078) Detected management node left, id:2, nodeIP:10.2.0.6 Kind regards, Paul Angus Regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Wido den Hollander [mailto:w...@widodh.nl] Sent: 14 April 2016 15:04 To: Paul Angus <paul.an...@shapeblue.com>; dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging > Op 14 april 2016 om 14:49 schreef Paul Angus <paul.an...@shapeblue.com>: > > > Hi All > > I think that we can all agree that CloudStack logs are very difficult > to read, especially for operational staff. > > I believe that the primary reason for this is that a large amount of > what should be INFO level events are categorised as DEBUG. The > logging therefore must be set at DEBUG for the logs to capture these > events. By defaulting the logging to DEBUG we include a lot of detail > which is counter-productive when using the logs to diagnose operational > issues. > Yes! > I think the situation can be greatly improved by reviewing the > categorisation of logged events and where necessary re-categorise then > such that INFO, WARN and ERROR logging would be sufficient for an > 'operational' admin. > > I think a phased approach where the default logging level remains at > DEBUG while re-categorisation occurs would mean that nothing > materially changes until we are happy with the categorisations. Then > we can default the logging level to INFO. > I created a issue for this a while ago: https://issues.apache.org/jira/browse/CLOUDSTACK-8645 Short: Yes. Just change this where needed. Wido > > > Thoughts everyone? > > > > > Kind regards, > > Paul Angus > > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
RE: [discuss] CloudStack logging
Grabbing a couple of examples - these should be debug - I can't 'do' anything with this information. INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4119d5bc) Begin cleanup expired async-jobs INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4119d5bc) End cleanup expired async-jobs INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-2:ctx-3209f65c) Scan hung worker VM to recycle INFO [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-113:ctx-56d9f9f2) Scan hung worker VM to recycle Whereas this should be WARN. I wouldn't want to lose this by switching off DEBUG DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-91226be7) Detected management node left, id:2, nodeIP:10.2.0.6 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-0bccd078) Detected management node left, id:2, nodeIP:10.2.0.6 Kind regards, Paul Angus Regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: Wido den Hollander [mailto:w...@widodh.nl] Sent: 14 April 2016 15:04 To: Paul Angus <paul.an...@shapeblue.com>; dev@cloudstack.apache.org Subject: Re: [discuss] CloudStack logging > Op 14 april 2016 om 14:49 schreef Paul Angus <paul.an...@shapeblue.com>: > > > Hi All > > I think that we can all agree that CloudStack logs are very difficult > to read, especially for operational staff. > > I believe that the primary reason for this is that a large amount of > what should be INFO level events are categorised as DEBUG. The > logging therefore must be set at DEBUG for the logs to capture these > events. By defaulting the logging to DEBUG we include a lot of detail > which is counter-productive when using the logs to diagnose operational > issues. > Yes! > I think the situation can be greatly improved by reviewing the > categorisation of logged events and where necessary re-categorise then > such that INFO, WARN and ERROR logging would be sufficient for an > 'operational' admin. > > I think a phased approach where the default logging level remains at > DEBUG while re-categorisation occurs would mean that nothing > materially changes until we are happy with the categorisations. Then > we can default the logging level to INFO. > I created a issue for this a while ago: https://issues.apache.org/jira/browse/CLOUDSTACK-8645 Short: Yes. Just change this where needed. Wido > > > Thoughts everyone? > > > > > Kind regards, > > Paul Angus > > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
Re: [discuss] CloudStack logging
> Op 14 april 2016 om 14:49 schreef Paul Angus: > > > Hi All > > I think that we can all agree that CloudStack logs are very difficult to read, > especially for operational staff. > > I believe that the primary reason for this is that a large amount of what > should be INFO level events are categorised as DEBUG. The logging therefore > must be set at DEBUG for the logs to capture these events. By defaulting the > logging to DEBUG we include a lot of detail which is counter-productive when > using the logs to diagnose operational issues. > Yes! > I think the situation can be greatly improved by reviewing the categorisation > of logged events and where necessary re-categorise then such that INFO, WARN > and ERROR logging would be sufficient for an 'operational' admin. > > I think a phased approach where the default logging level remains at DEBUG > while re-categorisation occurs would mean that nothing materially changes > until we are happy with the categorisations. Then we can default the logging > level to INFO. > I created a issue for this a while ago: https://issues.apache.org/jira/browse/CLOUDSTACK-8645 Short: Yes. Just change this where needed. Wido > > > Thoughts everyone? > > > > > Kind regards, > > Paul Angus > > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue
Re: [discuss] CloudStack logging
Do you have to recompile in order to turn off the logging for a specific package or class? If yes, that is a show stopper for almost everyone... If it only requires a management server restart, that is more realistic. *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Thu, Apr 14, 2016 at 9:48 AM, Daan Hooglandwrote: > On Thu, Apr 14, 2016 at 3:41 PM, Will Stevens > wrote: > > > Is there an easy way to do that Daan, or is it a tedious task you just > have > > to power through? > > > It is hard work, mostly tedious, sometimes hilarious and most definitely > > devops! It certainly wouldn't classify as development. > > > > > -- > Daan >
Re: [discuss] CloudStack logging
On Thu, Apr 14, 2016 at 3:41 PM, Will Stevenswrote: > Is there an easy way to do that Daan, or is it a tedious task you just have > to power through? > It is hard work, mostly tedious, sometimes hilarious and most definitely devops! It certainly wouldn't classify as development. -- Daan
Re: [discuss] CloudStack logging
Is there an easy way to do that Daan, or is it a tedious task you just have to power through? I agree this initiative would be helpful. On Apr 14, 2016 9:04 AM, "Daan Hoogland"wrote: > I largely agree with just one sneer to the operators amongst us: the level > can be set on a per package and even class basis. So you can work through > disabling anoying logging step by step. Makes sense to keep looking at the > source in the meanwhile to make sure sensible diagnostics remains > available. > > On Thu, Apr 14, 2016 at 2:49 PM, Paul Angus > wrote: > > > Hi All > > > > I think that we can all agree that CloudStack logs are very difficult to > > read, especially for operational staff. > > > > I believe that the primary reason for this is that a large amount of what > > should be INFO level events are categorised as DEBUG. The logging > > therefore must be set at DEBUG for the logs to capture these events. By > > defaulting the logging to DEBUG we include a lot of detail which is > > counter-productive when using the logs to diagnose operational issues. > > > > I think the situation can be greatly improved by reviewing the > > categorisation of logged events and where necessary re-categorise then > such > > that INFO, WARN and ERROR logging would be sufficient for an > 'operational' > > admin. > > > > I think a phased approach where the default logging level remains at > DEBUG > > while re-categorisation occurs would mean that nothing materially changes > > until we are happy with the categorisations. Then we can default the > > logging level to INFO. > > > > > > > > Thoughts everyone? > > > > > > > > > > Kind regards, > > > > Paul Angus > > > > > > Regards, > > > > Paul Angus > > > > paul.an...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > -- > Daan >
Re: [discuss] CloudStack logging
I largely agree with just one sneer to the operators amongst us: the level can be set on a per package and even class basis. So you can work through disabling anoying logging step by step. Makes sense to keep looking at the source in the meanwhile to make sure sensible diagnostics remains available. On Thu, Apr 14, 2016 at 2:49 PM, Paul Anguswrote: > Hi All > > I think that we can all agree that CloudStack logs are very difficult to > read, especially for operational staff. > > I believe that the primary reason for this is that a large amount of what > should be INFO level events are categorised as DEBUG. The logging > therefore must be set at DEBUG for the logs to capture these events. By > defaulting the logging to DEBUG we include a lot of detail which is > counter-productive when using the logs to diagnose operational issues. > > I think the situation can be greatly improved by reviewing the > categorisation of logged events and where necessary re-categorise then such > that INFO, WARN and ERROR logging would be sufficient for an 'operational' > admin. > > I think a phased approach where the default logging level remains at DEBUG > while re-categorisation occurs would mean that nothing materially changes > until we are happy with the categorisations. Then we can default the > logging level to INFO. > > > > Thoughts everyone? > > > > > Kind regards, > > Paul Angus > > > Regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > -- Daan