Re: [openstack-dev] [Fuel] Removing old node logs
Making diagnostic snapshot for a particular environment is a good idea. But the issue is still there. We often have the situation when user actually doesn't care of old logs at all. He downloads ISO, installs it and tries various installation options (Ubuntu, Centos, HA, Ceph, etc.). Sooner or later his hard drive is full and he even can not make the diagnostic snapshot. Dealing with that stuff about taking care of available free space inside shotgun seems to be not a good idea. But we still need to address this. The easiest way to do that is to delete old log directories (logrotate or nailgun itself). In this case the issue at least will be rather seldom. But, of course, the right way is to have a kind of monitoring system on the master node and notify user when disk is full or launch a kind of cleaning task. Ok, right place where we should deal with that stuff about removing old logs is logrotate. Currently it just moves files like this /var/log/remote/old-node.example.com/some.log - /var/log/remote/ old-node.example.com/some.log.1.gz. But what it actually should do is to remove the whole directories which are related to nonexistent nodes, right? Vladimir Kozhukalov On Tue, Jun 24, 2014 at 9:19 PM, Andrey Danin ada...@mirantis.com wrote: +1 to @Aleksandr On Tue, Jun 24, 2014 at 8:32 PM, Aleksandr Didenko adide...@mirantis.com wrote: Yes, of course, snapshot for all nodes at once (like currently) should also be available. On Tue, Jun 24, 2014 at 7:27 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hello, @Aleks, it's a good idea to make snapshot per environment, but I think we can keep functionality to make snapshot for all nodes at once too. - Igor On Tue, Jun 24, 2014 at 6:38 PM, Aleksandr Didenko adide...@mirantis.com wrote: Yeah, I thought about diagnostic snapshot too. Maybe it would be better to implement per-environment diagnostic snapshots? I.e. add diagnostic snapshot generate/download buttons/links in the environment actions tab. Such snapshot would contain info/logs about Fuel master node and nodes assigned to the environment only. On Tue, Jun 24, 2014 at 6:27 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hi guys, What about our diagnostic snapshot? I mean we're going to make snapshot of entire /var/log and obviously this old logs will be included in snapshot. Should we skip theem or such situation is ok? - Igor On Tue, Jun 24, 2014 at 5:57 PM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, If user runs some experiments with creating/deleting clusters, then taking care of old logs is under user's responsibility, I suppose. Fuel configures log rotation with compression for remote logs, so old logs will be gzipped and will not take much space. In case of additional boolean parameter, the default value should be 0-don't touch old logs. -- Regards, Alex On Tue, Jun 24, 2014 at 4:07 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Andrey Danin ada...@mirantis.com skype: gcon.monolake ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [Fuel] Removing old node logs
When nailgun remove a node it can gzip all the logs of this node to a special file, like: /var/log/remote/archive/node-3-timestamp.tgz And logrotate can keep these files for month, then delete them. Master node health monitor is another big discussion. On Thu, Jun 26, 2014 at 1:25 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Making diagnostic snapshot for a particular environment is a good idea. But the issue is still there. We often have the situation when user actually doesn't care of old logs at all. He downloads ISO, installs it and tries various installation options (Ubuntu, Centos, HA, Ceph, etc.). Sooner or later his hard drive is full and he even can not make the diagnostic snapshot. Dealing with that stuff about taking care of available free space inside shotgun seems to be not a good idea. But we still need to address this. The easiest way to do that is to delete old log directories (logrotate or nailgun itself). In this case the issue at least will be rather seldom. But, of course, the right way is to have a kind of monitoring system on the master node and notify user when disk is full or launch a kind of cleaning task. Ok, right place where we should deal with that stuff about removing old logs is logrotate. Currently it just moves files like this /var/log/remote/old-node.example.com/some.log - /var/log/remote/ old-node.example.com/some.log.1.gz. But what it actually should do is to remove the whole directories which are related to nonexistent nodes, right? Vladimir Kozhukalov On Tue, Jun 24, 2014 at 9:19 PM, Andrey Danin ada...@mirantis.com wrote: +1 to @Aleksandr On Tue, Jun 24, 2014 at 8:32 PM, Aleksandr Didenko adide...@mirantis.com wrote: Yes, of course, snapshot for all nodes at once (like currently) should also be available. On Tue, Jun 24, 2014 at 7:27 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hello, @Aleks, it's a good idea to make snapshot per environment, but I think we can keep functionality to make snapshot for all nodes at once too. - Igor On Tue, Jun 24, 2014 at 6:38 PM, Aleksandr Didenko adide...@mirantis.com wrote: Yeah, I thought about diagnostic snapshot too. Maybe it would be better to implement per-environment diagnostic snapshots? I.e. add diagnostic snapshot generate/download buttons/links in the environment actions tab. Such snapshot would contain info/logs about Fuel master node and nodes assigned to the environment only. On Tue, Jun 24, 2014 at 6:27 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hi guys, What about our diagnostic snapshot? I mean we're going to make snapshot of entire /var/log and obviously this old logs will be included in snapshot. Should we skip theem or such situation is ok? - Igor On Tue, Jun 24, 2014 at 5:57 PM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, If user runs some experiments with creating/deleting clusters, then taking care of old logs is under user's responsibility, I suppose. Fuel configures log rotation with compression for remote logs, so old logs will be gzipped and will not take much space. In case of additional boolean parameter, the default value should be 0-don't touch old logs. -- Regards, Alex On Tue, Jun 24, 2014 at 4:07 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
[openstack-dev] [Fuel] Removing old node logs
Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Removing old node logs
Hi, If user runs some experiments with creating/deleting clusters, then taking care of old logs is under user's responsibility, I suppose. Fuel configures log rotation with compression for remote logs, so old logs will be gzipped and will not take much space. In case of additional boolean parameter, the default value should be 0-don't touch old logs. -- Regards, Alex On Tue, Jun 24, 2014 at 4:07 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Removing old node logs
Yeah, I thought about diagnostic snapshot too. Maybe it would be better to implement per-environment diagnostic snapshots? I.e. add diagnostic snapshot generate/download buttons/links in the environment actions tab. Such snapshot would contain info/logs about Fuel master node and nodes assigned to the environment only. On Tue, Jun 24, 2014 at 6:27 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hi guys, What about our diagnostic snapshot? I mean we're going to make snapshot of entire /var/log and obviously this old logs will be included in snapshot. Should we skip theem or such situation is ok? - Igor On Tue, Jun 24, 2014 at 5:57 PM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, If user runs some experiments with creating/deleting clusters, then taking care of old logs is under user's responsibility, I suppose. Fuel configures log rotation with compression for remote logs, so old logs will be gzipped and will not take much space. In case of additional boolean parameter, the default value should be 0-don't touch old logs. -- Regards, Alex On Tue, Jun 24, 2014 at 4:07 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Removing old node logs
Hi guys, What about our diagnostic snapshot? I mean we're going to make snapshot of entire /var/log and obviously this old logs will be included in snapshot. Should we skip theem or such situation is ok? - Igor On Tue, Jun 24, 2014 at 5:57 PM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, If user runs some experiments with creating/deleting clusters, then taking care of old logs is under user's responsibility, I suppose. Fuel configures log rotation with compression for remote logs, so old logs will be gzipped and will not take much space. In case of additional boolean parameter, the default value should be 0-don't touch old logs. -- Regards, Alex On Tue, Jun 24, 2014 at 4:07 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Removing old node logs
What about to gzip old logs by Astute and place them to a special directory, which will be managed under logrotate.d, and logrotate will remove untouched logs after 1 month. On Tue, Jun 24, 2014 at 6:57 PM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, If user runs some experiments with creating/deleting clusters, then taking care of old logs is under user's responsibility, I suppose. Fuel configures log rotation with compression for remote logs, so old logs will be gzipped and will not take much space. In case of additional boolean parameter, the default value should be 0-don't touch old logs. -- Regards, Alex On Tue, Jun 24, 2014 at 4:07 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Andrey Danin ada...@mirantis.com skype: gcon.monolake ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Removing old node logs
Hello, @Aleks, it's a good idea to make snapshot per environment, but I think we can keep functionality to make snapshot for all nodes at once too. - Igor On Tue, Jun 24, 2014 at 6:38 PM, Aleksandr Didenko adide...@mirantis.com wrote: Yeah, I thought about diagnostic snapshot too. Maybe it would be better to implement per-environment diagnostic snapshots? I.e. add diagnostic snapshot generate/download buttons/links in the environment actions tab. Such snapshot would contain info/logs about Fuel master node and nodes assigned to the environment only. On Tue, Jun 24, 2014 at 6:27 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hi guys, What about our diagnostic snapshot? I mean we're going to make snapshot of entire /var/log and obviously this old logs will be included in snapshot. Should we skip theem or such situation is ok? - Igor On Tue, Jun 24, 2014 at 5:57 PM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, If user runs some experiments with creating/deleting clusters, then taking care of old logs is under user's responsibility, I suppose. Fuel configures log rotation with compression for remote logs, so old logs will be gzipped and will not take much space. In case of additional boolean parameter, the default value should be 0-don't touch old logs. -- Regards, Alex On Tue, Jun 24, 2014 at 4:07 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Removing old node logs
Yes, of course, snapshot for all nodes at once (like currently) should also be available. On Tue, Jun 24, 2014 at 7:27 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hello, @Aleks, it's a good idea to make snapshot per environment, but I think we can keep functionality to make snapshot for all nodes at once too. - Igor On Tue, Jun 24, 2014 at 6:38 PM, Aleksandr Didenko adide...@mirantis.com wrote: Yeah, I thought about diagnostic snapshot too. Maybe it would be better to implement per-environment diagnostic snapshots? I.e. add diagnostic snapshot generate/download buttons/links in the environment actions tab. Such snapshot would contain info/logs about Fuel master node and nodes assigned to the environment only. On Tue, Jun 24, 2014 at 6:27 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hi guys, What about our diagnostic snapshot? I mean we're going to make snapshot of entire /var/log and obviously this old logs will be included in snapshot. Should we skip theem or such situation is ok? - Igor On Tue, Jun 24, 2014 at 5:57 PM, Aleksandr Didenko adide...@mirantis.com wrote: Hi, If user runs some experiments with creating/deleting clusters, then taking care of old logs is under user's responsibility, I suppose. Fuel configures log rotation with compression for remote logs, so old logs will be gzipped and will not take much space. In case of additional boolean parameter, the default value should be 0-don't touch old logs. -- Regards, Alex On Tue, Jun 24, 2014 at 4:07 PM, Vladimir Kozhukalov vkozhuka...@mirantis.com wrote: Guys, What do you think of removing node logs on master node right after removing node from cluster? The issue is when user do experiments he creates and deletes clusters and old unused directories remain and take disk space. On the other hand, it is not so hard to imaging the situation when user would like to be able to take a look in old logs. My suggestion here is to add a boolean parameter into settings which will manage this piece of logic (1-remove old logs, 0-don't touch old logs). Thanks for your opinions. Vladimir Kozhukalov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
