Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Artem, [1] and [2] are changes that do exactly this - snapshot is created in /var/log. Please review these changes if you haven't already. Cheers, Maciej [1] https://review.openstack.org/#/c/270823/ [2] https://review.openstack.org/#/c/271179/ On Mon, Jan 25, 2016 at 5:09 PM, Artem Panchenko

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Guys, I want to pay your attention that we need to not only fix snapshots generation issue, but also prevent caused by it unexpected services failures (see details in a duplicate [0] of original [1] bug), which would become a challenge for not experienced users (for example he/she won't be

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Igor: It seems that fqdn -> ipaddr will indeed be resolved. Please share your feedback in review: https://review.openstack.org/#/c/266964/3 On Fri, Jan 15, 2016 at 4:25 PM, Igor Kalnitsky wrote: > Sheena - > > What do you mean by *targeted*? Shotgun's designed to be a

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

I’ve also seen the request multiple times to be able to provide more targeted snapshots which might also (partially) solve this problem as it would require significantly less disk space to grab logs from a subset of nodes for a specific window of time, instead of the more robust grab-all solution

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Sheena - What do you mean by *targeted*? Shotgun's designed to be a *targeted* solution. If someone wants more *precise* targets - it's easy to specify them in Nailgun's settings.yaml. - Igor On Fri, Jan 15, 2016 at 5:02 PM, Sheena Gregson wrote: > I’ve also seen the

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Thanks for your insight guys! I agree with Oleg, I will see what I can do to make this work this way. About hardlinks - wouldn't it be better to use symlinks? This way we don't occupy more space than necessary, and we can link to files and directories that are in other block device than /var.

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Hey Maceij - > About hardlinks - wouldn't it be better to use symlinks? > This way we don't occupy more space than necessary AFAIK, hardlinks won't occupy much space. They are the links, after all. :) As for symlinks, I'm afraid shotgun (and fabric underneath) won't resolve them and links are

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Igor, I meant that symlinks also give us the benefit of not using additional space (just as hardlinks do) while being able to link to files from different filesystems. Also, as Barłomiej pointed out the `h` switch for tar should do the trick [1]. Cheers, Maciej [1]

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Igor, I took a glance on Maciej's patch and it adds a switch to tar command to make it follow symbolic links, so it looks good to me. Bartłomiej On Thu, Jan 14, 2016 at 10:39 AM, Igor Kalnitsky wrote: > Hey Maceij - > > > About hardlinks - wouldn't it be better to use

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

> I took a glance on Maciej's patch and it adds a switch to tar command > to make it follow symbolic links Yeah, that should work. Except one thing - we previously had fqdn -> ipaddr links in snapshots. So now they will be resolved into full copy? > I meant that symlinks also give us the benefit

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Igor, I will investigate this, thanks! Artem, I guess that if we have an untrusted user on master node, he could just put something he wants to be in the snapshot in /var/log without having to time the attack carefully with tar execution. I want to use links for directories, this saves me the

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Hi, using symlinks is a bit dangerous, here is a quote from the man you mentioned [0]: > The`--dereference'option is unsafe if an untrusted user can modify directories while|tar|is running. Hard links usage is much safer, because you can't use them for directories. But at the same time

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Hi, > b) Make the snapshot location share the diskspace of /var/log? +1 for that. And +1 for using hard links to save space during snapshot creation. Regards, Alex On Tue, Jan 12, 2016 at 12:12 PM, Artem Panchenko wrote: > Hi, > > doesn't matter how /var partition is

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

I think we need to find a way to: 1) verify the size of snapshot without actually making it and compare to the available disk space beforehand. 2) refuse to create snapshot if space is insufficient and notify user (otherwise it breaks Admin node as we have seen) 3) provide a way to prioritize

[openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Hi! I need some advice on how to tackle this issue. There is a bug [1] describing the problem with creating a diagnostic snapshot. The issue is that /var/log has 100GB available, while /var (where diagnostic snapshot is being generated - /var/www/nailgun/dump/fuel-snapshot according to [2]) has

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

Hi, doesn't matter how /var partition is big, diagnostic snapshot still could occupy all its space if there are lots of logs on slave nodes. Although, we can try to control disk space usage by snapshot in shotgun, but IMHO it's much safer to keep all related to logs staff away from critical