On Thu, Jul 10, 2014 at 08:32:37AM -0400, Sean Dague wrote: > On 07/10/2014 05:03 AM, Daniel P. Berrange wrote: > > On Wed, Jul 09, 2014 at 06:23:27PM -0400, Sean Dague wrote: > >> The libvirt logs needed are huge, so we can't run them all the time. And > >> realistically, I don't think they provided us the info we needed. There > >> has been at least one fail on Dan's log hack patch for this scenario > >> today, so maybe it will be in there. > > > > I did finally get lucky and hit the failure, and the libvirtd.log has > > provided the info to narrow down the problem in QEMU I believe. I'm > > going to be talking with QEMU developers about it based on this info > > now. > > > > FYI, the logs are approximately 3 MB compressed for a full tempest > > run. If turned on this would be either the 3rd or 4th largest log > > file we'd be collecting, adding 8-10% to the total size of all. > > It's larger than anything other that the ceilometer logs, which are > their own issue. Remember that we are doing 20 - 30k runs a week. So 3MB > + 20k = 60 GB / week. We're currently trying to keep 6 months of logs. > So * 26 = 1.5 TB of libvirt logs. We're currently limited by having a > max of 14 x 1 TB volumes on our log server in Rax.
Could we simply expire the libvirtd.log file after 2 weeks while leaving everything else at 6 months ? > We're hoping to fix that with using swift for log storage, if we get > that in place, we could probably do that on every run. > > Is it possible to make libvirt log to 2 log files? One that is the > normal light load, and an enhanced error log? Then we could maybe make a > decision on cleanup time about if we need the error log saved or not. > Like if things failed we'd keep it. This all starts to get more > complicated, but might be worth exploring. Yes, you can easily do multiple log files at different levels Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
