Hello everyone, Back at the Portland summit the Infra team committed to archiving six months of test logs for Openstack. Since then we have managed to do just that. However, more recently we have seen the growth rate on those logs continue to grow beyond what is a currently sustainable level.
For reasons, we currently store logs on a filesystem backed by cinder volumes. Rackspace limits the size and number of volumes attached to a single host meaning the upper bound on the log archive filesystem is ~12TB and we are almost there. You can see real numbers and pretty graphs on our cacti server [0]. Long term we are trying to move to putting all of the logs in swift, but it turns out there are some use case issues we need to sort out around that before we can do so (but this is being worked on so should happen). Until that day arrives we need to work on logging more smartly, and if we can't do that we will have to reduce the log retention period. So what can you do? Well it appears that our log files may need a diet. I have listed the worst offenders below (after a small sampling, there may be more) and it would be great if we could go through those with a comb and figure out if we are logging actually useful data. The great thing about doing this is it will make lives better for deployers of Openstack too. Some initial checking indicates a lot of this noise may be related to ceilometer. It looks like it is logging AMQP stuff frequently and inflating the logs of individual services as it polls them. Offending files from tempest tests: screen-n-cond.txt.gz 7.3M screen-ceilometer-collector.txt.gz 6.0M screen-n-api.txt.gz 3.7M screen-n-cpu.txt.gz 3.6M tempest.txt.gz 2.7M screen-ceilometer-anotification.txt.gz 1.9M subunit_log.txt.gz 1.5M screen-g-api.txt.gz 1.4M screen-ceilometer-acentral.txt.gz 1.4M screen-n-net.txt.gz 1.4M from: http://logs.openstack.org/52/81252/2/gate/gate-tempest-dsvm-full/488bc4e/logs/?C=S;O=D Unittest offenders: Nova subunit_log.txt.gz 14M Neutron subunit_log.txt.gz 7.8M Keystone subunit_log.txt.gz 4.8M Note all of the above files are compressed with gzip -9 and the filesizes above reflect compressed file sizes. Debug logs are important to you guys when dealing with Jenkins results. We want your feedback on how we can make this better for everyone. [0] http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all Thank you, Clark Boylan _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
