Mattherw, Dmitry I would suggest to use: 1) multi-threaded utilities 2) use xz for small snapshots (<1gb) and lrzip for bigger snapshots
On Mon, Nov 24, 2014 at 2:25 PM, Matthew Mosesohn <mmoses...@mirantis.com> wrote: > I did this exercise over many iterations during Docker container > packing and found that as long as the data is under 1gb, it's going to > compress really well with xz. Over 1gb and lrzip looks more attractive > (but only on high memory systems). In reality, we're looking at log > footprints from OpenStack environments on the order of 500mb to 2gb. > > xz is very slow on single-core systems with 1.5gb of memory, but it's > quite a bit faster if you run it on a more powerful system. I've found > level 4 compression to be the best compromise that works well enough > that it's still far better than gzip. If increasing compression time > by 3-5x is too much for you guys, why not just go to bzip? You'll > still improve compression but be able to cut back on time. > > Best Regards, > Matthew Mosesohn > > On Mon, Nov 24, 2014 at 3:14 PM, Vladimir Kuklin <vkuk...@mirantis.com> > wrote: > > IMO, we should get a bunch of snapshots and decide which compression to > use > > according to the results of an experiment. XZ compression takes much > longer, > > so you will need to use parallel xz compression utility. > > > > On Fri, Nov 21, 2014 at 9:09 PM, Tomasz Napierala < > tnapier...@mirantis.com> > > wrote: > >> > >> > >> > On 21 Nov 2014, at 16:55, Dmitry Pyzhov <dpyz...@mirantis.com> wrote: > >> > > >> > We have a request for change compression from gz to xz. This simple > >> > change halfs our snapshots. Does anyone has any objections? Otherwise > we > >> > will include this change in 6.0 release. > >> > >> I agree with the change, but it shouldn’t be high > >> > >> Regards, > >> -- > >> Tomasz 'Zen' Napierala > >> Sr. OpenStack Engineer > >> tnapier...@mirantis.com > >> > >> > >> > >> > >> > >> > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > -- > > Yours Faithfully, > > Vladimir Kuklin, > > Fuel Library Tech Lead, > > Mirantis, Inc. > > +7 (495) 640-49-04 > > +7 (926) 702-39-68 > > Skype kuklinvv > > 45bk3, Vorontsovskaya Str. > > Moscow, Russia, > > www.mirantis.com > > www.mirantis.ru > > vkuk...@mirantis.com > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Yours Faithfully, Vladimir Kuklin, Fuel Library Tech Lead, Mirantis, Inc. +7 (495) 640-49-04 +7 (926) 702-39-68 Skype kuklinvv 45bk3, Vorontsovskaya Str. Moscow, Russia, www.mirantis.com <http://www.mirantis.ru/> www.mirantis.ru vkuk...@mirantis.com
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev