Hello Bryce,
Am 14.02.2014 00:36, schrieb Bryce Nordgren: > Hi all, > > I’m setting up bareos as a backup solution for my large (60TB) RAID. I’m > backing up to another RAID on a different machine, and the File Daemon and Storage Daemon are connected by GbE. > > The setup works well. With no software compression, I’m maxing out the GbE > link, sustaining 117 MB/s (10^6 *bytes/sec). I’m looking into compression because 7.4Tb takes almost 18 hours to backup, and there will be more. With compression, I have a question: When Bareos reports “[FD|SD] Bytes Written”, is this is a post-compression value? (e.g., the number of bytes read by the FD is larger by the “Software Compression” percentage, correct?) Yes, the calculation of the compression value is compression = (double)100 - 100.0 * ((double)jcr->JobBytes / (double)jcr->ReadBytes); , where uint64_t JobBytes; /* Number of bytes processed this job */ uint64_t ReadBytes; /* Bytes read -- before compression */ (see dird/backup.c and include/jcr.h) > > The short version is: assuming the above, it looks like my “effective backup > rates” (Bytes read by the FD/sec) is: > > No compression: 117Mb/sec (100%) > GZIP: 65.82 Mb/sec (56%) > LZOFAST: 236.44 Mb/sec (202%) > Pigz [1] projection: 532.76 Mb/sec (454%) you could also try lz4, lz4hc and lzfast. > > That last bit is a projection based on comparing the Parallel Implementation > of GZip (PIGZ)[1] to the standard gzip on my system. I started messing with parallel compression because I noted that during a GZIP run, my 24-thread system was ~95% idle. It is interesting to note that PIGZ compiles against versions of libz found on RHEL/CentOS/SL 6 machines. > > I would like to submit a feature request for PIGZ support on Linux-based FDs, > but I wanted to run it by the group first. It produces GZIP-compatible output files, it just does the compression in parallel if you have an SMP machine. (I checked by compressing with PIGZ and uncompressing with gunzip.) > > Thanks, > Bryce > > [1] http://zlib.net/pigz/ > What you can always do is split your fileset into chunks and run backups of each one in parallel. Best regards, Philipp -- Mit freundlichen Grüßen Philipp Storz [email protected] Bareos GmbH & Co. KG Phone: Phone: +49221630693-92 http://www.bareos.com Fax: +49221630693-10 Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646 Geschäftsführer: Stephan Dühr, M. Außendorf, J. Steffens, P. Storz, M. v. Wieringen -- You received this message because you are subscribed to the Google Groups "bareos-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
