[zfs-discuss] zfs compression algorithm : jpeg ??

2009-09-04 Thread Len Zaifman
We have groups generating terabytes a day of image data from lab instruments and saving them to an X4500. We have tried lzbj : compressratio = 1.13 in 11 hours , 1.3 TB - 1.1 TB gzip -9 : compress ratio = 1.68 in 37 hours, 1.3 TB - .75 TB The filesystem performance was

Re: [zfs-discuss] zfs compression algorithm : jpeg ??

2009-09-04 Thread Richard Elling
On Sep 4, 2009, at 12:23 PM, Len Zaifman wrote: We have groups generating terabytes a day of image data from lab instruments and saving them to an X4500. Wouldn't it be easier to compress at the application, or between the application and the archiving file system? We have tried lzbj :

Re: [zfs-discuss] zfs compression algorithm : jpeg ??

2009-09-04 Thread Louis-Frédéric Feuillette
On Fri, 2009-09-04 at 13:41 -0700, Richard Elling wrote: On Sep 4, 2009, at 12:23 PM, Len Zaifman wrote: We have groups generating terabytes a day of image data from lab instruments and saving them to an X4500. Wouldn't it be easier to compress at the application, or between the

Re: [zfs-discuss] zfs compression algorithm : jpeg ??

2009-09-04 Thread Nicolas Williams
On Fri, Sep 04, 2009 at 01:41:15PM -0700, Richard Elling wrote: On Sep 4, 2009, at 12:23 PM, Len Zaifman wrote: We have groups generating terabytes a day of image data from lab instruments and saving them to an X4500. Wouldn't it be easier to compress at the application, or between the

Re: [zfs-discuss] zfs compression algorithm : jpeg ??

2009-09-04 Thread Bob Friesenhahn
On Fri, 4 Sep 2009, Louis-Frédéric Feuillette wrote: JPEG2000 uses arithmetic encoding to do the final compression step. Arithmetic encoding has a higher compression rate (in general) than gzip-9, lzbj or others. There is an opensource implementation of jpeg2000 called jasper[1]. Jasper is