--- Daniel O'Connor <[EMAIL PROTECTED]> wrote: > Hi, > I work for a company which makes meteor and wind radar > (http://www.gsoft.com.au). > > On occasion (ie during meteor showers such as the Leonids) we > configure > the system to save raw data as it comes out of the acquisition > system, > the data rate for this varies (depends on acquisition parameters and > number of coherent integrations etc), but usually it is around > 600kb/sec. > > The data consists of 16 bit samples, one for each digital channel (in > phase and quad for each receiver) at each sampled height. Typically > there are 10 channels and 30 range gates, so for a single transmitter > pulse tick you get 10 * 30 = 300 16 bit samples. > > In an effort to compress the data I looked at lossless audio > compressors > (as well as gzip etc) and I ended up trying flac and shorten. > > > Program Compressed Size Compression time > Decompression time > gzip 358 Mb (84.9%) 147.96 sec 21.28 > sec > bzip2 330 Mb (78.2%) 589.01 sec 232.13 > sec > shorten -c 10 326 Mb (77.4%) 91.39 sec 55.89 > sec > shorten -c 300 313 Mb (74.3%) 102.08 sec 53.04 > sec > flac 5chn 328 Mb (77.8%) 131.22 sec 66.54 > sec > flac 8chn 332 Mb (78.7%) 139.57 sec 66.74 > sec > flac 10chn 327 Mb (77.5%) 126.32 sec n/a > sec
In every case I've ever seen, default FLAC compresses more than default shorten. The anomaly here is due to the number of channels; it appears that you really do have 300 uncorrelated channels of data (uncorrelated according to the linear predictor I mean). FLAC only supports 8 channels, but I would bet that the compression would be a lot better if you separated the 300 channels stream into several streams (<=8 channels each) and compressed them with FLAC. That may not be practical for your application. > Note the 10 channel version was done by me changing the format.h file > and recompiling, but alas I couldn't get decompression to go :( Yes, the hard limit of 8 channels is binding everywhere; it would take some more hacking at the code to make it handle more channels. > The command line I used for flac was > /usr/bin/time flac --lax --endian=little --channels=5 --bps=16 > --sample-rate=64320 --sign=signed --force-raw-format reallyraw -o > reallyraw.flac-5chn > > I needed the --lax otherwise it generated a > FLAC__STREAM_ENCODER_NOT_STREAMABLE error. Yes, by default the encoder tries to encode according to a subset of parameters that makes for easier decoding. You need --lax to turn that off. > I am basically letting you guys know about a possibly more > interesting > than normal application for your code and soliciting any tips you may > have :) You could try splitting the signal like I mentioned. It might take some experimenting to find channel combinations that have some exploitable correlation. See the format page for the kinds of inter-channel and intra-channel decorrelation FLAC does, it might give you some more ideas: http://flac.sourceforge.net/format.html Josh __________________________________________________ Do you Yahoo!? Yahoo! Web Hosting - Let the expert host your site http://webhosting.yahoo.com ------------------------------------------------------- This sf.net email is sponsored by: To learn the basics of securing your web site with SSL, click here to get a FREE TRIAL of a Thawte Server Certificate: http://www.gothawte.com/rd524.html _______________________________________________ Flac-dev mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/flac-dev
