On Mon, 3 Dec 2007, michel agoyan wrote: > In fact I'm looking for a binary format because it is more compact than > a text file. > I simple have an huge amount of waveforms to deal with coming from a > digital scope. > I want to process them (average correlation ...) and write the results > using a format > that is compact and that allows the use of waveform viewer such as gwave.
What exactly is "a huge amount" of waveforms? As long as an individual sim run isn't something like 10GB, it should be manageable. If you'd like to chop the amount of data down, the following typically works very well: 1) Separate the signals into different streams. Include the absolute time data as a separate stream. 2) Separate each individual floating point value change for a net as time/sign/exp/mantissa values. Note that for the time value in this 4-tuple, you want an index into an absolute time array, not the actual time value. 3) Emit the deltas (compared to previous value) of each of the time/sign/exp/mantissa to their respective (sub)streams as variable length integers. 4) Filter through a compressor such as zlib. Compression ratio will improve if like streams (e.g., all exponents) are compressed back-to-back but this will require some experimentation. I do something similar to this with the vcd recoder in gtkwave and can read 2GB of vcd into memory on my laptop. Granted, binary data is different from floating point, however, the grouping concept of bunching like data with like data applies. Note that compression will improve if you can pre-process the value changes in (2) through a predictor--in that case instead of deltas for (3) you would emit the error as compared to the predicted value. The sourcecode for FLAC probably could be modified or used for ideas. -t _______________________________________________ geda-dev mailing list [email protected] http://www.seul.org/cgi-bin/mailman/listinfo/geda-dev
