Jeff Squyres wrote: > On Apr 22, 2008, at 9:03 AM, Tomas Ukkonen wrote: > >> I read from somewhere that OpenMPI supports >> >> some kind of data compression but I couldn't find >> any information about it. >> >> Is this true and how it can be used? >> > Nope, sorry -- not true. > > This just came up in a different context, actually. We added some > preliminary compression on our startup/mpirun messages and found that > it really had no effect; any savings that you get in bandwidth (and > therefore overall wall clock time) are eaten up by the time necessary > to compress/uncompress the messages. There were a few more things we > could have tried, but frankly we had some higher priority items to > finish for the upcoming v1.3 series. :-( > Ok, so I have to do it myself. Not a problem really because there are only few places where the compression really seems to matter. >> Does anyone have any experiences about using it? >> >> Is it possible to use compression in just some >> subset of communications (communicator >> specific compression settings)? >> >> In our MPI application we are transferring large >> amounts of sparse/redundant data that compresses >> very well. Also my initial tests showed significant >> improvements in performance. >> > > If your particular data is well-suited for fast compression, you might > want to compress it before calling MPI_SEND / after calling MPI_RECV. > Use the MPI_BYTE datatype to send/receive the messages, and then MPI > won't do anything additional for datatype conversions, etc Yeah, already did something like this. I have a situation where all the nodes are sending large amounts of redundant data at once. The combination: "compress --> MPI_SEND --> MPI_RECV --> decompress" works of course, but it forces one to allocate large amounts of memory (or diskspace) for the compressed data. You can do it manually in parts of course, but it would be nice if MPI library could do it behind the scenes.
Thanks, -- Tomas Ukkonen
