When comparing compression performance I see the following performance, is anyone else getting significantly higher on any other systems?
Read Speeds: lz4 with null fill data, ~ 90MB/s lz4 with a SAS data set, ~40-50MB/s z with null fill data, ~ 15MB/s z with a SAS data set, ~ 5MB/s While on a 4G page pool I tested each of these file sizes and got roughly identical performance in all cases: 1 GB, 5 GB, and 10GB. This was on an S824 (p8) with read performance typically going to 1.2GB/s of read on a single thread (non-compressed). Doing a "very limited test" in Production hardware E850, 8gb Page Pool, with ~2.4 GB/s of read on a single thread (non-compressed) I got very similar results. In all cases the work was done from the NSD master, and due to the file sizes and the difference in page pool, i'd expect the 1gb files to move at a significantly faster pace if pagepool was a factor. If anyone could tell me what performance they get on their platform and what OS or Hardware they're using, I'd very much be interested. I'm debating if using GPFS to migrate the files to a .gz compressed version, and then providing a fifo mechanism to pipe through the compressed data wouldn't be a better solution. Alec On Wed, Jan 20, 2021 at 2:10 PM Alec <[email protected]> wrote: > I see a lot of references to the page pool. Our page pool is only 8 gb and > our files can be very large into the terrabytes. > > I will try increasing the page pool in dev to 2x a test file and see if > the problem resolves. > > Any documentation on the correlation here would be nice. > > I will see if I can get rights for the debug as well. > > Alec >
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
