Ilya, Zstd itself has default compression level 3. I just used that number to be consistent with the library defaults.
I will check if there is a significant difference in performance. Sergi пн, 19 нояб. 2018 г. в 14:59, Ilya Kasnacheev <ilya.kasnach...@gmail.com>: > Hello! > > You have zstd default level of 3. In my tests, zstd usually performed much > better with compression level 2. Please consider. > > I admire your effort! > > Regards, > -- > Ilya Kasnacheev > > > пн, 19 нояб. 2018 г. в 14:02, Sergi Vladykin <sergi.vlady...@gmail.com>: > > > Right now the functionality has nothing to do with WAL, but your idea > > definitely makes sense and worth being implemented as a next step. > > > > Sergi > > > > пн, 19 нояб. 2018 г. в 13:58, Andrey Mashenkov < > andrey.mashen...@gmail.com > > >: > > > > > Hi Sergi, > > > > > > It is not clear for me will your changes affect PageSnapshot WAL > record. > > > Is it possible to add compression support for PageSnapshot WAL record > as > > > well, to reduce WAL size? > > > > > > Thanks. > > > > > > On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin < > sergi.vlady...@gmail.com > > > > > > wrote: > > > > > > > Folks, > > > > > > > > I've implemented page compression for persistent store and going to > > merge > > > > it to master. > > > > > > > > https://github.com/apache/ignite/pull/5200 > > > > > > > > Some design notes: > > > > > > > > It employs "hole punching" approach, it means that the pages are kept > > > > uncompressed in memory, > > > > but when they get written to disk, they will be compressed and all > the > > > > extra file system blocks for the page will be released. Thus the > > storage > > > > files become sparse. > > > > > > > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY > and > > > > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, > > which > > > > basically just takes only meaningful data from half-filled pages but > > does > > > > not apply any compression. It is easy to add more if needed. > > > > > > > > Since we can release only full file system blocks which are typically > > 4k > > > > size, user must configure page size to be at least multiple FS > blocks, > > > e.g. > > > > 8k or 16k. It also means that max compression ratio here is > > fsBlockSize / > > > > pageSize = 4k / 16k = 0.25 > > > > > > > > It is possible to enable compression for existing databases if they > > were > > > > configured for large enough page size. In this case pages will be > > written > > > > to disk in compressed form when updated, and the database will become > > > > compressed gradually. > > > > > > > > There will be 2 new properties on CacheConfiguration > > > > (setDiskPageCompression and setDiskPageCompressionLevel) to setup > disk > > > page > > > > compression. > > > > > > > > Compression dictionaries are not supported at the time, but may in > the > > > > future. IMO it should be added as a separate feature if needed. > > > > > > > > The only supported platform for now is Linux. Since all popular file > > > > systems support sparse files, it must be relatively easy to support > > more > > > > platforms. > > > > > > > > Please take a look and provide your thoughts and suggestions. > > > > > > > > Thanks! > > > > > > > > Sergi > > > > > > > > > > > > > -- > > > Best regards, > > > Andrey V. Mashenkov > > > > > >
