Right now the functionality has nothing to do with WAL, but your idea definitely makes sense and worth being implemented as a next step.
Sergi пн, 19 нояб. 2018 г. в 13:58, Andrey Mashenkov <andrey.mashen...@gmail.com>: > Hi Sergi, > > It is not clear for me will your changes affect PageSnapshot WAL record. > Is it possible to add compression support for PageSnapshot WAL record as > well, to reduce WAL size? > > Thanks. > > On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin <sergi.vlady...@gmail.com> > wrote: > > > Folks, > > > > I've implemented page compression for persistent store and going to merge > > it to master. > > > > https://github.com/apache/ignite/pull/5200 > > > > Some design notes: > > > > It employs "hole punching" approach, it means that the pages are kept > > uncompressed in memory, > > but when they get written to disk, they will be compressed and all the > > extra file system blocks for the page will be released. Thus the storage > > files become sparse. > > > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and > > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which > > basically just takes only meaningful data from half-filled pages but does > > not apply any compression. It is easy to add more if needed. > > > > Since we can release only full file system blocks which are typically 4k > > size, user must configure page size to be at least multiple FS blocks, > e.g. > > 8k or 16k. It also means that max compression ratio here is fsBlockSize / > > pageSize = 4k / 16k = 0.25 > > > > It is possible to enable compression for existing databases if they were > > configured for large enough page size. In this case pages will be written > > to disk in compressed form when updated, and the database will become > > compressed gradually. > > > > There will be 2 new properties on CacheConfiguration > > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk > page > > compression. > > > > Compression dictionaries are not supported at the time, but may in the > > future. IMO it should be added as a separate feature if needed. > > > > The only supported platform for now is Linux. Since all popular file > > systems support sparse files, it must be relatively easy to support more > > platforms. > > > > Please take a look and provide your thoughts and suggestions. > > > > Thanks! > > > > Sergi > > > > > -- > Best regards, > Andrey V. Mashenkov >
