Re: [PATCH v1 0/7] writeback incompressible pages to storage
On Thu, Jun 29, 2017 at 06:17:13PM +0900, Sergey Senozhatsky wrote: > Hello, > > On (06/29/17 17:47), Minchan Kim wrote: > [..] > > > > This patch supports writeback feature of zram so admin can set up > > > > a block device and with it, zram can save the memory via writing > > > > out the incompressile pages once it found it's incompressible pages > > > > (1/4 comp ratio) instead of keeping the page in memory. > > > > > > hm, alternative idea. just an idea. can we try compressing the page > > > with another algorithm? example: downcast from lz4 to zlib? we can > > > set up a fallback "worst case" algorithm, so each entry can contain > > > additional flag that would tell if the src page was compressed with > > > the fast or slow algorithm. that sounds to me easier than "create a > > > new block device and bond it to zram, etc". but I may be wrong. > > > > We tried it although it was static not dynamic adatation you suggested. > > could you please explain more? I'm not sure I understand what > was the configuration (what is static adaptation?). echo inflate > /sys/block/zramX/comp_algorighm > > > However problem was media-stream data so zlib, lzam added just pointless > > overhead. > > would that overhead be bigger than a full-blown I/O request to > another block device (potentially slow, or under load, etc. etc.)? The problem is not a overhead but memeory saving. Although we use higher compression algorithm like zlib, lzma, the comp ratio was not different with lzo and lz4 so it was added pointless overhead without any saving.
Re: [PATCH v1 0/7] writeback incompressible pages to storage
On Thu, Jun 29, 2017 at 06:17:13PM +0900, Sergey Senozhatsky wrote: > Hello, > > On (06/29/17 17:47), Minchan Kim wrote: > [..] > > > > This patch supports writeback feature of zram so admin can set up > > > > a block device and with it, zram can save the memory via writing > > > > out the incompressile pages once it found it's incompressible pages > > > > (1/4 comp ratio) instead of keeping the page in memory. > > > > > > hm, alternative idea. just an idea. can we try compressing the page > > > with another algorithm? example: downcast from lz4 to zlib? we can > > > set up a fallback "worst case" algorithm, so each entry can contain > > > additional flag that would tell if the src page was compressed with > > > the fast or slow algorithm. that sounds to me easier than "create a > > > new block device and bond it to zram, etc". but I may be wrong. > > > > We tried it although it was static not dynamic adatation you suggested. > > could you please explain more? I'm not sure I understand what > was the configuration (what is static adaptation?). echo inflate > /sys/block/zramX/comp_algorighm > > > However problem was media-stream data so zlib, lzam added just pointless > > overhead. > > would that overhead be bigger than a full-blown I/O request to > another block device (potentially slow, or under load, etc. etc.)? The problem is not a overhead but memeory saving. Although we use higher compression algorithm like zlib, lzma, the comp ratio was not different with lzo and lz4 so it was added pointless overhead without any saving.
Re: [PATCH v1 0/7] writeback incompressible pages to storage
Hello, On (06/29/17 17:47), Minchan Kim wrote: [..] > > > This patch supports writeback feature of zram so admin can set up > > > a block device and with it, zram can save the memory via writing > > > out the incompressile pages once it found it's incompressible pages > > > (1/4 comp ratio) instead of keeping the page in memory. > > > > hm, alternative idea. just an idea. can we try compressing the page > > with another algorithm? example: downcast from lz4 to zlib? we can > > set up a fallback "worst case" algorithm, so each entry can contain > > additional flag that would tell if the src page was compressed with > > the fast or slow algorithm. that sounds to me easier than "create a > > new block device and bond it to zram, etc". but I may be wrong. > > We tried it although it was static not dynamic adatation you suggested. could you please explain more? I'm not sure I understand what was the configuration (what is static adaptation?). > However problem was media-stream data so zlib, lzam added just pointless > overhead. would that overhead be bigger than a full-blown I/O request to another block device (potentially slow, or under load, etc. etc.)? -ss
Re: [PATCH v1 0/7] writeback incompressible pages to storage
Hello, On (06/29/17 17:47), Minchan Kim wrote: [..] > > > This patch supports writeback feature of zram so admin can set up > > > a block device and with it, zram can save the memory via writing > > > out the incompressile pages once it found it's incompressible pages > > > (1/4 comp ratio) instead of keeping the page in memory. > > > > hm, alternative idea. just an idea. can we try compressing the page > > with another algorithm? example: downcast from lz4 to zlib? we can > > set up a fallback "worst case" algorithm, so each entry can contain > > additional flag that would tell if the src page was compressed with > > the fast or slow algorithm. that sounds to me easier than "create a > > new block device and bond it to zram, etc". but I may be wrong. > > We tried it although it was static not dynamic adatation you suggested. could you please explain more? I'm not sure I understand what was the configuration (what is static adaptation?). > However problem was media-stream data so zlib, lzam added just pointless > overhead. would that overhead be bigger than a full-blown I/O request to another block device (potentially slow, or under load, etc. etc.)? -ss
Re: [PATCH v1 0/7] writeback incompressible pages to storage
Hi Sergey, On Thu, Jun 29, 2017 at 12:41:57AM +0900, Sergey Senozhatsky wrote: > Hello, > > On (06/26/17 15:52), Minchan Kim wrote: > [..] > > zRam is useful for memory saving with compressible pages but sometime, > > workload can be changed and system has lots of incompressible pages > > which is very harmful for zram. > > could do. that makes zram quite complicated, to be honest. no offense, > but the whole zram's "good compression" margin looks to me completely > random and quite unreasonable. building a complex logic atop of random > logic is a bit tricky. but I see what problem you are trying to address. > > > This patch supports writeback feature of zram so admin can set up > > a block device and with it, zram can save the memory via writing > > out the incompressile pages once it found it's incompressible pages > > (1/4 comp ratio) instead of keeping the page in memory. > > hm, alternative idea. just an idea. can we try compressing the page > with another algorithm? example: downcast from lz4 to zlib? we can > set up a fallback "worst case" algorithm, so each entry can contain > additional flag that would tell if the src page was compressed with > the fast or slow algorithm. that sounds to me easier than "create a > new block device and bond it to zram, etc". but I may be wrong. We tried it although it was static not dynamic adatation you suggested. However problem was media-stream data so zlib, lzam added just pointless overhead. Thanks.
Re: [PATCH v1 0/7] writeback incompressible pages to storage
Hi Sergey, On Thu, Jun 29, 2017 at 12:41:57AM +0900, Sergey Senozhatsky wrote: > Hello, > > On (06/26/17 15:52), Minchan Kim wrote: > [..] > > zRam is useful for memory saving with compressible pages but sometime, > > workload can be changed and system has lots of incompressible pages > > which is very harmful for zram. > > could do. that makes zram quite complicated, to be honest. no offense, > but the whole zram's "good compression" margin looks to me completely > random and quite unreasonable. building a complex logic atop of random > logic is a bit tricky. but I see what problem you are trying to address. > > > This patch supports writeback feature of zram so admin can set up > > a block device and with it, zram can save the memory via writing > > out the incompressile pages once it found it's incompressible pages > > (1/4 comp ratio) instead of keeping the page in memory. > > hm, alternative idea. just an idea. can we try compressing the page > with another algorithm? example: downcast from lz4 to zlib? we can > set up a fallback "worst case" algorithm, so each entry can contain > additional flag that would tell if the src page was compressed with > the fast or slow algorithm. that sounds to me easier than "create a > new block device and bond it to zram, etc". but I may be wrong. We tried it although it was static not dynamic adatation you suggested. However problem was media-stream data so zlib, lzam added just pointless overhead. Thanks.
Re: [PATCH v1 0/7] writeback incompressible pages to storage
Hello, On (06/26/17 15:52), Minchan Kim wrote: [..] > zRam is useful for memory saving with compressible pages but sometime, > workload can be changed and system has lots of incompressible pages > which is very harmful for zram. could do. that makes zram quite complicated, to be honest. no offense, but the whole zram's "good compression" margin looks to me completely random and quite unreasonable. building a complex logic atop of random logic is a bit tricky. but I see what problem you are trying to address. > This patch supports writeback feature of zram so admin can set up > a block device and with it, zram can save the memory via writing > out the incompressile pages once it found it's incompressible pages > (1/4 comp ratio) instead of keeping the page in memory. hm, alternative idea. just an idea. can we try compressing the page with another algorithm? example: downcast from lz4 to zlib? we can set up a fallback "worst case" algorithm, so each entry can contain additional flag that would tell if the src page was compressed with the fast or slow algorithm. that sounds to me easier than "create a new block device and bond it to zram, etc". but I may be wrong. -ss
Re: [PATCH v1 0/7] writeback incompressible pages to storage
Hello, On (06/26/17 15:52), Minchan Kim wrote: [..] > zRam is useful for memory saving with compressible pages but sometime, > workload can be changed and system has lots of incompressible pages > which is very harmful for zram. could do. that makes zram quite complicated, to be honest. no offense, but the whole zram's "good compression" margin looks to me completely random and quite unreasonable. building a complex logic atop of random logic is a bit tricky. but I see what problem you are trying to address. > This patch supports writeback feature of zram so admin can set up > a block device and with it, zram can save the memory via writing > out the incompressile pages once it found it's incompressible pages > (1/4 comp ratio) instead of keeping the page in memory. hm, alternative idea. just an idea. can we try compressing the page with another algorithm? example: downcast from lz4 to zlib? we can set up a fallback "worst case" algorithm, so each entry can contain additional flag that would tell if the src page was compressed with the fast or slow algorithm. that sounds to me easier than "create a new block device and bond it to zram, etc". but I may be wrong. -ss
