subject:"\[dm\-devel\] \[PATCH v4 1\/1\] DM\: inplace compressed DM target"

Re: [dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target (fwd)

2017-02-14 Thread Ram Pai

On Tue, Feb 14, 2017 at 07:09:04AM +0100, Julia Lawall wrote:
> On line 1759, since ret is unsigned it will not be less than 0.

Thanks fixed it. Infact noticed that dm_icomp_io_range_compress() had missed a 
case where
it was supposed to return error; a negative number.  Fixed that aswell.

RP

> 
> julia
> 
> -- Forwarded message --
> Date: Tue, 14 Feb 2017 09:00:39 +0800
> From: kbuild test robot 
> To: kbu...@01.org
> Cc: Julia Lawall 
> Subject: Re: [PATCH v4 1/1] DM: inplace compressed DM target
> 
> CC: kbuild-...@01.org
> In-Reply-To: <1487018545-5061-2-git-send-email-linux...@us.ibm.com>
> 
> Hi Ram,
> 
> [auto build test WARNING on dm/for-next]
> [also build test WARNING on v4.10-rc8 next-20170213]
> [if your patch is applied to the wrong git tree, please drop us a note to 
> help improve the system]
> 
> url:
> https://github.com/0day-ci/linux/commits/Ram-Pai/DM-inplace-compressed-DM-target/20170214-055727
> base:   
> https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git 
> for-next
> :: branch date: 3 hours ago
> :: commit date: 3 hours ago
> 
> >> drivers/md/dm-inplace-compress.c:1759:5-8: WARNING: Unsigned expression 
> >> compared with zero: ret < 0
> 
> git remote add linux-review https://github.com/0day-ci/linux
> git remote update linux-review
> git checkout e7924efaaba5efdcd28f32efbb949ed1153c932c
> vim +1759 drivers/md/dm-inplace-compress.c
> 
> e7924efa Ram Pai 2017-02-13  1743   * @io : io range
> e7924efa Ram Pai 2017-02-13  1744   * @sector_start : the sector on backing 
> storage to which the
> e7924efa Ram Pai 2017-02-13  1745   * compressed data needs to be written.
> e7924efa Ram Pai 2017-02-13  1746   * @meta_start: the page index of the bits 
> corresponding to
> e7924efa Ram Pai 2017-02-13  1747   * @meta_end  : start and end blocks.
> e7924efa Ram Pai 2017-02-13  1748   */
> e7924efa Ram Pai 2017-02-13  1749  static int dm_icomp_compress_write(struct 
> dm_icomp_io_range *io,
> e7924efa Ram Pai 2017-02-13  1750 sector_t sector_start, u64 
> *meta_start, u64 *meta_end)
> e7924efa Ram Pai 2017-02-13  1751  {
> e7924efa Ram Pai 2017-02-13  1752 struct dm_icomp_req *req = io->req;
> e7924efa Ram Pai 2017-02-13  1753 sector_t count = 
> DMCP_BYTES_TO_SECTOR(io->decomp_len);
> e7924efa Ram Pai 2017-02-13  1754 unsigned int comp_len, ret;
> e7924efa Ram Pai 2017-02-13  1755 u64 page_index;
> e7924efa Ram Pai 2017-02-13  1756
> e7924efa Ram Pai 2017-02-13  1757 /* comp_data must be able to 
> accommadate a larger compress buffer */
> e7924efa Ram Pai 2017-02-13  1758 ret = 
> dm_icomp_io_range_compress(req->info, io, _len);
> e7924efa Ram Pai 2017-02-13 @1759 if (ret < 0) {
> e7924efa Ram Pai 2017-02-13  1760 req->result = -EIO;
> e7924efa Ram Pai 2017-02-13  1761 return -EIO;
> e7924efa Ram Pai 2017-02-13  1762 }
> e7924efa Ram Pai 2017-02-13  1763 WARN_ON(comp_len > io->comp_len);
> e7924efa Ram Pai 2017-02-13  1764
> e7924efa Ram Pai 2017-02-13  1765 dm_icomp_get_req(req);
> e7924efa Ram Pai 2017-02-13  1766
> e7924efa Ram Pai 2017-02-13  1767 io->io_req.bi_op = REQ_OP_WRITE;
> 
> ---
> 0-DAY kernel test infrastructureOpen Source Technology Center
> https://lists.01.org/pipermail/kbuild-all   Intel Corporation

-- 
Ram Pai

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target (fwd)

2017-02-14 Thread Julia Lawall

On line 1759, since ret is unsigned it will not be less than 0.

julia

-- Forwarded message --
Date: Tue, 14 Feb 2017 09:00:39 +0800
From: kbuild test robot 
To: kbu...@01.org
Cc: Julia Lawall 
Subject: Re: [PATCH v4 1/1] DM: inplace compressed DM target

CC: kbuild-...@01.org
In-Reply-To: <1487018545-5061-2-git-send-email-linux...@us.ibm.com>

Hi Ram,

[auto build test WARNING on dm/for-next]
[also build test WARNING on v4.10-rc8 next-20170213]
[if your patch is applied to the wrong git tree, please drop us a note to help 
improve the system]

url:
https://github.com/0day-ci/linux/commits/Ram-Pai/DM-inplace-compressed-DM-target/20170214-055727
base:   
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git 
for-next
:: branch date: 3 hours ago
:: commit date: 3 hours ago

>> drivers/md/dm-inplace-compress.c:1759:5-8: WARNING: Unsigned expression 
>> compared with zero: ret < 0

git remote add linux-review https://github.com/0day-ci/linux
git remote update linux-review
git checkout e7924efaaba5efdcd28f32efbb949ed1153c932c
vim +1759 drivers/md/dm-inplace-compress.c

e7924efa Ram Pai 2017-02-13  1743   * @io : io range
e7924efa Ram Pai 2017-02-13  1744   * @sector_start : the sector on backing 
storage to which the
e7924efa Ram Pai 2017-02-13  1745   *   compressed data needs to be written.
e7924efa Ram Pai 2017-02-13  1746   * @meta_start: the page index of the bits 
corresponding to
e7924efa Ram Pai 2017-02-13  1747   * @meta_end  : start and end blocks.
e7924efa Ram Pai 2017-02-13  1748   */
e7924efa Ram Pai 2017-02-13  1749  static int dm_icomp_compress_write(struct 
dm_icomp_io_range *io,
e7924efa Ram Pai 2017-02-13  1750   sector_t sector_start, u64 
*meta_start, u64 *meta_end)
e7924efa Ram Pai 2017-02-13  1751  {
e7924efa Ram Pai 2017-02-13  1752   struct dm_icomp_req *req = io->req;
e7924efa Ram Pai 2017-02-13  1753   sector_t count = 
DMCP_BYTES_TO_SECTOR(io->decomp_len);
e7924efa Ram Pai 2017-02-13  1754   unsigned int comp_len, ret;
e7924efa Ram Pai 2017-02-13  1755   u64 page_index;
e7924efa Ram Pai 2017-02-13  1756
e7924efa Ram Pai 2017-02-13  1757   /* comp_data must be able to 
accommadate a larger compress buffer */
e7924efa Ram Pai 2017-02-13  1758   ret = 
dm_icomp_io_range_compress(req->info, io, _len);
e7924efa Ram Pai 2017-02-13 @1759   if (ret < 0) {
e7924efa Ram Pai 2017-02-13  1760   req->result = -EIO;
e7924efa Ram Pai 2017-02-13  1761   return -EIO;
e7924efa Ram Pai 2017-02-13  1762   }
e7924efa Ram Pai 2017-02-13  1763   WARN_ON(comp_len > io->comp_len);
e7924efa Ram Pai 2017-02-13  1764
e7924efa Ram Pai 2017-02-13  1765   dm_icomp_get_req(req);
e7924efa Ram Pai 2017-02-13  1766
e7924efa Ram Pai 2017-02-13  1767   io->io_req.bi_op = REQ_OP_WRITE;

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation

--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target

2017-02-13 Thread kbuild test robot

Hi Ram,

[auto build test WARNING on dm/for-next]
[also build test WARNING on v4.10-rc8 next-20170213]
[if your patch is applied to the wrong git tree, please drop us a note to help 
improve the system]

url:
https://github.com/0day-ci/linux/commits/Ram-Pai/DM-inplace-compressed-DM-target/20170214-055727
base:   
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git 
for-next
config: openrisc-allyesconfig (attached as .config)
compiler: or32-linux-gcc (GCC) 4.5.1-or32-1.0rc1
reproduce:
wget 
https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross
 -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=openrisc 

All warnings (new ones prefixed by >>):

   drivers/md/dm-inplace-compress.c: In function 'dm_icomp_status':
>> drivers/md/dm-inplace-compress.c:2096:3: warning: format '%ld' expects type 
>> 'long int', but argument 4 has type 'long long int'
   drivers/md/dm-inplace-compress.c:2096:3: warning: format '%ld' expects type 
'long int', but argument 5 has type 'long long int'
   drivers/md/dm-inplace-compress.c:2096:3: warning: format '%ld' expects type 
'long int', but argument 6 has type 'long long int'

vim +2096 drivers/md/dm-inplace-compress.c

  2080  req->locked_locks = 0;
  2081  
  2082  req->cpu = raw_smp_processor_id();
  2083  dm_icomp_queue_req(info, req);
  2084  
  2085  return DM_MAPIO_SUBMITTED;
  2086  }
  2087  
  2088  static void dm_icomp_status(struct dm_target *ti, status_type_t type,
  2089unsigned int status_flags, char *result, unsigned int maxlen)
  2090  {
  2091  struct dm_icomp_info *info = ti->private;
  2092  unsigned int sz = 0;
  2093  
  2094  switch (type) {
  2095  case STATUSTYPE_INFO:
> 2096  DMEMIT("%ld %ld %ld",
  2097  atomic64_read(>uncompressed_write_size),
  2098  atomic64_read(>compressed_write_size),
  2099  atomic64_read(>meta_write_size));
  2100  break;
  2101  case STATUSTYPE_TABLE:
  2102  if (info->write_mode == DMCP_WRITE_BACK)
  2103  DMEMIT("%s %s:%d %s:%s %s:%d", info->dev->name,
  2104  "writeback", info->writeback_delay,

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: [dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target

2017-02-13 Thread kbuild test robot

Hi Ram,

[auto build test WARNING on dm/for-next]
[also build test WARNING on v4.10-rc8 next-20170213]
[if your patch is applied to the wrong git tree, please drop us a note to help 
improve the system]

url:
https://github.com/0day-ci/linux/commits/Ram-Pai/DM-inplace-compressed-DM-target/20170214-055727
base:   
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git 
for-next
config: m68k-allmodconfig (attached as .config)
compiler: m68k-linux-gcc (GCC) 4.9.0
reproduce:
wget 
https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross
 -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=m68k 

All warnings (new ones prefixed by >>):

   drivers/md/dm-inplace-compress.c: In function 'dm_icomp_status':
>> drivers/md/dm-inplace-compress.c:2096:3: warning: format '%ld' expects 
>> argument of type 'long int', but argument 4 has type 'long long int' 
>> [-Wformat=]
  DMEMIT("%ld %ld %ld",
  ^
   drivers/md/dm-inplace-compress.c:2096:3: warning: format '%ld' expects 
argument of type 'long int', but argument 5 has type 'long long int' [-Wformat=]
   drivers/md/dm-inplace-compress.c:2096:3: warning: format '%ld' expects 
argument of type 'long int', but argument 6 has type 'long long int' [-Wformat=]

vim +2096 drivers/md/dm-inplace-compress.c

  2080  req->locked_locks = 0;
  2081  
  2082  req->cpu = raw_smp_processor_id();
  2083  dm_icomp_queue_req(info, req);
  2084  
  2085  return DM_MAPIO_SUBMITTED;
  2086  }
  2087  
  2088  static void dm_icomp_status(struct dm_target *ti, status_type_t type,
  2089unsigned int status_flags, char *result, unsigned int maxlen)
  2090  {
  2091  struct dm_icomp_info *info = ti->private;
  2092  unsigned int sz = 0;
  2093  
  2094  switch (type) {
  2095  case STATUSTYPE_INFO:
> 2096  DMEMIT("%ld %ld %ld",
  2097  atomic64_read(>uncompressed_write_size),
  2098  atomic64_read(>compressed_write_size),
  2099  atomic64_read(>meta_write_size));
  2100  break;
  2101  case STATUSTYPE_TABLE:
  2102  if (info->write_mode == DMCP_WRITE_BACK)
  2103  DMEMIT("%s %s:%d %s:%s %s:%d", info->dev->name,
  2104  "writeback", info->writeback_delay,

---
0-DAY kernel test infrastructureOpen Source Technology Center
https://lists.01.org/pipermail/kbuild-all   Intel Corporation


.config.gz
Description: application/gzip
--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

[dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target

2017-02-13 Thread Ram Pai

This is a simple DM target supporting inplace compression. Its best
suited for SSD. The underlying disk must support 512B sector size.
The target only supports 4k sector size.

Disk layout:
|super|...meta...|..data...|

Store unit is 4k (a block). Super is 1 block, which stores meta  and
data size and compression algorithm. Meta is a bitmap. For each data
 block, there are 5 bits meta.

Data:

Data of   a block is compressed. Compressed  data  is round up to 512B,
which is the payload. In disk, payload is  stored at  the beginning of
logical sector  of the block. Let's look  at an example.  Say we store
data to block A, which  is in sector  B(A*8), its orginal  size is 4k,
compressed size is  1500.Compressed data (CD)  will  use three
sectors (512B). The three  sectors  are the  payload. Payload  will be
stored at sector B.

---
... | CD1 | CD2 | CD3 |   |   |   |   || ...
---
^B^B+1  ^B+2  ^B+7 ^B+8

For this block, we will not use sector B+3 to B+7 (a hole). We use four
meta  bits  to  present payload  size. The compressed size (1500) isn't
stored in meta directly. Instead, we  store  it  at  the last 32bits of
payload. In this  example, we store it at the  end  of  sector  B+2. If
compressed size + sizeof(32bits)  crosses a   sector, payload size will
increase one sector.  If payload  uses 8 sectors, we store uncompressed
data directly.

If IO size is bigger than one block, we can store the data as an extent.
Data of the  whole extent will compressed and stored in the similar way
like above.  The first  block of the extent is the head, all others are
the tail.  If extent is 1 block,  the  block  is head. We have 1 bit of
meta to present if a  block  is  head  or  tail. If 4 meta bits of head
block can't  store  extent payload size, we will borrow tail block meta
bits to  store  payload  size.   Max  allowd extent size is 128k, so we
don't compress/decompress too big size data.

Meta:
Modifying   data   will modify meta too. Meta will be written(flush) to
disk   depending   on   meta   write   policy. We support writeback and
writethrough mode.  In  writeback mode, meta will be written to disk in
an interval or a  FLUSH  request.  In  writethrough mode, data and meta
data will be written to disk together.

Advantages:

1. Simple. Since  we  store  compressed  data  in-place,  we don't need
   complicated disk data management.
2. Efficient. For  each  4k, we only need 5 bits meta. 1T data will use
less than 200M meta, so we  can  load  all meta into memory. And actual
compression size is in payload. So   if  IO doesn't need RMW and we use
writeback meta flush, we don't  need  extra IO for meta.

Disadvantages:

1. hole. Since we   store  compressed data in-place, there are a lot of
   holes (in above  example,  B+3 - B+7) Hole can impact IO, because we
   can't do IO merge.

2. 1:1 size. Compression  doesn't  change disk  size. If disk is 1T, we
   can only store 1T data even we do compression.

But this is for SSD only. Generally SSD firmware has a FTL layer to map
disk  sectors  to flash nand. High end SSD firmware has filesystem-like
FTL.

1. hole. Disk has a lot of holes, but SSD FTL   can   still  store data
   contiguous in nand. Even if we can't do IO   merge in  OS layer, SSD
   firmware can do it.

2. 1:1 size. On one side, we write compressed data to SSD, which means
   less  data is  written to SSD. This will be very helpful to improve
   SSD garbage collection, and  so write speed and life cycle. So even
   this is a problem, the target  is still helpful. On the other side,
   advanced SSD FTL can easily do thin provision. For example, if nand
   is   1T   and   we   let   SSD   report   it   as   2T,   and   use
   the  SSD  as  compressed target. In such SSD, we don't have the 1:1
   size issue.

So even if   SSD   FTL   cannot   map   non-contiguous disk sectors to
contiguous nand, the compression target can still function well.


Signed-off-by: Shaohua Li 
Signed-off-by: Ram Pai 
---
 .../device-mapper/dm-inplace-compress.txt  |  155 ++
 drivers/md/Kconfig |6 +
 drivers/md/Makefile|2 +
 drivers/md/dm-inplace-compress.c   | 2220 
 drivers/md/dm-inplace-compress.h   |  187 ++
 5 files changed, 2570 insertions(+)
 create mode 100644 Documentation/device-mapper/dm-inplace-compress.txt
 create mode 100644 drivers/md/dm-inplace-compress.c
 create mode 100644 drivers/md/dm-inplace-compress.h

diff --git a/Documentation/device-mapper/dm-inplace-compress.txt 
b/Documentation/device-mapper/dm-inplace-compress.txt
new file mode 100644
index 000..1835369
--- /dev/null
+++ b/Documentation/device-mapper/dm-inplace-compress.txt
@@ -0,0 +1,155 @@
+Device-Mapper's "inplace-compress" target provides

Re: [dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target (fwd)

Re: [dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target (fwd)

Re: [dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target

Re: [dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target

[dm-devel] [PATCH v4 1/1] DM: inplace compressed DM target

5 matches

Site Navigation

Mail list logo

Footer information