Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
On Tue, Feb 18, 2014 at 10:38:18PM +0300, Sergey Senozhatsky wrote:
> On (02/18/14 15:04), Minchan Kim wrote:
> [..]
> > 1.8.5.3
> >
> > As you can see, your patch made zram too much regressed when it
> > used one stream buffer. The reason I guessed is overhead of
> > scheduling by sleep/wakeup when others couldn't find a idle stream
> > so I had an experiment with below simple patch to restore old behavior
> > so it works well again. The reason old behaivor was really good is
> > it uses mutex with spin_on_owner so that it could avoid frequent
> > sleep/wakeup.
> >
> > A solution I could think is that we could grow up the number of
> > buffer dynamically up to max_buffers(ex, 2 * number of CPU) so we
> > could grab a idle stream easily for each CPU while we could release
> > buffers by shrinker when the memory pressure is severe.
> > Of course, we should keep one buffer to work.
> >
> > Another solution I can imagaine is to define CONFIG_ZRAM_MULTI_STREAM
> > and CONFIG_ZRAM_SINGLE_STREAM(ie, default) so we couldn't break good
> > performance for old cases and new users who want write parallel
> > could use ZRAM_MULTI which should be several streams at a default
> > rather than a stream. And we could ehhacne it further by dynamic control
> > as I mentioned.
> >
> > Thoughts?
> >
>
> like the following one (this patch limits the number of streams to
> num_online_cpus(). I really don't mind to limit zstrm to
> N * num_online_cpus() where N could be 2)
>
> iozone -t 3 -R -r 16K -s 60M -I +Z
>
> write test:
> (old core i5 laptop, 2 cpus + 2 HyperThreading)
>
>basemulti buffer
>
> " Initial write " 574938.14 716623.64
> "Rewrite " 573541.22 1499074.62
> " Random write " 587016.14 1393996.66
> " Pwrite " 595616.45 711028.70
> " Fwrite " 1482843.62 1493398.12
>
I guess you tested it for zram-block, not zram-swap.
If we use it as zram-swap and happens parallel swapout,
write performance would be dropped due to frequent sleep/wakeup.
>
> zcomp_strm_get() and zcomp_strm_put() were updated.
> zcomp keeps the number allocated and still active (not freed)
> streams (atomic_t). zcomp performs zstrm free() only if
> current number of streams exceeds the number of online
> cpus (cpu went offline).
>
>
> not a properly `aligned' patch, just to demonstrate the idea.
>
> thoughts?
>
> thanks.
>
> ---
>
> >From 0e16f26480933d22339675ca938049dfb21557e2 Mon Sep 17 00:00:00 2001
> From: Sergey Senozhatsky
> Date: Tue, 11 Feb 2014 00:28:46 +0300
> Subject: [PATCH] zram: multi stream compressing backend abstraction
>
> Signed-off-by: Sergey Senozhatsky
> ---
> drivers/block/zram/zcomp.c | 174
> +
> drivers/block/zram/zcomp.h | 58 ++
> drivers/block/zram/zcomp_lzo.c | 48
> 3 files changed, 280 insertions(+)
> create mode 100644 drivers/block/zram/zcomp.c
> create mode 100644 drivers/block/zram/zcomp.h
> create mode 100644 drivers/block/zram/zcomp_lzo.c
>
> diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
> new file mode 100644
> index 000..6bf49fc
> --- /dev/null
> +++ b/drivers/block/zram/zcomp.c
> @@ -0,0 +1,174 @@
> +/*
> + * Copyright (C) 2014 Sergey Senozhatsky.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +
> +#include
> +#include
> +#include
> +#include
> +#include
> +
> +#include "zcomp.h"
> +
> +extern struct zcomp_backend zcomp_lzo;
> +
> +static struct zcomp_backend *find_backend(const char *compress)
> +{
> + if (strncmp(compress, "lzo", 3) == 0)
> + return _lzo;
> + return NULL;
> +}
> +
> +static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
> +{
> + if (zstrm->private)
> + comp->backend->destroy(zstrm->private);
> + free_pages((unsigned long)zstrm->buffer, 1);
> + kfree(zstrm);
> +}
> +
> +/*
> + * allocate new zcomp_strm structure with ->private initialized by
> + * backend, return NULL on error
> + */
> +static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp)
> +{
> + struct zcomp_strm *zstrm = kmalloc(sizeof(*zstrm), GFP_KERNEL);
> + if (!zstrm)
> + return NULL;
> +
> + zstrm->private = comp->backend->create();
> + /*
> + * allocate 2 pages. 1 for compressed data, plus 1 extra for the
> + * case when compressed size is larger than the original one
> + */
> + zstrm->buffer = (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, 1);
> + if (!zstrm->private || !zstrm->buffer) {
> + zcomp_strm_free(comp, zstrm);
> + zstrm = NULL;
> + }
> + return zstrm;
> +}
> +
> +/*
> + * get
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
Hello Sergey, On Tue, Feb 18, 2014 at 12:20:17PM +0300, Sergey Senozhatsky wrote: > Hello, > > On (02/18/14 15:04), Minchan Kim wrote: > > Hello Sergey, > > > > On Thu, Feb 13, 2014 at 08:43:18PM +0300, Sergey Senozhatsky wrote: > > > This patchset introduces zcomp compression backend abstraction > > > adding ability to support compression algorithms other than LZO; > > > support for multi compression streams, making parallel compressions > > > possible. > > > > > > Diff from v4 (reviewed by Minchan Kim): > > > -- renamed zcomp buffer_lock; removed src len and dst len from > > >compress() and decompress(); not using term `buffer' and > > >`workmem' in code and documentation; define compress() and > > >decompress() functions for LZO backend; not using goto's; > > >do not put idle zcomp_strm to idle list tail. > > > > > > Diff from v3: > > > -- renamed compression backend and working memory structs as requested > > >by Minchan Kim; fixed several issues noted by Minchan Kim. > > > > > > Sergey Senozhatsky (4): > > > zram: introduce compressing backend abstraction > > > zram: use zcomp compressing backends > > > zram: zcomp support multi compressing streams > > > zram: document max_comp_streams > > > > > > Documentation/ABI/testing/sysfs-block-zram | 9 +- > > > Documentation/blockdev/zram.txt| 20 +++- > > > drivers/block/zram/Makefile| 2 +- > > > drivers/block/zram/zcomp.c | 155 > > > + > > > drivers/block/zram/zcomp.h | 56 +++ > > > drivers/block/zram/zcomp_lzo.c | 48 + > > > drivers/block/zram/zram_drv.c | 102 --- > > > drivers/block/zram/zram_drv.h | 10 +- > > > 8 files changed, 355 insertions(+), 47 deletions(-) > > > create mode 100644 drivers/block/zram/zcomp.c > > > create mode 100644 drivers/block/zram/zcomp.h > > > create mode 100644 drivers/block/zram/zcomp_lzo.c > > > > I reviewed patchset and implement looked good to me but > > I found severe regression in my test which was one stream. > > > > Base is current upstream and zcomp-multi is yours. > > At last, zcom-single is where I tweaked to see schedule overhead > > cause by event. > > > >Base zcomp-multi zcomp-single > > > > ==Initial write ==Initial write ==Initial write > > records: 5 records: 5 records: 5 > > avg: 1642424.35avg: 699610.40 avg: > > 1655583.71 > > wow... spinlock vs mutex 3 times slower? that's ugly. > can you please provide iozone arg list? iozone -t -T -l 12 -u 12 -r 16K -s 60M -I +Z -V 0 > > > std: 39890.95(2.43%) std: 232014.19(33.16%) std: > > 52293.96(3.16%) > > max: 1690170.94max: 1163473.45 max: > > 1697164.75 > > min: 1568669.52min: 573429.88 min: > > 1553410.23 > > ==Rewrite ==Rewrite ==Rewrite > > records: 5 records: 5 records: 5 > > avg: 1611775.39avg: 501406.64 avg: > > 1684419.11 > > std: 17144.58(1.06%) std: 15354.41(3.06%)std: > > 18367.42(1.09%) > > max: 1641800.95max: 531356.78 max: > > 1706445.84 > > min: 1593515.27min: 488817.78 min: > > 1655335.73 > > ==Read ==Read==Read > > records: 5 records: 5 records: 5 > > avg: 2418916.31avg: 2385573.68 avg: > > 2316542.26 > > std: 55324.98(2.29%) std: 64475.37(2.70%)std: > > 50621.10(2.19%) > > max: 2517417.72max: 2444138.89 max: > > 2383321.09 > > min: 2364690.92min: 2263763.77 min: > > 2233198.12 > > ==Re-read ==Re-read ==Re-read > > records: 5 records: 5 records: 5 > > avg: 2351292.92avg: 2333131.90 avg: > > 2336306.89 > > std: 73358.82(3.12%) std: 34726.09(1.49%)std: > > 74001.47(3.17%) > > max: 2444053.92max: 2396357.19 max: > > 2432697.06 > > min: 2255477.41min: 2299239.74 min: > > 2231400.94 > > ==Reverse Read ==Reverse Read ==Reverse Read > > records: 5 records: 5 records: 5 > > avg: 2383917.40avg: 2267700.38 avg: > > 2328689.00 > > std: 51389.78(2.16%) std: 57018.67(2.51%)std: > > 44955.21(1.93%) > > max: 2435560.11max: 2346465.31 max: > > 2375047.77 > > min: 2290726.91min: 2188208.84
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
On (02/18/14 15:04), Minchan Kim wrote:
[..]
> 1.8.5.3
>
> As you can see, your patch made zram too much regressed when it
> used one stream buffer. The reason I guessed is overhead of
> scheduling by sleep/wakeup when others couldn't find a idle stream
> so I had an experiment with below simple patch to restore old behavior
> so it works well again. The reason old behaivor was really good is
> it uses mutex with spin_on_owner so that it could avoid frequent
> sleep/wakeup.
>
> A solution I could think is that we could grow up the number of
> buffer dynamically up to max_buffers(ex, 2 * number of CPU) so we
> could grab a idle stream easily for each CPU while we could release
> buffers by shrinker when the memory pressure is severe.
> Of course, we should keep one buffer to work.
>
> Another solution I can imagaine is to define CONFIG_ZRAM_MULTI_STREAM
> and CONFIG_ZRAM_SINGLE_STREAM(ie, default) so we couldn't break good
> performance for old cases and new users who want write parallel
> could use ZRAM_MULTI which should be several streams at a default
> rather than a stream. And we could ehhacne it further by dynamic control
> as I mentioned.
>
> Thoughts?
>
like the following one (this patch limits the number of streams to
num_online_cpus(). I really don't mind to limit zstrm to
N * num_online_cpus() where N could be 2)
iozone -t 3 -R -r 16K -s 60M -I +Z
write test:
(old core i5 laptop, 2 cpus + 2 HyperThreading)
basemulti buffer
" Initial write " 574938.14 716623.64
"Rewrite " 573541.22 1499074.62
" Random write " 587016.14 1393996.66
" Pwrite " 595616.45 711028.70
" Fwrite " 1482843.62 1493398.12
zcomp_strm_get() and zcomp_strm_put() were updated.
zcomp keeps the number allocated and still active (not freed)
streams (atomic_t). zcomp performs zstrm free() only if
current number of streams exceeds the number of online
cpus (cpu went offline).
not a properly `aligned' patch, just to demonstrate the idea.
thoughts?
thanks.
---
>From 0e16f26480933d22339675ca938049dfb21557e2 Mon Sep 17 00:00:00 2001
From: Sergey Senozhatsky
Date: Tue, 11 Feb 2014 00:28:46 +0300
Subject: [PATCH] zram: multi stream compressing backend abstraction
Signed-off-by: Sergey Senozhatsky
---
drivers/block/zram/zcomp.c | 174 +
drivers/block/zram/zcomp.h | 58 ++
drivers/block/zram/zcomp_lzo.c | 48
3 files changed, 280 insertions(+)
create mode 100644 drivers/block/zram/zcomp.c
create mode 100644 drivers/block/zram/zcomp.h
create mode 100644 drivers/block/zram/zcomp_lzo.c
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
new file mode 100644
index 000..6bf49fc
--- /dev/null
+++ b/drivers/block/zram/zcomp.c
@@ -0,0 +1,174 @@
+/*
+ * Copyright (C) 2014 Sergey Senozhatsky.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include
+#include
+#include
+#include
+#include
+
+#include "zcomp.h"
+
+extern struct zcomp_backend zcomp_lzo;
+
+static struct zcomp_backend *find_backend(const char *compress)
+{
+ if (strncmp(compress, "lzo", 3) == 0)
+ return _lzo;
+ return NULL;
+}
+
+static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
+{
+ if (zstrm->private)
+ comp->backend->destroy(zstrm->private);
+ free_pages((unsigned long)zstrm->buffer, 1);
+ kfree(zstrm);
+}
+
+/*
+ * allocate new zcomp_strm structure with ->private initialized by
+ * backend, return NULL on error
+ */
+static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp)
+{
+ struct zcomp_strm *zstrm = kmalloc(sizeof(*zstrm), GFP_KERNEL);
+ if (!zstrm)
+ return NULL;
+
+ zstrm->private = comp->backend->create();
+ /*
+* allocate 2 pages. 1 for compressed data, plus 1 extra for the
+* case when compressed size is larger than the original one
+*/
+ zstrm->buffer = (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, 1);
+ if (!zstrm->private || !zstrm->buffer) {
+ zcomp_strm_free(comp, zstrm);
+ zstrm = NULL;
+ }
+ return zstrm;
+}
+
+/*
+ * get available idle stream or allocate a new one
+ */
+struct zcomp_strm *zcomp_strm_get(struct zcomp *comp)
+{
+ struct zcomp_strm *zstrm;
+
+ while (1) {
+ spin_lock(>strm_lock);
+ if (!list_empty(>idle_strm)) {
+ zstrm = list_entry(comp->idle_strm.next,
+ struct zcomp_strm, list);
+ list_del(>list);
+ spin_unlock(>strm_lock);
+
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
Hello, On (02/18/14 15:04), Minchan Kim wrote: > Hello Sergey, > > On Thu, Feb 13, 2014 at 08:43:18PM +0300, Sergey Senozhatsky wrote: > > This patchset introduces zcomp compression backend abstraction > > adding ability to support compression algorithms other than LZO; > > support for multi compression streams, making parallel compressions > > possible. > > > > Diff from v4 (reviewed by Minchan Kim): > > -- renamed zcomp buffer_lock; removed src len and dst len from > >compress() and decompress(); not using term `buffer' and > >`workmem' in code and documentation; define compress() and > >decompress() functions for LZO backend; not using goto's; > >do not put idle zcomp_strm to idle list tail. > > > > Diff from v3: > > -- renamed compression backend and working memory structs as requested > >by Minchan Kim; fixed several issues noted by Minchan Kim. > > > > Sergey Senozhatsky (4): > > zram: introduce compressing backend abstraction > > zram: use zcomp compressing backends > > zram: zcomp support multi compressing streams > > zram: document max_comp_streams > > > > Documentation/ABI/testing/sysfs-block-zram | 9 +- > > Documentation/blockdev/zram.txt| 20 +++- > > drivers/block/zram/Makefile| 2 +- > > drivers/block/zram/zcomp.c | 155 > > + > > drivers/block/zram/zcomp.h | 56 +++ > > drivers/block/zram/zcomp_lzo.c | 48 + > > drivers/block/zram/zram_drv.c | 102 --- > > drivers/block/zram/zram_drv.h | 10 +- > > 8 files changed, 355 insertions(+), 47 deletions(-) > > create mode 100644 drivers/block/zram/zcomp.c > > create mode 100644 drivers/block/zram/zcomp.h > > create mode 100644 drivers/block/zram/zcomp_lzo.c > > I reviewed patchset and implement looked good to me but > I found severe regression in my test which was one stream. > > Base is current upstream and zcomp-multi is yours. > At last, zcom-single is where I tweaked to see schedule overhead > cause by event. > >Base zcomp-multi zcomp-single > > ==Initial write ==Initial write ==Initial write > records: 5 records: 5 records: 5 > avg: 1642424.35avg: 699610.40 avg: > 1655583.71 wow... spinlock vs mutex 3 times slower? that's ugly. can you please provide iozone arg list? > std: 39890.95(2.43%) std: 232014.19(33.16%) std: > 52293.96(3.16%) > max: 1690170.94max: 1163473.45 max: > 1697164.75 > min: 1568669.52min: 573429.88 min: > 1553410.23 > ==Rewrite ==Rewrite ==Rewrite > records: 5 records: 5 records: 5 > avg: 1611775.39avg: 501406.64 avg: > 1684419.11 > std: 17144.58(1.06%) std: 15354.41(3.06%)std: > 18367.42(1.09%) > max: 1641800.95max: 531356.78 max: > 1706445.84 > min: 1593515.27min: 488817.78 min: > 1655335.73 > ==Read ==Read==Read > records: 5 records: 5 records: 5 > avg: 2418916.31avg: 2385573.68 avg: > 2316542.26 > std: 55324.98(2.29%) std: 64475.37(2.70%)std: > 50621.10(2.19%) > max: 2517417.72max: 2444138.89 max: > 2383321.09 > min: 2364690.92min: 2263763.77 min: > 2233198.12 > ==Re-read ==Re-read ==Re-read > records: 5 records: 5 records: 5 > avg: 2351292.92avg: 2333131.90 avg: > 2336306.89 > std: 73358.82(3.12%) std: 34726.09(1.49%)std: > 74001.47(3.17%) > max: 2444053.92max: 2396357.19 max: > 2432697.06 > min: 2255477.41min: 2299239.74 min: > 2231400.94 > ==Reverse Read ==Reverse Read ==Reverse Read > records: 5 records: 5 records: 5 > avg: 2383917.40avg: 2267700.38 avg: > 2328689.00 > std: 51389.78(2.16%) std: 57018.67(2.51%)std: > 44955.21(1.93%) > max: 2435560.11max: 2346465.31 max: > 2375047.77 > min: 2290726.91min: 2188208.84 min: > 2251465.20 > ==Stride read ==Stride read ==Stride read > records: 5 records: 5 records: 5 > avg: 2361656.52avg: 2292182.65 avg: > 2302384.26 > std: 64730.61(2.74%) std: 34649.38(1.51%)std: >
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
Hello, On (02/18/14 15:04), Minchan Kim wrote: Hello Sergey, On Thu, Feb 13, 2014 at 08:43:18PM +0300, Sergey Senozhatsky wrote: This patchset introduces zcomp compression backend abstraction adding ability to support compression algorithms other than LZO; support for multi compression streams, making parallel compressions possible. Diff from v4 (reviewed by Minchan Kim): -- renamed zcomp buffer_lock; removed src len and dst len from compress() and decompress(); not using term `buffer' and `workmem' in code and documentation; define compress() and decompress() functions for LZO backend; not using goto's; do not put idle zcomp_strm to idle list tail. Diff from v3: -- renamed compression backend and working memory structs as requested by Minchan Kim; fixed several issues noted by Minchan Kim. Sergey Senozhatsky (4): zram: introduce compressing backend abstraction zram: use zcomp compressing backends zram: zcomp support multi compressing streams zram: document max_comp_streams Documentation/ABI/testing/sysfs-block-zram | 9 +- Documentation/blockdev/zram.txt| 20 +++- drivers/block/zram/Makefile| 2 +- drivers/block/zram/zcomp.c | 155 + drivers/block/zram/zcomp.h | 56 +++ drivers/block/zram/zcomp_lzo.c | 48 + drivers/block/zram/zram_drv.c | 102 --- drivers/block/zram/zram_drv.h | 10 +- 8 files changed, 355 insertions(+), 47 deletions(-) create mode 100644 drivers/block/zram/zcomp.c create mode 100644 drivers/block/zram/zcomp.h create mode 100644 drivers/block/zram/zcomp_lzo.c I reviewed patchset and implement looked good to me but I found severe regression in my test which was one stream. Base is current upstream and zcomp-multi is yours. At last, zcom-single is where I tweaked to see schedule overhead cause by event. Base zcomp-multi zcomp-single ==Initial write ==Initial write ==Initial write records: 5 records: 5 records: 5 avg: 1642424.35avg: 699610.40 avg: 1655583.71 wow... spinlock vs mutex 3 times slower? that's ugly. can you please provide iozone arg list? std: 39890.95(2.43%) std: 232014.19(33.16%) std: 52293.96(3.16%) max: 1690170.94max: 1163473.45 max: 1697164.75 min: 1568669.52min: 573429.88 min: 1553410.23 ==Rewrite ==Rewrite ==Rewrite records: 5 records: 5 records: 5 avg: 1611775.39avg: 501406.64 avg: 1684419.11 std: 17144.58(1.06%) std: 15354.41(3.06%)std: 18367.42(1.09%) max: 1641800.95max: 531356.78 max: 1706445.84 min: 1593515.27min: 488817.78 min: 1655335.73 ==Read ==Read==Read records: 5 records: 5 records: 5 avg: 2418916.31avg: 2385573.68 avg: 2316542.26 std: 55324.98(2.29%) std: 64475.37(2.70%)std: 50621.10(2.19%) max: 2517417.72max: 2444138.89 max: 2383321.09 min: 2364690.92min: 2263763.77 min: 2233198.12 ==Re-read ==Re-read ==Re-read records: 5 records: 5 records: 5 avg: 2351292.92avg: 2333131.90 avg: 2336306.89 std: 73358.82(3.12%) std: 34726.09(1.49%)std: 74001.47(3.17%) max: 2444053.92max: 2396357.19 max: 2432697.06 min: 2255477.41min: 2299239.74 min: 2231400.94 ==Reverse Read ==Reverse Read ==Reverse Read records: 5 records: 5 records: 5 avg: 2383917.40avg: 2267700.38 avg: 2328689.00 std: 51389.78(2.16%) std: 57018.67(2.51%)std: 44955.21(1.93%) max: 2435560.11max: 2346465.31 max: 2375047.77 min: 2290726.91min: 2188208.84 min: 2251465.20 ==Stride read ==Stride read ==Stride read records: 5 records: 5 records: 5 avg: 2361656.52avg: 2292182.65 avg: 2302384.26 std: 64730.61(2.74%) std: 34649.38(1.51%)std: 51145.23(2.22%) max: 2471425.77max: 2341282.19 max: 2375936.66 min: 2279913.31min:
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
On (02/18/14 15:04), Minchan Kim wrote:
[..]
1.8.5.3
As you can see, your patch made zram too much regressed when it
used one stream buffer. The reason I guessed is overhead of
scheduling by sleep/wakeup when others couldn't find a idle stream
so I had an experiment with below simple patch to restore old behavior
so it works well again. The reason old behaivor was really good is
it uses mutex with spin_on_owner so that it could avoid frequent
sleep/wakeup.
A solution I could think is that we could grow up the number of
buffer dynamically up to max_buffers(ex, 2 * number of CPU) so we
could grab a idle stream easily for each CPU while we could release
buffers by shrinker when the memory pressure is severe.
Of course, we should keep one buffer to work.
Another solution I can imagaine is to define CONFIG_ZRAM_MULTI_STREAM
and CONFIG_ZRAM_SINGLE_STREAM(ie, default) so we couldn't break good
performance for old cases and new users who want write parallel
could use ZRAM_MULTI which should be several streams at a default
rather than a stream. And we could ehhacne it further by dynamic control
as I mentioned.
Thoughts?
like the following one (this patch limits the number of streams to
num_online_cpus(). I really don't mind to limit zstrm to
N * num_online_cpus() where N could be 2)
iozone -t 3 -R -r 16K -s 60M -I +Z
write test:
(old core i5 laptop, 2 cpus + 2 HyperThreading)
basemulti buffer
Initial write574938.14 716623.64
Rewrite573541.22 1499074.62
Random write587016.14 1393996.66
Pwrite595616.45 711028.70
Fwrite 1482843.62 1493398.12
zcomp_strm_get() and zcomp_strm_put() were updated.
zcomp keeps the number allocated and still active (not freed)
streams (atomic_t). zcomp performs zstrm free() only if
current number of streams exceeds the number of online
cpus (cpu went offline).
not a properly `aligned' patch, just to demonstrate the idea.
thoughts?
thanks.
---
From 0e16f26480933d22339675ca938049dfb21557e2 Mon Sep 17 00:00:00 2001
From: Sergey Senozhatsky sergey.senozhat...@gmail.com
Date: Tue, 11 Feb 2014 00:28:46 +0300
Subject: [PATCH] zram: multi stream compressing backend abstraction
Signed-off-by: Sergey Senozhatsky sergey.senozhat...@gmail.com
---
drivers/block/zram/zcomp.c | 174 +
drivers/block/zram/zcomp.h | 58 ++
drivers/block/zram/zcomp_lzo.c | 48
3 files changed, 280 insertions(+)
create mode 100644 drivers/block/zram/zcomp.c
create mode 100644 drivers/block/zram/zcomp.h
create mode 100644 drivers/block/zram/zcomp_lzo.c
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
new file mode 100644
index 000..6bf49fc
--- /dev/null
+++ b/drivers/block/zram/zcomp.c
@@ -0,0 +1,174 @@
+/*
+ * Copyright (C) 2014 Sergey Senozhatsky.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include linux/kernel.h
+#include linux/string.h
+#include linux/slab.h
+#include linux/wait.h
+#include linux/sched.h
+
+#include zcomp.h
+
+extern struct zcomp_backend zcomp_lzo;
+
+static struct zcomp_backend *find_backend(const char *compress)
+{
+ if (strncmp(compress, lzo, 3) == 0)
+ return zcomp_lzo;
+ return NULL;
+}
+
+static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
+{
+ if (zstrm-private)
+ comp-backend-destroy(zstrm-private);
+ free_pages((unsigned long)zstrm-buffer, 1);
+ kfree(zstrm);
+}
+
+/*
+ * allocate new zcomp_strm structure with -private initialized by
+ * backend, return NULL on error
+ */
+static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp)
+{
+ struct zcomp_strm *zstrm = kmalloc(sizeof(*zstrm), GFP_KERNEL);
+ if (!zstrm)
+ return NULL;
+
+ zstrm-private = comp-backend-create();
+ /*
+* allocate 2 pages. 1 for compressed data, plus 1 extra for the
+* case when compressed size is larger than the original one
+*/
+ zstrm-buffer = (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, 1);
+ if (!zstrm-private || !zstrm-buffer) {
+ zcomp_strm_free(comp, zstrm);
+ zstrm = NULL;
+ }
+ return zstrm;
+}
+
+/*
+ * get available idle stream or allocate a new one
+ */
+struct zcomp_strm *zcomp_strm_get(struct zcomp *comp)
+{
+ struct zcomp_strm *zstrm;
+
+ while (1) {
+ spin_lock(comp-strm_lock);
+ if (!list_empty(comp-idle_strm)) {
+ zstrm = list_entry(comp-idle_strm.next,
+ struct zcomp_strm, list);
+
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
Hello Sergey, On Tue, Feb 18, 2014 at 12:20:17PM +0300, Sergey Senozhatsky wrote: Hello, On (02/18/14 15:04), Minchan Kim wrote: Hello Sergey, On Thu, Feb 13, 2014 at 08:43:18PM +0300, Sergey Senozhatsky wrote: This patchset introduces zcomp compression backend abstraction adding ability to support compression algorithms other than LZO; support for multi compression streams, making parallel compressions possible. Diff from v4 (reviewed by Minchan Kim): -- renamed zcomp buffer_lock; removed src len and dst len from compress() and decompress(); not using term `buffer' and `workmem' in code and documentation; define compress() and decompress() functions for LZO backend; not using goto's; do not put idle zcomp_strm to idle list tail. Diff from v3: -- renamed compression backend and working memory structs as requested by Minchan Kim; fixed several issues noted by Minchan Kim. Sergey Senozhatsky (4): zram: introduce compressing backend abstraction zram: use zcomp compressing backends zram: zcomp support multi compressing streams zram: document max_comp_streams Documentation/ABI/testing/sysfs-block-zram | 9 +- Documentation/blockdev/zram.txt| 20 +++- drivers/block/zram/Makefile| 2 +- drivers/block/zram/zcomp.c | 155 + drivers/block/zram/zcomp.h | 56 +++ drivers/block/zram/zcomp_lzo.c | 48 + drivers/block/zram/zram_drv.c | 102 --- drivers/block/zram/zram_drv.h | 10 +- 8 files changed, 355 insertions(+), 47 deletions(-) create mode 100644 drivers/block/zram/zcomp.c create mode 100644 drivers/block/zram/zcomp.h create mode 100644 drivers/block/zram/zcomp_lzo.c I reviewed patchset and implement looked good to me but I found severe regression in my test which was one stream. Base is current upstream and zcomp-multi is yours. At last, zcom-single is where I tweaked to see schedule overhead cause by event. Base zcomp-multi zcomp-single ==Initial write ==Initial write ==Initial write records: 5 records: 5 records: 5 avg: 1642424.35avg: 699610.40 avg: 1655583.71 wow... spinlock vs mutex 3 times slower? that's ugly. can you please provide iozone arg list? iozone -t -T -l 12 -u 12 -r 16K -s 60M -I +Z -V 0 std: 39890.95(2.43%) std: 232014.19(33.16%) std: 52293.96(3.16%) max: 1690170.94max: 1163473.45 max: 1697164.75 min: 1568669.52min: 573429.88 min: 1553410.23 ==Rewrite ==Rewrite ==Rewrite records: 5 records: 5 records: 5 avg: 1611775.39avg: 501406.64 avg: 1684419.11 std: 17144.58(1.06%) std: 15354.41(3.06%)std: 18367.42(1.09%) max: 1641800.95max: 531356.78 max: 1706445.84 min: 1593515.27min: 488817.78 min: 1655335.73 ==Read ==Read==Read records: 5 records: 5 records: 5 avg: 2418916.31avg: 2385573.68 avg: 2316542.26 std: 55324.98(2.29%) std: 64475.37(2.70%)std: 50621.10(2.19%) max: 2517417.72max: 2444138.89 max: 2383321.09 min: 2364690.92min: 2263763.77 min: 2233198.12 ==Re-read ==Re-read ==Re-read records: 5 records: 5 records: 5 avg: 2351292.92avg: 2333131.90 avg: 2336306.89 std: 73358.82(3.12%) std: 34726.09(1.49%)std: 74001.47(3.17%) max: 2444053.92max: 2396357.19 max: 2432697.06 min: 2255477.41min: 2299239.74 min: 2231400.94 ==Reverse Read ==Reverse Read ==Reverse Read records: 5 records: 5 records: 5 avg: 2383917.40avg: 2267700.38 avg: 2328689.00 std: 51389.78(2.16%) std: 57018.67(2.51%)std: 44955.21(1.93%) max: 2435560.11max: 2346465.31 max: 2375047.77 min: 2290726.91min: 2188208.84 min: 2251465.20 ==Stride read ==Stride read ==Stride read records: 5 records: 5 records: 5 avg: 2361656.52avg: 2292182.65
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
On Tue, Feb 18, 2014 at 10:38:18PM +0300, Sergey Senozhatsky wrote:
On (02/18/14 15:04), Minchan Kim wrote:
[..]
1.8.5.3
As you can see, your patch made zram too much regressed when it
used one stream buffer. The reason I guessed is overhead of
scheduling by sleep/wakeup when others couldn't find a idle stream
so I had an experiment with below simple patch to restore old behavior
so it works well again. The reason old behaivor was really good is
it uses mutex with spin_on_owner so that it could avoid frequent
sleep/wakeup.
A solution I could think is that we could grow up the number of
buffer dynamically up to max_buffers(ex, 2 * number of CPU) so we
could grab a idle stream easily for each CPU while we could release
buffers by shrinker when the memory pressure is severe.
Of course, we should keep one buffer to work.
Another solution I can imagaine is to define CONFIG_ZRAM_MULTI_STREAM
and CONFIG_ZRAM_SINGLE_STREAM(ie, default) so we couldn't break good
performance for old cases and new users who want write parallel
could use ZRAM_MULTI which should be several streams at a default
rather than a stream. And we could ehhacne it further by dynamic control
as I mentioned.
Thoughts?
like the following one (this patch limits the number of streams to
num_online_cpus(). I really don't mind to limit zstrm to
N * num_online_cpus() where N could be 2)
iozone -t 3 -R -r 16K -s 60M -I +Z
write test:
(old core i5 laptop, 2 cpus + 2 HyperThreading)
basemulti buffer
Initial write574938.14 716623.64
Rewrite573541.22 1499074.62
Random write587016.14 1393996.66
Pwrite595616.45 711028.70
Fwrite 1482843.62 1493398.12
I guess you tested it for zram-block, not zram-swap.
If we use it as zram-swap and happens parallel swapout,
write performance would be dropped due to frequent sleep/wakeup.
zcomp_strm_get() and zcomp_strm_put() were updated.
zcomp keeps the number allocated and still active (not freed)
streams (atomic_t). zcomp performs zstrm free() only if
current number of streams exceeds the number of online
cpus (cpu went offline).
not a properly `aligned' patch, just to demonstrate the idea.
thoughts?
thanks.
---
From 0e16f26480933d22339675ca938049dfb21557e2 Mon Sep 17 00:00:00 2001
From: Sergey Senozhatsky sergey.senozhat...@gmail.com
Date: Tue, 11 Feb 2014 00:28:46 +0300
Subject: [PATCH] zram: multi stream compressing backend abstraction
Signed-off-by: Sergey Senozhatsky sergey.senozhat...@gmail.com
---
drivers/block/zram/zcomp.c | 174
+
drivers/block/zram/zcomp.h | 58 ++
drivers/block/zram/zcomp_lzo.c | 48
3 files changed, 280 insertions(+)
create mode 100644 drivers/block/zram/zcomp.c
create mode 100644 drivers/block/zram/zcomp.h
create mode 100644 drivers/block/zram/zcomp_lzo.c
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
new file mode 100644
index 000..6bf49fc
--- /dev/null
+++ b/drivers/block/zram/zcomp.c
@@ -0,0 +1,174 @@
+/*
+ * Copyright (C) 2014 Sergey Senozhatsky.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include linux/kernel.h
+#include linux/string.h
+#include linux/slab.h
+#include linux/wait.h
+#include linux/sched.h
+
+#include zcomp.h
+
+extern struct zcomp_backend zcomp_lzo;
+
+static struct zcomp_backend *find_backend(const char *compress)
+{
+ if (strncmp(compress, lzo, 3) == 0)
+ return zcomp_lzo;
+ return NULL;
+}
+
+static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
+{
+ if (zstrm-private)
+ comp-backend-destroy(zstrm-private);
+ free_pages((unsigned long)zstrm-buffer, 1);
+ kfree(zstrm);
+}
+
+/*
+ * allocate new zcomp_strm structure with -private initialized by
+ * backend, return NULL on error
+ */
+static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp)
+{
+ struct zcomp_strm *zstrm = kmalloc(sizeof(*zstrm), GFP_KERNEL);
+ if (!zstrm)
+ return NULL;
+
+ zstrm-private = comp-backend-create();
+ /*
+ * allocate 2 pages. 1 for compressed data, plus 1 extra for the
+ * case when compressed size is larger than the original one
+ */
+ zstrm-buffer = (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, 1);
+ if (!zstrm-private || !zstrm-buffer) {
+ zcomp_strm_free(comp, zstrm);
+ zstrm = NULL;
+ }
+ return zstrm;
+}
+
+/*
+ * get available idle stream or allocate a new one
+ */
+struct zcomp_strm
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
Hello Sergey, On Thu, Feb 13, 2014 at 08:43:18PM +0300, Sergey Senozhatsky wrote: > This patchset introduces zcomp compression backend abstraction > adding ability to support compression algorithms other than LZO; > support for multi compression streams, making parallel compressions > possible. > > Diff from v4 (reviewed by Minchan Kim): > -- renamed zcomp buffer_lock; removed src len and dst len from >compress() and decompress(); not using term `buffer' and >`workmem' in code and documentation; define compress() and >decompress() functions for LZO backend; not using goto's; >do not put idle zcomp_strm to idle list tail. > > Diff from v3: > -- renamed compression backend and working memory structs as requested >by Minchan Kim; fixed several issues noted by Minchan Kim. > > Sergey Senozhatsky (4): > zram: introduce compressing backend abstraction > zram: use zcomp compressing backends > zram: zcomp support multi compressing streams > zram: document max_comp_streams > > Documentation/ABI/testing/sysfs-block-zram | 9 +- > Documentation/blockdev/zram.txt| 20 +++- > drivers/block/zram/Makefile| 2 +- > drivers/block/zram/zcomp.c | 155 > + > drivers/block/zram/zcomp.h | 56 +++ > drivers/block/zram/zcomp_lzo.c | 48 + > drivers/block/zram/zram_drv.c | 102 --- > drivers/block/zram/zram_drv.h | 10 +- > 8 files changed, 355 insertions(+), 47 deletions(-) > create mode 100644 drivers/block/zram/zcomp.c > create mode 100644 drivers/block/zram/zcomp.h > create mode 100644 drivers/block/zram/zcomp_lzo.c I reviewed patchset and implement looked good to me but I found severe regression in my test which was one stream. Base is current upstream and zcomp-multi is yours. At last, zcom-single is where I tweaked to see schedule overhead cause by event. Base zcomp-multi zcomp-single ==Initial write ==Initial write ==Initial write records: 5 records: 5 records: 5 avg: 1642424.35avg: 699610.40 avg: 1655583.71 std: 39890.95(2.43%) std: 232014.19(33.16%) std: 52293.96(3.16%) max: 1690170.94max: 1163473.45 max: 1697164.75 min: 1568669.52min: 573429.88 min: 1553410.23 ==Rewrite ==Rewrite ==Rewrite records: 5 records: 5 records: 5 avg: 1611775.39avg: 501406.64 avg: 1684419.11 std: 17144.58(1.06%) std: 15354.41(3.06%)std: 18367.42(1.09%) max: 1641800.95max: 531356.78 max: 1706445.84 min: 1593515.27min: 488817.78 min: 1655335.73 ==Read ==Read==Read records: 5 records: 5 records: 5 avg: 2418916.31avg: 2385573.68 avg: 2316542.26 std: 55324.98(2.29%) std: 64475.37(2.70%)std: 50621.10(2.19%) max: 2517417.72max: 2444138.89 max: 2383321.09 min: 2364690.92min: 2263763.77 min: 2233198.12 ==Re-read ==Re-read ==Re-read records: 5 records: 5 records: 5 avg: 2351292.92avg: 2333131.90 avg: 2336306.89 std: 73358.82(3.12%) std: 34726.09(1.49%)std: 74001.47(3.17%) max: 2444053.92max: 2396357.19 max: 2432697.06 min: 2255477.41min: 2299239.74 min: 2231400.94 ==Reverse Read ==Reverse Read ==Reverse Read records: 5 records: 5 records: 5 avg: 2383917.40avg: 2267700.38 avg: 2328689.00 std: 51389.78(2.16%) std: 57018.67(2.51%)std: 44955.21(1.93%) max: 2435560.11max: 2346465.31 max: 2375047.77 min: 2290726.91min: 2188208.84 min: 2251465.20 ==Stride read ==Stride read ==Stride read records: 5 records: 5 records: 5 avg: 2361656.52avg: 2292182.65 avg: 2302384.26 std: 64730.61(2.74%) std: 34649.38(1.51%)std: 51145.23(2.22%) max: 2471425.77max: 2341282.19 max: 2375936.66 min: 2279913.31min: 2242688.59 min: 2224134.48 ==Random read ==Random read ==Random read records: 5 records: 5 records: 5 avg: 2369086.95avg:
Re: [PATCHv5 0/4] add compressing abstraction and multi stream support
Hello Sergey, On Thu, Feb 13, 2014 at 08:43:18PM +0300, Sergey Senozhatsky wrote: This patchset introduces zcomp compression backend abstraction adding ability to support compression algorithms other than LZO; support for multi compression streams, making parallel compressions possible. Diff from v4 (reviewed by Minchan Kim): -- renamed zcomp buffer_lock; removed src len and dst len from compress() and decompress(); not using term `buffer' and `workmem' in code and documentation; define compress() and decompress() functions for LZO backend; not using goto's; do not put idle zcomp_strm to idle list tail. Diff from v3: -- renamed compression backend and working memory structs as requested by Minchan Kim; fixed several issues noted by Minchan Kim. Sergey Senozhatsky (4): zram: introduce compressing backend abstraction zram: use zcomp compressing backends zram: zcomp support multi compressing streams zram: document max_comp_streams Documentation/ABI/testing/sysfs-block-zram | 9 +- Documentation/blockdev/zram.txt| 20 +++- drivers/block/zram/Makefile| 2 +- drivers/block/zram/zcomp.c | 155 + drivers/block/zram/zcomp.h | 56 +++ drivers/block/zram/zcomp_lzo.c | 48 + drivers/block/zram/zram_drv.c | 102 --- drivers/block/zram/zram_drv.h | 10 +- 8 files changed, 355 insertions(+), 47 deletions(-) create mode 100644 drivers/block/zram/zcomp.c create mode 100644 drivers/block/zram/zcomp.h create mode 100644 drivers/block/zram/zcomp_lzo.c I reviewed patchset and implement looked good to me but I found severe regression in my test which was one stream. Base is current upstream and zcomp-multi is yours. At last, zcom-single is where I tweaked to see schedule overhead cause by event. Base zcomp-multi zcomp-single ==Initial write ==Initial write ==Initial write records: 5 records: 5 records: 5 avg: 1642424.35avg: 699610.40 avg: 1655583.71 std: 39890.95(2.43%) std: 232014.19(33.16%) std: 52293.96(3.16%) max: 1690170.94max: 1163473.45 max: 1697164.75 min: 1568669.52min: 573429.88 min: 1553410.23 ==Rewrite ==Rewrite ==Rewrite records: 5 records: 5 records: 5 avg: 1611775.39avg: 501406.64 avg: 1684419.11 std: 17144.58(1.06%) std: 15354.41(3.06%)std: 18367.42(1.09%) max: 1641800.95max: 531356.78 max: 1706445.84 min: 1593515.27min: 488817.78 min: 1655335.73 ==Read ==Read==Read records: 5 records: 5 records: 5 avg: 2418916.31avg: 2385573.68 avg: 2316542.26 std: 55324.98(2.29%) std: 64475.37(2.70%)std: 50621.10(2.19%) max: 2517417.72max: 2444138.89 max: 2383321.09 min: 2364690.92min: 2263763.77 min: 2233198.12 ==Re-read ==Re-read ==Re-read records: 5 records: 5 records: 5 avg: 2351292.92avg: 2333131.90 avg: 2336306.89 std: 73358.82(3.12%) std: 34726.09(1.49%)std: 74001.47(3.17%) max: 2444053.92max: 2396357.19 max: 2432697.06 min: 2255477.41min: 2299239.74 min: 2231400.94 ==Reverse Read ==Reverse Read ==Reverse Read records: 5 records: 5 records: 5 avg: 2383917.40avg: 2267700.38 avg: 2328689.00 std: 51389.78(2.16%) std: 57018.67(2.51%)std: 44955.21(1.93%) max: 2435560.11max: 2346465.31 max: 2375047.77 min: 2290726.91min: 2188208.84 min: 2251465.20 ==Stride read ==Stride read ==Stride read records: 5 records: 5 records: 5 avg: 2361656.52avg: 2292182.65 avg: 2302384.26 std: 64730.61(2.74%) std: 34649.38(1.51%)std: 51145.23(2.22%) max: 2471425.77max: 2341282.19 max: 2375936.66 min: 2279913.31min: 2242688.59 min: 2224134.48 ==Random read ==Random read ==Random read records: 5 records: 5 records: 5 avg: 2369086.95avg: 2315431.27 avg: 2352314.50
[PATCHv5 0/4] add compressing abstraction and multi stream support
This patchset introduces zcomp compression backend abstraction adding ability to support compression algorithms other than LZO; support for multi compression streams, making parallel compressions possible. Diff from v4 (reviewed by Minchan Kim): -- renamed zcomp buffer_lock; removed src len and dst len from compress() and decompress(); not using term `buffer' and `workmem' in code and documentation; define compress() and decompress() functions for LZO backend; not using goto's; do not put idle zcomp_strm to idle list tail. Diff from v3: -- renamed compression backend and working memory structs as requested by Minchan Kim; fixed several issues noted by Minchan Kim. Sergey Senozhatsky (4): zram: introduce compressing backend abstraction zram: use zcomp compressing backends zram: zcomp support multi compressing streams zram: document max_comp_streams Documentation/ABI/testing/sysfs-block-zram | 9 +- Documentation/blockdev/zram.txt| 20 +++- drivers/block/zram/Makefile| 2 +- drivers/block/zram/zcomp.c | 155 + drivers/block/zram/zcomp.h | 56 +++ drivers/block/zram/zcomp_lzo.c | 48 + drivers/block/zram/zram_drv.c | 102 --- drivers/block/zram/zram_drv.h | 10 +- 8 files changed, 355 insertions(+), 47 deletions(-) create mode 100644 drivers/block/zram/zcomp.c create mode 100644 drivers/block/zram/zcomp.h create mode 100644 drivers/block/zram/zcomp_lzo.c -- 1.9.0.rc3.260.g4cf525c -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCHv5 0/4] add compressing abstraction and multi stream support
This patchset introduces zcomp compression backend abstraction adding ability to support compression algorithms other than LZO; support for multi compression streams, making parallel compressions possible. Diff from v4 (reviewed by Minchan Kim): -- renamed zcomp buffer_lock; removed src len and dst len from compress() and decompress(); not using term `buffer' and `workmem' in code and documentation; define compress() and decompress() functions for LZO backend; not using goto's; do not put idle zcomp_strm to idle list tail. Diff from v3: -- renamed compression backend and working memory structs as requested by Minchan Kim; fixed several issues noted by Minchan Kim. Sergey Senozhatsky (4): zram: introduce compressing backend abstraction zram: use zcomp compressing backends zram: zcomp support multi compressing streams zram: document max_comp_streams Documentation/ABI/testing/sysfs-block-zram | 9 +- Documentation/blockdev/zram.txt| 20 +++- drivers/block/zram/Makefile| 2 +- drivers/block/zram/zcomp.c | 155 + drivers/block/zram/zcomp.h | 56 +++ drivers/block/zram/zcomp_lzo.c | 48 + drivers/block/zram/zram_drv.c | 102 --- drivers/block/zram/zram_drv.h | 10 +- 8 files changed, 355 insertions(+), 47 deletions(-) create mode 100644 drivers/block/zram/zcomp.c create mode 100644 drivers/block/zram/zcomp.h create mode 100644 drivers/block/zram/zcomp_lzo.c -- 1.9.0.rc3.260.g4cf525c -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
