[LSF/MM TOPIC] Springfield: smart and automated storage

Springfield is a collection of projects unifying multiple levels of the storage stack and providing a general API for automation, health and status monitoring, as well as sane and easy configuration across multiple levels of the storage stack. It is a scalable solution, working from a single node

[PATCH v4 10/13] bcache: add backing_request_endio() for bi_end_io of attached backing device I/O

In order to catch I/O error of backing device, a separate bi_end_io call back is required. Then a per backing device counter can record I/O errors number and retire the backing device if the counter reaches a per backing device I/O error limit. This patch adds backing_request_endio() to bcache

[PATCH v4 04/13] bcache: quit dc->writeback_thread when BCACHE_DEV_DETACHING is set

In patch "bcache: fix cached_dev->count usage for bch_cache_set_error()", cached_dev_get() is called when creating dc->writeback_thread, and cached_dev_put() is called when exiting dc->writeback_thread. This modification works well unless people detach the bcache device manually by 'echo 1 >

[PATCH v4 02/13] bcache: properly set task state in bch_writeback_thread()

Kernel thread routine bch_writeback_thread() has the following code block, 447 down_write(>writeback_lock); 448~450 if (check conditions) { 451 up_write(>writeback_lock); 452 set_current_state(TASK_INTERRUPTIBLE); 453 454 if

[PATCH v4 09/13] bcache: fix inaccurate io state for detached bcache devices

From: Tang Junhui When we run IO in a detached device, and run iostat to shows IO status, normally it will show like bellow (Omitted some fields): Device: ... avgrq-sz avgqu-sz await r_await w_await svctm %util sdd... 15.89 0.531.820.202.23

[PATCH v4 05/13] bcache: stop dc->writeback_rate_update properly

struct delayed_work writeback_rate_update in struct cache_dev is a delayed worker to call function update_writeback_rate() in period (the interval is defined by dc->writeback_rate_update_seconds). When a metadate I/O error happens on cache device, bcache error handling routine

Re: [LSF/MM TOPIC] Springfield: smart and automated storage

On Sat, 2018-01-27 at 09:37 +0100, Jan Tulak wrote: > Springfield is a collection of projects unifying multiple levels of > the storage stack and providing a general API for automation, health > and status monitoring, as well as sane and easy configuration across > multiple levels of the storage

[PATCH v4 12/13] bcache: stop bcache device when backing device is offline

Currently bcache does not handle backing device failure, if backing device is offline and disconnected from system, its bcache device can still be accessible. If the bcache device is in writeback mode, I/O requests even can success if the requests hit on cache device. That is to say, when and how

Re: [PATCH V3] blk-mq: introduce BLK_STS_DEV_RESOURCE

On Tue, Jan 23 2018 at 10:31pm -0500, Ming Lei wrote: > On Tue, Jan 23, 2018 at 04:57:34PM +, Bart Van Assche wrote: > > On Wed, 2018-01-24 at 00:37 +0800, Ming Lei wrote: > > > On Tue, Jan 23, 2018 at 04:24:20PM +, Bart Van Assche wrote: > > > > My opinion about

Re: [PATCH V3] blk-mq: introduce BLK_STS_DEV_RESOURCE

On Sat, 2018-01-27 at 14:09 -0500, Mike Snitzer wrote: > Ming let me know that he successfully tested this V3 patch using both > your test (fio to both mpath and underlying path) and Bart's (02-mq with > can_queue in guest). > > Would be great if you'd review and verify this fix works for you

[PATCH v4 01/13] bcache: set writeback_rate_update_seconds in range [1, 60] seconds

dc->writeback_rate_update_seconds can be set via sysfs and its value can be set to [1, ULONG_MAX]. It does not make sense to set such a large value, 60 seconds is long enough value considering the default 5 seconds works well for long time. Because dc->writeback_rate_update is a special delayed

[PATCH v4 00/13] bcache: device failure handling improvement

Hi maintainers and folks, This patch set tries to improve bcache device failure handling, includes cache device and backing device failures. The basic idea to handle failed cache device is, - Unregister cache set - Detach all backing devices which are attached to this cache set - Stop all the

[PATCH v4 03/13] bcache: fix cached_dev->count usage for bch_cache_set_error()

When bcache metadata I/O fails, bcache will call bch_cache_set_error() to retire the whole cache set. The expected behavior to retire a cache set is to unregister the cache set, and unregister all backing device attached to this cache set, then remove sysfs entries of the cache set and all

[PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev

Current bcache failure handling code will stop all attached bcache devices when the cache set is broken or disconnected. This is desired behavior for most of enterprise or cloud use cases, but maybe not for low end configuration. Nix points out, users may still want to access

Re: [PATCH V3] blk-mq: introduce BLK_STS_DEV_RESOURCE

On Sat, Jan 27, 2018 at 10:12:43PM +, Bart Van Assche wrote: > On Sat, 2018-01-27 at 14:09 -0500, Mike Snitzer wrote: > > Ming let me know that he successfully tested this V3 patch using both > > your test (fio to both mpath and underlying path) and Bart's (02-mq with > > can_queue in guest).

Re: [PATCH V3] blk-mq: introduce BLK_STS_DEV_RESOURCE

On Sat, Jan 27 2018 at 5:12pm -0500, Bart Van Assche wrote: > On Sat, 2018-01-27 at 14:09 -0500, Mike Snitzer wrote: > > Ming let me know that he successfully tested this V3 patch using both > > your test (fio to both mpath and underlying path) and Bart's (02-mq with > >

[PATCH v4 04/13] bcache: quit dc->writeback_thread when BCACHE_DEV_DETACHING is set

In patch "bcache: fix cached_dev->count usage for bch_cache_set_error()", cached_dev_get() is called when creating dc->writeback_thread, and cached_dev_put() is called when exiting dc->writeback_thread. This modification works well unless people detach the bcache device manually by 'echo 1 >

[PATCH v4 06/13] bcache: set error_limit correctly

Struct cache uses io_errors for two purposes, - Error decay: when cache set error_decay is set, io_errors is used to generate a small piece of delay when I/O error happens. - I/O errors counter: in order to generate big enough value for error decay, I/O errors counter value is stored by left

[PATCH v4 02/13] bcache: properly set task state in bch_writeback_thread()

Kernel thread routine bch_writeback_thread() has the following code block, 447 down_write(>writeback_lock); 448~450 if (check conditions) { 451 up_write(>writeback_lock); 452 set_current_state(TASK_INTERRUPTIBLE); 453 454 if

Re: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev

Hello Coly, Saturday, January 27, 2018, 5:24:06 PM, you wrote: > Current bcache failure handling code will stop all attached bcache devices > when the cache set is broken or disconnected. This is desired behavior for > most of enterprise or cloud use cases, but maybe not for low end >

Re: [PATCH V3] blk-mq: introduce BLK_STS_DEV_RESOURCE

On Sat, Jan 27 2018 at 7:54pm -0500, Bart Van Assche wrote: > On Sat, 2018-01-27 at 19:23 -0500, Mike Snitzer wrote: > > Your contributions do _not_ make up for your inability to work well with > > others. Tiresome doesn't begin to describe these interactions. > > > >

Re: [PATCH V3] blk-mq: introduce BLK_STS_DEV_RESOURCE

On Sat, 2018-01-27 at 19:23 -0500, Mike Snitzer wrote: > Your contributions do _not_ make up for your inability to work well with > others. Tiresome doesn't begin to describe these interactions. > > Life is too short to continue enduring your bullshit. > > But do let us know when you have

[PATCH v4 01/13] bcache: set writeback_rate_update_seconds in range [1, 60] seconds

dc->writeback_rate_update_seconds can be set via sysfs and its value can be set to [1, ULONG_MAX]. It does not make sense to set such a large value, 60 seconds is long enough value considering the default 5 seconds works well for long time. Because dc->writeback_rate_update is a special delayed

Re: [PATCH V3] blk-mq: introduce BLK_STS_DEV_RESOURCE

On Sat, 2018-01-27 at 21:03 -0500, Mike Snitzer wrote: > You cannot even be forthcoming about the technical merit of a change you > authored (commit 6077c2d70) that I'm left to clean up in the face of > performance bottlenecks it unwittingly introduced? If you were being > honest: you'd grant

[PATCH v4 08/13] bcache: stop all attached bcache devices for a retired cache set

When there are too many I/O errors on cache device, current bcache code will retire the whole cache set, and detach all bcache devices. But the detached bcache devices are not stopped, which is problematic when bcache is in writeback mode. If the retired cache set has dirty data of backing

[PATCH v4 11/13] bcache: add io_disable to struct cached_dev

If a bcache device is configured to writeback mode, current code does not handle write I/O errors on backing devices properly. In writeback mode, write request is written to cache device, and latter being flushed to backing device. If I/O failed when writing from cache device to the backing

Re: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev

On 28/01/2018 11:33 AM, Pavel Goran wrote: > Hello Coly, > > Saturday, January 27, 2018, 5:24:06 PM, you wrote: > >> Current bcache failure handling code will stop all attached bcache devices >> when the cache set is broken or disconnected. This is desired behavior for >> most of enterprise or

Re: [PATCH V3] blk-mq: introduce BLK_STS_DEV_RESOURCE

On Sat, Jan 27 2018 at 10:00pm -0500, Bart Van Assche wrote: > On Sat, 2018-01-27 at 21:03 -0500, Mike Snitzer wrote: > > You cannot even be forthcoming about the technical merit of a change you > > authored (commit 6077c2d70) that I'm left to clean up in the face of > >

[PATCH v4 08/13] bcache: stop all attached bcache devices for a retired cache set

When there are too many I/O errors on cache device, current bcache code will retire the whole cache set, and detach all bcache devices. But the detached bcache devices are not stopped, which is problematic when bcache is in writeback mode. If the retired cache set has dirty data of backing

[PATCH v4 10/13] bcache: add backing_request_endio() for bi_end_io of attached backing device I/O

In order to catch I/O error of backing device, a separate bi_end_io call back is required. Then a per backing device counter can record I/O errors number and retire the backing device if the counter reaches a per backing device I/O error limit. This patch adds backing_request_endio() to bcache

[PATCH v4 09/13] bcache: fix inaccurate io state for detached bcache devices

From: Tang Junhui When we run IO in a detached device, and run iostat to shows IO status, normally it will show like bellow (Omitted some fields): Device: ... avgrq-sz avgqu-sz await r_await w_await svctm %util sdd... 15.89 0.531.820.202.23

Re[2]: [PATCH v4 13/13] bcache: add stop_when_cache_set_failed to struct cached_dev

Hello Coly, Sunday, January 28, 2018, 7:32:09 AM, you wrote: >>> Current bcache failure handling code will stop all attached bcache devices >>> when the cache set is broken or disconnected. This is desired behavior for >>> most of enterprise or cloud use cases, but maybe not for low end >>>

[PATCH v4 03/13] bcache: fix cached_dev->count usage for bch_cache_set_error()

When bcache metadata I/O fails, bcache will call bch_cache_set_error() to retire the whole cache set. The expected behavior to retire a cache set is to unregister the cache set, and unregister all backing device attached to this cache set, then remove sysfs entries of the cache set and all

[PATCH v4 05/13] bcache: stop dc->writeback_rate_update properly

struct delayed_work writeback_rate_update in struct cache_dev is a delayed worker to call function update_writeback_rate() in period (the interval is defined by dc->writeback_rate_update_seconds). When a metadate I/O error happens on cache device, bcache error handling routine

[PATCH v4 06/13] bcache: set error_limit correctly

Struct cache uses io_errors for two purposes, - Error decay: when cache set error_decay is set, io_errors is used to generate a small piece of delay when I/O error happens. - I/O errors counter: in order to generate big enough value for error decay, I/O errors counter value is stored by left

[PATCH v4 11/13] bcache: add io_disable to struct cached_dev

If a bcache device is configured to writeback mode, current code does not handle write I/O errors on backing devices properly. In writeback mode, write request is written to cache device, and latter being flushed to backing device. If I/O failed when writing from cache device to the backing