Re: [dm-devel] DM's filesystem lookup in dm_get_dev_t() [was: Re: linux-next: manual merge of the device-mapper tree with Linus' tree]

[Alasdair G Kergon]

On Tue, Dec 22, 2020 at 06:24:09PM +0100, Hannes Reinecke wrote:
> However, lookup_bdev() now always recurses into the filesystem, causing 
> multipath to stall in an all-paths-down scenario.
 
I have not read the background about whatever the new problem is - I'm
jumping in cold seeing this message - but from the very beginning of
device-mapper we have strongly recommended that userspace supplies the
block device in the form MAJOR:MINOR and all our own tools do that.  We
guarantee not to deadlock in these places when this is done.

We also accept the device in the form of a path name as we know there
are times when this is safe and convenient, but then we offer no
guarantees - we place the responsibility upon userspace only to do this
when it knows it is safe to do so i.e. no race and no deadlock.

Alasdair

Re: [PATCH v2 2/2] dm unstripe: Add documentation for unstripe target

[Alasdair G Kergon]

On Tue, Dec 12, 2017 at 07:45:56AM -0700, Keith Busch wrote:
> Ah, this device's makers call the "stripe" size what should be called
> "chunk".

If this target is to go anywhere, let's try to define it as 'undoing'
the existing dm-stripe target using primary terminology, field names
etc. as close as possible to our existing target.  We can still mention
any alternative terminology encountered in the documentation.  The
first (simplest) documented example could be a dm-stripe target being
'unstriped'.

Alasdair

Re: [PATCH v2 2/2] dm unstripe: Add documentation for unstripe target

[Alasdair G Kergon]

On Tue, Dec 12, 2017 at 07:45:56AM -0700, Keith Busch wrote:
> Ah, this device's makers call the "stripe" size what should be called
> "chunk".

If this target is to go anywhere, let's try to define it as 'undoing'
the existing dm-stripe target using primary terminology, field names
etc. as close as possible to our existing target.  We can still mention
any alternative terminology encountered in the documentation.  The
first (simplest) documented example could be a dm-stripe target being
'unstriped'.

Alasdair

Re: [PATCH 4/4] dm: convert table_device.count from atomic_t to refcount_t

[Alasdair G Kergon]

On Fri, Nov 24, 2017 at 08:29:42AM +, Reshetova, Elena wrote:
> By looking at the code, I don't see where the change in the reference counting
> could have caused this. 
 
The cause was the bug I identified in patch 3, not this patch.

The regression is easily hit - tables that reference the same underlying device
more than once are very common.

Alasdair

Re: [PATCH 4/4] dm: convert table_device.count from atomic_t to refcount_t

[Alasdair G Kergon]

On Fri, Nov 24, 2017 at 08:29:42AM +, Reshetova, Elena wrote:
> By looking at the code, I don't see where the change in the reference counting
> could have caused this. 
 
The cause was the bug I identified in patch 3, not this patch.

The regression is easily hit - tables that reference the same underlying device
more than once are very common.

Alasdair

Re: [dm-devel] [PATCH 3/4] dm: convert dm_dev_internal.count from atomic_t to refcount_t

[Alasdair G Kergon]

On Fri, Oct 20, 2017 at 10:37:38AM +0300, Elena Reshetova wrote:
>   } else if (dd->dm_dev->mode != (mode | dd->dm_dev->mode)) {
>   r = upgrade_mode(dd, mode, t->md);
>   if (r)
>   return r;
> + refcount_inc(>count);
>   }

Missing here:

else
refcount_inc(>count);

?

Alasdair

Re: [dm-devel] [PATCH 3/4] dm: convert dm_dev_internal.count from atomic_t to refcount_t

[Alasdair G Kergon]

On Fri, Oct 20, 2017 at 10:37:38AM +0300, Elena Reshetova wrote:
>   } else if (dd->dm_dev->mode != (mode | dd->dm_dev->mode)) {
>   r = upgrade_mode(dd, mode, t->md);
>   if (r)
>   return r;
> + refcount_inc(>count);
>   }

Missing here:

else
refcount_inc(>count);

?

Alasdair

Re: [dm-devel] [PATCH v6 0/3] dm: boot a mapped device without an initramfs

[Alasdair G Kergon]

Some more thoughts with your example, dmsetup might look like:

# dmsetup create --bootformat "lroot:uuid,rw,0 2097152 linear 8:2 0, \
2097152 2097152 linear 8:3 0, 4194304 2097152 linear 8:4 0"

- also supporting creating multiple devices if the semi-colon is used

- colon to separate name from uuid, like we already do major:minor 
- colon to separate other flags from rw if we need them in future

- splitting first on a unescaped semi-colons, then on the first two
unescaped commas, and then on unescaped commas and then unescaped spaces
within the table

- backslash escapes the following character so it is never a treated
as a separator
  - lroot\:uuid\;\\\ \"\, would be a device with no uuid and the name
lroot:uuid;\ ",  (on a non-udev system without name mangling)

# dmsetup ls --bootformat lroot
dm="lroot:uuid,rw,0 2097152 linear 8:2 0, 2097152 2097152 \
linear 8:3 0, 4194304 2097152 linear 8:4 0"

# dmsetup ls --bootformat
(all devices on one output line)

While the code also supports devices in the /dev/sda2 format for 
convenience, please use the preferred 8:2 format in any implementation
and documented examples (to avoid the unnecessary dependency on /dev and
its dependencies).

Or with some alternative name for the option
--boot[format|param]
--short[format]
--kernelparam
--condensed
other suggestions?

dmsetup create --condensed
dmsetup ls --condensed

Alasdair

Re: [dm-devel] [PATCH v6 0/3] dm: boot a mapped device without an initramfs

[Alasdair G Kergon]

Some more thoughts with your example, dmsetup might look like:

# dmsetup create --bootformat "lroot:uuid,rw,0 2097152 linear 8:2 0, \
2097152 2097152 linear 8:3 0, 4194304 2097152 linear 8:4 0"

- also supporting creating multiple devices if the semi-colon is used

- colon to separate name from uuid, like we already do major:minor 
- colon to separate other flags from rw if we need them in future

- splitting first on a unescaped semi-colons, then on the first two
unescaped commas, and then on unescaped commas and then unescaped spaces
within the table

- backslash escapes the following character so it is never a treated
as a separator
  - lroot\:uuid\;\\\ \"\, would be a device with no uuid and the name
lroot:uuid;\ ",  (on a non-udev system without name mangling)

# dmsetup ls --bootformat lroot
dm="lroot:uuid,rw,0 2097152 linear 8:2 0, 2097152 2097152 \
linear 8:3 0, 4194304 2097152 linear 8:4 0"

# dmsetup ls --bootformat
(all devices on one output line)

While the code also supports devices in the /dev/sda2 format for 
convenience, please use the preferred 8:2 format in any implementation
and documented examples (to avoid the unnecessary dependency on /dev and
its dependencies).

Or with some alternative name for the option
--boot[format|param]
--short[format]
--kernelparam
--condensed
other suggestions?

dmsetup create --condensed
dmsetup ls --condensed

Alasdair

Re: [dm-devel] [PATCH v6 0/3] dm: boot a mapped device without an initramfs

[Alasdair G Kergon]

On Thu, May 04, 2017 at 01:18:41PM +0200, Enric Balletbo Serra wrote:
> I'm wondering if a command line like this would be acceptable.

1) Make sure the implementation continues to support backslash quoting
so that any characters you introduce with special meanings (comma,
semi-colon, double-quote in that example) can still be used if required.

2) "none" is of course a valid uuid:)  More comma-separation or
re-ordering, perhaps?
 
3) Whatever final format is agreed here should be supported by dmsetup
as well, so you can both supply the format to dmsetup and ask dmsetup
to display your existing devices in this format.  Choose a format
that makes this easy.

Alasdair

Re: [dm-devel] [PATCH v6 0/3] dm: boot a mapped device without an initramfs

[Alasdair G Kergon]

On Thu, May 04, 2017 at 01:18:41PM +0200, Enric Balletbo Serra wrote:
> I'm wondering if a command line like this would be acceptable.

1) Make sure the implementation continues to support backslash quoting
so that any characters you introduce with special meanings (comma,
semi-colon, double-quote in that example) can still be used if required.

2) "none" is of course a valid uuid:)  More comma-separation or
re-ordering, perhaps?
 
3) Whatever final format is agreed here should be supported by dmsetup
as well, so you can both supply the format to dmsetup and ask dmsetup
to display your existing devices in this format.  Choose a format
that makes this easy.

Alasdair

Re: [dm-devel] [PATCH 0/9] Generate uevents for all DM events

[Alasdair G Kergon]

On Wed, Oct 05, 2016 at 10:06:41AM -0700, Andy Grover wrote:
> My project *would* like this added sooner, so I'll work on a revised  
> patchset that uses netlink instead of uevents, and will also work on a  
> revision to uevents.txt that talks about KOBJ_CHANGE and when it should  
> and should not be used, as described by agk, to help out the next person.

netlink might still be overkill for this at this stage, if it's only
making one thread able to monitor multiple devices efficiently.

Let's find out your requirements and see if there's anything different from
the ones we've debated before.

Alasdair

Re: [dm-devel] [PATCH 0/9] Generate uevents for all DM events

[Alasdair G Kergon]

On Wed, Oct 05, 2016 at 10:06:41AM -0700, Andy Grover wrote:
> My project *would* like this added sooner, so I'll work on a revised  
> patchset that uses netlink instead of uevents, and will also work on a  
> revision to uevents.txt that talks about KOBJ_CHANGE and when it should  
> and should not be used, as described by agk, to help out the next person.

netlink might still be overkill for this at this stage, if it's only
making one thread able to monitor multiple devices efficiently.

Let's find out your requirements and see if there's anything different from
the ones we've debated before.

Alasdair

Re: [dm-devel] [PATCH 0/9] Generate uevents for all DM events

[Alasdair G Kergon]

On Tue, Oct 04, 2016 at 04:39:28PM -0700, Andy Grover wrote:
> devicemapper is using uevents for:
> a. dm-verity detected corruption
> b. dm-multipath: path failed or reinstated
> c. dm device renamed
> d. there's also some use in md and bcache.
>
> devicemapper uses DM_EVENT ioctl (yuck) for:
> 1. dm-thin pool data/metadata filling up (hit a threshold)
> 2. dm-cache is now clean
> 3. dm-log flushed or log failed
> 4. dm-raid error detected or sync complete

> there doesn't seem to be much technical differentiation between the  
> two lists.

The distinction in dm is that events in the first category may affect
the availability of the device: they represent major (and hopefully
rare) changes.

Events in the second category are just notifications: no impact on /dev,
no need to trigger udev rules, and their use will continue to be
extended, and (rarely at the moment) could be frequent (which is no
problem for the existing polling-based mechanism).

> Instead of using uevent for everything, we could go to a separate  
> genetlink for 1-4 instead of making them use uevent like a-d, but we'd  
> end up with two different userspace notification techniques.

We see these as two different categories of notifications, and prefer
the greater flexibility a mechanism independent of uevents would
provide.  The team has discussed several alternatives over the years but
didn't make a decision as we've not yet reached a point where we're
straining the existing mechanism too far.

Alasdair

Re: [dm-devel] [PATCH 0/9] Generate uevents for all DM events

[Alasdair G Kergon]

On Tue, Oct 04, 2016 at 04:39:28PM -0700, Andy Grover wrote:
> devicemapper is using uevents for:
> a. dm-verity detected corruption
> b. dm-multipath: path failed or reinstated
> c. dm device renamed
> d. there's also some use in md and bcache.
>
> devicemapper uses DM_EVENT ioctl (yuck) for:
> 1. dm-thin pool data/metadata filling up (hit a threshold)
> 2. dm-cache is now clean
> 3. dm-log flushed or log failed
> 4. dm-raid error detected or sync complete

> there doesn't seem to be much technical differentiation between the  
> two lists.

The distinction in dm is that events in the first category may affect
the availability of the device: they represent major (and hopefully
rare) changes.

Events in the second category are just notifications: no impact on /dev,
no need to trigger udev rules, and their use will continue to be
extended, and (rarely at the moment) could be frequent (which is no
problem for the existing polling-based mechanism).

> Instead of using uevent for everything, we could go to a separate  
> genetlink for 1-4 instead of making them use uevent like a-d, but we'd  
> end up with two different userspace notification techniques.

We see these as two different categories of notifications, and prefer
the greater flexibility a mechanism independent of uevents would
provide.  The team has discussed several alternatives over the years but
didn't make a decision as we've not yet reached a point where we're
straining the existing mechanism too far.

Alasdair

Re: [PATCH] MAINTAINERS: correct entry for LVM

[Alasdair G Kergon]

On Mon, Apr 11, 2016 at 09:45:01PM +0530, Sudip Mukherjee wrote:
> L stands for "Mailing list that is relevant to this area", and this is a  
> mailing list. :)

Your proposed patch isn't changing the L entry, so this is of no relevance.

Alasdair

Re: [PATCH] MAINTAINERS: correct entry for LVM

[Alasdair G Kergon]

On Mon, Apr 11, 2016 at 09:45:01PM +0530, Sudip Mukherjee wrote:
> L stands for "Mailing list that is relevant to this area", and this is a  
> mailing list. :)

Your proposed patch isn't changing the L entry, so this is of no relevance.

Alasdair

Re: [PATCH] MAINTAINERS: correct entry for LVM

[Alasdair G Kergon]

On Mon, Apr 11, 2016 at 08:50:39PM +0530, Sudip Mukherjee wrote:
> The entry of dm-de...@redhat.com was duplicated and the duplicate entry
> was marked as a Maintainer but it appears from the email address that it
> is a List. So remove the entry of M and only keep the L entry.
 
M and L are not mutually exclusive!

The definition of M is:
  M: Mail patches to: FullName 
and since we want patches to be sent to the mailing list, this entry is correct
as it stands.

Alasdair

Re: [PATCH] MAINTAINERS: correct entry for LVM

[Alasdair G Kergon]

On Mon, Apr 11, 2016 at 08:50:39PM +0530, Sudip Mukherjee wrote:
> The entry of dm-de...@redhat.com was duplicated and the duplicate entry
> was marked as a Maintainer but it appears from the email address that it
> is a List. So remove the entry of M and only keep the L entry.
 
M and L are not mutually exclusive!

The definition of M is:
  M: Mail patches to: FullName 
and since we want patches to be sent to the mailing list, this entry is correct
as it stands.

Alasdair

Re: [PATCH v5 0/3] init: add support to directly boot to a mapped device

[Alasdair G Kergon]

On Sat, Feb 20, 2016 at 10:13:49AM -0800, Kees Cook wrote:
> This is a resurrection of a patch series from a few years back, first
> brought to the dm maintainers in 2010. It creates a way to define dm
> devices on the kernel command line for systems that do not use an
> initramfs, or otherwise need a dm running before init starts.
> 
> This has been used by Chrome OS for several years, and now by Brillo
> (and likely Android soon).
> 
> The last version was v4:
> https://patchwork.kernel.org/patch/104860/
> https://patchwork.kernel.org/patch/104861/
 
Inconsistencies in the terminology here can be sorted out during review,
and I see that you've taken on board some of my review comments from
2010, but what are your responses to the rest of them?

Alasdair

Re: [PATCH v5 0/3] init: add support to directly boot to a mapped device

[Alasdair G Kergon]

On Sat, Feb 20, 2016 at 10:13:49AM -0800, Kees Cook wrote:
> This is a resurrection of a patch series from a few years back, first
> brought to the dm maintainers in 2010. It creates a way to define dm
> devices on the kernel command line for systems that do not use an
> initramfs, or otherwise need a dm running before init starts.
> 
> This has been used by Chrome OS for several years, and now by Brillo
> (and likely Android soon).
> 
> The last version was v4:
> https://patchwork.kernel.org/patch/104860/
> https://patchwork.kernel.org/patch/104861/
 
Inconsistencies in the terminology here can be sorted out during review,
and I see that you've taken on board some of my review comments from
2010, but what are your responses to the rest of them?

Alasdair

Re: reinstate dm target local ioctl support

[Alasdair G Kergon]

On Thu, Jan 28, 2016 at 01:50:19PM +, Andy Whitcroft wrote:
> However this also removed the possibility of a dm target having target
> specific ioctls.  Currently this is not used by any in-tree targets but
> was utilised by the flashcache out-of-tree module.
 
What data is being passed in and out?

The method we support is passing it through DM_TARGET_MSG (as text).
For examples look at dm-switch.c and dm-stats.c, which handled
pretty large quantities of data.

Would that work?

Alasdair

Re: reinstate dm target local ioctl support

[Alasdair G Kergon]

On Thu, Jan 28, 2016 at 01:50:19PM +, Andy Whitcroft wrote:
> However this also removed the possibility of a dm target having target
> specific ioctls.  Currently this is not used by any in-tree targets but
> was utilised by the flashcache out-of-tree module.
 
What data is being passed in and out?

The method we support is passing it through DM_TARGET_MSG (as text).
For examples look at dm-switch.c and dm-stats.c, which handled
pretty large quantities of data.

Would that work?

Alasdair

Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt

[Alasdair G Kergon]

On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote:
> These are the benchmarks for request based dm-crypt. Please check it.
 
Now please put request-based dm-crypt completely to one side and focus
just on the existing bio-based code.  Why is it slower and what can be
adjusted to improve this?

People aren't going to take a request-based solution seriously until
you can explain in full detail *why* bio-based is slower AND why it's
impossible to improve its performance.

> For request based things, some sequential bios/requests can merged
> into one request to expand the IO size to be a big block handled by
> hardware engine at one time. 

Bio-based also merges I/O so that does not provide justification.
Investigate in much more detail the actual merging and scheduling
involved in the cases you need to optimise.  See if blktrace gives you
any clues, or add your own instrumentation.  You could even look at some
of the patches we've had in the list archives for optimising bio-based
crypt in different situations.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH 0/2] Introduce the request handling for dm-crypt

[Alasdair G Kergon]

On Wed, Dec 02, 2015 at 08:46:54PM +0800, Baolin Wang wrote:
> These are the benchmarks for request based dm-crypt. Please check it.
 
Now please put request-based dm-crypt completely to one side and focus
just on the existing bio-based code.  Why is it slower and what can be
adjusted to improve this?

People aren't going to take a request-based solution seriously until
you can explain in full detail *why* bio-based is slower AND why it's
impossible to improve its performance.

> For request based things, some sequential bios/requests can merged
> into one request to expand the IO size to be a big block handled by
> hardware engine at one time. 

Bio-based also merges I/O so that does not provide justification.
Investigate in much more detail the actual merging and scheduling
involved in the cases you need to optimise.  See if blktrace gives you
any clues, or add your own instrumentation.  You could even look at some
of the patches we've had in the list archives for optimising bio-based
crypt in different situations.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH v4 01/11] block: make generic_make_request handle arbitrarily sized bios

[Alasdair G Kergon]

On Wed, May 27, 2015 at 04:42:44PM -0700, Ming Lin wrote:
> Here are fio results of XFS on a DM stripped target with 2 SSDs + 1 HDD.
> Does it make sense?

To stripe across devices with different characteristics?

Some suggestions.

Prepare 3 kernels.
  O - Old kernel.
  M - Old kernel with merge_bvec_fn disabled.
  N - New kernel.

You're trying to search for counter-examples to the hypothesis that 
"Kernel N always outperforms Kernel O".  Then if you find any, trying 
to show either that the performance impediment is small enough that
it doesn't matter or that the cases are sufficiently rare or obscure
that they may be ignored because of the greater benefits of N in much more
common cases.

(1) You're looking to set up configurations where kernel O performs noticeably
better than M.  Then you're comparing the performance of O and N in those
situations.

(2) You're looking at other sensible configurations where O and M have
similar performance, and comparing that with the performance of N.

In each case you find, you expect to be able to vary some parameter (such as
stripe size) to show a progression of the effect.

When running tests you've to take care the system is reset into the same
initial state before each test, so you'll normally also try to include some
baseline test between tests that should give the same results each time
and also take the average of a number of runs (while also reporting some
measure of the variation within each set to make sure that remains low,
typically a low single digit percentage).

Since we're mostly concerned about splitting, you'll want to monitor
iostat to see if that gives you enough information to home in on 
suitable configurations for (1).  Failing that, you might need to
instrument the kernels to tell you the sizes of the bios being
created and the amount of splitting actually happening.

Striping was mentioned because it forces splitting.  So show the progression
from tiny stripes to huge stripes.  (Ensure all the devices providing the
stripes have identical characteristics, but you can test with slow and
fast underlying devices.)

You may also want to test systems with a restricted amount of available
memory to show how the splitting via worker thread performs.  (Again,
instrument to prove the extent to which the new code is being exercised.)

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH v4 01/11] block: make generic_make_request handle arbitrarily sized bios

[Alasdair G Kergon]

On Wed, May 27, 2015 at 04:42:44PM -0700, Ming Lin wrote:
 Here are fio results of XFS on a DM stripped target with 2 SSDs + 1 HDD.
 Does it make sense?

To stripe across devices with different characteristics?

Some suggestions.

Prepare 3 kernels.
  O - Old kernel.
  M - Old kernel with merge_bvec_fn disabled.
  N - New kernel.

You're trying to search for counter-examples to the hypothesis that 
Kernel N always outperforms Kernel O.  Then if you find any, trying 
to show either that the performance impediment is small enough that
it doesn't matter or that the cases are sufficiently rare or obscure
that they may be ignored because of the greater benefits of N in much more
common cases.

(1) You're looking to set up configurations where kernel O performs noticeably
better than M.  Then you're comparing the performance of O and N in those
situations.

(2) You're looking at other sensible configurations where O and M have
similar performance, and comparing that with the performance of N.

In each case you find, you expect to be able to vary some parameter (such as
stripe size) to show a progression of the effect.

When running tests you've to take care the system is reset into the same
initial state before each test, so you'll normally also try to include some
baseline test between tests that should give the same results each time
and also take the average of a number of runs (while also reporting some
measure of the variation within each set to make sure that remains low,
typically a low single digit percentage).

Since we're mostly concerned about splitting, you'll want to monitor
iostat to see if that gives you enough information to home in on 
suitable configurations for (1).  Failing that, you might need to
instrument the kernels to tell you the sizes of the bios being
created and the amount of splitting actually happening.

Striping was mentioned because it forces splitting.  So show the progression
from tiny stripes to huge stripes.  (Ensure all the devices providing the
stripes have identical characteristics, but you can test with slow and
fast underlying devices.)

You may also want to test systems with a restricted amount of available
memory to show how the splitting via worker thread performs.  (Again,
instrument to prove the extent to which the new code is being exercised.)

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH v4 01/11] block: make generic_make_request handle arbitrarily sized bios

[Alasdair G Kergon]

On Wed, May 27, 2015 at 09:06:40AM +1000, Neil Brown wrote:
> Because we don't know what the "right" size is.  And the "right" size can
> change when array reconfiguration happens.
 
In certain configurations today, device-mapper does report back a sensible
maximum bio size smaller than would otherwise be used and thereby avoids
retrospective splitting.  (In tests, the overhead of the duplicate calculation
was found to be negligible so we never restructured the code to optimise it 
away.)

> Splitting has to happen somewhere, if only in bio_addpage where it decides to
> create a new bio rather than add another page to the current one.  So moving
> the split to a different level of the stack shouldn't necessarily change the
> performance profile.
 
It does sometimes make a significant difference to device-mapper stacks.
DM only uses it for performance reasons - it can already split bios when
it needs to.  I tried to remove merge_bvec_fn from DM several years ago but
couldn't because of the adverse performance impact of lots of splitting 
activity.

The overall cost of splitting ought to be less in many (but not necessarily
all) cases now as a result of all these patches, so exactly where the best
balance lies now needs to be reassessed empirically.  It is hard to reach
conclusions theoretically because of the complex interplay between the various
factors at different levels.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH v4 01/11] block: make generic_make_request handle arbitrarily sized bios

[Alasdair G Kergon]

On Tue, May 26, 2015 at 08:02:08AM -0700, Ming Lin wrote:
> Now bio_add_page() can build lager bios.
> And blk_queue_split() can split the bios in ->make_request() if needed.

But why not try to make the bio the right size in the first place so you
don't have to incur the performance impact of splitting?

What performance testing have you yet done to demonstrate the *actual* impact
of this patchset in situations where merge_bvec_fn is currently a net benefit?

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH v4 01/11] block: make generic_make_request handle arbitrarily sized bios

[Alasdair G Kergon]

On Wed, May 27, 2015 at 09:06:40AM +1000, Neil Brown wrote:
 Because we don't know what the right size is.  And the right size can
 change when array reconfiguration happens.
 
In certain configurations today, device-mapper does report back a sensible
maximum bio size smaller than would otherwise be used and thereby avoids
retrospective splitting.  (In tests, the overhead of the duplicate calculation
was found to be negligible so we never restructured the code to optimise it 
away.)

 Splitting has to happen somewhere, if only in bio_addpage where it decides to
 create a new bio rather than add another page to the current one.  So moving
 the split to a different level of the stack shouldn't necessarily change the
 performance profile.
 
It does sometimes make a significant difference to device-mapper stacks.
DM only uses it for performance reasons - it can already split bios when
it needs to.  I tried to remove merge_bvec_fn from DM several years ago but
couldn't because of the adverse performance impact of lots of splitting 
activity.

The overall cost of splitting ought to be less in many (but not necessarily
all) cases now as a result of all these patches, so exactly where the best
balance lies now needs to be reassessed empirically.  It is hard to reach
conclusions theoretically because of the complex interplay between the various
factors at different levels.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH v4 01/11] block: make generic_make_request handle arbitrarily sized bios

[Alasdair G Kergon]

On Tue, May 26, 2015 at 08:02:08AM -0700, Ming Lin wrote:
 Now bio_add_page() can build lager bios.
 And blk_queue_split() can split the bios in -make_request() if needed.

But why not try to make the bio the right size in the first place so you
don't have to incur the performance impact of splitting?

What performance testing have you yet done to demonstrate the *actual* impact
of this patchset in situations where merge_bvec_fn is currently a net benefit?

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] Regression: Disk corruption with dm-crypt and kernels >= 4.0

[Alasdair G Kergon]

On Fri, May 01, 2015 at 12:37:07AM -0400, Abelardo Ricart III wrote:
> # first bad commit: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt: don't
> allocate pages for a partial request
 
That's not a particularly good commit to identify.

If you didn't already, can you confirm whether or not the code works at the
patch immediately following?

  7145c241a1bf2841952c3e297c4080b357b3e52d

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] Regression: Disk corruption with dm-crypt and kernels = 4.0

[Alasdair G Kergon]

On Fri, May 01, 2015 at 12:37:07AM -0400, Abelardo Ricart III wrote:
 # first bad commit: [cf2f1abfbd0dba701f7f16ef619e4d2485de3366] dm crypt: don't
 allocate pages for a partial request
 
That's not a particularly good commit to identify.

If you didn't already, can you confirm whether or not the code works at the
patch immediately following?

  7145c241a1bf2841952c3e297c4080b357b3e52d

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 07/10] block: kill merge_bvec_fn() completely

[Alasdair G Kergon]

On Mon, Apr 27, 2015 at 11:48:34PM -0700, Ming Lin wrote:
> As generic_make_request() is now able to handle arbitrarily sized bios,
> it's no longer necessary for each individual block driver to define its
> own ->merge_bvec_fn() callback. Remove every invocation completely.

merge_bvec_fn is also about telling the code building up bios what optimum
maximum size to use.
 
Have you got some test data to demonstrate that we won't find some situations
where performance is now lost by code building up bios that are far too big
and are always going to have to be split up later?

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 07/10] block: kill merge_bvec_fn() completely

[Alasdair G Kergon]

On Mon, Apr 27, 2015 at 11:48:34PM -0700, Ming Lin wrote:
 As generic_make_request() is now able to handle arbitrarily sized bios,
 it's no longer necessary for each individual block driver to define its
 own -merge_bvec_fn() callback. Remove every invocation completely.

merge_bvec_fn is also about telling the code building up bios what optimum
maximum size to use.
 
Have you got some test data to demonstrate that we won't find some situations
where performance is now lost by code building up bios that are far too big
and are always going to have to be split up later?

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH 3/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation

[Alasdair G Kergon]

On Sun, Apr 05, 2015 at 07:20:19PM +0200, Pali Rohár wrote:
> This patch adds dm message commands and option strings to optionally wipe key
> from dm-crypt device before entering suspend or hibernate state.
 
Try to avoid 0/1 - use descriptive options instead. 
E.g. key wipe_on_hibernation / key retain_on_hibernation  (message)
 wipe_key_on_hiberation ('dmsetup table' - don't forget the reporting 
interface!)

Have you tested against every state the driver might be in at the time of
suspend/hibernation?

> +static void crypt_suspend_and_wipe_key(struct crypt_config *cc)
> +{
> + dm_suspend_md(dm_table_get_md(cc->ti->table));

I'm not particularly keen on this - silently ignoring expected error states
like -EINVAL rather than checking first and not calling the function at all
when it's known not to be needed.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH 3/3] dm-crypt: Adds support for wiping key when doing suspend/hibernation

[Alasdair G Kergon]

On Sun, Apr 05, 2015 at 07:20:19PM +0200, Pali Rohár wrote:
 This patch adds dm message commands and option strings to optionally wipe key
 from dm-crypt device before entering suspend or hibernate state.
 
Try to avoid 0/1 - use descriptive options instead. 
E.g. key wipe_on_hibernation / key retain_on_hibernation  (message)
 wipe_key_on_hiberation ('dmsetup table' - don't forget the reporting 
interface!)

Have you tested against every state the driver might be in at the time of
suspend/hibernation?

 +static void crypt_suspend_and_wipe_key(struct crypt_config *cc)
 +{
 + dm_suspend_md(dm_table_get_md(cc-ti-table));

I'm not particularly keen on this - silently ignoring expected error states
like -EINVAL rather than checking first and not calling the function at all
when it's known not to be needed.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 15/45] dm-log-userspace.h: include stdint.h in userspace

[Alasdair G Kergon]

On Tue, Feb 17, 2015 at 10:08:56AM +0100, Arnd Bergmann wrote:
> The normal way to do this in kernel headers is to use linux/types.h
> but change the data structures to use __u64 instead of uint64_t
> to avoid the build error.
 
That's what happened to dm-ioctl.h.
(Or someone could adjust linux/types.h to include these as standard.)

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 15/45] dm-log-userspace.h: include stdint.h in userspace

[Alasdair G Kergon]

On Tue, Feb 17, 2015 at 10:08:56AM +0100, Arnd Bergmann wrote:
 The normal way to do this in kernel headers is to use linux/types.h
 but change the data structures to use __u64 instead of uint64_t
 to avoid the build error.
 
That's what happened to dm-ioctl.h.
(Or someone could adjust linux/types.h to include these as standard.)

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH RESEND] md:Remove no longer needed fix me comment in dm-region-hash.c for removing the structure pointer, *rh from the definiton of dm_region

[Alasdair G Kergon]

On Tue, Jan 20, 2015 at 06:36:10AM -0500, nick wrote:
> Unless we use a union or something similar there seems to be no easy 
> way without rewriting a lot of an already function driver file.

The comment does not restrict itself to 'easy ways'.

Alasdair


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH RESEND] md:Remove no longer needed fix me comment in dm-region-hash.c for removing the structure pointer, *rh from the definiton of dm_region

[Alasdair G Kergon]

On Mon, Jan 19, 2015 at 10:16:21PM -0500, Nicholas Krause wrote:
> Removes the no longer needed fix me comment questioning if we need to remove
> the structure pointer,rh as defined as a structure of type dsm_region_hash
> from the definition of the structure, dm_region. Furthermore this  pointer is
> needed in order for various functions to run and build successfully in the 
> file,
> dm-region-hash.c.
 
You've misunderstood the comment!
Is there a better alternative to having every dm_region struct store its own 
copy of
this pointer?

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH RESEND] md:Remove no longer needed fix me comment in dm-region-hash.c for removing the structure pointer, *rh from the definiton of dm_region

[Alasdair G Kergon]

On Tue, Jan 20, 2015 at 06:36:10AM -0500, nick wrote:
 Unless we use a union or something similar there seems to be no easy 
 way without rewriting a lot of an already function driver file.

The comment does not restrict itself to 'easy ways'.

Alasdair


--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH RESEND] md:Remove no longer needed fix me comment in dm-region-hash.c for removing the structure pointer, *rh from the definiton of dm_region

[Alasdair G Kergon]

On Mon, Jan 19, 2015 at 10:16:21PM -0500, Nicholas Krause wrote:
 Removes the no longer needed fix me comment questioning if we need to remove
 the structure pointer,rh as defined as a structure of type dsm_region_hash
 from the definition of the structure, dm_region. Furthermore this  pointer is
 needed in order for various functions to run and build successfully in the 
 file,
 dm-region-hash.c.
 
You've misunderstood the comment!
Is there a better alternative to having every dm_region struct store its own 
copy of
this pointer?

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] md/dm-ioctl.c: optimize memory allocation in copy_params

[Alasdair G Kergon]

On Wed, Jul 23, 2014 at 08:16:58AM -0400, Mikulas Patocka wrote:
> So, it means that you do not use device mapper at all. So, why are you 
> trying to change memory allocation in device mapper?
 
So the *test* they run is asking device-mapper to briefly reserve a 64KB
buffer when there is no data to report:  The answer is not to run that
pointless test:)

And if a single 64KB allocation really is a big deal, then patch 'vold'
in userspace so it doesn't ask for 64KB when it clearly doesn't need it!

+ int Devmapper::dumpState(SocketClient *c) {
+char *buffer = (char *) malloc(1024 * 64);

The code has just
#define DEVMAPPER_BUFFER_SIZE 4096
for all the other dm ioctls it issues.

Only use a larger value when it is needed i.e. if DM_BUFFER_FULL_FLAG gets set.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] md/dm-ioctl.c: optimize memory allocation in copy_params

[Alasdair G Kergon]

On Wed, Jul 23, 2014 at 08:16:58AM -0400, Mikulas Patocka wrote:
 So, it means that you do not use device mapper at all. So, why are you 
 trying to change memory allocation in device mapper?
 
So the *test* they run is asking device-mapper to briefly reserve a 64KB
buffer when there is no data to report:  The answer is not to run that
pointless test:)

And if a single 64KB allocation really is a big deal, then patch 'vold'
in userspace so it doesn't ask for 64KB when it clearly doesn't need it!

+ int Devmapper::dumpState(SocketClient *c) {
+char *buffer = (char *) malloc(1024 * 64);

The code has just
#define DEVMAPPER_BUFFER_SIZE 4096
for all the other dm ioctls it issues.

Only use a larger value when it is needed i.e. if DM_BUFFER_FULL_FLAG gets set.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] md/dm-ioctl.c: optimize memory allocation in copy_params

[Alasdair G Kergon]

On Tue, Jul 22, 2014 at 02:23:52AM +0100, Alasdair G Kergon wrote:
> Unanswered.  Let's ask the same question in a different way:
 
A quick search for 'vold' returns:
  https://android.googlesource.com/platform/system/vold/
and the code there requests a fixed 64K allocation to hold the names of
active volumes.

So unlike libdevmapper-based applications where a smaller allocation is
used at first and only extended if needed, Android just assumes that
64KB is enough for everyone.

So is your claim that:

1. This 64KB allocation for the brief duration of the ioctl to store the
names of active device-mapper volumes leads to memory problems?
[Mustn't the system *already* be in a bad state if this pushes it over
the limit?]

and

2. The systems on which this memory shortage occurs have so many volumes
(with long names?) that a smaller allocation would not suffice?

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] md/dm-ioctl.c: optimize memory allocation in copy_params

[Alasdair G Kergon]

>>> On 2014/7/9 6:39, Mikulas Patocka wrote:
>> Which ioctl with more than 16kB
>> arguments do you use? 

Unanswered.  Let's ask the same question in a different way:

Please supply the output of these three commands on the real-world system on
which you believe that that particular allocation is causing you a problem:
  dmsetup info -c
  dmsetup table
  dmsetup status

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] md/dm-ioctl.c: optimize memory allocation in copy_params

[Alasdair G Kergon]

 On 2014/7/9 6:39, Mikulas Patocka wrote:
 Which ioctl with more than 16kB
 arguments do you use? 

Unanswered.  Let's ask the same question in a different way:

Please supply the output of these three commands on the real-world system on
which you believe that that particular allocation is causing you a problem:
  dmsetup info -c
  dmsetup table
  dmsetup status

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] md/dm-ioctl.c: optimize memory allocation in copy_params

[Alasdair G Kergon]

On Tue, Jul 22, 2014 at 02:23:52AM +0100, Alasdair G Kergon wrote:
 Unanswered.  Let's ask the same question in a different way:
 
A quick search for 'vold' returns:
  https://android.googlesource.com/platform/system/vold/
and the code there requests a fixed 64K allocation to hold the names of
active volumes.

So unlike libdevmapper-based applications where a smaller allocation is
used at first and only extended if needed, Android just assumes that
64KB is enough for everyone.

So is your claim that:

1. This 64KB allocation for the brief duration of the ioctl to store the
names of active device-mapper volumes leads to memory problems?
[Mustn't the system *already* be in a bad state if this pushes it over
the limit?]

and

2. The systems on which this memory shortage occurs have so many volumes
(with long names?) that a smaller allocation would not suffice?

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCG]DM: dm-compression: a compressed DM target for SSD

[Alasdair G Kergon]

On Sat, Dec 28, 2013 at 12:57:25PM +, Alasdair G Kergon wrote:
> To use this compression target above dm-thin (likely to prefer larger
> block sizes), for example, could the block sizes be adapatable /
> configurable?

So a key feature of the underlying thin provisioning needs to be
efficient tracking of a small sector size (which doesn't suit the
existing dm-thin target).  This compression target could sometimes
benefit from a larger block size though, particularly where the 
data above has large blocks.

(Could we name this something like 'microcompression'?)
 
Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCG]DM: dm-compression: a compressed DM target for SSD

[Alasdair G Kergon]

I've not looked at this in any depth, but here are some first
impressions:

On Fri, Dec 27, 2013 at 02:24:21PM +0800, Shaohua Li wrote:
> This is a simple DM target supporting compression for SSD only.

Presumably there'll be other disk layouts and other types of compression
in future, so if you want to grab the generic name "compression" then
please make sure the interface to the code supports such extensions.
Use of the term "SSD" may also be too narrow as there could be other
technologies that are not labelled "SSD" that could benefit from the
target.  At best, we say "for example, ssd" leaving things open for
other uses.

IOW EITHER you should make it modular and supply a name to the ctr that
tells it to use this specific combination OR if you don't think there'll
need to be shared code with other compression types/disk layouts, rename
this particular one to something more specific.

For this naming, focus on the key feature of the code, which seems to me
to be the "in-place" or "in situ" nature of the so-called compression.
- If you don't have some form of thin provisioning underneath, why would
you use this?  
  => dm-compress-inplace / insitu
  => dm-compinsitu
  => dm-compress-thin  (sub-module loaded from dm-compress)
  => dm-compressthin   (standalone target)
 -lzo ?

To use this compression target above dm-thin (likely to prefer larger
block sizes), for example, could the block sizes be adapatable /
configurable?

Please use dm_ / DM_ prefixes - with underscore - and choose one prefix
to use consistently.  I see "cp" (makes me think "copy") as well as
"comp".

We don't label the fields in the STATUSTYPE_INFO output.

Do write Documentation/device-mapper/.txt.
E.g. what you wrote in the patch header should be moved into that file
instead.  (Use a recent documentation file as a model for the format of
the file, such as verity or thin-provisioning.)

And don't be afraid to include more comments in the code for the benefit
of people who are unfamiliar with the nuances of device-mapper
targets:)

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCG]DM: dm-compression: a compressed DM target for SSD

[Alasdair G Kergon]

I've not looked at this in any depth, but here are some first
impressions:

On Fri, Dec 27, 2013 at 02:24:21PM +0800, Shaohua Li wrote:
 This is a simple DM target supporting compression for SSD only.

Presumably there'll be other disk layouts and other types of compression
in future, so if you want to grab the generic name compression then
please make sure the interface to the code supports such extensions.
Use of the term SSD may also be too narrow as there could be other
technologies that are not labelled SSD that could benefit from the
target.  At best, we say for example, ssd leaving things open for
other uses.

IOW EITHER you should make it modular and supply a name to the ctr that
tells it to use this specific combination OR if you don't think there'll
need to be shared code with other compression types/disk layouts, rename
this particular one to something more specific.

For this naming, focus on the key feature of the code, which seems to me
to be the in-place or in situ nature of the so-called compression.
- If you don't have some form of thin provisioning underneath, why would
you use this?  
  = dm-compress-inplace / insitu
  = dm-compinsitu
  = dm-compress-thin  (sub-module loaded from dm-compress)
  = dm-compressthin   (standalone target)
 -lzo ?

To use this compression target above dm-thin (likely to prefer larger
block sizes), for example, could the block sizes be adapatable /
configurable?

Please use dm_ / DM_ prefixes - with underscore - and choose one prefix
to use consistently.  I see cp (makes me think copy) as well as
comp.

We don't label the fields in the STATUSTYPE_INFO output.

Do write Documentation/device-mapper/target_name_without_leading_dm.txt.
E.g. what you wrote in the patch header should be moved into that file
instead.  (Use a recent documentation file as a model for the format of
the file, such as verity or thin-provisioning.)

And don't be afraid to include more comments in the code for the benefit
of people who are unfamiliar with the nuances of device-mapper
targets:)

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCG]DM: dm-compression: a compressed DM target for SSD

[Alasdair G Kergon]

On Sat, Dec 28, 2013 at 12:57:25PM +, Alasdair G Kergon wrote:
 To use this compression target above dm-thin (likely to prefer larger
 block sizes), for example, could the block sizes be adapatable /
 configurable?

So a key feature of the underlying thin provisioning needs to be
efficient tracking of a small sector size (which doesn't suit the
existing dm-thin target).  This compression target could sometimes
benefit from a larger block size though, particularly where the 
data above has large blocks.

(Could we name this something like 'microcompression'?)
 
Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] FW: Re: [BUG 3.12.rc4] Oops: unable to handle kernel paging request during shutdown

[Alasdair G Kergon]

On Wed, Oct 30, 2013 at 10:18:44AM -0700, Greg KH wrote:
> The "module unload" issue is rare, thankfully, but yes, this type of
> function will be showing up in 3.13-rc1 through the btrfs tree as it
> needs that functionality, so feel free to use it to resolve this issue
> if you need it.
 
Excellent!  Though rare, yes, the number of bug reports was still
noticeable so we'll definitely take a look at this now.

Thanks,
Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] FW: Re: [BUG 3.12.rc4] Oops: unable to handle kernel paging request during shutdown

[Alasdair G Kergon]

On Wed, Oct 30, 2013 at 10:18:44AM -0700, Greg KH wrote:
 The module unload issue is rare, thankfully, but yes, this type of
 function will be showing up in 3.13-rc1 through the btrfs tree as it
 needs that functionality, so feel free to use it to resolve this issue
 if you need it.
 
Excellent!  Though rare, yes, the number of bug reports was still
noticeable so we'll definitely take a look at this now.

Thanks,
Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Alasdair G Kergon]

On Fri, Aug 30, 2013 at 09:33:29PM +0900, Akira Hayakawa wrote:
> I have three candidates on my mind.
> I am thinking of removing the dm- prefix as bcache and enhanceio do.
 
Yes, that's fine for the general way of referring to this.
(Within the codebase, the filename and module name would still need the
prefix: we use this for auto-module loading, function name/error message
scoping etc.)

>a) It writes in-coming writes in log-structured manner.
>b) It is extremely fast in write.
>c) It is caching software.

Another candidate: 'lcache' - contrast with 'bcache
 
> 3) writeboost or wboost
> Concentrating on the characteristic b)
> but doesn't mention cache.

I quite like that option.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Alasdair G Kergon]

On Fri, Aug 30, 2013 at 09:33:29PM +0900, Akira Hayakawa wrote:
 I have three candidates on my mind.
 I am thinking of removing the dm- prefix as bcache and enhanceio do.
 
Yes, that's fine for the general way of referring to this.
(Within the codebase, the filename and module name would still need the
prefix: we use this for auto-module loading, function name/error message
scoping etc.)

a) It writes in-coming writes in log-structured manner.
b) It is extremely fast in write.
c) It is caching software.

Another candidate: 'lcache' - contrast with 'bcache
 
 3) writeboost or wboost
 Concentrating on the characteristic b)
 but doesn't mention cache.

I quite like that option.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Alasdair G Kergon]

On Wed, Aug 28, 2013 at 07:05:55PM -0700, Greg KH wrote:
> For staging drivers, I need a TODO file that lists
> what needs to be done to the code to get it into a mergable state for
> the "real" part of the kernel,

Two simple requirements before putting your proof-of-concept into staging
if you want to work that way:

1) Drop the major version number to 0.  Version 1 is reserved for
supported modules.

2) Agree a new and meaningful target name with us so you don't have to
change it later.  "lc" means nothing, I'm afraid.

Then in general terms, you should continue to compare your device-mapper
target with the existing targets and where there are differences, either
change your target to be like something that already exists, or be ready
to explain why that can't or shouldn't be done.

In particular, the interface and architecture will need substantial
changes and working these out should be your highest priority.

For example, you write:

  Be careful, you MUST create all the LVs as the destinations of
  the dirty blocks on the cache device before this operation.  Otherwise,
  the kernel may crash.

I read a statement like that as an indication of an interface or
architectural problem.  The device-mapper approach is to 'design out'
problems, rather than relying on users not doing bad things.
Study the existing interfaces used by other targets to understand
some approaches that proved successful, then decide which ones
come closest to your needs.

(Your code also needs many more comments inline to explain what it does
and how it works.)

The documentation file will eventually need rewriting to follow the same
format as the other targets recently added to the kernel.  We document
the kernel interface rather than any particular userspace tools, which
just have the status of convenient examples.

Another little thing I noticed: look into using something like
__dm_bless_for_disk() for your metadata and clearly segregate your
on-disk structures and document the layout.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [RFC] dm-lc: plan to go to staging

[Alasdair G Kergon]

On Wed, Aug 28, 2013 at 07:05:55PM -0700, Greg KH wrote:
 For staging drivers, I need a TODO file that lists
 what needs to be done to the code to get it into a mergable state for
 the real part of the kernel,

Two simple requirements before putting your proof-of-concept into staging
if you want to work that way:

1) Drop the major version number to 0.  Version 1 is reserved for
supported modules.

2) Agree a new and meaningful target name with us so you don't have to
change it later.  lc means nothing, I'm afraid.

Then in general terms, you should continue to compare your device-mapper
target with the existing targets and where there are differences, either
change your target to be like something that already exists, or be ready
to explain why that can't or shouldn't be done.

In particular, the interface and architecture will need substantial
changes and working these out should be your highest priority.

For example, you write:

  Be careful, you MUST create all the LVs as the destinations of
  the dirty blocks on the cache device before this operation.  Otherwise,
  the kernel may crash.

I read a statement like that as an indication of an interface or
architectural problem.  The device-mapper approach is to 'design out'
problems, rather than relying on users not doing bad things.
Study the existing interfaces used by other targets to understand
some approaches that proved successful, then decide which ones
come closest to your needs.

(Your code also needs many more comments inline to explain what it does
and how it works.)

The documentation file will eventually need rewriting to follow the same
format as the other targets recently added to the kernel.  We document
the kernel interface rather than any particular userspace tools, which
just have the status of convenient examples.

Another little thing I noticed: look into using something like
__dm_bless_for_disk() for your metadata and clearly segregate your
on-disk structures and document the layout.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] dm: Make MIN_IOS, et al, tunable via sysctl.

[Alasdair G Kergon]

On Fri, Aug 16, 2013 at 03:55:21PM -0700, Frank Mayhar wrote:
> This patch fixes that by changing the hardcoded MIN_IOS (and certain
> other) #defines in dm-crypt, dm-io, dm-mpath, dm-snap and dm itself to
> sysctl-modifiable values.  This lets us change the size of these pools
> on the fly, we can reduce the size of the pools and reduce memory
> pressure.
 
Did you consider using module_param_named()?
(E.g. dm_verity_prefetch_cluster in dm-verity.)

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] dm: Make MIN_IOS, et al, tunable via sysctl.

[Alasdair G Kergon]

On Fri, Aug 16, 2013 at 03:55:21PM -0700, Frank Mayhar wrote:
 This patch fixes that by changing the hardcoded MIN_IOS (and certain
 other) #defines in dm-crypt, dm-io, dm-mpath, dm-snap and dm itself to
 sysctl-modifiable values.  This lets us change the size of these pools
 on the fly, we can reduce the size of the pools and reduce memory
 pressure.
 
Did you consider using module_param_named()?
(E.g. dm_verity_prefetch_cluster in dm-verity.)

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] dm cache: Avoid conflicting remove_mapping() in mq policy

[Alasdair G Kergon]

On Fri, Aug 16, 2013 at 01:31:18AM -0700, Guenter Roeck wrote:
> Is this patch going anyhere ? The problem breaks the sparc32:allmodconfig 
> build,
> so it would be useful to get it fixed.

We're pulling together a batch of patches today, including this one.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] dm cache: Avoid conflicting remove_mapping() in mq policy

[Alasdair G Kergon]

On Fri, Aug 16, 2013 at 01:31:18AM -0700, Guenter Roeck wrote:
 Is this patch going anyhere ? The problem breaks the sparc32:allmodconfig 
 build,
 so it would be useful to get it fixed.

We're pulling together a batch of patches today, including this one.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/5] Add aix lvm partitions support files

[Alasdair G Kergon]

On Mon, Aug 12, 2013 at 04:52:34AM -0700, Christoph Hellwig wrote:
> lvm is mostly about the Linux specific formats.  If you want to create
> a separate userspace consumer for device mapper that's probably also
> fine, but if you need support for complex layours device mapper is the
> way to go, and the partitioning layer isn't getting you anywhere.
 
If you can map the AIX volume manager objects onto the linux ones, you
can write a 'format' module for lvm2 to handle them.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 2/5] Add aix lvm partitions support files

[Alasdair G Kergon]

On Mon, Aug 12, 2013 at 04:52:34AM -0700, Christoph Hellwig wrote:
 lvm is mostly about the Linux specific formats.  If you want to create
 a separate userspace consumer for device mapper that's probably also
 fine, but if you need support for complex layours device mapper is the
 way to go, and the partitioning layer isn't getting you anywhere.
 
If you can map the AIX volume manager objects onto the linux ones, you
can write a 'format' module for lvm2 to handle them.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [git pull] device-mapper changes for 3.11

[Alasdair G Kergon]

On Thu, Jul 11, 2013 at 04:20:40PM +1000, Stephen Rothwell wrote:
> Alistair, is there some reason that you cannot maintain some relatively
> stable git tree for me to fetch (instead of the quilt series).  

Well Mike Snitzer is becoming a joint maintainer and this also requires
changes to the workflow so watch this space.

> Also,
> there is not real reason to keep rebasing this stuff (especially just
> before sending it to Linus).
 
Most of the time the patches are decoupled from the rest of the tree
and the final linux-next kernel is the same regardless of where the
quilt tree is attached.  I maintain a single sequence of patches 
where the ones ready for upstream float up to the top and the
linux-next export is a by-product and not the primary output.

Anyway, like I say, we'll be changing things during the next couple of
cycles and we'll bear your suggestions in mind.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [git pull] device-mapper changes for 3.11

[Alasdair G Kergon]

On Thu, Jul 11, 2013 at 04:20:40PM +1000, Stephen Rothwell wrote:
 Alistair, is there some reason that you cannot maintain some relatively
 stable git tree for me to fetch (instead of the quilt series).  

Well Mike Snitzer is becoming a joint maintainer and this also requires
changes to the workflow so watch this space.

 Also,
 there is not real reason to keep rebasing this stuff (especially just
 before sending it to Linus).
 
Most of the time the patches are decoupled from the rest of the tree
and the final linux-next kernel is the same regardless of where the
quilt tree is attached.  I maintain a single sequence of patches 
where the ones ready for upstream float up to the top and the
linux-next export is a by-product and not the primary output.

Anyway, like I say, we'll be changing things during the next couple of
cycles and we'll bear your suggestions in mind.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: linux-next: rebase of the device-mapper quilt series

[Alasdair G Kergon]

On Thu, Jul 11, 2013 at 10:31:16AM +1000, Stephen Rothwell wrote:
> You have
> just invalidated your testing 

On the contrary, it is part of my testing.

> Your whole series was already
> based after v3.10 (i.e. released or rebased after the merge window
> opened), so why move it again?
 
I usually base it either on the actual final release, v3.10 in this
case, or, if there are non-trivial conflicts, a suitable commit from a
point after the inclusion of the tree (normally block) leading to the
conflict.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: linux-next: rebase of the device-mapper quilt series

[Alasdair G Kergon]

On Thu, Jul 11, 2013 at 10:31:16AM +1000, Stephen Rothwell wrote:
> Why have you just rebased the device-mapper series

I was confirming there weren't any unexpected conflicts or other issues to
deal with from a point after the block tree went in before generating
the pull request.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper changes for 3.11

[Alasdair G Kergon]

Please pull from:
 
  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm 
tags/dm-3.11-changes

to get the following device-mapper changes for 3.11.

Thanks,
Alasdair



Add a device-mapper target called dm-switch to provide a multipath
framework for storage arrays that dynamically reconfigure their
preferred paths for different device regions.

Fix a bug in the verity target that prevented its use with some
specific sizes of devices.

Improve some locking mechanisms in the device-mapper core and bufio.

Add Mike Snitzer as a device-mapper maintainer.

A few more clean-ups and fixes.


Alasdair G Kergon (2):
  dm flakey: correct ctr alloc failure mesg
  dm: update maintainers

Hannes Reinecke (1):
  dm mpath: fix ioctl deadlock when no paths

Jim Ramsay (1):
  dm: add switch target

Mikulas Patocka (9):
  dm ioctl: set noio flag to avoid __vmalloc deadlock
  dm verity: fix inability to use a few specific devices sizes
  dm: use __GFP_HIGHMEM in __vmalloc
  dm verity: remove pointless comparison
  dm verity: use __ffs and __fls
  dm cache: fix arm link errors with inline
  dm bufio: submit writes outside lock
  dm: optimize use SRCU and RCU
  dm: optimize reorder structure

 Documentation/device-mapper/switch.txt |  126 
 MAINTAINERS|2 +
 drivers/md/Kconfig |   14 +
 drivers/md/Makefile|1 +
 drivers/md/dm-bufio.c  |   75 -
 drivers/md/dm-cache-target.c   |4 +
 drivers/md/dm-flakey.c |2 +-
 drivers/md/dm-ioctl.c  |  127 +---
 drivers/md/dm-mpath.c  |8 +-
 drivers/md/dm-switch.c |  538 
 drivers/md/dm-table.c  |   35 ---
 drivers/md/dm-verity.c |   17 +-
 drivers/md/dm.c|  177 ++-
 include/linux/device-mapper.h  |6 +-
 include/uapi/linux/dm-ioctl.h  |4 +-
 15 files changed, 951 insertions(+), 185 deletions(-)
 create mode 100644 Documentation/device-mapper/switch.txt
 create mode 100644 drivers/md/dm-switch.c

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper changes for 3.11

[Alasdair G Kergon]

Please pull from:
 
  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm 
tags/dm-3.11-changes

to get the following device-mapper changes for 3.11.

Thanks,
Alasdair



Add a device-mapper target called dm-switch to provide a multipath
framework for storage arrays that dynamically reconfigure their
preferred paths for different device regions.

Fix a bug in the verity target that prevented its use with some
specific sizes of devices.

Improve some locking mechanisms in the device-mapper core and bufio.

Add Mike Snitzer as a device-mapper maintainer.

A few more clean-ups and fixes.


Alasdair G Kergon (2):
  dm flakey: correct ctr alloc failure mesg
  dm: update maintainers

Hannes Reinecke (1):
  dm mpath: fix ioctl deadlock when no paths

Jim Ramsay (1):
  dm: add switch target

Mikulas Patocka (9):
  dm ioctl: set noio flag to avoid __vmalloc deadlock
  dm verity: fix inability to use a few specific devices sizes
  dm: use __GFP_HIGHMEM in __vmalloc
  dm verity: remove pointless comparison
  dm verity: use __ffs and __fls
  dm cache: fix arm link errors with inline
  dm bufio: submit writes outside lock
  dm: optimize use SRCU and RCU
  dm: optimize reorder structure

 Documentation/device-mapper/switch.txt |  126 
 MAINTAINERS|2 +
 drivers/md/Kconfig |   14 +
 drivers/md/Makefile|1 +
 drivers/md/dm-bufio.c  |   75 -
 drivers/md/dm-cache-target.c   |4 +
 drivers/md/dm-flakey.c |2 +-
 drivers/md/dm-ioctl.c  |  127 +---
 drivers/md/dm-mpath.c  |8 +-
 drivers/md/dm-switch.c |  538 
 drivers/md/dm-table.c  |   35 ---
 drivers/md/dm-verity.c |   17 +-
 drivers/md/dm.c|  177 ++-
 include/linux/device-mapper.h  |6 +-
 include/uapi/linux/dm-ioctl.h  |4 +-
 15 files changed, 951 insertions(+), 185 deletions(-)
 create mode 100644 Documentation/device-mapper/switch.txt
 create mode 100644 drivers/md/dm-switch.c

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: linux-next: rebase of the device-mapper quilt series

[Alasdair G Kergon]

On Thu, Jul 11, 2013 at 10:31:16AM +1000, Stephen Rothwell wrote:
 Why have you just rebased the device-mapper series

I was confirming there weren't any unexpected conflicts or other issues to
deal with from a point after the block tree went in before generating
the pull request.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: linux-next: rebase of the device-mapper quilt series

[Alasdair G Kergon]

On Thu, Jul 11, 2013 at 10:31:16AM +1000, Stephen Rothwell wrote:
 You have
 just invalidated your testing 

On the contrary, it is part of my testing.

 Your whole series was already
 based after v3.10 (i.e. released or rebased after the merge window
 opened), so why move it again?
 
I usually base it either on the actual final release, v3.10 in this
case, or, if there are non-trivial conflicts, a suitable commit from a
point after the inclusion of the tree (normally block) leading to the
conflict.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] MAINTAINERS: dm-de...@redhat.com is a list

[Alasdair G Kergon]

On Mon, Jul 08, 2013 at 10:21:43AM -0700, Joe Perches wrote:
> I think this Q entry isn't as useful as it could be
> because it doesn't show the status of any patch.
 
When kernel.org patchwork was unavailable for many months we migrated
over to a private instance - and we still haven't moved back!  I
discussed this with Mike a couple of weeks ago and we decided to have a
go at moving back to the public instance again before the next merge
window.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] MAINTAINERS: dm-de...@redhat.com is a list

[Alasdair G Kergon]

On Tue, Jul 02, 2013 at 05:01:47PM +0200, Yann Droneaud wrote:
> This patch changes type of dm-de...@redhat.com
> The output of ./scripts/get_maintainer.pl is
> modified to report dm-de...@redhat.com as a list:
 
>  DEVICE-MAPPER  (LVM)
>  M:   Alasdair Kergon 
> -M:   dm-de...@redhat.com
>  L:   dm-de...@redhat.com
>  W:   http://sources.redhat.com/dm
>  Q:   http://patchwork.kernel.org/project/dm-devel/list/

Is this part of some other changes to the definition of M and L?

Otherwise I don't agree with the change.

MAINTAINERS states:

M: Mail patches to: FullName 
L: Mailing list that is relevant to this area

In the case of dm-devel, it's not merely a "relevant" list, but it is
essential for us: patches to dm must be sent to the list for patchwork
to collect.  I shall also add Mike Snitzer as a joint maintainer.

And did you notice this bit with uapi?
F:  include/linux/dm-*.h

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] MAINTAINERS: dm-de...@redhat.com is a list

[Alasdair G Kergon]

On Tue, Jul 02, 2013 at 05:01:47PM +0200, Yann Droneaud wrote:
 This patch changes type of dm-de...@redhat.com
 The output of ./scripts/get_maintainer.pl is
 modified to report dm-de...@redhat.com as a list:
 
  DEVICE-MAPPER  (LVM)
  M:   Alasdair Kergon a...@redhat.com
 -M:   dm-de...@redhat.com
  L:   dm-de...@redhat.com
  W:   http://sources.redhat.com/dm
  Q:   http://patchwork.kernel.org/project/dm-devel/list/

Is this part of some other changes to the definition of M and L?

Otherwise I don't agree with the change.

MAINTAINERS states:

M: Mail patches to: FullName address@domain
L: Mailing list that is relevant to this area

In the case of dm-devel, it's not merely a relevant list, but it is
essential for us: patches to dm must be sent to the list for patchwork
to collect.  I shall also add Mike Snitzer as a joint maintainer.

And did you notice this bit with uapi?
F:  include/linux/dm-*.h

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] MAINTAINERS: dm-de...@redhat.com is a list

[Alasdair G Kergon]

On Mon, Jul 08, 2013 at 10:21:43AM -0700, Joe Perches wrote:
 I think this Q entry isn't as useful as it could be
 because it doesn't show the status of any patch.
 
When kernel.org patchwork was unavailable for many months we migrated
over to a private instance - and we still haven't moved back!  I
discussed this with Mike a couple of weeks ago and we decided to have a
go at moving back to the public instance again before the next merge
window.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] dm: verity: Add support for emitting uevents on dm-verity errors.

[Alasdair G Kergon]

On Thu, Jun 20, 2013 at 06:12:07PM -0700, Geremy Condra wrote:
> Assuming nobody currently using DM_UEVENT minds it doesn't cause any
> problems for me. Any current users object?
 
DM_UEVENT toggles uevents on or off across the whole of device-mapper.

If there is no particular need for verity *not* to generate uevents
while other parts of device-mapper *are* generating them, then we can
make do with just the one Kconfig parameter.  (Userspace is still free
to ignore particular events.)

So let's drop the second config parameter.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] dm: verity: Add support for emitting uevents on dm-verity errors.

[Alasdair G Kergon]

On Thu, Jun 20, 2013 at 02:21:00PM -0700, Geremy Condra wrote:
> With this change and config option, dm-verity errors
> will cause uevents to be sent to userspace, notifying
> it that an error has occurred and potentially
> triggering recovery actions.
 
> +config DM_VERITY_ERROR_NOTIFY
> + tristate "Notify userspace of verity errors through uevents"
> + depends on DM_VERITY && DM_UEVENT
> + ---help---
> +   This option instructs dm-verity to send uevents on error.
> +

Would it cause any problems if we had this feature enabled all the time,
controlled only by DM_UEVENT?

In other words, if we delete DM_VERITY_ERROR_NOTIFY from the upstream patch?

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] dm: verity: Add support for emitting uevents on dm-verity errors.

[Alasdair G Kergon]

On Thu, Jun 20, 2013 at 02:21:00PM -0700, Geremy Condra wrote:
 With this change and config option, dm-verity errors
 will cause uevents to be sent to userspace, notifying
 it that an error has occurred and potentially
 triggering recovery actions.
 
 +config DM_VERITY_ERROR_NOTIFY
 + tristate Notify userspace of verity errors through uevents
 + depends on DM_VERITY  DM_UEVENT
 + ---help---
 +   This option instructs dm-verity to send uevents on error.
 +

Would it cause any problems if we had this feature enabled all the time,
controlled only by DM_UEVENT?

In other words, if we delete DM_VERITY_ERROR_NOTIFY from the upstream patch?

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] dm: verity: Add support for emitting uevents on dm-verity errors.

[Alasdair G Kergon]

On Thu, Jun 20, 2013 at 06:12:07PM -0700, Geremy Condra wrote:
 Assuming nobody currently using DM_UEVENT minds it doesn't cause any
 problems for me. Any current users object?
 
DM_UEVENT toggles uevents on or off across the whole of device-mapper.

If there is no particular need for verity *not* to generate uevents
while other parts of device-mapper *are* generating them, then we can
make do with just the one Kconfig parameter.  (Userspace is still free
to ignore particular events.)

So let's drop the second config parameter.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper fix for 3.10

[Alasdair G Kergon]

Please pull from:
 
  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm tags/dm-3.10-fixes

to get the following device-mapper fix for 3.10.

Thanks,
Alasdair



A patch to fix metadata resizing with device-mapper thin devices.


Alasdair G Kergon (1):
  dm thin: fix metadata dev resize detection

 drivers/md/dm-thin.c |4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper fix for 3.10

[Alasdair G Kergon]

Please pull from:
 
  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm tags/dm-3.10-fixes

to get the following device-mapper fix for 3.10.

Thanks,
Alasdair



A patch to fix metadata resizing with device-mapper thin devices.


Alasdair G Kergon (1):
  dm thin: fix metadata dev resize detection

 drivers/md/dm-thin.c |4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [git pull] device-mapper updates for 3.10

[Alasdair G Kergon]

On Fri, May 10, 2013 at 11:20:29AM +0100, Alasdair G Kergon wrote:
> Please pull from:
>   git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm 
> tags/dm-3.10-changes

And the kbuild test robot just spotted that I lost a line from one of
the patches which meant the thin threshold notification wouldn't
happen, sorry.

Take two...

Alasdair


Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm 
tags/dm-3.10-changes-2

to get the following device-mapper updates for 3.10.


Allow devices that hold metadata for the device-mapper thin
provisioning target to be extended easily; allow WRITE SAME on
multipath devices; an assortment of little fixes and clean-ups.


Alasdair G Kergon (3):
  dm: document iterate_devices
  dm cache policy: fix description of lookup fn
  dm cache: move config fns

Joe Thornber (12):
  dm cache: tune migration throttling
  dm persistent data: fix error message typos
  dm cache: fix typos in comments
  dm cache: replace memcpy with struct assignment
  dm thin: refactor data dev resize
  dm thin: open dev read only when possible
  dm persistent data: support space map resizing
  dm thin: detect metadata device resizing
  dm persistent data: add threshold callback to space map
  dm persistent metadata: add space map threshold callback
  dm thin: generate event when metadata threshold passed
  dm cache: set config value

Mike Snitzer (3):
  dm stripe: fix regression in stripe_width calculation
  dm table: fix write same support
  dm mpath: enable WRITE SAME support

Mikulas Patocka (1):
  dm bufio: avoid a possible __vmalloc deadlock

Wei Yongjun (2):
  dm cache: fix error return code in cache_create
  dm snapshot: fix error return code in snapshot_ctr

 drivers/md/dm-bufio.c  |   24 ++-
 drivers/md/dm-cache-metadata.c |4 +-
 drivers/md/dm-cache-policy.h   |4 +-
 drivers/md/dm-cache-target.c   |  100 +-
 drivers/md/dm-mpath.c  |1 +
 drivers/md/dm-snap.c   |1 +
 drivers/md/dm-stripe.c |   11 +-
 drivers/md/dm-table.c  |2 +-
 drivers/md/dm-thin-metadata.c  |   36 +++-
 drivers/md/dm-thin-metadata.h  |7 +
 drivers/md/dm-thin.c   |  200 
 drivers/md/persistent-data/dm-space-map-disk.c |3 +-
 drivers/md/persistent-data/dm-space-map-metadata.c |  127 +++--
 drivers/md/persistent-data/dm-space-map.h  |   23 +++
 include/linux/device-mapper.h  |   15 ++
 15 files changed, 442 insertions(+), 116 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper updates for 3.10

[Alasdair G Kergon]

Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm 
tags/dm-3.10-changes

to get the following device-mapper updates for 3.10.

Thanks,
Alasdair


Allow devices that hold metadata for the device-mapper thin
provisioning target to be extended easily; allow WRITE SAME on
multipath devices; an assortment of little fixes and clean-ups.


Alasdair G Kergon (3):
  dm: document iterate_devices
  dm cache policy: fix description of lookup fn
  dm cache: move config fns

Joe Thornber (12):
  dm cache: tune migration throttling
  dm persistent data: fix error message typos
  dm cache: fix typos in comments
  dm cache: replace memcpy with struct assignment
  dm thin: refactor data dev resize
  dm thin: open dev read only when possible
  dm persistent data: support space map resizing
  dm thin: detect metadata device resizing
  dm persistent data: add threshold callback to space map
  dm persistent metadata: add space map threshold callback
  dm thin: generate event when metadata threshold passed
  dm cache: set config value

Mike Snitzer (3):
  dm stripe: fix regression in stripe_width calculation
  dm table: fix write same support
  dm mpath: enable WRITE SAME support

Mikulas Patocka (1):
  dm bufio: avoid a possible __vmalloc deadlock

Wei Yongjun (2):
  dm cache: fix error return code in cache_create
  dm snapshot: fix error return code in snapshot_ctr

 drivers/md/dm-bufio.c  |   24 ++-
 drivers/md/dm-cache-metadata.c |4 +-
 drivers/md/dm-cache-policy.h   |4 +-
 drivers/md/dm-cache-target.c   |  100 +-
 drivers/md/dm-mpath.c  |1 +
 drivers/md/dm-snap.c   |1 +
 drivers/md/dm-stripe.c |   11 +-
 drivers/md/dm-table.c  |2 +-
 drivers/md/dm-thin-metadata.c  |   36 +++-
 drivers/md/dm-thin-metadata.h  |7 +
 drivers/md/dm-thin.c   |  200 
 drivers/md/persistent-data/dm-space-map-disk.c |3 +-
 drivers/md/persistent-data/dm-space-map-metadata.c |  127 +++--
 drivers/md/persistent-data/dm-space-map.h  |   23 +++
 include/linux/device-mapper.h  |   15 ++
 15 files changed, 442 insertions(+), 116 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper updates for 3.10

[Alasdair G Kergon]

Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm 
tags/dm-3.10-changes

to get the following device-mapper updates for 3.10.

Thanks,
Alasdair


Allow devices that hold metadata for the device-mapper thin
provisioning target to be extended easily; allow WRITE SAME on
multipath devices; an assortment of little fixes and clean-ups.


Alasdair G Kergon (3):
  dm: document iterate_devices
  dm cache policy: fix description of lookup fn
  dm cache: move config fns

Joe Thornber (12):
  dm cache: tune migration throttling
  dm persistent data: fix error message typos
  dm cache: fix typos in comments
  dm cache: replace memcpy with struct assignment
  dm thin: refactor data dev resize
  dm thin: open dev read only when possible
  dm persistent data: support space map resizing
  dm thin: detect metadata device resizing
  dm persistent data: add threshold callback to space map
  dm persistent metadata: add space map threshold callback
  dm thin: generate event when metadata threshold passed
  dm cache: set config value

Mike Snitzer (3):
  dm stripe: fix regression in stripe_width calculation
  dm table: fix write same support
  dm mpath: enable WRITE SAME support

Mikulas Patocka (1):
  dm bufio: avoid a possible __vmalloc deadlock

Wei Yongjun (2):
  dm cache: fix error return code in cache_create
  dm snapshot: fix error return code in snapshot_ctr

 drivers/md/dm-bufio.c  |   24 ++-
 drivers/md/dm-cache-metadata.c |4 +-
 drivers/md/dm-cache-policy.h   |4 +-
 drivers/md/dm-cache-target.c   |  100 +-
 drivers/md/dm-mpath.c  |1 +
 drivers/md/dm-snap.c   |1 +
 drivers/md/dm-stripe.c |   11 +-
 drivers/md/dm-table.c  |2 +-
 drivers/md/dm-thin-metadata.c  |   36 +++-
 drivers/md/dm-thin-metadata.h  |7 +
 drivers/md/dm-thin.c   |  200 
 drivers/md/persistent-data/dm-space-map-disk.c |3 +-
 drivers/md/persistent-data/dm-space-map-metadata.c |  127 +++--
 drivers/md/persistent-data/dm-space-map.h  |   23 +++
 include/linux/device-mapper.h  |   15 ++
 15 files changed, 442 insertions(+), 116 deletions(-)

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [git pull] device-mapper updates for 3.10

[Alasdair G Kergon]

On Fri, May 10, 2013 at 11:20:29AM +0100, Alasdair G Kergon wrote:
 Please pull from:
   git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm 
 tags/dm-3.10-changes

And the kbuild test robot just spotted that I lost a line from one of
the patches which meant the thin threshold notification wouldn't
happen, sorry.

Take two...

Alasdair


Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm 
tags/dm-3.10-changes-2

to get the following device-mapper updates for 3.10.


Allow devices that hold metadata for the device-mapper thin
provisioning target to be extended easily; allow WRITE SAME on
multipath devices; an assortment of little fixes and clean-ups.


Alasdair G Kergon (3):
  dm: document iterate_devices
  dm cache policy: fix description of lookup fn
  dm cache: move config fns

Joe Thornber (12):
  dm cache: tune migration throttling
  dm persistent data: fix error message typos
  dm cache: fix typos in comments
  dm cache: replace memcpy with struct assignment
  dm thin: refactor data dev resize
  dm thin: open dev read only when possible
  dm persistent data: support space map resizing
  dm thin: detect metadata device resizing
  dm persistent data: add threshold callback to space map
  dm persistent metadata: add space map threshold callback
  dm thin: generate event when metadata threshold passed
  dm cache: set config value

Mike Snitzer (3):
  dm stripe: fix regression in stripe_width calculation
  dm table: fix write same support
  dm mpath: enable WRITE SAME support

Mikulas Patocka (1):
  dm bufio: avoid a possible __vmalloc deadlock

Wei Yongjun (2):
  dm cache: fix error return code in cache_create
  dm snapshot: fix error return code in snapshot_ctr

 drivers/md/dm-bufio.c  |   24 ++-
 drivers/md/dm-cache-metadata.c |4 +-
 drivers/md/dm-cache-policy.h   |4 +-
 drivers/md/dm-cache-target.c   |  100 +-
 drivers/md/dm-mpath.c  |1 +
 drivers/md/dm-snap.c   |1 +
 drivers/md/dm-stripe.c |   11 +-
 drivers/md/dm-table.c  |2 +-
 drivers/md/dm-thin-metadata.c  |   36 +++-
 drivers/md/dm-thin-metadata.h  |7 +
 drivers/md/dm-thin.c   |  200 
 drivers/md/persistent-data/dm-space-map-disk.c |3 +-
 drivers/md/persistent-data/dm-space-map-metadata.c |  127 +++--
 drivers/md/persistent-data/dm-space-map.h  |   23 +++
 include/linux/device-mapper.h  |   15 ++
 15 files changed, 442 insertions(+), 116 deletions(-)
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper fixes for 3.9-rc6

[Alasdair G Kergon]

Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm tags/dm-3.9-fixes-2
 
to get the following device-mapper fixes for 3.9.

Thanks,
Alasdair
 

A pair of patches to fix the writethrough mode of the device-mapper
cache target when the device being cached is not itself wrapped with
device-mapper.


Darrick J. Wong (1):
  dm cache: fix writes to cache device in writethrough mode

Mike Snitzer (1):
  dm cache: reduce bio front_pad size in writeback mode

 drivers/md/dm-cache-target.c |   51 +++---
 1 file changed, 38 insertions(+), 13 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper fixes for 3.9-rc6

[Alasdair G Kergon]

Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm tags/dm-3.9-fixes-2
 
to get the following device-mapper fixes for 3.9.

Thanks,
Alasdair
 

A pair of patches to fix the writethrough mode of the device-mapper
cache target when the device being cached is not itself wrapped with
device-mapper.


Darrick J. Wong (1):
  dm cache: fix writes to cache device in writethrough mode

Mike Snitzer (1):
  dm cache: reduce bio front_pad size in writeback mode

 drivers/md/dm-cache-target.c |   51 +++---
 1 file changed, 38 insertions(+), 13 deletions(-)
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] dm-crypt performance

[Alasdair G Kergon]

[Adding dm-crypt + linux-kernel]

On Mon, Mar 25, 2013 at 11:47:22PM -0400, Mikulas Patocka wrote:
> I performed some dm-crypt performance tests as Mike suggested.
> 
> It turns out that unbound workqueue performance has improved somewhere 
> between kernel 3.2 (when I made the dm-crypt patches) and 3.8, so the 
> patches for hand-built dispatch are no longer needed.
> 
> For RAID-0 composed of two disks with total throughput 260MB/s, the 
> unbound workqueue performs as well as the hand-built dispatch (both 
> sustain the 260MB/s transfer rate).
> 
> For ramdisk, unbound workqueue performs better than hand-built dispatch 
> (620MB/s vs 400MB/s). Unbound workqueue with the patch that Mike suggested 
> (git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git) improves 
> performance slighlty on ramdisk compared to 3.8 (700MB/s vs. 620MB/s).
> 
> 
> 
> However, there is still the problem with request ordering. Milan found out 
> that under some circumstances parallel dm-crypt has worse performance than 
> the previous dm-crypt code. I found out that this is not caused by 
> deficiencies in the code that distributes work to individual processors. 
> Performance drop is caused by the fact that distributing write bios to 
> multiple processors causes the encryption to finish out of order and the 
> I/O scheduler is unable to merge these out-of-order bios.
> 
> The deadline and noop schedulers perform better (only 50% slowdown 
> compared to old dm-crypt), CFQ performs very badly (8 times slowdown).
> 
> 
> If I sort the requests in dm-crypt to come out in the same order as they 
> were received, there is no longer any slowdown, the new crypt performs as 
> well as the old crypt, but the last time I submitted the patches, people 
> objected to sorting requests in dm-crypt, saying that the I/O scheduler 
> should sort them. But it doesn't. This problem still persists in the 
> current kernels.
> 
> 
> For best performance we could use the unbound workqueue implementation 
> with request sorting, if people don't object to the request sorting being 
> done in dm-crypt.


On Tue, Mar 26, 2013 at 02:52:29AM -0400, Christoph Hellwig wrote:
> FYI, XFS also does it's own request ordering for the metadata buffers,
> because it knows the needed ordering and has a bigger view than than
> than especially CFQ.  You at least have precedence in a widely used
> subsystem for this code.


So please post this updated version of the patches for a wider group of
people to try out.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] dm-crypt performance

[Alasdair G Kergon]

[Adding dm-crypt + linux-kernel]

On Mon, Mar 25, 2013 at 11:47:22PM -0400, Mikulas Patocka wrote:
 I performed some dm-crypt performance tests as Mike suggested.
 
 It turns out that unbound workqueue performance has improved somewhere 
 between kernel 3.2 (when I made the dm-crypt patches) and 3.8, so the 
 patches for hand-built dispatch are no longer needed.
 
 For RAID-0 composed of two disks with total throughput 260MB/s, the 
 unbound workqueue performs as well as the hand-built dispatch (both 
 sustain the 260MB/s transfer rate).
 
 For ramdisk, unbound workqueue performs better than hand-built dispatch 
 (620MB/s vs 400MB/s). Unbound workqueue with the patch that Mike suggested 
 (git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git) improves 
 performance slighlty on ramdisk compared to 3.8 (700MB/s vs. 620MB/s).
 
 
 
 However, there is still the problem with request ordering. Milan found out 
 that under some circumstances parallel dm-crypt has worse performance than 
 the previous dm-crypt code. I found out that this is not caused by 
 deficiencies in the code that distributes work to individual processors. 
 Performance drop is caused by the fact that distributing write bios to 
 multiple processors causes the encryption to finish out of order and the 
 I/O scheduler is unable to merge these out-of-order bios.
 
 The deadline and noop schedulers perform better (only 50% slowdown 
 compared to old dm-crypt), CFQ performs very badly (8 times slowdown).
 
 
 If I sort the requests in dm-crypt to come out in the same order as they 
 were received, there is no longer any slowdown, the new crypt performs as 
 well as the old crypt, but the last time I submitted the patches, people 
 objected to sorting requests in dm-crypt, saying that the I/O scheduler 
 should sort them. But it doesn't. This problem still persists in the 
 current kernels.
 
 
 For best performance we could use the unbound workqueue implementation 
 with request sorting, if people don't object to the request sorting being 
 done in dm-crypt.


On Tue, Mar 26, 2013 at 02:52:29AM -0400, Christoph Hellwig wrote:
 FYI, XFS also does it's own request ordering for the metadata buffers,
 because it knows the needed ordering and has a bigger view than than
 than especially CFQ.  You at least have precedence in a widely used
 subsystem for this code.


So please post this updated version of the patches for a wider group of
people to try out.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] dm: dm-cache fails to write the cache device in writethrough mode

[Alasdair G Kergon]

On Fri, Mar 22, 2013 at 10:15:49PM -0700, Darrick J. Wong wrote:
> Interesting ... if I set up this:
> # echo "0 67108864 linear /dev/sda 0" | dmsetup create origin
> And then repeat the test, but using /dev/mapper/origin as the origin instead
> of /dev/sda, the problem goes away.
 
So that looks like an easy and correct workaround for now.

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] dm: dm-cache fails to write the cache device in writethrough mode

[Alasdair G Kergon]

On Fri, Mar 22, 2013 at 10:15:49PM -0700, Darrick J. Wong wrote:
 Interesting ... if I set up this:
 # echo 0 67108864 linear /dev/sda 0 | dmsetup create origin
 And then repeat the test, but using /dev/mapper/origin as the origin instead
 of /dev/sda, the problem goes away.
 
So that looks like an easy and correct workaround for now.

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] dm: dm-cache fails to write the cache device in writethrough mode

[Alasdair G Kergon]

On Fri, Mar 22, 2013 at 11:27:16PM -0400, Mike Snitzer wrote:
> So the big difference is the thinp-test-suite uses intermediate linear
> DM layers above the slower sd device (spindle/data) -- whereas in your
> setup the origin device is direct to sd (8:0).
 
I would think the difference is because DM issues (edited) copies of
incoming bios to the layers underneath, so changes don't propagate back
up.

Darrick: if you retest with a dm linear device, does the problem go
away?

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] dm: dm-cache fails to write the cache device in writethrough mode

[Alasdair G Kergon]

On Fri, Mar 22, 2013 at 01:11:51PM -0700, Darrick J. Wong wrote:
> The new writethrough strategy for dm-cache issues a bio to the origin device,
> remaps the bio to the cache device, and issues the bio to the cache device.
> However, the block layer modifies bi_sector and bi_size, so we need to 
> preserve
> these or else nothing gets written to the cache (bi_size == 0).  This fixes 
> the
> problem where someone writes a block through the cache, but a subsequent 
> reread
> (from the cache) returns old contents.

If this needs doing here, please use dm_bio_record() and dm_bio_restore().
E.g. Look at how dm-raid1.c does something similar.
 
Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] [PATCH] dm: dm-cache fails to write the cache device in writethrough mode

[Alasdair G Kergon]

On Fri, Mar 22, 2013 at 01:11:51PM -0700, Darrick J. Wong wrote:
 The new writethrough strategy for dm-cache issues a bio to the origin device,
 remaps the bio to the cache device, and issues the bio to the cache device.
 However, the block layer modifies bi_sector and bi_size, so we need to 
 preserve
 these or else nothing gets written to the cache (bi_size == 0).  This fixes 
 the
 problem where someone writes a block through the cache, but a subsequent 
 reread
 (from the cache) returns old contents.

If this needs doing here, please use dm_bio_record() and dm_bio_restore().
E.g. Look at how dm-raid1.c does something similar.
 
Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [dm-devel] dm: dm-cache fails to write the cache device in writethrough mode

[Alasdair G Kergon]

On Fri, Mar 22, 2013 at 11:27:16PM -0400, Mike Snitzer wrote:
 So the big difference is the thinp-test-suite uses intermediate linear
 DM layers above the slower sd device (spindle/data) -- whereas in your
 setup the origin device is direct to sd (8:0).
 
I would think the difference is because DM issues (edited) copies of
incoming bios to the layers underneath, so changes don't propagate back
up.

Darrick: if you retest with a dm linear device, does the problem go
away?

Alasdair

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper fixes for 3.9

[Alasdair G Kergon]

Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm tags/dm-3.9-fixes

to get the following device-mapper fixes for 3.9.

Thanks,
Alasdair


A set of device-mapper fixes for 3.9.

Fix reported data loss with discards and thin snapshots; avoid a
deadlock observed in dm verity; fix a race in the new dm cache code
along with some other minor bugs; store the cache policy version on disk
to make the stored hints format future-proof.


Heinz Mauelshagen (2):
  dm cache: detect cache_create failure
  dm cache: avoid calling policy destructor twice on error

Joe Thornber (5):
  dm thin: fix discard corruption
  dm thin: fix non power of two discard granularity calc
  dm cache: avoid 64 bit division on 32 bit
  dm cache: metadata clear dirty bits on clean shutdown
  dm cache: fix race in writethrough implementation

Mike Snitzer (2):
  dm cache: policy change version from string to integer set
  dm cache: policy ignore hints if generated by different version

Mikulas Patocka (1):
  dm verity: avoid deadlock

 drivers/md/dm-bufio.c|2 +
 drivers/md/dm-cache-metadata.c   |   64 --
 drivers/md/dm-cache-metadata.h   |2 +-
 drivers/md/dm-cache-policy-cleaner.c |7 +-
 drivers/md/dm-cache-policy-internal.h|2 +
 drivers/md/dm-cache-policy-mq.c  |8 +-
 drivers/md/dm-cache-policy.c |8 ++
 drivers/md/dm-cache-policy.h |2 +
 drivers/md/dm-cache-target.c |  169 +-
 drivers/md/dm-thin.c |   11 +-
 drivers/md/dm-verity.c   |   39 +-
 drivers/md/persistent-data/dm-btree-remove.c |   46 +++
 12 files changed, 252 insertions(+), 108 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[git pull] device-mapper fixes for 3.9

[Alasdair G Kergon]

Please pull from:

  git://git.kernel.org/pub/scm/linux/kernel/git/agk/linux-dm tags/dm-3.9-fixes

to get the following device-mapper fixes for 3.9.

Thanks,
Alasdair


A set of device-mapper fixes for 3.9.

Fix reported data loss with discards and thin snapshots; avoid a
deadlock observed in dm verity; fix a race in the new dm cache code
along with some other minor bugs; store the cache policy version on disk
to make the stored hints format future-proof.


Heinz Mauelshagen (2):
  dm cache: detect cache_create failure
  dm cache: avoid calling policy destructor twice on error

Joe Thornber (5):
  dm thin: fix discard corruption
  dm thin: fix non power of two discard granularity calc
  dm cache: avoid 64 bit division on 32 bit
  dm cache: metadata clear dirty bits on clean shutdown
  dm cache: fix race in writethrough implementation

Mike Snitzer (2):
  dm cache: policy change version from string to integer set
  dm cache: policy ignore hints if generated by different version

Mikulas Patocka (1):
  dm verity: avoid deadlock

 drivers/md/dm-bufio.c|2 +
 drivers/md/dm-cache-metadata.c   |   64 --
 drivers/md/dm-cache-metadata.h   |2 +-
 drivers/md/dm-cache-policy-cleaner.c |7 +-
 drivers/md/dm-cache-policy-internal.h|2 +
 drivers/md/dm-cache-policy-mq.c  |8 +-
 drivers/md/dm-cache-policy.c |8 ++
 drivers/md/dm-cache-policy.h |2 +
 drivers/md/dm-cache-target.c |  169 +-
 drivers/md/dm-thin.c |   11 +-
 drivers/md/dm-verity.c   |   39 +-
 drivers/md/persistent-data/dm-btree-remove.c |   46 +++
 12 files changed, 252 insertions(+), 108 deletions(-)
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: linux-next: error importing the device-mapper quilt series

[Alasdair G Kergon]

On Fri, Mar 15, 2013 at 10:14:41AM +1100, Stephen Rothwell wrote:
> The patch in question has no commit message or headers at all ...
 
Now fixed.  It escaped before I'd finished it...

Alasdair

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/