Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

On Wed, Feb 22, 2017 at 8:12 AM, Przemek Klosowski
 wrote:
> On 02/22/2017 09:37 AM, Vratislav Podzimek wrote:
>
> On Tue, 2017-02-21 at 14:29 -0700, Chris Murphy wrote:
>
> Tested Fedora 24 and Fedora 23, there's no notification with either
> one of those. I have no idea if this is a VM thing. Or if it's a
> regression.
>
> Maybe the notification was depending on smartd rather than mdadm.
> Kinda need someone who knows more about how GNOME Shell handles faulty
> devices - how it's intended to work at least.
>
> SMART-signaled failures are propagated/signaled by udisksd.
>
> SMART is very good but not conclusive: we've seen RAID failures where disks
> just die or become unresponsive, without anything wrong indicated in SMART.
> Backblaze did a series of hard drive reliability reports that document that
> as well; they are a great read if you haven't seen them:
> https://www.backblaze.com/blog/?s=reliability

I'm pretty certain the notification I got from GNOME Shell was related
to the array itself. That suggests it's not a smartd initiated
notification. But at the moment I can't reproduce this by deleting an
array member device using sysfs. The array does go degraded, there are
numerous kernel messages as such, but no GNOME notification.

-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Przemek Klosowski]

On 02/22/2017 09:37 AM, Vratislav Podzimek wrote:

On Tue, 2017-02-21 at 14:29 -0700, Chris Murphy wrote:

Tested Fedora 24 and Fedora 23, there's no notification with either
one of those. I have no idea if this is a VM thing. Or if it's a
regression.

Maybe the notification was depending on smartd rather than mdadm.
Kinda need someone who knows more about how GNOME Shell handles faulty
devices - how it's intended to work at least.

SMART-signaled failures are propagated/signaled by udisksd.
SMART is very good but not conclusive: we've seen RAID failures where 
disks just die or become unresponsive, without anything wrong indicated 
in SMART. Backblaze did a series of hard drive reliability reports that 
document that as well; they are a great read if you haven't seen them: 
https://www.backblaze.com/blog/?s=reliability
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-02-21 at 12:32 -0700, Chris Murphy wrote:
> On Tue, Feb 21, 2017 at 6:34 AM, Vratislav Podzimek  
> wrote:
> > 
> > On Wed, 2017-02-01 at 12:25 -0700, Chris Murphy wrote:
> > > 
> > > Actually, I've got a concern about this feature.
> > > 
> > > I'm not sure what gnome-shell depends on for monitoring mdadm arrays,
> > > but I know it will put up a notification if an mdadm member becomes
> > > faulty; because I've seen this notification. Maybe it's getting this
> > > information from udisksd? In any case, I'm wondering whether the user
> > > will still be notified of device failures in gnome-shell with this
> > > change?
> > I'm not seeing any code taking care of MD RAID device monitoring in
> > udisks2. So it has to be somewhere else. Any ideas where to look? gvfs
> > maybe?
> 
> I'm not sure. With Fedora 25 this notification functionality is not
> happening. I'm not sure what or when the regression happens, or even
> if it's flawed testing. I did this in a virt-manager VM, and removing
> the virtual block device, booting I get no notification. If I boot
> with both devices and inside the VM I do:
> 
> # echo 1 > /sys/block/sdb/device/delete
> 
> I see complaints in kernel messages indicating the array is now
> degraded, and udisks does pick up this fact:
> 
> [   80.911812] localhost.localdomain udisksd[1393]: Unable to resolve
> /sys/devices/virtual/block/md127/md/dev-sdb2/block symlink
> [   80.912156] localhost.localdomain udisksd[1393]: Unable to resolve
> /sys/devices/virtual/block/md127/md/dev-sdb2/block symlink
> [   80.912284] localhost.localdomain udisksd[1393]: Unable to resolve
> /sys/devices/virtual/block/md126/md/dev-sdb1/block symlink
> [   80.912414] localhost.localdomain udisksd[1393]: Unable to resolve
> /sys/devices/virtual/block/md126/md/dev-sdb1/block symlink
Yeah, but that's not really figuring out that RAID is degraded. These
are just error messages generated by udisksd still thinking the RAID
is okay. So no signal emitted by udisksd here.

> 
> But still no notification in GNOME Shell. I don't know if there's
> something GNOME Shell is expecting to get additionally from udisksd,
> if so it might actually be a storaged regression. In that case the
> solution might need to be abstracted from GNOME Shell in storaged
> anyway, because why reinvent this particular wheel in the DE? We have
> degradedness notifications needed for mdadm, LVM, and Btrfs (and even
> ZFS if we're asking for unicorns).
Yeah, this needs some unified, generic solution. My suggestion was to use
journal for this with some special key-value pair indicating that the
message is actually a storage failure. That should be easy to do now that
we have structured logging. And it could benefit from all the existing
mechanisms developed for logging (outside of VM, log gathering,...).

> 
> I guess I need to try this with Fedora 24 and see if it might be a
> storaged regression.
That's very unlikely.

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-02-21 at 14:29 -0700, Chris Murphy wrote:
> Tested Fedora 24 and Fedora 23, there's no notification with either
> one of those. I have no idea if this is a VM thing. Or if it's a
> regression.
> 
> Maybe the notification was depending on smartd rather than mdadm.
> Kinda need someone who knows more about how GNOME Shell handles faulty
> devices - how it's intended to work at least.
SMART-signaled failures are propagated/signaled by udisksd.

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-02-21 at 07:52 -0600, Chris Adams wrote:
> Once upon a time, Vratislav Podzimek  said:
> > 
> > On Tue, 2017-02-07 at 14:50 -0700, Chris Murphy wrote:
> > > 
> > > My opinion of the change is limited to whether gnome-shell will still
> > > notify the user of device failures; if not, or if unknown then I think
> > > the change should be rejected as lack of device failure notifications
> > > in the DE is a regression that outweighs the benefit. It's
> > Any idea how this is working right now? AFAICT, there's no mechanism for
> > this other than 
> > a) running 'mdadm --monitor' or
> > b) watching journal for log messages of particular format
> > 
> > We could implement a similar thing for LVM RAID.
> 
> Well, all the world is not GNOME or any DE for that matter; mdadm has
> the mdmonitor service (enabled by default I believe) that will send
> email about MD RAID failures.  This is perfectly function for a server,
> so should have some functionally equivalent replacement for LVM RAID.
Sure thing, could you please file an RFE/bug report at

https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper

Thanks!

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

Tested Fedora 24 and Fedora 23, there's no notification with either
one of those. I have no idea if this is a VM thing. Or if it's a
regression.

Maybe the notification was depending on smartd rather than mdadm.
Kinda need someone who knows more about how GNOME Shell handles faulty
devices - how it's intended to work at least.


Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

On Tue, Feb 21, 2017 at 6:34 AM, Vratislav Podzimek  wrote:
> On Wed, 2017-02-01 at 12:25 -0700, Chris Murphy wrote:
>> Actually, I've got a concern about this feature.
>>
>> I'm not sure what gnome-shell depends on for monitoring mdadm arrays,
>> but I know it will put up a notification if an mdadm member becomes
>> faulty; because I've seen this notification. Maybe it's getting this
>> information from udisksd? In any case, I'm wondering whether the user
>> will still be notified of device failures in gnome-shell with this
>> change?
> I'm not seeing any code taking care of MD RAID device monitoring in
> udisks2. So it has to be somewhere else. Any ideas where to look? gvfs
> maybe?

I'm not sure. With Fedora 25 this notification functionality is not
happening. I'm not sure what or when the regression happens, or even
if it's flawed testing. I did this in a virt-manager VM, and removing
the virtual block device, booting I get no notification. If I boot
with both devices and inside the VM I do:

# echo 1 > /sys/block/sdb/device/delete

I see complaints in kernel messages indicating the array is now
degraded, and udisks does pick up this fact:

[   80.911812] localhost.localdomain udisksd[1393]: Unable to resolve
/sys/devices/virtual/block/md127/md/dev-sdb2/block symlink
[   80.912156] localhost.localdomain udisksd[1393]: Unable to resolve
/sys/devices/virtual/block/md127/md/dev-sdb2/block symlink
[   80.912284] localhost.localdomain udisksd[1393]: Unable to resolve
/sys/devices/virtual/block/md126/md/dev-sdb1/block symlink
[   80.912414] localhost.localdomain udisksd[1393]: Unable to resolve
/sys/devices/virtual/block/md126/md/dev-sdb1/block symlink

But still no notification in GNOME Shell. I don't know if there's
something GNOME Shell is expecting to get additionally from udisksd,
if so it might actually be a storaged regression. In that case the
solution might need to be abstracted from GNOME Shell in storaged
anyway, because why reinvent this particular wheel in the DE? We have
degradedness notifications needed for mdadm, LVM, and Btrfs (and even
ZFS if we're asking for unicorns).

I guess I need to try this with Fedora 24 and see if it might be a
storaged regression.



-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Adams]

Once upon a time, Vratislav Podzimek  said:
> On Tue, 2017-02-07 at 14:50 -0700, Chris Murphy wrote:
> > My opinion of the change is limited to whether gnome-shell will still
> > notify the user of device failures; if not, or if unknown then I think
> > the change should be rejected as lack of device failure notifications
> > in the DE is a regression that outweighs the benefit. It's
> Any idea how this is working right now? AFAICT, there's no mechanism for
> this other than 
> a) running 'mdadm --monitor' or
> b) watching journal for log messages of particular format
> 
> We could implement a similar thing for LVM RAID.

Well, all the world is not GNOME or any DE for that matter; mdadm has
the mdmonitor service (enabled by default I believe) that will send
email about MD RAID failures.  This is perfectly function for a server,
so should have some functionally equivalent replacement for LVM RAID.
-- 
Chris Adams 
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-01-31 at 14:28 -0500, Matthew Miller wrote:
> I would like the "User Experience" section to be fleshed-out a bit
> more. Currently it says "There should be no visible change for
> non-expert users. Expert users could make use of the new LVM RAID's
> features." 
> 
> I think, though, there's plenty of middle ground here: users who are
> not experts in LVM RAID yet would like to have data redundancy and need
> to manage that.
Sure, for them we need some nice overview of common 'mdadm' commands and
their 'lvm' equivalents. CC'ing LVM guys here.

> 
> The Documentation section mentions the installation guide, which is
> great, but what about documentation for replacing disks or adding to
> the set?
ditto

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-01-31 at 15:52 -0400, Robert Marcano wrote:
> On 01/31/2017 03:44 PM, Chris Adams wrote:
> > 
> > 
> > Also: does this apply to /boot partition RAID 1?  IIRC that didn't work
> > with LVM RAID at one time.
> > 
> 
> This is important, that mdraid is still available in anaconda and not 
> entirely replaced by LVM RAID. Not only /boot partition but UEFI boot 
> partition on mdraid 1
That's not going to happen. This change is only concerning the scenario
when user chooses to have LVM on top of RAID (IOW, sets a RAID level for
their VG).

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-01-31 at 13:44 -0600, Chris Adams wrote:
> Once upon a time, Matthew Miller  said:
> > 
> > I would like the "User Experience" section to be fleshed-out a bit
> > more. Currently it says "There should be no visible change for
> > non-expert users. Expert users could make use of the new LVM RAID's
> > features." 
> 
> At a minimum, it would be nice to have a list of common tasks,
> especially in a comparison format (e.g. "Replace a failed drive" with
> MD and LVM commands).  Maybe this already exists somewhere and could
> just be referenced.
I'm not aware of any such document. CC'ing LVM guys to correct me if
I'm wrong.


> 
> I think one thing that's a bit of an annoyance is that LVM RAID is done
> at the LV layer, so you have to remember to set it up for every LV you
> create, vs. MD RAID where the whole VG will have RAID (this can be
> useful in some situations, but probably not the common case).  This
> should be pointed out in the release notes IMHO.
It's a good idea to mention this in the release notes, thanks! However,
a linear LV can be simply converted to a RAID LV later so at least it's
not that fatal.

> 
> Also: does this apply to /boot partition RAID 1?  IIRC that didn't work
> with LVM RAID at one time.
/boot on LVM is not supported right now so this is not concerning /boot at
all.

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-01-31 at 07:05 -0800, Andrew Lutomirski wrote:
> 
> 
> On Jan 31, 2017 6:34 AM, "Chris Adams"  wrote:
> Once upon a time, Jan Kurik  said:
> > LVM RAID provides same functionality as MD
> > RAID (it shares the same kernel code) with better flexibility and
> > additional features expected in future.
> 
> How do LVM RAID volumes get tested?  There's a regular cron job for
> testing MD RAID volumes, but I'm not aware of something like that for
> LVM RAID.
> 
> For that matter, how do you administer it?  For MD RAID, there's mdadm.  Does 
> LVM RAID have a fully-featured equivalent?
Yes, it's administered with the LVM commands as described in
'man lvmraid'. I understand that some guide comparing the common
'mdadm' commands with 'lvm' commands could be useful though
(CC'ing LVM guys).

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Wed, 2017-02-01 at 12:25 -0700, Chris Murphy wrote:
> Actually, I've got a concern about this feature.
> 
> I'm not sure what gnome-shell depends on for monitoring mdadm arrays,
> but I know it will put up a notification if an mdadm member becomes
> faulty; because I've seen this notification. Maybe it's getting this
> information from udisksd? In any case, I'm wondering whether the user
> will still be notified of device failures in gnome-shell with this
> change?
I'm not seeing any code taking care of MD RAID device monitoring in
udisks2. So it has to be somewhere else. Any ideas where to look? gvfs
maybe?

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-01-31 at 08:33 -0600, Chris Adams wrote:
> Once upon a time, Jan Kurik  said:
> > 
> > LVM RAID provides same functionality as MD
> > RAID (it shares the same kernel code) with better flexibility and
> > additional features expected in future.
> 
> How do LVM RAID volumes get tested?  There's a regular cron job for
> testing MD RAID volumes, but I'm not aware of something like that for
> LVM RAID.
Already answered by Chris Murphy I believe. There's no cron job for
LVM RAID specifically right now, but it should be easy to add it.

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Vratislav Podzimek]

On Tue, 2017-02-07 at 14:50 -0700, Chris Murphy wrote:
> My opinion of the change is limited to whether gnome-shell will still
> notify the user of device failures; if not, or if unknown then I think
> the change should be rejected as lack of device failure notifications
> in the DE is a regression that outweighs the benefit. It's
Any idea how this is working right now? AFAICT, there's no mechanism for
this other than 
a) running 'mdadm --monitor' or
b) watching journal for log messages of particular format

We could implement a similar thing for LVM RAID.

-- 
Vratislav Podzimek

Anaconda Rider | Red Hat, Inc. | Brno - Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

My opinion of the change is limited to whether gnome-shell will still
notify the user of device failures; if not, or if unknown then I think
the change should be rejected as lack of device failure notifications
in the DE is a regression that outweighs the benefit. It's


Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Kevin Fenzi]

On Tue, 7 Feb 2017 12:27:20 -0800
Samuel Sieb  wrote:

> On 01/31/2017 12:02 PM, Chris Murphy wrote:
> > On Tue, Jan 31, 2017 at 7:33 AM, Chris Adams 
> > wrote:  
> >> How do LVM RAID volumes get tested?  There's a regular cron job for
> >> testing MD RAID volumes, but I'm not aware of something like that
> >> for LVM RAID.  
> >
> > I'm not aware of an upstream cron job for this; nor one in Fedora.
> >
> > 'echo check > /sys/block/mdX/md/sync_action' works for either mdadm
> > or LVM RAID, as it's kernel code doing the scrub; but LVM does have
> > a command for it 'lvchange --syncaction {check|repair) vg/raid_lv'
> >  
> There is something in Fedora that does this.  I don't know whether
> it's automatic in the kernel or there's a cron job.  But my md RAID
> arrays get scanned every once in a while and I didn't do anything to
> set that up.

% rpm -qf /etc/cron.d/raid-check 
mdadm-3.4-2.fc25.x86_64
% cat /etc/cron.d/raid-check 
# Run system wide raid-check once a week on Sunday at 1am by default
0 1 * * Sun root /usr/sbin/raid-check

/etc/sysconfig/raid-check has the config. I think by default it's not
set to do anything, but perhaps something updates this on install. 

kevin


pgpT2YAkSHPDd.pgp
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Josh Stone]

On 02/07/2017 12:27 PM, Samuel Sieb wrote:
> On 01/31/2017 12:02 PM, Chris Murphy wrote:
>> On Tue, Jan 31, 2017 at 7:33 AM, Chris Adams  wrote:
>>> How do LVM RAID volumes get tested?  There's a regular cron job for
>>> testing MD RAID volumes, but I'm not aware of something like that for
>>> LVM RAID.
>>
>> I'm not aware of an upstream cron job for this; nor one in Fedora.
>>
>> 'echo check > /sys/block/mdX/md/sync_action' works for either mdadm or
>> LVM RAID, as it's kernel code doing the scrub; but LVM does have a
>> command for it 'lvchange --syncaction {check|repair) vg/raid_lv'
>>
> There is something in Fedora that does this.  I don't know whether it's 
> automatic in the kernel or there's a cron job.  But my md RAID arrays 
> get scanned every once in a while and I didn't do anything to set that up.

It's /etc/cron.d/raid-check from mdadm.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Samuel Sieb]

On 01/31/2017 12:02 PM, Chris Murphy wrote:

On Tue, Jan 31, 2017 at 7:33 AM, Chris Adams  wrote:

How do LVM RAID volumes get tested?  There's a regular cron job for
testing MD RAID volumes, but I'm not aware of something like that for
LVM RAID.


I'm not aware of an upstream cron job for this; nor one in Fedora.

'echo check > /sys/block/mdX/md/sync_action' works for either mdadm or
LVM RAID, as it's kernel code doing the scrub; but LVM does have a
command for it 'lvchange --syncaction {check|repair) vg/raid_lv'

There is something in Fedora that does this.  I don't know whether it's 
automatic in the kernel or there's a cron job.  But my md RAID arrays 
get scanned every once in a while and I didn't do anything to set that up.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Samuel Sieb]

On 02/06/2017 03:16 AM, den...@ausil.us wrote:

Fedup has not existed for a few releases now. The upgrades are handled by dnf 
alone.


Yes, but the command does still exist as a functional alias.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[dennis]

On 2 February 2017 9:06:32 pm GMT+00:00, Chris Murphy  
wrote:
>On Thu, Feb 2, 2017 at 3:38 AM, David Woodhouse 
>wrote:
>
>> I'm not sure the upgrade method matters, does it? In both cases I
>think
>> it was changes to dracut and the way raid was assembled (perhaps
>moving
>> from automatically in the kernel to doing it in userspace, or vice
>> versa).
>
>The upgrade method does matter because fedup had its own dracut
>variant. Systemd offline updates (used also by upgrades) uses the
>existing initramfs prior to the upgrade and a special, minimalist,
>boot target. So if the system boots normally before the update or
>upgrade, it should boot and assemble fine for the offline update. It
>could fail following a successful upgrade however - if there's some
>new previously undiscovered bug.

Fedup has not existed for a few releases now. The upgrades are handled by dnf 
alone.

Dennis
>
>>
>>> > So let's please ensure that we have proper
>>> > test coverage for existing systems.
>>>
>>> Please describe your proposal for ensuring proper test coverage, in
>>> particular if you personally aren't able to test what is by
>definition
>>> a custom layout?
>>
>> Nono, this *isn't* a custom layout. It's a fairly standard RAID
>setup.
>
>All RAID setups are custom layouts insofar as they're only created
>with custom partitioning. There is only one default layout used by the
>installer.
>
>
>> But if we change the defaults, then I suppose that retrospetively
>> *makes* it a "custom layout"... at least in the sense that we can
>> reasonably expect it to keep breaking every release or two :(
>
>It suggests it's not getting enough testing and asking someone else to
>test it probably will have no effect. Again, if there's a particular
>layout you want to make sure is working, you need to test it, or maybe
>write up an openQA test for it, so it can get automatically tested by
>a bot.
>
>
>-- 
>Chris Murphy
>___
>devel mailing list -- devel@lists.fedoraproject.org
>To unsubscribe send an email to devel-le...@lists.fedoraproject.org

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

On Thu, Feb 2, 2017 at 3:38 AM, David Woodhouse  wrote:

> I'm not sure the upgrade method matters, does it? In both cases I think
> it was changes to dracut and the way raid was assembled (perhaps moving
> from automatically in the kernel to doing it in userspace, or vice
> versa).

The upgrade method does matter because fedup had its own dracut
variant. Systemd offline updates (used also by upgrades) uses the
existing initramfs prior to the upgrade and a special, minimalist,
boot target. So if the system boots normally before the update or
upgrade, it should boot and assemble fine for the offline update. It
could fail following a successful upgrade however - if there's some
new previously undiscovered bug.


>
>> > So let's please ensure that we have proper
>> > test coverage for existing systems.
>>
>> Please describe your proposal for ensuring proper test coverage, in
>> particular if you personally aren't able to test what is by definition
>> a custom layout?
>
> Nono, this *isn't* a custom layout. It's a fairly standard RAID setup.

All RAID setups are custom layouts insofar as they're only created
with custom partitioning. There is only one default layout used by the
installer.


> But if we change the defaults, then I suppose that retrospetively
> *makes* it a "custom layout"... at least in the sense that we can
> reasonably expect it to keep breaking every release or two :(

It suggests it's not getting enough testing and asking someone else to
test it probably will have no effect. Again, if there's a particular
layout you want to make sure is working, you need to test it, or maybe
write up an openQA test for it, so it can get automatically tested by
a bot.


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[David Woodhouse]

Please don't drop me from Cc when replying. I know the list has a
misguided setup, but mailers can be configured to ignore that. Thanks.

http://david.woodhou.se/reply-to-list.html

On Wed, 2017-02-01 at 12:13 -0700, Chris Murphy wrote:
> On Wed, Feb 1, 2017 at 4:55 AM, David Woodhouse  wrote:
> > My server at home broke on upgrading to Fedora 22 (#1201962), and also
> > on upgrading to Fedora 20 before that (IIRC).
>
> That's a while ago, the system upgrade method is different now. At
> least on workstation it's using systemd offline update to do the major
> version upgrade, same as minor updates. So if it's able to do minor
> updates without breaking, it should be able to do the major one
> successfully. Whether there may be a bug that prevents a successfully
> upgraded system from booting - well that's what testing is for.

I'm not sure the upgrade method matters, does it? In both cases I think
it was changes to dracut and the way raid was assembled (perhaps moving
from automatically in the kernel to doing it in userspace, or vice
versa).

> > So let's please ensure that we have proper
> > test coverage for existing systems.
>
> Please describe your proposal for ensuring proper test coverage, in
> particular if you personally aren't able to test what is by definition
> a custom layout?

Nono, this *isn't* a custom layout. It's a fairly standard RAID setup.
But if we change the defaults, then I suppose that retrospectively
*makes* it a "custom layout"... at least in the sense that we can
reasonably expect it to keep breaking every release or two :(


smime.p7s
Description: S/MIME cryptographic signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

Actually, I've got a concern about this feature.

I'm not sure what gnome-shell depends on for monitoring mdadm arrays,
but I know it will put up a notification if an mdadm member becomes
faulty; because I've seen this notification. Maybe it's getting this
information from udisksd? In any case, I'm wondering whether the user
will still be notified of device failures in gnome-shell with this
change?

Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

On Wed, Feb 1, 2017 at 4:55 AM, David Woodhouse  wrote:

>
> Please ensure upgrades of systems using MD RAID are properly tested.

Please help test upgrades if there's a layout you want to work. The
only way anything gets tested is if someone does the test.


> My server at home broke on upgrading to Fedora 22 (#1201962), and also
> on upgrading to Fedora 20 before that (IIRC).

That's a while ago, the system upgrade method is different now. At
least on workstation it's using systemd offline update to do the major
version upgrade, same as minor updates. So if it's able to do minor
updates without breaking, it should be able to do the major one
successfully. Whether there may be a bug that prevents a successfully
upgraded system from booting - well that's what testing is for.

>So let's please ensure that we have proper
> test coverage for existing systems.

Please describe your proposal for ensuring proper test coverage, in
particular if you personally aren't able to test what is by definition
a custom layout?


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[David Woodhouse]

On Tue, 2017-01-31 at 13:13 +0100, Jan Kurik wrote:
> = Proposed Self Contained Change: Anaconda LVM RAID =
> https://fedoraproject.org/wiki/Changes/AnacondaLVMRAID
> 
> Change owner(s):
> * Vratislav Podzimek (Anaconda/Blivet) 
> * Heinz Mauelshagen (LVM) 
> 
> Use LVM RAID instead of LVM of top of MD RAID in the Anaconda
> installer.
> 
> 
> == Detailed Description ==
> In the current situation when a user chooses LVM (or Thin LVM)
> partitioning in the Custom Spoke and then sets RAID level for the VG
> Anaconda (and Blivet) create an MD RAID device which is used as a PV
> for the VG. With this change we are going to use LVM RAID directly
> instead. That means that all the LVs in that VG will be RAID LVs with
> the specified RAID level. LVM RAID provides same functionality as MD
> RAID (it shares the same kernel code) with better flexibility and
> additional features expected in future.
> 
> 
> 
> == Scope ==
> * Proposal owners:
> -- Blivet developers: Support creation of LVM RAID in a similar way
> as
> LVM on top of MD RAID. (Creation of RAID LVs is already supported.)
> -- Anaconda developers: Use the new way to create LVM RAID instead of
> creating LVM on top of MD RAID.
> -- LVM developers: LVM RAID already has all features required by this
> change.
> 
> * Other developers:
> N/A (not a System Wide Change)
> 
> * Release engineering:

Please ensure upgrades of systems using MD RAID are properly tested.

My server at home broke on upgrading to Fedora 22 (#1201962), and also
on upgrading to Fedora 20 before that (IIRC). This implies that even
when MD RAID was still being used by default, upgrades weren't very
well-tested. With a move away from MD to LVM RAID, I'm concerned that
things will only get worse. So let's please ensure that we have proper
test coverage for existing systems.


smime.p7s
Description: S/MIME cryptographic signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Matthew Miller]

I would like the "User Experience" section to be fleshed-out a bit
more. Currently it says "There should be no visible change for
non-expert users. Expert users could make use of the new LVM RAID's
features." 

I think, though, there's plenty of middle ground here: users who are
not experts in LVM RAID yet would like to have data redundancy and need
to manage that.

The Documentation section mentions the installation guide, which is
great, but what about documentation for replacing disks or adding to
the set?

-- 
Matthew Miller

Fedora Project Leader
___
devel-announce mailing list -- devel-announce@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

On Tue, Jan 31, 2017 at 12:52 PM, Robert Marcano
 wrote:
> On 01/31/2017 03:44 PM, Chris Adams wrote:
>>
>>
>> Also: does this apply to /boot partition RAID 1?  IIRC that didn't work
>> with LVM RAID at one time.
>>
>
> This is important, that mdraid is still available in anaconda and not
> entirely replaced by LVM RAID.

> Not only /boot partition but UEFI boot
> partition on mdraid 1

Yeah the mdadm raid1 for EFI System partitions is broken 8 ways to
Sunday and is explicitly not recommended by upstream. It's an accepted
hideously bad hack only because the alternative is more work than
anyone is willing to volunteer time to do it correctly. The best
production way of doing this is firmware RAID (i.e. imsm metadata,
which mdadm supports and likewise uses md kernel support).


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Murphy]

On Tue, Jan 31, 2017 at 6:52 AM, Dominik 'Rathann' Mierzejewski
 wrote:

> I'd like to see (a link to) a more comprehensive discussion of the
> purported advantages of LVM RAID over LVM on MD RAID here.

If the user never interacts with the storage stack, it's a wash.

Otherwise, the advantage is RAID level is an LV attribute, set at the
time the LV is created, which means LV's can have different RAID
levels, and are resizeable within the VG they belong to. So a VG with
three disks can have a raid0 LV, raid1 LV, and raid5 LV - and they're
all resizable.

I think the resize benefit is minor because only Btrfs has online
shrink, ext4 can only shrink offline, and XFS doesn't support shrink.
To mitigate this means leaving some unused space in the VG (and on all
PVs).

As for drawbacks, as a practical matter very few people are familiar
with managing LVM RAIDs with LVM tools. While it uses md kernel code,
it uses LVM metadata, not mdadm metadata, so mdadm cannot be used at
all to manage them.


On Tue, Jan 31, 2017 at 7:33 AM, Chris Adams  wrote:
> How do LVM RAID volumes get tested?  There's a regular cron job for
> testing MD RAID volumes, but I'm not aware of something like that for
> LVM RAID.

I'm not aware of an upstream cron job for this; nor one in Fedora.

'echo check > /sys/block/mdX/md/sync_action' works for either mdadm or
LVM RAID, as it's kernel code doing the scrub; but LVM does have a
command for it 'lvchange --syncaction {check|repair) vg/raid_lv'


-- 
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[David Lehman]

On Tue, 2017-01-31 at 15:52 -0400, Robert Marcano wrote:
> On 01/31/2017 03:44 PM, Chris Adams wrote:
> > 
> > Also: does this apply to /boot partition RAID 1?  IIRC that didn't
> > work
> > with LVM RAID at one time.
> > 
> 
> This is important, that mdraid is still available in anaconda and
> not 
> entirely replaced by LVM RAID. Not only /boot partition but UEFI
> boot 
> partition on mdraid 1

My understanding is that this only applies to "LVM on RAID" layouts. If
you create /boot in anaconda and set the device type to "Software RAID"
it will be created and managed with mdadm. It's only if you set a RAID
level on an LVM volume group that you will be affected by this change.

Change owners/sponsors, please correct me if I'm wrong.

David

> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Robert Marcano]

On 01/31/2017 03:44 PM, Chris Adams wrote:


Also: does this apply to /boot partition RAID 1?  IIRC that didn't work
with LVM RAID at one time.



This is important, that mdraid is still available in anaconda and not 
entirely replaced by LVM RAID. Not only /boot partition but UEFI boot 
partition on mdraid 1

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Adams]

Once upon a time, Matthew Miller  said:
> I would like the "User Experience" section to be fleshed-out a bit
> more. Currently it says "There should be no visible change for
> non-expert users. Expert users could make use of the new LVM RAID's
> features." 

At a minimum, it would be nice to have a list of common tasks,
especially in a comparison format (e.g. "Replace a failed drive" with
MD and LVM commands).  Maybe this already exists somewhere and could
just be referenced.

I think one thing that's a bit of an annoyance is that LVM RAID is done
at the LV layer, so you have to remember to set it up for every LV you
create, vs. MD RAID where the whole VG will have RAID (this can be
useful in some situations, but probably not the common case).  This
should be pointed out in the release notes IMHO.

Also: does this apply to /boot partition RAID 1?  IIRC that didn't work
with LVM RAID at one time.

-- 
Chris Adams 
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Matthew Miller]

I would like the "User Experience" section to be fleshed-out a bit
more. Currently it says "There should be no visible change for
non-expert users. Expert users could make use of the new LVM RAID's
features." 

I think, though, there's plenty of middle ground here: users who are
not experts in LVM RAID yet would like to have data redundancy and need
to manage that.

The Documentation section mentions the installation guide, which is
great, but what about documentation for replacing disks or adding to
the set?

-- 
Matthew Miller

Fedora Project Leader
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Andrew Lutomirski]

On Jan 31, 2017 6:34 AM, "Chris Adams"  wrote:

Once upon a time, Jan Kurik  said:
> LVM RAID provides same functionality as MD
> RAID (it shares the same kernel code) with better flexibility and
> additional features expected in future.

How do LVM RAID volumes get tested?  There's a regular cron job for
testing MD RAID volumes, but I'm not aware of something like that for
LVM RAID.


For that matter, how do you administer it?  For MD RAID, there's mdadm.
Does LVM RAID have a fully-featured equivalent?
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Chris Adams]

Once upon a time, Jan Kurik  said:
> LVM RAID provides same functionality as MD
> RAID (it shares the same kernel code) with better flexibility and
> additional features expected in future.

How do LVM RAID volumes get tested?  There's a regular cron job for
testing MD RAID volumes, but I'm not aware of something like that for
LVM RAID.
-- 
Chris Adams 
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: F26 Self Contained Change: Anaconda LVM RAID

[Dominik 'Rathann' Mierzejewski]

On Tuesday, 31 January 2017 at 13:13, Jan Kurik wrote:
> = Proposed Self Contained Change: Anaconda LVM RAID =
> https://fedoraproject.org/wiki/Changes/AnacondaLVMRAID
> 
> Change owner(s):
> * Vratislav Podzimek (Anaconda/Blivet) 
> * Heinz Mauelshagen (LVM) 
> 
> Use LVM RAID instead of LVM of top of MD RAID in the Anaconda installer.
> 
> 
> == Detailed Description ==
> In the current situation when a user chooses LVM (or Thin LVM)
> partitioning in the Custom Spoke and then sets RAID level for the VG
> Anaconda (and Blivet) create an MD RAID device which is used as a PV
> for the VG. With this change we are going to use LVM RAID directly
> instead. That means that all the LVs in that VG will be RAID LVs with
> the specified RAID level. LVM RAID provides same functionality as MD
> RAID (it shares the same kernel code) with better flexibility and
> additional features expected in future.

I'd like to see (a link to) a more comprehensive discussion of the
purported advantages of LVM RAID over LVM on MD RAID here.

Regards,
Dominik
-- 
Fedora http://fedoraproject.org/wiki/User:Rathann
RPMFusion http://rpmfusion.org
"Faith manages."
-- Delenn to Lennier in Babylon 5:"Confessions and Lamentations"
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

F26 Self Contained Change: Anaconda LVM RAID

[Jan Kurik]

= Proposed Self Contained Change: Anaconda LVM RAID =
https://fedoraproject.org/wiki/Changes/AnacondaLVMRAID

Change owner(s):
* Vratislav Podzimek (Anaconda/Blivet) 
* Heinz Mauelshagen (LVM) 

Use LVM RAID instead of LVM of top of MD RAID in the Anaconda installer.


== Detailed Description ==
In the current situation when a user chooses LVM (or Thin LVM)
partitioning in the Custom Spoke and then sets RAID level for the VG
Anaconda (and Blivet) create an MD RAID device which is used as a PV
for the VG. With this change we are going to use LVM RAID directly
instead. That means that all the LVs in that VG will be RAID LVs with
the specified RAID level. LVM RAID provides same functionality as MD
RAID (it shares the same kernel code) with better flexibility and
additional features expected in future.



== Scope ==
* Proposal owners:
-- Blivet developers: Support creation of LVM RAID in a similar way as
LVM on top of MD RAID. (Creation of RAID LVs is already supported.)
-- Anaconda developers: Use the new way to create LVM RAID instead of
creating LVM on top of MD RAID.
-- LVM developers: LVM RAID already has all features required by this change.

* Other developers:
N/A (not a System Wide Change)

* Release engineering:

* List of deliverables:
N/A (not a System Wide Change)

* Policies and guidelines:

* Trademark approval:
-- 
Jan Kuřík
Platform & Fedora Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic
___
devel-announce mailing list -- devel-announce@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org

F26 Self Contained Change: Anaconda LVM RAID

[Jan Kurik]

= Proposed Self Contained Change: Anaconda LVM RAID =
https://fedoraproject.org/wiki/Changes/AnacondaLVMRAID

Change owner(s):
* Vratislav Podzimek (Anaconda/Blivet) 
* Heinz Mauelshagen (LVM) 

Use LVM RAID instead of LVM of top of MD RAID in the Anaconda installer.


== Detailed Description ==
In the current situation when a user chooses LVM (or Thin LVM)
partitioning in the Custom Spoke and then sets RAID level for the VG
Anaconda (and Blivet) create an MD RAID device which is used as a PV
for the VG. With this change we are going to use LVM RAID directly
instead. That means that all the LVs in that VG will be RAID LVs with
the specified RAID level. LVM RAID provides same functionality as MD
RAID (it shares the same kernel code) with better flexibility and
additional features expected in future.



== Scope ==
* Proposal owners:
-- Blivet developers: Support creation of LVM RAID in a similar way as
LVM on top of MD RAID. (Creation of RAID LVs is already supported.)
-- Anaconda developers: Use the new way to create LVM RAID instead of
creating LVM on top of MD RAID.
-- LVM developers: LVM RAID already has all features required by this change.

* Other developers:
N/A (not a System Wide Change)

* Release engineering:

* List of deliverables:
N/A (not a System Wide Change)

* Policies and guidelines:

* Trademark approval:
-- 
Jan Kuřík
Platform & Fedora Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org