Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
- Original Message -
> From: "Gianluca Cecchi"
> To: "Nir Soffer"
> Cc: "Dan Kenigsberg" , "users" , "Yeela
> Kaplan" , "Benjamin
> Marzinski"
> Sent: Monday, January 26, 2015 8:46:52 PM
> Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
>
> On Mon, Jan 26, 2015 at 1:37 PM, Nir Soffer wrote:
>
> >
> > > > Any suggestion appreciated.
> > > >
> > > > Current multipath.conf (where I also commented out the getuid_callout
> > that
> > > > is not used anymore):
> > > >
> > > > [root@tekkaman setup]# cat /etc/multipath.conf
> > > > # RHEV REVISION 1.1
> > > >
> > > > blacklist {
> > > > devnode "^(sda|sdb)[0-9]*"
> > > > }
> >
> >
> > I think what happened is:
> >
> > 1. 3.5.1 had new multipath version
> >
>
> what do you mean with "new multipath version"?
I mean new multipath.conf version:
# RHEV REVISION 1.1
When vdsm finds that its current configuration version is different from current
multipath.conf version, it replaces the current multipath.conf with a new
version.
> I currently have device-mapper-multipath-0.4.9-56.fc20.x86_64
> The system came from f18 to f19 and then to f20 via fedup in both cases
> In my yum.log files I see this about January 2013 when I was in Fedora 18
> Jan 07 00:11:44 Installed: device-mapper-multipath-0.4.9-36.fc18.x86_64
>
> I then upgraded to f19 on 24th November 2013 and device-mapper-multipath
> was the one pushed during the fedup update:
> device-mapper-multipath-libs-0.4.9-51.fc19.x86_64
>
> Then on 12th of November 2014 I passed form f19 to f20 and fedup pushed
> device-mapper-multipath-0.4.9-56.fc20.x86_64
> that is my current version
> At that time I also passed from oVirt 3.4.4 to 3.5.0.
> And I didn't register any problem with my internal disk
>
> It was sufficient to keep inside
> blacklist {
>devnode "sd[a-b]"
> }
>
> At the head of the file I only had:
> # RHEV REVISION 1.0
This is why vdsm replaced the file.
>
> No reference about
> # RHEV PRIVATE
>
> But right now that I'm writing I notice that my rule for blacklist after
> migration to 3.5.1 was
>
> devnode "^(sda|sdb)[0-9]*"
>
> probably blacklists only partitions and not the full disks ;-)
> So I'm going to check with the old blacklist option and/or the PRIVATE
> label as suggested by Dan...
>
> Probably passing from RHEV REVISION 1.0 to 1.1 the original blacklist part
> was thrown away...
>
> 2. So vdsm upgraded the local file
> > 3. blacklist above was removed
> >(it should exists in /etc/multipath.bak)
> >
> >
> It seems it didn't generate any multipath.conf.bak ...
>
In 3.5.1 it should rotate multipath.conf, saving multipath.conf.1 ...
If you don't find the backup file after multipath.conf was updated, this is a
bug.
Can you open a a bug about it?
Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
On Mon, Jan 26, 2015 at 1:37 PM, Nir Soffer wrote:
>
> > > Any suggestion appreciated.
> > >
> > > Current multipath.conf (where I also commented out the getuid_callout
> that
> > > is not used anymore):
> > >
> > > [root@tekkaman setup]# cat /etc/multipath.conf
> > > # RHEV REVISION 1.1
> > >
> > > blacklist {
> > > devnode "^(sda|sdb)[0-9]*"
> > > }
>
>
> I think what happened is:
>
> 1. 3.5.1 had new multipath version
>
what do you mean with "new multipath version"?
I currently have device-mapper-multipath-0.4.9-56.fc20.x86_64
The system came from f18 to f19 and then to f20 via fedup in both cases
In my yum.log files I see this about January 2013 when I was in Fedora 18
Jan 07 00:11:44 Installed: device-mapper-multipath-0.4.9-36.fc18.x86_64
I then upgraded to f19 on 24th November 2013 and device-mapper-multipath
was the one pushed during the fedup update:
device-mapper-multipath-libs-0.4.9-51.fc19.x86_64
Then on 12th of November 2014 I passed form f19 to f20 and fedup pushed
device-mapper-multipath-0.4.9-56.fc20.x86_64
that is my current version
At that time I also passed from oVirt 3.4.4 to 3.5.0.
And I didn't register any problem with my internal disk
It was sufficient to keep inside
blacklist {
devnode "sd[a-b]"
}
At the head of the file I only had:
# RHEV REVISION 1.0
No reference about
# RHEV PRIVATE
But right now that I'm writing I notice that my rule for blacklist after
migration to 3.5.1 was
devnode "^(sda|sdb)[0-9]*"
probably blacklists only partitions and not the full disks ;-)
So I'm going to check with the old blacklist option and/or the PRIVATE
label as suggested by Dan...
Probably passing from RHEV REVISION 1.0 to 1.1 the original blacklist part
was thrown away...
2. So vdsm upgraded the local file
> 3. blacklist above was removed
>(it should exists in /etc/multipath.bak)
>
>
It seems it didn't generate any multipath.conf.bak ...
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
On Mon, Jan 26, 2015 at 10:27:23AM -0500, Fabian Deutsch wrote:
>
>
> - Original Message -
> > - Original Message -
> > > From: "Dan Kenigsberg"
> > > To: "Gianluca Cecchi" , nsof...@redhat.com
> > > Cc: "users" , ykap...@redhat.com
> > > Sent: Monday, January 26, 2015 2:09:23 PM
> > > Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
> > >
> > > On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote:
> > > > Hello,
> > > > on my all-in-one installation @home I had 3.5.0 with F20.
> > > > Today I updated to 3.5.1.
> > > >
> > > > it seems it modified /etc/multipath.conf preventing me from using my
> > > > second
> > > > disk at all...
> > > >
> > > > My system has internal ssd disk (sda) for OS and one local storage
> > > > domain
> > > > and another disk (sdb) with some partitions (on one of them there is
> > > > also
> > > > another local storage domain).
> > > >
> > > > At reboot I was put in emergency boot because partitions at sdb disk
> > > > could
> > > > not be mounted (they were busy).
> > > > it took me some time to understand that the problem was due to sdb gone
> > > > managed as multipath device and so busy for partitions to be mounted.
> > > >
> > > > Here you can find how multipath became after update and reboot
> > > > https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharing
> > > >
> > > > No device-mapper-multipath update in yum.log
> > > >
> > > > Also it seems that after changing it, it was then reverted at boot again
> > > > (I
> > > > don't know if the responsible was initrd/dracut or vdsmd) so in the mean
> > > > time the only thing I could do was to make the file immutable with
> > > >
> > > > chattr +i /etc/multipath.conf
> > >
> > > The "supported" method of achieving this is to place "# RHEV PRIVATE" in
> > > the second line of your hand-modified multipath.conf
> > >
> > > I do not understand why this has happened only after upgrade to 3.5.1 -
> > > 3.5.0's should have reverted you multipath.conf just as well during each
> > > vdsm startup.
> > >
> > > The good thing is that this annoying behavior has been dropped from the
> > > master branch, so that 3.6 is not going to have it. Vdsm is not to mess
> > > with other services config file while it is running. The logic moved to
> > > `vdsm-tool configure`
> > >
> > > >
> > > > and so I was able to reboot and verify that my partitions on sdb were ok
> > > > and I was able to mount them (for safe I also ran an fsck against them)
> > > >
> > > > Update ran around 19:20 and finished at 19:34
> > > > here the log in gzip format
> > > > https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharing
> > > >
> > > > Reboot was done around 21:10-21:14
> > > >
> > > > Here my /var/log/messages in gzip format, where you can see latest days.
> > > > https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharing
> > > >
> > > >
> > > > Any suggestion appreciated.
> > > >
> > > > Current multipath.conf (where I also commented out the getuid_callout
> > > > that
> > > > is not used anymore):
> > > >
> > > > [root@tekkaman setup]# cat /etc/multipath.conf
> > > > # RHEV REVISION 1.1
> > > >
> > > > blacklist {
> > > > devnode "^(sda|sdb)[0-9]*"
> > > > }
> >
> >
> > I think what happened is:
> >
> > 1. 3.5.1 had new multipath version
> > 2. So vdsm upgraded the local file
> > 3. blacklist above was removed
> >(it should exists in /etc/multipath.bak)
> >
> > To prevent local changes, you have to mark the file as private
> > as Dan suggests.
> >
> > Seems to be related to the find_multiapth = "yes" bug:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1173290
>
> The symptoms above sound exactly liek this issue.
> When find_multipahts is no (the default when the directive is not present)
> I sthat all non-blacklisted devices are tried to get claimed, and this
> happened a
Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
- Original Message -
> - Original Message -
> > From: "Dan Kenigsberg"
> > To: "Gianluca Cecchi" , nsof...@redhat.com
> > Cc: "users" , ykap...@redhat.com
> > Sent: Monday, January 26, 2015 2:09:23 PM
> > Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
> >
> > On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote:
> > > Hello,
> > > on my all-in-one installation @home I had 3.5.0 with F20.
> > > Today I updated to 3.5.1.
> > >
> > > it seems it modified /etc/multipath.conf preventing me from using my
> > > second
> > > disk at all...
> > >
> > > My system has internal ssd disk (sda) for OS and one local storage domain
> > > and another disk (sdb) with some partitions (on one of them there is also
> > > another local storage domain).
> > >
> > > At reboot I was put in emergency boot because partitions at sdb disk
> > > could
> > > not be mounted (they were busy).
> > > it took me some time to understand that the problem was due to sdb gone
> > > managed as multipath device and so busy for partitions to be mounted.
> > >
> > > Here you can find how multipath became after update and reboot
> > > https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharing
> > >
> > > No device-mapper-multipath update in yum.log
> > >
> > > Also it seems that after changing it, it was then reverted at boot again
> > > (I
> > > don't know if the responsible was initrd/dracut or vdsmd) so in the mean
> > > time the only thing I could do was to make the file immutable with
> > >
> > > chattr +i /etc/multipath.conf
> >
> > The "supported" method of achieving this is to place "# RHEV PRIVATE" in
> > the second line of your hand-modified multipath.conf
> >
> > I do not understand why this has happened only after upgrade to 3.5.1 -
> > 3.5.0's should have reverted you multipath.conf just as well during each
> > vdsm startup.
> >
> > The good thing is that this annoying behavior has been dropped from the
> > master branch, so that 3.6 is not going to have it. Vdsm is not to mess
> > with other services config file while it is running. The logic moved to
> > `vdsm-tool configure`
> >
> > >
> > > and so I was able to reboot and verify that my partitions on sdb were ok
> > > and I was able to mount them (for safe I also ran an fsck against them)
> > >
> > > Update ran around 19:20 and finished at 19:34
> > > here the log in gzip format
> > > https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharing
> > >
> > > Reboot was done around 21:10-21:14
> > >
> > > Here my /var/log/messages in gzip format, where you can see latest days.
> > > https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharing
> > >
> > >
> > > Any suggestion appreciated.
> > >
> > > Current multipath.conf (where I also commented out the getuid_callout
> > > that
> > > is not used anymore):
> > >
> > > [root@tekkaman setup]# cat /etc/multipath.conf
> > > # RHEV REVISION 1.1
> > >
> > > blacklist {
> > > devnode "^(sda|sdb)[0-9]*"
> > > }
>
>
> I think what happened is:
>
> 1. 3.5.1 had new multipath version
> 2. So vdsm upgraded the local file
> 3. blacklist above was removed
>(it should exists in /etc/multipath.bak)
>
> To prevent local changes, you have to mark the file as private
> as Dan suggests.
>
> Seems to be related to the find_multiapth = "yes" bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1173290
The symptoms above sound exactly liek this issue.
When find_multipahts is no (the default when the directive is not present)
I sthat all non-blacklisted devices are tried to get claimed, and this happened
above.
Blacklisting the devices works, or adding "find_mutlipaths yes" should also
work, because
in that case only device which have more than one path (or are explicitly
named) will be
claimed by multipath.
My 2ct.
- fabian
> Ben, can you confirm that this is the same issue?
>
> > >
> > > defaults {
> > > polling_interval5
> > > #getuid_callout "/usr/lib/udev/scsi_id --whitelisted
> > > --replace-whitespace --device=/dev/%n"
> > > no_path_retry fail
> > > user_friendly_names no
> > > flush_on_last_del yes
> > > fast_io_fail_tmo5
> > > dev_loss_tmo30
> > > max_fds 4096
> > > }
> >
>
> Regards,
> Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
- Original Message -
> From: "Dan Kenigsberg"
> To: "Gianluca Cecchi" , nsof...@redhat.com
> Cc: "users" , ykap...@redhat.com
> Sent: Monday, January 26, 2015 2:09:23 PM
> Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
>
> On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote:
> > Hello,
> > on my all-in-one installation @home I had 3.5.0 with F20.
> > Today I updated to 3.5.1.
> >
> > it seems it modified /etc/multipath.conf preventing me from using my second
> > disk at all...
> >
> > My system has internal ssd disk (sda) for OS and one local storage domain
> > and another disk (sdb) with some partitions (on one of them there is also
> > another local storage domain).
> >
> > At reboot I was put in emergency boot because partitions at sdb disk could
> > not be mounted (they were busy).
> > it took me some time to understand that the problem was due to sdb gone
> > managed as multipath device and so busy for partitions to be mounted.
> >
> > Here you can find how multipath became after update and reboot
> > https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharing
> >
> > No device-mapper-multipath update in yum.log
> >
> > Also it seems that after changing it, it was then reverted at boot again (I
> > don't know if the responsible was initrd/dracut or vdsmd) so in the mean
> > time the only thing I could do was to make the file immutable with
> >
> > chattr +i /etc/multipath.conf
>
> The "supported" method of achieving this is to place "# RHEV PRIVATE" in
> the second line of your hand-modified multipath.conf
>
> I do not understand why this has happened only after upgrade to 3.5.1 -
> 3.5.0's should have reverted you multipath.conf just as well during each
> vdsm startup.
>
> The good thing is that this annoying behavior has been dropped from the
> master branch, so that 3.6 is not going to have it. Vdsm is not to mess
> with other services config file while it is running. The logic moved to
> `vdsm-tool configure`
>
> >
> > and so I was able to reboot and verify that my partitions on sdb were ok
> > and I was able to mount them (for safe I also ran an fsck against them)
> >
> > Update ran around 19:20 and finished at 19:34
> > here the log in gzip format
> > https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharing
> >
> > Reboot was done around 21:10-21:14
> >
> > Here my /var/log/messages in gzip format, where you can see latest days.
> > https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharing
> >
> >
> > Any suggestion appreciated.
> >
> > Current multipath.conf (where I also commented out the getuid_callout that
> > is not used anymore):
> >
> > [root@tekkaman setup]# cat /etc/multipath.conf
> > # RHEV REVISION 1.1
> >
> > blacklist {
> > devnode "^(sda|sdb)[0-9]*"
> > }
I think what happened is:
1. 3.5.1 had new multipath version
2. So vdsm upgraded the local file
3. blacklist above was removed
(it should exists in /etc/multipath.bak)
To prevent local changes, you have to mark the file as private
as Dan suggests.
Seems to be related to the find_multiapth = "yes" bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1173290
Ben, can you confirm that this is the same issue?
> >
> > defaults {
> > polling_interval5
> > #getuid_callout "/usr/lib/udev/scsi_id --whitelisted
> > --replace-whitespace --device=/dev/%n"
> > no_path_retry fail
> > user_friendly_names no
> > flush_on_last_del yes
> > fast_io_fail_tmo5
> > dev_loss_tmo30
> > max_fds 4096
> > }
>
Regards,
Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote:
> Hello,
> on my all-in-one installation @home I had 3.5.0 with F20.
> Today I updated to 3.5.1.
>
> it seems it modified /etc/multipath.conf preventing me from using my second
> disk at all...
>
> My system has internal ssd disk (sda) for OS and one local storage domain
> and another disk (sdb) with some partitions (on one of them there is also
> another local storage domain).
>
> At reboot I was put in emergency boot because partitions at sdb disk could
> not be mounted (they were busy).
> it took me some time to understand that the problem was due to sdb gone
> managed as multipath device and so busy for partitions to be mounted.
>
> Here you can find how multipath became after update and reboot
> https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharing
>
> No device-mapper-multipath update in yum.log
>
> Also it seems that after changing it, it was then reverted at boot again (I
> don't know if the responsible was initrd/dracut or vdsmd) so in the mean
> time the only thing I could do was to make the file immutable with
>
> chattr +i /etc/multipath.conf
The "supported" method of achieving this is to place "# RHEV PRIVATE" in
the second line of your hand-modified multipath.conf
I do not understand why this has happened only after upgrade to 3.5.1 -
3.5.0's should have reverted you multipath.conf just as well during each
vdsm startup.
The good thing is that this annoying behavior has been dropped from the
master branch, so that 3.6 is not going to have it. Vdsm is not to mess
with other services config file while it is running. The logic moved to
`vdsm-tool configure`
>
> and so I was able to reboot and verify that my partitions on sdb were ok
> and I was able to mount them (for safe I also ran an fsck against them)
>
> Update ran around 19:20 and finished at 19:34
> here the log in gzip format
> https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharing
>
> Reboot was done around 21:10-21:14
>
> Here my /var/log/messages in gzip format, where you can see latest days.
> https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharing
>
>
> Any suggestion appreciated.
>
> Current multipath.conf (where I also commented out the getuid_callout that
> is not used anymore):
>
> [root@tekkaman setup]# cat /etc/multipath.conf
> # RHEV REVISION 1.1
>
> blacklist {
> devnode "^(sda|sdb)[0-9]*"
> }
>
> defaults {
> polling_interval5
> #getuid_callout "/usr/lib/udev/scsi_id --whitelisted
> --replace-whitespace --device=/dev/%n"
> no_path_retry fail
> user_friendly_names no
> flush_on_last_del yes
> fast_io_fail_tmo5
> dev_loss_tmo30
> max_fds 4096
> }
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Update to 3.5.1 scrambled multipath.conf?
Hello,
on my all-in-one installation @home I had 3.5.0 with F20.
Today I updated to 3.5.1.
it seems it modified /etc/multipath.conf preventing me from using my second
disk at all...
My system has internal ssd disk (sda) for OS and one local storage domain
and another disk (sdb) with some partitions (on one of them there is also
another local storage domain).
At reboot I was put in emergency boot because partitions at sdb disk could
not be mounted (they were busy).
it took me some time to understand that the problem was due to sdb gone
managed as multipath device and so busy for partitions to be mounted.
Here you can find how multipath became after update and reboot
https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharing
No device-mapper-multipath update in yum.log
Also it seems that after changing it, it was then reverted at boot again (I
don't know if the responsible was initrd/dracut or vdsmd) so in the mean
time the only thing I could do was to make the file immutable with
chattr +i /etc/multipath.conf
and so I was able to reboot and verify that my partitions on sdb were ok
and I was able to mount them (for safe I also ran an fsck against them)
Update ran around 19:20 and finished at 19:34
here the log in gzip format
https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharing
Reboot was done around 21:10-21:14
Here my /var/log/messages in gzip format, where you can see latest days.
https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharing
Any suggestion appreciated.
Current multipath.conf (where I also commented out the getuid_callout that
is not used anymore):
[root@tekkaman setup]# cat /etc/multipath.conf
# RHEV REVISION 1.1
blacklist {
devnode "^(sda|sdb)[0-9]*"
}
defaults {
polling_interval5
#getuid_callout "/usr/lib/udev/scsi_id --whitelisted
--replace-whitespace --device=/dev/%n"
no_path_retry fail
user_friendly_names no
flush_on_last_del yes
fast_io_fail_tmo5
dev_loss_tmo30
max_fds 4096
}
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
