Re: strange behavior on kernel update
Hi Konrad,
Konrad Rzeszutek schrieb:
> On Tue, Apr 28, 2009 at 04:00:08AM +0200, Maddin wrote:
>
>> Hi,
>>
>> i'm using centos with dm-multipath and iscsi as a database system. All
>> packages are normal centos packages from their mirrors.
>>
>> This night I was updating centos to the current release (5.3, previous
>> version was 5.2). So the first thing I have done was stopping the multipathd
>> and iscsi. After this I was doing the normal update stuff (yum
>> check-updates, yum clean all, yum update glibc\*, yum update). During the
>> last yum update the new kernel was installed (2.6.18-128.1.6.el5, previous
>> was 2.6.18-92.1.22.el5).
>>
>> During the installation process of the kernel, the yum process hangs.
>> Looking with ps for the hanging processes, I see that the kernel package was
>> building the initrd. After killing this process and repeating the yum update
>> process (this time only the kernel-package) about 5 times, the same strange
>> failure occurs (process hangs on building initrd). The sixth time I was
>> stracing the whole update process and see that the mkinitrd process does a
>> wait-syscall for sth.. After killing this process again I went down to the
>> server room to reboot the system and try another update process. I reboot
>> the systems, stops multipathd and iscsi and starting the update process. On
>> this update I could see that during the mkinitrd process, a kernel message
>> "device-mapper: failed path x:x:x:x" occurs. Hmm this is strange I thought,
>> because I've stopped multipathd and iscsi, so why means the kernel that a
>> path is failed? Once again killing the mkinitrd process and then starting
>>
>
> The device mapper (a kernel component) still has your multipath entries even
> thought you have
> killed multipathd and iscsi. It doesn't delete them, unless you
> specifically instructs the daemon (multipathd) to do so.
>
> From the standpoint of the kernel, the underlaying block devices got
> yanked out and the "brain" (multipathd) terminated. And any I/O that
> touches that multipath device (such as 'pvscan', 'lvscan' which mkinitrd
> uses to figure out the boot device) would cause the kernel to notice
> that the I/O's aren't going anywhere and mark them as failed.
>
>
>
Ok that sounds logical to me.
>> multipathd and iscsi I've tried another update process. And, yeha, it
>> works???
>>
>
> .. and depending on your multipath.conf file, the I/O can be queued forever
> (queue_if_no_path), or up to a certain time-limit ('no_path_retry'). Or
> error out immediately (if you don't have any of those options set). From
> your experiment it seems that those flags are set and the mkinitrd
> process is in the D state, waiting for the kernel to return a value
> after a read/write system call.
>
> If you had deleted the device mapper entries before terminating
> multipathd and iscsi this shouldn't have happend. Or if you had set the
> multipath configuration to error out immediately you wouldn't hit this.
>
>
I've set queue_if_no_path to all targets. I'll try to delete all path's
and then updating again.
So thanks in advance for your help.
>> I could reproduce this behavior on 2 machines, both centos with standard
>> packages.
>>
>
> Correclty so.
>
>> So can anybody comprehend this "failure" or any ideas?
>>
>
>
>> Cheers
>> Maddin
>>
>>
>>
>>
>>
>>
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
Re: strange behavior on kernel update
On Tue, Apr 28, 2009 at 04:00:08AM +0200, Maddin wrote:
>
> Hi,
>
> i'm using centos with dm-multipath and iscsi as a database system. All
> packages are normal centos packages from their mirrors.
>
> This night I was updating centos to the current release (5.3, previous
> version was 5.2). So the first thing I have done was stopping the multipathd
> and iscsi. After this I was doing the normal update stuff (yum
> check-updates, yum clean all, yum update glibc\*, yum update). During the
> last yum update the new kernel was installed (2.6.18-128.1.6.el5, previous
> was 2.6.18-92.1.22.el5).
>
> During the installation process of the kernel, the yum process hangs.
> Looking with ps for the hanging processes, I see that the kernel package was
> building the initrd. After killing this process and repeating the yum update
> process (this time only the kernel-package) about 5 times, the same strange
> failure occurs (process hangs on building initrd). The sixth time I was
> stracing the whole update process and see that the mkinitrd process does a
> wait-syscall for sth.. After killing this process again I went down to the
> server room to reboot the system and try another update process. I reboot
> the systems, stops multipathd and iscsi and starting the update process. On
> this update I could see that during the mkinitrd process, a kernel message
> "device-mapper: failed path x:x:x:x" occurs. Hmm this is strange I thought,
> because I've stopped multipathd and iscsi, so why means the kernel that a
> path is failed? Once again killing the mkinitrd process and then starting
The device mapper (a kernel component) still has your multipath entries even
thought you have
killed multipathd and iscsi. It doesn't delete them, unless you
specifically instructs the daemon (multipathd) to do so.
>From the standpoint of the kernel, the underlaying block devices got
yanked out and the "brain" (multipathd) terminated. And any I/O that
touches that multipath device (such as 'pvscan', 'lvscan' which mkinitrd
uses to figure out the boot device) would cause the kernel to notice
that the I/O's aren't going anywhere and mark them as failed.
> multipathd and iscsi I've tried another update process. And, yeha, it
> works???
.. and depending on your multipath.conf file, the I/O can be queued forever
(queue_if_no_path), or up to a certain time-limit ('no_path_retry'). Or
error out immediately (if you don't have any of those options set). From
your experiment it seems that those flags are set and the mkinitrd
process is in the D state, waiting for the kernel to return a value
after a read/write system call.
If you had deleted the device mapper entries before terminating
multipathd and iscsi this shouldn't have happend. Or if you had set the
multipath configuration to error out immediately you wouldn't hit this.
>
> I could reproduce this behavior on 2 machines, both centos with standard
> packages.
Correclty so.
>
> So can anybody comprehend this "failure" or any ideas?
>
> Cheers
> Maddin
>
>
>
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~--~~~~--~~--~--~---
strange behavior on kernel update
Hi, i'm using centos with dm-multipath and iscsi as a database system. All packages are normal centos packages from their mirrors. This night I was updating centos to the current release (5.3, previous version was 5.2). So the first thing I have done was stopping the multipathd and iscsi. After this I was doing the normal update stuff (yum check-updates, yum clean all, yum update glibc\*, yum update). During the last yum update the new kernel was installed (2.6.18-128.1.6.el5, previous was 2.6.18-92.1.22.el5). During the installation process of the kernel, the yum process hangs. Looking with ps for the hanging processes, I see that the kernel package was building the initrd. After killing this process and repeating the yum update process (this time only the kernel-package) about 5 times, the same strange failure occurs (process hangs on building initrd). The sixth time I was stracing the whole update process and see that the mkinitrd process does a wait-syscall for sth.. After killing this process again I went down to the server room to reboot the system and try another update process. I reboot the systems, stops multipathd and iscsi and starting the update process. On this update I could see that during the mkinitrd process, a kernel message "device-mapper: failed path x:x:x:x" occurs. Hmm this is strange I thought, because I've stopped multipathd and iscsi, so why means the kernel that a path is failed? Once again killing the mkinitrd process and then starting multipathd and iscsi I've tried another update process. And, yeha, it works??? I could reproduce this behavior on 2 machines, both centos with standard packages. So can anybody comprehend this "failure" or any ideas? Cheers Maddin --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
