On Tue, Apr 28, 2009 at 04:00:08AM +0200, Maddin wrote:
> Hi,
> i'm using centos with dm-multipath and iscsi as a database system. All
> packages are normal centos packages from their mirrors.
> This night I was updating centos to the current release (5.3, previous
> version was 5.2). So the first thing I have done was stopping the multipathd
> and iscsi. After this I was doing the normal update stuff (yum
> check-updates, yum clean all, yum update glibc\*, yum update). During the
> last yum update the new kernel was installed (2.6.18-128.1.6.el5, previous
> was 2.6.18-92.1.22.el5).
> During the installation process of the kernel, the yum process hangs.
> Looking with ps for the hanging processes, I see that the kernel package was
> building the initrd. After killing this process and repeating the yum update
> process (this time only the kernel-package) about 5 times, the same strange
> failure occurs (process hangs on building initrd). The sixth time I was
> stracing the whole update process and see that the mkinitrd process does a
> wait-syscall for sth.. After killing this process again I went down to the
> server room to reboot the system and try another update process. I reboot
> the systems, stops multipathd and iscsi and starting the update process. On
> this update I could see that during the mkinitrd process, a kernel message
> "device-mapper: failed path x:x:x:x" occurs. Hmm this is strange I thought,
> because I've stopped multipathd and iscsi, so why means the kernel that a
> path is failed? Once again killing the mkinitrd process and then starting

The device mapper (a kernel component) still has your multipath entries even 
thought you have
killed multipathd and iscsi. It doesn't delete them, unless you
specifically instructs the daemon (multipathd) to do so.

>From the standpoint of the kernel, the underlaying block devices got
yanked out and the "brain" (multipathd) terminated. And any I/O that
touches that multipath device (such as 'pvscan', 'lvscan' which mkinitrd
uses to figure out the boot device) would cause the kernel to notice
that the I/O's aren't going anywhere and mark them as failed.

> multipathd and iscsi I've tried another update process. And, yeha, it
> works???

.. and depending on your multipath.conf file, the I/O can be queued forever
(queue_if_no_path), or up to a certain time-limit ('no_path_retry'). Or
error out immediately (if you don't have any of those options set). From
your experiment it seems that those flags are set and the mkinitrd
process is in the D state, waiting for the kernel to return a value
after a read/write system call.

If you had deleted the device mapper entries before terminating
multipathd and iscsi this shouldn't have happend. Or if you had set the
multipath configuration to error out immediately you wouldn't hit this.

> I could reproduce this behavior on 2 machines, both centos with standard
> packages.

Correclty so.
> So can anybody comprehend this "failure" or any ideas?

> Cheers
> Maddin

You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to 
For more options, visit this group at http://groups.google.com/group/open-iscsi

Reply via email to