Mike, here are the answers to your questions.
Can you post the contents of the clone script? Is it based on the code
starting on p. 528?
>>>> Yes, we're using that one with no modifications
When you say " corrupts one or more of the copy's file systems", can you be
more specific?
>>>> These are the errors I'm seeing (multiple occurrences in most cases but
>>>> I'll skip the duplication)
This happens when the LNXADMIN machine is copying the root file system disk
(100), then mounting the clone's root file system, doing the IP update, and
unmounting it. The' FORMAT F - Operation terminated' errors are somewhat
expected - we're using replicated DASD devices and Linux doesn't like the delay
in acknowledging writes but deals with it ok.
dasd-eckd 0.0.ffff: New DASD 3390/0C (CU 3990/01) with 5008 cylinders, 1
dasd-eckd 0.0.ffff: DASD with 4 KB/block, 3605760 KB total size, 48 KB/t
dasdd:VOL1/ 0X0100: dasdd1 dasdd2
DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
DCVML91 : dasd-eckd 0.0.ffff: DASD with 4 KB/block, 3605760 KB total size, 48
KB/track, compatible disk layout
dasdd:detected capacity change from 0 to 3692298240
dasdd:Expected Label VOL1 not found, treating as CDL formated Disk
dasdd:VOL1/ 0XFFFF:
DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
DCVML91 : __ratelimit: 760 callbacks suppressed
__ratelimit: 78441 callbacks suppressed
Buffer I/O error on device dasdd, logical block 824597 lost
page write due to I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824598 lost
page write due to I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824599 lost
page write due to I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824600 lost
page write due to I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824601 lost page write due to
I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824602 lost page write due to
I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824603 lost page write due to
I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824604 lost page write due to
I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824605 lost page write due to
I/O error on dasdd
Buffer I/O error on device dasdd, logical block 824606 lost page write due to
I/O error on dasdd
DCVML91 :
end_request: I/O error, dev dasdd, sector 6607016
end_request: I/O error, dev dasdd, sector 6608040
end_request: I/O error, dev dasdd, sector 6609064
<several dozen more I/O errors>
When the cloned guest boots up I get this, though the exact errors vary from
corrupting two or three file systems to just the root. I've noticed the last
few times I've test this that's it pretty consistently the same INODE getting
hit.
Enabling local filesystem quotas: EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0)
DCVML99 :
EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in inode
#5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error
(device dm-0): __ext4_ext_check_block: bad header/extent in inode #5136:
invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
.......
DCVML99 : bad header/extent in inode #5136: invalid magic - magic 0, entries 0,
max 0(0), depth 0(0)
/etc/init.d/functions: line 568: 1116 Bus error "$@"
[FAILED]
/etc/rc.d/rc.sysinit: line 546: 1130 Bus error cat
/var/lib/random-seed > /dev/urandom
EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in inode
#5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error
(device dm-0): __ext4_ext_check_block: bad header/extent in inode #5136:
invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device
dm-0): __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic
- magic 0, entries 0, max 0(0), depth 0(0)
DCVML99 : /etc/rc.d/rc.sysinit: line 550: 1131 Bus error chmod
600 /var/lib/random-seed
/etc/rc.d/rc.sysinit: line 550: 1132 Bus error dd
if=/dev/urandom of=/var/lib/random-seed count=1 bs=4096 2> /dev/null
/etc/rc.d/rc.sysinit: line 595: 1133 Bus error rm -f /fastboot
/fsckoptions /forcefsck /.autofsck /forcequotacheck /halt /poweroff /.suspended
&>/dev/null
/etc/rc.d/rc.sysinit: line 602: 1134 Bus error rm -rf
/var/lock/cvs/* /var/run/screen/*
EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in inode
#5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error
(device dm-0): __ext4_ext_check_block: bad header/extent in inode #5136:
invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device
dm-0): __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic
- magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0)
DCVML99 :
EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in inode
#5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error
(device dm-0): __ext4_ext_check_block: bad header/extent in inode #5136:
invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device
dm-0): __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic
- magic 0, entries 0, max 0(0), depth 0(0)
/etc/rc.d/rc.sysinit: line 603: 1135 Bus error find /var/lock
/var/run ! -type d -exec rm -f {} \;
/etc/rc.d/rc.sysinit: line 604: 1136 Bus error rm -f
/var/lib/rpm/__db* &>/dev/null
/etc/rc.d/rc.sysinit: line 605: 1137 Bus error rm -f
/var/gdm/.gdmfifo &>/dev/null
/etc/rc.d/rc.sysinit: line 611: 1139 Bus error touch
/var/log/wtmp
/etc/rc.d/rc.sysinit: line 612: 1140 Bus error chgrp utmp
/var/run/utmp /var/log/wtmp
/etc/rc.d/rc.sysinit: line 613: 1141 Bus error chmod 0664
/var/run/utmp /var/log/wtmp
EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in inode
#5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error
(device dm-0): __ext4_ext_check_block: bad header/extent in inode #5136:
invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device
dm-0): __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic
- magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
__ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - magic
0, entries 0, max 0(0), depth 0(0)
/etc/rc.d/rc.sysinit: line 624: 1142 Bus error rm -f
/tmp/.X*-lock /tmp/.lock.* /tmp/.gdm_socket /tmp/.s.PGSQL.*
/etc/rc.d/rc.sysinit: line 628: 1143 Bus error rm -rf
/tmp/.X*-unix /tmp/.ICE-unix /tmp/.font-unix /tmp/hsperfdata_*
/tmp/kde-* /tmp/ksocket-* /tmp/mc-* /tmp/mcop-* /tmp/orbit-*
/tmp/scrollkeeper-* /tmp/ssh-* /dev/.in_sysinit
/etc/rc.d/rc.sysinit: line 631: 1144 Bus error mkdir -m 1777 -p
/tmp/.ICE-unix > /dev/null 2>&1
/etc/rc.d/rc.sysinit: line 632: 1145 Bus error chown root:root
/tmp/.ICE-unix
Enabling /etc/fstab swaps: /etc/init.d/functions: line 568: 1147 Bus error
"$@"
[FAILED]
/etc/rc.d/rc.sysinit: line 651: 1149 Bus error /bin/mount -t
binfmt_misc none /proc/sys/fs/binfmt_misc > /dev/null 2>&1
/etc/rc.d/rc.sysinit: line 666: 1150 Bus error mv -f
/var/log/dmesg /var/log/dmesg.old
EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in inode
#5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error
(device dm-0): __ext4_ext_check_block: bad header/extent in inode #5136:
invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
/etc/rc.d/rc.sysinit: line 667: 1151 Bus error dmesg -s 131072 >
/var/log/dmesg
/etc/rc.d/rc.sysinit: line 670: 1152 Bus error touch /.autofsck
&>/dev/null
DCVML99 :
EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in inode
#5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
init: rc main process (1158) killed by BUS signal
Thanks..
Paul Feller
AGT Mainframe Technical Support
-----Original Message-----
From: Feller, Paul
Sent: Wednesday, July 25, 2018 4:15 PM
To: 'Linux on 390 Port' <[email protected]>
Subject: RE: Issue cloning RHEL server
Mike, I'll get with the team member. He does more of the zLinux type stuff and
I do more of the z/VM type stuff.
Thanks..
Paul Feller
AGT Mainframe Technical Support
-----Original Message-----
From: Linux on 390 Port [mailto:[email protected]] On Behalf Of Michael
MacIsaac
Sent: Wednesday, July 25, 2018 1:19 PM
To: [email protected]
Subject: Re: Issue cloning RHEL server
Paul,
Can you post the contents of the clone script? Is it based on the code
starting on p. 528?
When you say " corrupts one or more of the copy's file systems", can you be
more specific?
Thanks.
-Mike
On Wed, Jul 25, 2018 at 1:08 PM Feller, Paul <[email protected]>
wrote:
> One of my team members is trying to run the cloning process described
> in "The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and SLES 11 SP3
> October 2013, January 2014 (SG-248147-00)". I believe the process has
> worked in the past. But as things go the environment has changed over time.
>
>
> Here is the question from the team member.
>
> We are still using the cloning process documented in, and scripts
> provided with, "The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and
> SLES 11 SP3 October 2013, January 2014 (SG-248147-00)". It's been
> sometime since I last used the process to create a new zLinux system
> and when I try now I'm finding that the procedure to update the IP
> information on the clone copy corrupts one or more of the copy's file
> systems. I can use DDR to build a bootable system from the gold image
> so I'm fairly sure the dasd copy process is working ok. The gold
> image is not corrupted. It boots without an error and so does the DDR copy.
>
> We're currently running z/VM 6.4 (RSU1801). The 'LNXADMIN' system is
> running Linux version 2.6.32-504.8.1.el6.s390x (Red Hat 4.4.7-4) and
> that's what we're cloning. Has anyone experienced this problem and
> found a work around?
>
>
> Thanks..
>
> Paul Feller
> AGT Mainframe Technical Support
>
>
>
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions, send
> email to [email protected] with the message: INFO LINUX-390 or
> visit
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htb
> in_wlvindex-3FLINUX-2D390&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWj
> ONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu
> 7ZzXBMAn_2Q&s=8YEDwSUUW6k1Eic6YXb-tpis6DEin60TNeH_D5rLyjM&e=
> ----------------------------------------------------------------------
> For more information on Linux on System z, visit
> https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_&;
> d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9cc
> yeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=lgGYDNMH5l_L
> 4-JL7-a3S4TlQNxD4Qfyx6hB8rvH1zQ&e=
>
--
-Mike MacIsaac
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions, send email to
[email protected] with the message: INFO LINUX-390 or visit
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htbin_wlvindex-3FLINUX-2D390&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=8YEDwSUUW6k1Eic6YXb-tpis6DEin60TNeH_D5rLyjM&e=
----------------------------------------------------------------------
For more information on Linux on System z, visit
https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=lgGYDNMH5l_L4-JL7-a3S4TlQNxD4Qfyx6hB8rvH1zQ&e=
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
