Paul,
I don't want to point fingers, but is it possible there is a defect with
your disk controller? That code has (had) been fairly well used, and I've
never seen errors with file system corruption. Not good.
Hope this helps.
-Mike
On Thu, Jul 26, 2018 at 9:46 AM Feller, Paul <[email protected]>
wrote:
> Mike, here are the answers to your questions.
>
>
> Can you post the contents of the clone script? Is it based on the code
> starting on p. 528?
>
> >>>> Yes, we're using that one with no modifications
>
> When you say " corrupts one or more of the copy's file systems", can you
> be more specific?
>
> >>>> These are the errors I'm seeing (multiple occurrences in most cases
> but I'll skip the duplication)
>
> This happens when the LNXADMIN machine is copying the root file system
> disk (100), then mounting the clone's root file system, doing the IP
> update, and unmounting it. The' FORMAT F - Operation terminated' errors are
> somewhat expected - we're using replicated DASD devices and Linux doesn't
> like the delay in acknowledging writes but deals with it ok.
>
> dasd-eckd 0.0.ffff: New DASD 3390/0C (CU 3990/01) with 5008 cylinders, 1
> dasd-eckd 0.0.ffff: DASD with 4 KB/block, 3605760 KB total size, 48 KB/t
> dasdd:VOL1/ 0X0100: dasdd1 dasdd2
> DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
> DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
> DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
> DCVML91 : dasd-eckd 0.0.ffff: DASD with 4 KB/block, 3605760 KB total
> size, 48 KB/track, compatible disk layout
> dasdd:detected capacity change from 0 to 3692298240
> dasdd:Expected Label VOL1 not found, treating as CDL formated Disk
>
> dasdd:VOL1/ 0XFFFF:
> DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
> DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
> DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated
> DCVML91 : __ratelimit: 760 callbacks suppressed
> __ratelimit: 78441 callbacks suppressed
> Buffer I/O error on device dasdd, logical block 824597
> lost page write due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824598
> lost page write due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824599
> lost page write due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824600
> lost page write due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824601 lost page write
> due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824602 lost page write
> due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824603 lost page write
> due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824604 lost page write
> due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824605 lost page write
> due to I/O error on dasdd
> Buffer I/O error on device dasdd, logical block 824606 lost page write
> due to I/O error on dasdd
> DCVML91 :
> end_request: I/O error, dev dasdd, sector 6607016
> end_request: I/O error, dev dasdd, sector 6608040
> end_request: I/O error, dev dasdd, sector 6609064
> <several dozen more I/O errors>
>
> When the cloned guest boots up I get this, though the exact errors vary
> from corrupting two or three file systems to just the root. I've noticed
> the last few times I've test this that's it pretty consistently the same
> INODE getting hit.
>
> Enabling local filesystem quotas: EXT4-fs error (device dm-0):
> __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic -
> magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
> __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic -
> magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
> __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic -
> magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
> __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic -
> magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
> __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic -
> magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0):
> __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic -
> magic 0, entries 0, max 0(0), depth 0(0)
> DCVML99 :
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
>
> .......
>
> DCVML99 : bad header/extent in inode #5136: invalid magic - magic 0,
> entries 0, max 0(0), depth 0(0)
> /etc/init.d/functions: line 568: 1116 Bus error "$@"
> [FAILED]
> /etc/rc.d/rc.sysinit: line 546: 1130 Bus error cat
> /var/lib/random-seed > /dev/urandom
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> DCVML99 : /etc/rc.d/rc.sysinit: line 550: 1131 Bus error
> chmod 600 /var/lib/random-seed
> /etc/rc.d/rc.sysinit: line 550: 1132 Bus error dd
> if=/dev/urandom of=/var/lib/random-seed count=1 bs=4096 2> /dev/null
> /etc/rc.d/rc.sysinit: line 595: 1133 Bus error rm -f
> /fastboot /fsckoptions /forcefsck /.autofsck /forcequotacheck /halt
> /poweroff /.suspended &>/dev/null
> /etc/rc.d/rc.sysinit: line 602: 1134 Bus error rm -rf
> /var/lock/cvs/* /var/run/screen/*
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> DCVML99 :
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> /etc/rc.d/rc.sysinit: line 603: 1135 Bus error find
> /var/lock /var/run ! -type d -exec rm -f {} \;
> /etc/rc.d/rc.sysinit: line 604: 1136 Bus error rm -f
> /var/lib/rpm/__db* &>/dev/null
> /etc/rc.d/rc.sysinit: line 605: 1137 Bus error rm -f
> /var/gdm/.gdmfifo &>/dev/null
> /etc/rc.d/rc.sysinit: line 611: 1139 Bus error touch
> /var/log/wtmp
> /etc/rc.d/rc.sysinit: line 612: 1140 Bus error chgrp utmp
> /var/run/utmp /var/log/wtmp
> /etc/rc.d/rc.sysinit: line 613: 1141 Bus error chmod 0664
> /var/run/utmp /var/log/wtmp
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> /etc/rc.d/rc.sysinit: line 624: 1142 Bus error rm -f
> /tmp/.X*-lock /tmp/.lock.* /tmp/.gdm_socket /tmp/.s.PGSQL.*
> /etc/rc.d/rc.sysinit: line 628: 1143 Bus error rm -rf
> /tmp/.X*-unix /tmp/.ICE-unix /tmp/.font-unix /tmp/hsperfdata_*
> /tmp/kde-* /tmp/ksocket-* /tmp/mc-* /tmp/mcop-* /tmp/orbit-*
> /tmp/scrollkeeper-* /tmp/ssh-* /dev/.in_sysinit
> /etc/rc.d/rc.sysinit: line 631: 1144 Bus error mkdir -m
> 1777 -p /tmp/.ICE-unix > /dev/null 2>&1
> /etc/rc.d/rc.sysinit: line 632: 1145 Bus error chown
> root:root /tmp/.ICE-unix
> Enabling /etc/fstab swaps: /etc/init.d/functions: line 568: 1147 Bus
> error "$@"
> [FAILED]
> /etc/rc.d/rc.sysinit: line 651: 1149 Bus error /bin/mount
> -t binfmt_misc none /proc/sys/fs/binfmt_misc > /dev/null 2>&1
> /etc/rc.d/rc.sysinit: line 666: 1150 Bus error mv -f
> /var/log/dmesg /var/log/dmesg.old
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> /etc/rc.d/rc.sysinit: line 667: 1151 Bus error dmesg -s
> 131072 > /var/log/dmesg
> /etc/rc.d/rc.sysinit: line 670: 1152 Bus error touch
> /.autofsck &>/dev/null
> DCVML99 :
> EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in
> inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0)
> init: rc main process (1158) killed by BUS signal
>
> Thanks..
>
> Paul Feller
> AGT Mainframe Technical Support
>
> -----Original Message-----
> From: Feller, Paul
> Sent: Wednesday, July 25, 2018 4:15 PM
> To: 'Linux on 390 Port' <[email protected]>
> Subject: RE: Issue cloning RHEL server
>
> Mike, I'll get with the team member. He does more of the zLinux type
> stuff and I do more of the z/VM type stuff.
>
> Thanks..
>
> Paul Feller
> AGT Mainframe Technical Support
>
> -----Original Message-----
> From: Linux on 390 Port [mailto:[email protected]] On Behalf Of
> Michael MacIsaac
> Sent: Wednesday, July 25, 2018 1:19 PM
> To: [email protected]
> Subject: Re: Issue cloning RHEL server
>
> Paul,
>
> Can you post the contents of the clone script? Is it based on the code
> starting on p. 528?
>
> When you say " corrupts one or more of the copy's file systems", can you
> be more specific?
>
> Thanks.
>
> -Mike
>
> On Wed, Jul 25, 2018 at 1:08 PM Feller, Paul <[email protected]
> >
> wrote:
>
> > One of my team members is trying to run the cloning process described
> > in "The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and SLES 11 SP3
> > October 2013, January 2014 (SG-248147-00)". I believe the process has
> > worked in the past. But as things go the environment has changed over
> time.
> >
> >
> > Here is the question from the team member.
> >
> > We are still using the cloning process documented in, and scripts
> > provided with, "The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and
> > SLES 11 SP3 October 2013, January 2014 (SG-248147-00)". It's been
> > sometime since I last used the process to create a new zLinux system
> > and when I try now I'm finding that the procedure to update the IP
> > information on the clone copy corrupts one or more of the copy's file
> > systems. I can use DDR to build a bootable system from the gold image
> > so I'm fairly sure the dasd copy process is working ok. The gold
> > image is not corrupted. It boots without an error and so does the DDR
> copy.
> >
> > We're currently running z/VM 6.4 (RSU1801). The 'LNXADMIN' system is
> > running Linux version 2.6.32-504.8.1.el6.s390x (Red Hat 4.4.7-4) and
> > that's what we're cloning. Has anyone experienced this problem and
> > found a work around?
> >
> >
> > Thanks..
> >
> > Paul Feller
> > AGT Mainframe Technical Support
> >
> >
> >
> >
> > ----------------------------------------------------------------------
> > For LINUX-390 subscribe / signoff / archive access instructions, send
> > email to [email protected] with the message: INFO LINUX-390 or
> > visit
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htb
> > in_wlvindex-3FLINUX-2D390&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWj
> > ONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu
> > 7ZzXBMAn_2Q&s=8YEDwSUUW6k1Eic6YXb-tpis6DEin60TNeH_D5rLyjM&e=
> > ----------------------------------------------------------------------
> > For more information on Linux on System z, visit
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_&;
> > d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9cc
> > yeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=lgGYDNMH5l_L
> > 4-JL7-a3S4TlQNxD4Qfyx6hB8rvH1zQ&e=
> >
>
>
> --
> -Mike MacIsaac
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions, send
> email to [email protected] with the message: INFO LINUX-390 or visit
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htbin_wlvindex-3FLINUX-2D390&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=8YEDwSUUW6k1Eic6YXb-tpis6DEin60TNeH_D5rLyjM&e=
> ----------------------------------------------------------------------
> For more information on Linux on System z, visit
> https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=lgGYDNMH5l_L4-JL7-a3S4TlQNxD4Qfyx6hB8rvH1zQ&e=
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions,
> send email to [email protected] with the message: INFO LINUX-390 or
> visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> ----------------------------------------------------------------------
> For more information on Linux on System z, visit
> http://wiki.linuxvm.org/
>
--
-Mike MacIsaac
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
