Paul, I don't want to point fingers, but is it possible there is a defect with your disk controller? That code has (had) been fairly well used, and I've never seen errors with file system corruption. Not good.
Hope this helps. -Mike On Thu, Jul 26, 2018 at 9:46 AM Feller, Paul <paul.fel...@transamerica.com> wrote: > Mike, here are the answers to your questions. > > > Can you post the contents of the clone script? Is it based on the code > starting on p. 528? > > >>>> Yes, we're using that one with no modifications > > When you say " corrupts one or more of the copy's file systems", can you > be more specific? > > >>>> These are the errors I'm seeing (multiple occurrences in most cases > but I'll skip the duplication) > > This happens when the LNXADMIN machine is copying the root file system > disk (100), then mounting the clone's root file system, doing the IP > update, and unmounting it. The' FORMAT F - Operation terminated' errors are > somewhat expected - we're using replicated DASD devices and Linux doesn't > like the delay in acknowledging writes but deals with it ok. > > dasd-eckd 0.0.ffff: New DASD 3390/0C (CU 3990/01) with 5008 cylinders, 1 > dasd-eckd 0.0.ffff: DASD with 4 KB/block, 3605760 KB total size, 48 KB/t > dasdd:VOL1/ 0X0100: dasdd1 dasdd2 > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: DASD with 4 KB/block, 3605760 KB total > size, 48 KB/track, compatible disk layout > dasdd:detected capacity change from 0 to 3692298240 > dasdd:Expected Label VOL1 not found, treating as CDL formated Disk > > dasdd:VOL1/ 0XFFFF: > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : __ratelimit: 760 callbacks suppressed > __ratelimit: 78441 callbacks suppressed > Buffer I/O error on device dasdd, logical block 824597 > lost page write due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824598 > lost page write due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824599 > lost page write due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824600 > lost page write due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824601 lost page write > due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824602 lost page write > due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824603 lost page write > due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824604 lost page write > due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824605 lost page write > due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824606 lost page write > due to I/O error on dasdd > DCVML91 : > end_request: I/O error, dev dasdd, sector 6607016 > end_request: I/O error, dev dasdd, sector 6608040 > end_request: I/O error, dev dasdd, sector 6609064 > <several dozen more I/O errors> > > When the cloned guest boots up I get this, though the exact errors vary > from corrupting two or three file systems to just the root. I've noticed > the last few times I've test this that's it pretty consistently the same > INODE getting hit. > > Enabling local filesystem quotas: EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - > magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - > magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - > magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - > magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - > magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid magic - > magic 0, entries 0, max 0(0), depth 0(0) > DCVML99 : > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > > ....... > > DCVML99 : bad header/extent in inode #5136: invalid magic - magic 0, > entries 0, max 0(0), depth 0(0) > /etc/init.d/functions: line 568: 1116 Bus error "$@" > [FAILED] > /etc/rc.d/rc.sysinit: line 546: 1130 Bus error cat > /var/lib/random-seed > /dev/urandom > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > DCVML99 : /etc/rc.d/rc.sysinit: line 550: 1131 Bus error > chmod 600 /var/lib/random-seed > /etc/rc.d/rc.sysinit: line 550: 1132 Bus error dd > if=/dev/urandom of=/var/lib/random-seed count=1 bs=4096 2> /dev/null > /etc/rc.d/rc.sysinit: line 595: 1133 Bus error rm -f > /fastboot /fsckoptions /forcefsck /.autofsck /forcequotacheck /halt > /poweroff /.suspended &>/dev/null > /etc/rc.d/rc.sysinit: line 602: 1134 Bus error rm -rf > /var/lock/cvs/* /var/run/screen/* > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > DCVML99 : > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > /etc/rc.d/rc.sysinit: line 603: 1135 Bus error find > /var/lock /var/run ! -type d -exec rm -f {} \; > /etc/rc.d/rc.sysinit: line 604: 1136 Bus error rm -f > /var/lib/rpm/__db* &>/dev/null > /etc/rc.d/rc.sysinit: line 605: 1137 Bus error rm -f > /var/gdm/.gdmfifo &>/dev/null > /etc/rc.d/rc.sysinit: line 611: 1139 Bus error touch > /var/log/wtmp > /etc/rc.d/rc.sysinit: line 612: 1140 Bus error chgrp utmp > /var/run/utmp /var/log/wtmp > /etc/rc.d/rc.sysinit: line 613: 1141 Bus error chmod 0664 > /var/run/utmp /var/log/wtmp > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > /etc/rc.d/rc.sysinit: line 624: 1142 Bus error rm -f > /tmp/.X*-lock /tmp/.lock.* /tmp/.gdm_socket /tmp/.s.PGSQL.* > /etc/rc.d/rc.sysinit: line 628: 1143 Bus error rm -rf > /tmp/.X*-unix /tmp/.ICE-unix /tmp/.font-unix /tmp/hsperfdata_* > /tmp/kde-* /tmp/ksocket-* /tmp/mc-* /tmp/mcop-* /tmp/orbit-* > /tmp/scrollkeeper-* /tmp/ssh-* /dev/.in_sysinit > /etc/rc.d/rc.sysinit: line 631: 1144 Bus error mkdir -m > 1777 -p /tmp/.ICE-unix > /dev/null 2>&1 > /etc/rc.d/rc.sysinit: line 632: 1145 Bus error chown > root:root /tmp/.ICE-unix > Enabling /etc/fstab swaps: /etc/init.d/functions: line 568: 1147 Bus > error "$@" > [FAILED] > /etc/rc.d/rc.sysinit: line 651: 1149 Bus error /bin/mount > -t binfmt_misc none /proc/sys/fs/binfmt_misc > /dev/null 2>&1 > /etc/rc.d/rc.sysinit: line 666: 1150 Bus error mv -f > /var/log/dmesg /var/log/dmesg.old > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > /etc/rc.d/rc.sysinit: line 667: 1151 Bus error dmesg -s > 131072 > /var/log/dmesg > /etc/rc.d/rc.sysinit: line 670: 1152 Bus error touch > /.autofsck &>/dev/null > DCVML99 : > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent in > inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > init: rc main process (1158) killed by BUS signal > > Thanks.. > > Paul Feller > AGT Mainframe Technical Support > > -----Original Message----- > From: Feller, Paul > Sent: Wednesday, July 25, 2018 4:15 PM > To: 'Linux on 390 Port' <LINUX-390@VM.MARIST.EDU> > Subject: RE: Issue cloning RHEL server > > Mike, I'll get with the team member. He does more of the zLinux type > stuff and I do more of the z/VM type stuff. > > Thanks.. > > Paul Feller > AGT Mainframe Technical Support > > -----Original Message----- > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of > Michael MacIsaac > Sent: Wednesday, July 25, 2018 1:19 PM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: Issue cloning RHEL server > > Paul, > > Can you post the contents of the clone script? Is it based on the code > starting on p. 528? > > When you say " corrupts one or more of the copy's file systems", can you > be more specific? > > Thanks. > > -Mike > > On Wed, Jul 25, 2018 at 1:08 PM Feller, Paul <paul.fel...@transamerica.com > > > wrote: > > > One of my team members is trying to run the cloning process described > > in "The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and SLES 11 SP3 > > October 2013, January 2014 (SG-248147-00)". I believe the process has > > worked in the past. But as things go the environment has changed over > time. > > > > > > Here is the question from the team member. > > > > We are still using the cloning process documented in, and scripts > > provided with, "The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and > > SLES 11 SP3 October 2013, January 2014 (SG-248147-00)". It's been > > sometime since I last used the process to create a new zLinux system > > and when I try now I'm finding that the procedure to update the IP > > information on the clone copy corrupts one or more of the copy's file > > systems. I can use DDR to build a bootable system from the gold image > > so I'm fairly sure the dasd copy process is working ok. The gold > > image is not corrupted. It boots without an error and so does the DDR > copy. > > > > We're currently running z/VM 6.4 (RSU1801). The 'LNXADMIN' system is > > running Linux version 2.6.32-504.8.1.el6.s390x (Red Hat 4.4.7-4) and > > that's what we're cloning. Has anyone experienced this problem and > > found a work around? > > > > > > Thanks.. > > > > Paul Feller > > AGT Mainframe Technical Support > > > > > > > > > > ---------------------------------------------------------------------- > > For LINUX-390 subscribe / signoff / archive access instructions, send > > email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > > visit > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htb > > in_wlvindex-3FLINUX-2D390&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWj > > ONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu > > 7ZzXBMAn_2Q&s=8YEDwSUUW6k1Eic6YXb-tpis6DEin60TNeH_D5rLyjM&e= > > ---------------------------------------------------------------------- > > For more information on Linux on System z, visit > > https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_& > > d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9cc > > yeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=lgGYDNMH5l_L > > 4-JL7-a3S4TlQNxD4Qfyx6hB8rvH1zQ&e= > > > > > -- > -Mike MacIsaac > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, send > email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htbin_wlvindex-3FLINUX-2D390&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=8YEDwSUUW6k1Eic6YXb-tpis6DEin60TNeH_D5rLyjM&e= > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=lgGYDNMH5l_L4-JL7-a3S4TlQNxD4Qfyx6hB8rvH1zQ&e= > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > -- -Mike MacIsaac ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/