Mike, interesting question. The DASD subsystem is an EMC VMAX3 all flash array. I think this is the first time we have tried to do the cloning process while on this DASD box. I'll have to talk to the team member about what all is done during the cloning process.
Thanks.. Paul Feller AGT Mainframe Technical Support -----Original Message----- From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Michael MacIsaac Sent: Thursday, July 26, 2018 9:12 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: Issue cloning RHEL server Paul, I don't want to point fingers, but is it possible there is a defect with your disk controller? That code has (had) been fairly well used, and I've never seen errors with file system corruption. Not good. Hope this helps. -Mike On Thu, Jul 26, 2018 at 9:46 AM Feller, Paul <paul.fel...@transamerica.com> wrote: > Mike, here are the answers to your questions. > > > Can you post the contents of the clone script? Is it based on the > code starting on p. 528? > > >>>> Yes, we're using that one with no modifications > > When you say " corrupts one or more of the copy's file systems", can > you be more specific? > > >>>> These are the errors I'm seeing (multiple occurrences in most > >>>> cases > but I'll skip the duplication) > > This happens when the LNXADMIN machine is copying the root file system > disk (100), then mounting the clone's root file system, doing the IP > update, and unmounting it. The' FORMAT F - Operation terminated' > errors are somewhat expected - we're using replicated DASD devices and > Linux doesn't like the delay in acknowledging writes but deals with it ok. > > dasd-eckd 0.0.ffff: New DASD 3390/0C (CU 3990/01) with 5008 > cylinders, 1 dasd-eckd 0.0.ffff: DASD with 4 KB/block, 3605760 KB > total size, 48 KB/t dasdd:VOL1/ 0X0100: dasdd1 dasdd2 > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: DASD with 4 KB/block, 3605760 KB total > size, 48 KB/track, compatible disk layout > dasdd:detected capacity change from 0 to 3692298240 > dasdd:Expected Label VOL1 not found, treating as CDL formated Disk > > dasdd:VOL1/ 0XFFFF: > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : dasd-eckd 0.0.ffff: FORMAT F - Operation Terminated > DCVML91 : __ratelimit: 760 callbacks suppressed > __ratelimit: 78441 callbacks suppressed Buffer I/O error on device > dasdd, logical block 824597 lost page write due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824598 lost page > write due to I/O error on dasdd Buffer I/O error on device dasdd, > logical block 824599 lost page write due to I/O error on dasdd > Buffer I/O error on device dasdd, logical block 824600 lost page > write due to I/O error on dasdd Buffer I/O error on device dasdd, > logical block 824601 lost page write due to I/O error on dasdd Buffer > I/O error on device dasdd, logical block 824602 lost page write due > to I/O error on dasdd Buffer I/O error on device dasdd, logical block > 824603 lost page write due to I/O error on dasdd Buffer I/O error on > device dasdd, logical block 824604 lost page write due to I/O error > on dasdd Buffer I/O error on device dasdd, logical block 824605 lost > page write due to I/O error on dasdd Buffer I/O error on device dasdd, > logical block 824606 lost page write due to I/O error on dasdd > DCVML91 : > end_request: I/O error, dev dasdd, sector 6607016 > end_request: I/O error, dev dasdd, sector 6608040 > end_request: I/O error, dev dasdd, sector 6609064 <several dozen more > I/O errors> > > When the cloned guest boots up I get this, though the exact errors > vary from corrupting two or three file systems to just the root. I've > noticed the last few times I've test this that's it pretty > consistently the same INODE getting hit. > > Enabling local filesystem quotas: EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid > magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid > magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid > magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid > magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid > magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid > magic - magic 0, entries 0, max 0(0), depth 0(0) > DCVML99 : > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent > in inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth > 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: bad > header/extent in inode #5136: invalid magic - magic 0, entries 0, max > 0(0), depth 0(0) > > ....... > > DCVML99 : bad header/extent in inode #5136: invalid magic - magic 0, > entries 0, max 0(0), depth 0(0) > /etc/init.d/functions: line 568: 1116 Bus error "$@" > [FAILED] > /etc/rc.d/rc.sysinit: line 546: 1130 Bus error cat > /var/lib/random-seed > /dev/urandom > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent > in inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth > 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: bad > header/extent in inode #5136: invalid magic - magic 0, entries 0, max > 0(0), depth 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: > bad header/extent in inode #5136: invalid magic - magic 0, entries 0, > max 0(0), depth 0(0) > DCVML99 : /etc/rc.d/rc.sysinit: line 550: 1131 Bus error chmod 600 > /var/lib/random-seed > /etc/rc.d/rc.sysinit: line 550: 1132 Bus error dd > if=/dev/urandom of=/var/lib/random-seed count=1 bs=4096 2> /dev/null > /etc/rc.d/rc.sysinit: line 595: 1133 Bus error rm -f > /fastboot /fsckoptions /forcefsck /.autofsck /forcequotacheck /halt > /poweroff /.suspended &>/dev/null > /etc/rc.d/rc.sysinit: line 602: 1134 Bus error rm -rf > /var/lock/cvs/* /var/run/screen/* > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent > in inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth > 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: bad > header/extent in inode #5136: invalid magic - magic 0, entries 0, max > 0(0), depth 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: > bad header/extent in inode #5136: invalid magic - magic 0, entries 0, > max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid > magic - magic 0, entries 0, max 0(0), depth 0(0) > DCVML99 : > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent > in inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth > 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: bad > header/extent in inode #5136: invalid magic - magic 0, entries 0, max > 0(0), depth 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: > bad header/extent in inode #5136: invalid magic - magic 0, entries 0, max > 0(0), depth 0(0) > /etc/rc.d/rc.sysinit: line 603: 1135 Bus error find > /var/lock /var/run ! -type d -exec rm -f {} \; > /etc/rc.d/rc.sysinit: line 604: 1136 Bus error rm -f > /var/lib/rpm/__db* &>/dev/null > /etc/rc.d/rc.sysinit: line 605: 1137 Bus error rm -f > /var/gdm/.gdmfifo &>/dev/null > /etc/rc.d/rc.sysinit: line 611: 1139 Bus error touch > /var/log/wtmp > /etc/rc.d/rc.sysinit: line 612: 1140 Bus error chgrp utmp > /var/run/utmp /var/log/wtmp > /etc/rc.d/rc.sysinit: line 613: 1141 Bus error chmod 0664 > /var/run/utmp /var/log/wtmp > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent > in inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth > 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: bad > header/extent in inode #5136: invalid magic - magic 0, entries 0, max > 0(0), depth 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: > bad header/extent in inode #5136: invalid magic - magic 0, entries 0, > max 0(0), depth 0(0) EXT4-fs error (device dm-0): > __ext4_ext_check_block: bad header/extent in inode #5136: invalid > magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error (device > dm-0): __ext4_ext_check_block: bad header/extent in inode #5136: > invalid magic - magic 0, entries 0, max 0(0), depth 0(0) EXT4-fs error > (device dm-0): __ext4_ext_check_block: bad header/extent in inode #5136: > invalid magic - magic 0, entries 0, max 0(0), depth 0(0) > /etc/rc.d/rc.sysinit: line 624: 1142 Bus error rm -f > /tmp/.X*-lock /tmp/.lock.* /tmp/.gdm_socket /tmp/.s.PGSQL.* > /etc/rc.d/rc.sysinit: line 628: 1143 Bus error rm -rf > /tmp/.X*-unix /tmp/.ICE-unix /tmp/.font-unix /tmp/hsperfdata_* > /tmp/kde-* /tmp/ksocket-* /tmp/mc-* /tmp/mcop-* /tmp/orbit-* > /tmp/scrollkeeper-* /tmp/ssh-* /dev/.in_sysinit > /etc/rc.d/rc.sysinit: line 631: 1144 Bus error mkdir -m > 1777 -p /tmp/.ICE-unix > /dev/null 2>&1 > /etc/rc.d/rc.sysinit: line 632: 1145 Bus error chown > root:root /tmp/.ICE-unix > Enabling /etc/fstab swaps: /etc/init.d/functions: line 568: 1147 Bus > error "$@" > [FAILED] > /etc/rc.d/rc.sysinit: line 651: 1149 Bus error /bin/mount > -t binfmt_misc none /proc/sys/fs/binfmt_misc > /dev/null 2>&1 > /etc/rc.d/rc.sysinit: line 666: 1150 Bus error mv -f > /var/log/dmesg /var/log/dmesg.old > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent > in inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth > 0(0) EXT4-fs error (device dm-0): __ext4_ext_check_block: bad > header/extent in inode #5136: invalid magic - magic 0, entries 0, max 0(0), > depth 0(0) > /etc/rc.d/rc.sysinit: line 667: 1151 Bus error dmesg -s > 131072 > /var/log/dmesg > /etc/rc.d/rc.sysinit: line 670: 1152 Bus error touch > /.autofsck &>/dev/null > DCVML99 : > EXT4-fs error (device dm-0): __ext4_ext_check_block: bad header/extent > in inode #5136: invalid magic - magic 0, entries 0, max 0(0), depth > 0(0) > init: rc main process (1158) killed by BUS signal > > Thanks.. > > Paul Feller > AGT Mainframe Technical Support > > -----Original Message----- > From: Feller, Paul > Sent: Wednesday, July 25, 2018 4:15 PM > To: 'Linux on 390 Port' <LINUX-390@VM.MARIST.EDU> > Subject: RE: Issue cloning RHEL server > > Mike, I'll get with the team member. He does more of the zLinux type > stuff and I do more of the z/VM type stuff. > > Thanks.. > > Paul Feller > AGT Mainframe Technical Support > > -----Original Message----- > From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of > Michael MacIsaac > Sent: Wednesday, July 25, 2018 1:19 PM > To: LINUX-390@VM.MARIST.EDU > Subject: Re: Issue cloning RHEL server > > Paul, > > Can you post the contents of the clone script? Is it based on the > code starting on p. 528? > > When you say " corrupts one or more of the copy's file systems", can > you be more specific? > > Thanks. > > -Mike > > On Wed, Jul 25, 2018 at 1:08 PM Feller, Paul > <paul.fel...@transamerica.com > > > wrote: > > > One of my team members is trying to run the cloning process > > described in "The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 and > > SLES 11 SP3 October 2013, January 2014 (SG-248147-00)". I believe > > the process has worked in the past. But as things go the > > environment has changed over > time. > > > > > > Here is the question from the team member. > > > > We are still using the cloning process documented in, and scripts > > provided with, "The Virtualization Cookbook for z/VM 6.3, RHEL 6.4 > > and SLES 11 SP3 October 2013, January 2014 (SG-248147-00)". It's > > been sometime since I last used the process to create a new zLinux > > system and when I try now I'm finding that the procedure to update > > the IP information on the clone copy corrupts one or more of the > > copy's file systems. I can use DDR to build a bootable system from > > the gold image so I'm fairly sure the dasd copy process is working > > ok. The gold image is not corrupted. It boots without an error and > > so does the DDR > copy. > > > > We're currently running z/VM 6.4 (RSU1801). The 'LNXADMIN' system > > is running Linux version 2.6.32-504.8.1.el6.s390x (Red Hat 4.4.7-4) > > and that's what we're cloning. Has anyone experienced this problem > > and found a work around? > > > > > > Thanks.. > > > > Paul Feller > > AGT Mainframe Technical Support > > > > > > > > > > -------------------------------------------------------------------- > > -- For LINUX-390 subscribe / signoff / archive access instructions, > > send email to lists...@vm.marist.edu with the message: INFO > > LINUX-390 or visit > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_h > > tb > > in_wlvindex-3FLINUX-2D390&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8x > > Wj > > ONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2w > > Ru 7ZzXBMAn_2Q&s=8YEDwSUUW6k1Eic6YXb-tpis6DEin60TNeH_D5rLyjM&e= > > -------------------------------------------------------------------- > > -- For more information on Linux on System z, visit > > https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org > > _& > > d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9 > > cc > > yeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=lgGYDNMH5l > > _L > > 4-JL7-a3S4TlQNxD4Qfyx6hB8rvH1zQ&e= > > > > > -- > -Mike MacIsaac > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, send > email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htb > in_wlvindex-3FLINUX-2D390&d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWj > ONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu > 7ZzXBMAn_2Q&s=8YEDwSUUW6k1Eic6YXb-tpis6DEin60TNeH_D5rLyjM&e= > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_&; > d=DwIBaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9cc > yeMRtzQHc&m=aAy6Wme7j7nn5JDewpSlMrLzWigB2wRu7ZzXBMAn_2Q&s=lgGYDNMH5l_L > 4-JL7-a3S4TlQNxD4Qfyx6hB8rvH1zQ&e= > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, send > email to lists...@vm.marist.edu with the message: INFO LINUX-390 or > visit > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htb > in_wlvindex-3FLINUX-2D390&d=DwIFaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWj > ONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=4k_izCyOFzgDPXe-P9bLEy4twS9z5tM7 > 46Po9pxKCtU&s=dnzwMogPKP83EFmXonTZIxLcUbt0ToT88sWXjRdzCq8&e= > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_&; > d=DwIFaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9cc > yeMRtzQHc&m=4k_izCyOFzgDPXe-P9bLEy4twS9z5tM746Po9pxKCtU&s=H2Gkkg0qTd4b > DffT9T7tdSfdyoMAgqfXy8Z_wyihLpc&e= > -- -Mike MacIsaac ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit https://urldefense.proofpoint.com/v2/url?u=http-3A__www.marist.edu_htbin_wlvindex-3FLINUX-2D390&d=DwIFaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=4k_izCyOFzgDPXe-P9bLEy4twS9z5tM746Po9pxKCtU&s=dnzwMogPKP83EFmXonTZIxLcUbt0ToT88sWXjRdzCq8&e= ---------------------------------------------------------------------- For more information on Linux on System z, visit https://urldefense.proofpoint.com/v2/url?u=http-3A__wiki.linuxvm.org_&d=DwIFaQ&c=9g4MJkl2VjLjS6R4ei18BA&r=Ikyt8xWjONqWHqSm_SmciXJmsnpmyFU9ccyeMRtzQHc&m=4k_izCyOFzgDPXe-P9bLEy4twS9z5tM746Po9pxKCtU&s=H2Gkkg0qTd4bDffT9T7tdSfdyoMAgqfXy8Z_wyihLpc&e= ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
