Thanks! I did some additional testing, and this seems to be an NFS read issue. Here's some more information:


This seems to be an NFS read issue for files that have hard links. I have tested the following scenarios with these results:

Local disk source to NFS archive = good
NFS source to NFS archive = broken
NFS source to local disk archive = broken

 
 
How to reproduce with some examples:
 
Create our folder with files that have hard links on an NFS mount. We will use this as our source for creating the archive. Copying /usr/bin from an image is a good one to test with.

[root@box3 bin]# pwd
/install/cpio-test/bin
 

[root@box3 bin]# df -k /install/cpio-test/bin

Filesystem           1K-blocks      Used Available Use% Mounted on
vault1:/ifs/hpc/box3/install
                     209149081088 121456335360 81445108224  60% /install

 

[root@box3 bin]# ll

total 61766
-rwxr-xr-x 1 root root     33408 Jun 22  2012 [
-rwxr-xr-x 1 root root    106792 Jun 22  2012 a2p
-rwxr-xr-x 1 root root     23488 Nov 11  2010 addftinfo
-rwxr-xr-x 1 root root     24904 Jun 22  2012 addr2line
-rwxr-xr-x 1 root root     56624 Jun 22  2012 ar
-rwxr-xr-x 1 root root    328392 Jun 22  2012 as
-rwxr-xr-x 1 root root     10400 Sep 23  2011 attr
<snip>
-rwxr-xr-x 1 root root      2605 Nov 11  2010 zmore
-rwxr-xr-x 1 root root      5246 Nov 11  2010 znew
lrwxrwxrwx 1 root root         6 May 14 09:13 zsoelim -> soelim
 
 
Take note that we have no 0 byte files in this folder

[root@box3 bin]# find . -type f -size 0b
[root@box3 bin]#


Create our CPIO archive, using the “NEWC” archive format. This format allows for a disk image with inode numbers greater than 65535.
 
[root@box3 bin]# find . | cpio -H newc -o --verbose > /install/cpio-test2/archive.cpio
.
./ar
./indxbib
./gnroff
./colcrt
./locale
<snip>
./tclsh8.5
./showkey
./a2p
43155 blocks
 

 Extract our CPIO archive

[root@box3 bin]# cd /install/cpio-test2

[root@box3 cpio-test2]# cpio -id < archive.cpio

43155 blocks

 
 Check our files. Notice that problem files are 0 byte size, and they also have hard links

 
[root@box3 cpio-test2]# stat gcc
  File: `gcc'
  Size: 0               Blocks: 3          IO Block: 524288 regular empty file
Device: 16h/22d Inode: 4373676595  Links: 2
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-05-14 13:07:00.258996240 -0700
Modify: 2013-05-14 13:07:00.258996240 -0700
Change: 2013-05-14 13:07:00.263049282 -0700
 

[root@box3 cpio-test2]# find . -type f -size 0b
./gcc
./perl
./x86_64-redhat-linux-gcc
./c2ph
./skill
./python
./pgrep
./sudoedit
./python2.6
./snice
./psed
./sudo
./s2p
./pstruct
./perlbug
./perlthanks
./perl5.10.1
./pkill

 

 It seems that all 0 byte files have hard links, and all hard linked files have 0 bytes:

[root@box3 cpio-test2]# for file in `find .`; do LINKS=`stat $file | grep Links | gawk '{print $6}'`; if [ $LINKS -gt 1 ]; then SIZE=`stat $file | grep Size`; echo "More than 1 link: $file   SIZE: $SIZE"; fi; done
 
More than 1 link: .   SIZE:   Size: 12665       Blocks: 260        IO Block: 524288 directory
More than 1 link: ./gcc   SIZE:   Size: 0               Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./perl   SIZE:   Size: 0              Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./x86_64-redhat-linux-gcc   SIZE:   Size: 0           Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./c2ph   SIZE:   Size: 0              Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./skill   SIZE:   Size: 0             Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./python   SIZE:   Size: 0            Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./pgrep   SIZE:   Size: 0             Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./sudoedit   SIZE:   Size: 0          Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./python2.6   SIZE:   Size: 0                 Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./snice   SIZE:   Size: 0             Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./psed   SIZE:   Size: 0              Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./sudo   SIZE:   Size: 0              Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./s2p   SIZE:   Size: 0               Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./pstruct   SIZE:   Size: 0           Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./perlbug   SIZE:   Size: 0           Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./perlthanks   SIZE:   Size: 0                Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./perl5.10.1   SIZE:   Size: 0                Blocks: 3          IO Block: 524288 regular empty file
More than 1 link: ./pkill   SIZE:   Size: 0             Blocks: 3          IO Block: 524288 regular empty file





On 5/16/2013 9:34 PM, Xiao Peng Wang wrote:

Russell,

    As you mentioned that 'cpio -H bin' works but 'cpio -H newc/crc' does not work, it should be the issue of cpio that has problem to work with NAS server.

    From xCAT point of view, maybe we could try to make the cpio write file to local instead of NFS sever.

    Could you change the 'cpio -H newc -o | gzip -c - > ../rootimg.gz' to 'cpio -H newc -o | gzip -c - > /tmp/rootimg.gz' or 'cpio -H newc -o > /tmp/rootimg.gz', and take a try that whether /tmp/rotimg.gz could get the correct file size?
    If yes, we could work around it that cpio output to local first, then copy it from local to remote /install/netboot/...

 
Thanks
Best Regards
----------------------------------------------------------------------
Wang Xiaopeng (王晓朋)
IBM China System Technology Laboratory
Tel: 86-10-82453455
Email: [email protected]
Address: 28,ZhongGuanCun Software Park,No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193


Inactive
        hide details for Russell Jones ---2013/05/15 00:47:59---On
        5/9/2013 8:54 AM, Xiao Peng Wang wrote: > If possible couldRussell Jones ---2013/05/15 00:47:59---On 5/9/2013 8:54 AM, Xiao Peng Wang wrote: > If possible could you try to do the following change wh

From: Russell Jones <[email protected]>
To: [email protected],
Date: 2013/05/15 00:47
Subject: Re: [xcat-user] xCAT 2.7 - packimage and 0 bytes files





On 5/9/2013 8:54 AM, Xiao Peng Wang wrote:
    If possible could you try to do the following change when you could recreate the issue on NetApp or Isilon. 
      Replace the '|cpio -H newc -o |' part with '|cpio -H newc -o --verbose 2>/tmp/cpiolog |' and to see what we could get in the '/tmp/cpiolog' when issue happens. 

    And please let us know the result of the trying of squashfs or the trying of James's suggestion that using 'cpio -o' replace 'cpio -H newc -o'.


Here's the results of my testing:

Redirecting errors to cpiolog with archive format as NEWC:
  • No errors recorded, but binaries are still 0 bytes. Example:

    [root@box3 bin]# pwd
    /install/netboot/centos6.3/x86_64/compute/rootimg/usr/bin

    [root@box3 bin]# ll gcc
    -rwxr-xr-x 2 root root 263824 Jun 22  2012 gcc


    (Extracted rootimg.gz here)

    [root@box3 bin]# pwd
    /install/netboot/centos6.3/x86_64/compute/tmp/usr/bin

    [root@box3 bin]# ll gcc

    -rwxr-xr-x 2 root root 0 May 13 10:38 gcc

[root@box3 bin]# find . -type f -size 0b
./gcc
./perl
./x86_64-redhat-linux-gcc
./c2ph
./skill
./python
./pgrep
./sudoedit
./python2.6
./snice
./psed
./sudo
./s2p
./pstruct
./perlbug
./perlthanks
./perl5.10.1
./pkill

  • Output from /tmp/cpiolog for that file. No errors recorded, and no errors around it. I surfed it and saw nothing out of the ordinary in this log, just a list of files it packed and the final size at the end.
    ./usr/bin/bzip2
    ./usr/bin/factor

    ./usr/bin/gcc

    ./usr/bin/mpdsigjob
    ./usr/bin/yes

Changing archive format to BIN (default in copy-out mode for cpio in CentOS 6) in packimage plugin

  • Seems to work fine:

    [root@box3 bin]# pwd
    /install/netboot/centos6.3/x86_64/compute/tmp2/usr/bin

    [root@box3 bin]# ll gcc
    -rwxr-xr-x 2 root root 263824 May 13 10:55 gcc

    [root@box3 bin]# find . -type f -size 0b
    [root@box3 bin]#

  • According to the man page the bin format has a limit of 65,000 inodes though, so if the rootimg is going to need to have a lot of small files in it, it's not practical.

I also tried the "crc" format, which according to the man page is NEWC with checksums added, and it showed the same problem - 0 byte files.


I have not tried squashfs yet as I am working out the best way of getting the aufs module compiled. I expect it to work fine however.
------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user



------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d


_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user


------------------------------------------------------------------------------
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user

Reply via email to