Re: [gentoo-user] Ext3 FS File Size Limits
On Sun, 2 Jun 2013 13:01:22 + Alan Mackenzie a...@muc.de wrote: On Sun, Jun 02, 2013 at 04:55:40AM -0700, Fast Turtle wrote: I've been going around with this little problem for a while. I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped. One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there? I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems [defaults] base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr default_mntopts = acl,user_xattr enable_periodic_fsck = 0 blocksize = 4096 inode_size = 256 inode_ratio = 16384 Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum file size is 16GB, but with a 2kB blocksize it's 256GB. Could it be you've somehow actually got a 1kB blocksize on the partition? I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. Sounds like laziness on the part of the author of the copy program you're using. if it is, then everyone is in trouble as it's the cp command - been part of *nix for many years -- Alan Mackenzie (Nuremberg, Germany).
Re: [gentoo-user] Ext3 FS File Size Limits
On Sun, 2 Jun 2013 13:01:22 + Alan Mackenzie a...@muc.de wrote: On Sun, Jun 02, 2013 at 04:55:40AM -0700, Fast Turtle wrote: I've been going around with this little problem for a while. I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped. One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there? I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems [defaults] base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr default_mntopts = acl,user_xattr enable_periodic_fsck = 0 blocksize = 4096 inode_size = 256 inode_ratio = 16384 Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum file size is 16GB, but with a 2kB blocksize it's 256GB. Could it be you've somehow actually got a 1kB blocksize on the partition? I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. Sounds like laziness on the part of the author of the copy program you're using. -- Alan Mackenzie (Nuremberg, Germany). Thanks for the wikipedia info (could very well be the problem) Don't recall if I used 1k block sizes on the partition but it could be. Would certainly explain the failure right on 16GB - filesize was to large (never had problems copying between ntfs formatted drives under Windows).
Re: [gentoo-user] Ext3 FS File Size Limits
On Sun, 02 Jun 2013 07:11:11 -0500 Dale rdalek1...@gmail.com wrote: Fast Turtle wrote: I've been going around with this little problem for a while. I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped. One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there? I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems [defaults] base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr default_mntopts = acl,user_xattr enable_periodic_fsck = 0 blocksize = 4096 inode_size = 256 inode_ratio = 16384 Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. I can offer this: df -i shows inodes. root@fireball / # df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 1525920 22728 15031922% / /dev/sda6 1525920 22728 15031922% / devtmpfs2049540593 20489471% /dev tmpfs 2058249654 20575951% /run shm 2058249 2 20582471% /dev/shm /dev/sda1 98392794 975981% /boot /dev/mapper/OS-usr 1638400 462712 1175688 29% /usr /dev/mapper/OS-var 1703936 259049 1444887 16% /var /dev/mapper/home-home 183148544 316215 1828323291% /home /dev/mapper/backup-backup 61046784 5818 610409661% /backup tmpfs 2058249 122993 19352566% /var/tmp/portage root@fireball / # Hope that helps on that part at least. Dale :-) :-) -- I am only responsible for what I said ... Not for what you understood or how you interpreted my words! Thanks Dale - Interesting what info you can get All: Accidently discovered that tune2fs -l /dev/sda# lists all of the config info so I shouldn't have any problems Filesystem volume name: home Last mounted on: not available Filesystem UUID: d4102f68-defd-497b-84e4-f165fd171ed7 Filesystem magic number: 0xEF53 Filesystem revision #:1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file Filesystem flags: signed_directory_hash Default mount options:user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 39075840 Block count: 156282705 Reserved block count: 7814135 Free blocks: 104269552 Free inodes: 38399824 First block: 0 Block size: 4096 Fragment size:4096 Had KDE lockup the other day and it forced a clean install w/o it. Seems I inadvertently formatted /home using the standard 4k block size this time around. Would explain why the 30GB files copied over w/o issue this time I'll repost this: Thanks for the Wiki Entry Alan (keep forgetting to check it) as that certainly explains the issue and I'm going to put that into my System Log book (important notes section). When I started playing with Linux back in 2000, I never thought much about file size limits as ext2 could handle files larger then Win98 could. Now it's come full circle as you do have to be aware of the formatting because it can affect file size limits as I discovered.
Re: [gentoo-user] Ext3 FS File Size Limits
Fast Turtle wrote: I've been going around with this little problem for a while. I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped. One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there? I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems [defaults] base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr default_mntopts = acl,user_xattr enable_periodic_fsck = 0 blocksize = 4096 inode_size = 256 inode_ratio = 16384 Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. I can offer this: df -i shows inodes. root@fireball / # df -i Filesystem Inodes IUsed IFree IUse% Mounted on rootfs 1525920 22728 15031922% / /dev/sda6 1525920 22728 15031922% / devtmpfs2049540593 20489471% /dev tmpfs 2058249654 20575951% /run shm 2058249 2 20582471% /dev/shm /dev/sda1 98392794 975981% /boot /dev/mapper/OS-usr 1638400 462712 1175688 29% /usr /dev/mapper/OS-var 1703936 259049 1444887 16% /var /dev/mapper/home-home 183148544 316215 1828323291% /home /dev/mapper/backup-backup 61046784 5818 610409661% /backup tmpfs 2058249 122993 19352566% /var/tmp/portage root@fireball / # Hope that helps on that part at least. Dale :-) :-) -- I am only responsible for what I said ... Not for what you understood or how you interpreted my words!
Re: [gentoo-user] Ext3 FS File Size Limits
2013/6/2 Fast Turtle ftur...@gmail.com: One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there? You can try `touch`ing a file on that partition and see if it fails. If so, you've run out of inodes; no need to peek at fs internals :)
Re: [gentoo-user] Ext3 FS File Size Limits
On Sun, Jun 02, 2013 at 04:55:40AM -0700, Fast Turtle wrote: I've been going around with this little problem for a while. I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped. One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there? I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems [defaults] base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr default_mntopts = acl,user_xattr enable_periodic_fsck = 0 blocksize = 4096 inode_size = 256 inode_ratio = 16384 Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum file size is 16GB, but with a 2kB blocksize it's 256GB. Could it be you've somehow actually got a 1kB blocksize on the partition? I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. Sounds like laziness on the part of the author of the copy program you're using. -- Alan Mackenzie (Nuremberg, Germany).
Re: [gentoo-user] Ext3 FS File Size Limits
On Sun, 2 Jun 2013 04:55:40 -0700, Fast Turtle wrote: I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped. Are you sure its the destination causing the problem? What happens if you try to copy to another directory on the NTFS filesystem? -- Neil Bothwick Mosquito - designed to make houseflies look better. signature.asc Description: PGP signature
Re: [gentoo-user] Ext3 FS File Size Limits
On Sun, Jun 02, 2013 at 04:55:40AM -0700, Fast Turtle wrote: I've been going around with this little problem for a while. I have several 30GB files I'm trying to restore from an NTFS formatted external backup to an ext3 partition, yet every attempt has failed right after 16GB of copying without fail or error message. They silently failing and I'm stumped. One of the possible causes I've thought of was running out of innodes but don't know how to check that or any of the other options used to create the file system on - anyone want to help there? I've also decided to look at the mke2f.conf file in /etc and see some default options being passed that may be causing the problems [defaults] base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr default_mntopts = acl,user_xattr enable_periodic_fsck = 0 blocksize = 4096 inode_size = 256 inode_ratio = 16384 Normally I use either a 1024 for most everything due to the many small files though for the partition I'm attempting to restore the files to, I've used 2048 as a compromise due to the number of larger files (music/videos) and critical backups from /etc I've also tried it with a default 4096 size on a 32GB ext2 formatted flash drive but even then, it's failing at 16GB w/o any error message. What command are you using to copy this data? Try: rsync -av /source/ /destination/ and see if you get any output when it fails. There have been other guesses, but with the information you've provided, that's all we can do. My guess is you're using cp and it gets to a large, corrupt file and fails. Post back with your command and it's output, then the rsync command above and it's output; as well as the inode command Dale gave you, and any other applicable commands you've had suggested. -- Happy Penguin Computers ') 126 Fenco Drive ( \ Tupelo, MS 38801 ^^ supp...@happypenguincomputers.com 662-269-2706 662-205-6424 http://happypenguincomputers.com/ A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting
Re: [gentoo-user] Ext3 FS File Size Limits
Alan Mackenzie a...@muc.de wrote: The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum file size is 16GB, but with a 2kB blocksize it's 256GB. Could it be you've somehow actually got a 1kB blocksize on the partition? Where does such a strange limitation come from? Ext* started as a UFS clone and UFS filesize is limited to 2**63 while UFS filesystem size is limited to 1 TB. That is much more than you claim for Ext3 Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Re: [gentoo-user] Ext3 FS File Size Limits
Hello, Jörg. On Sun, Jun 02, 2013 at 04:06:11PM +0200, Joerg Schilling wrote: Alan Mackenzie a...@muc.de wrote: The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum file size is 16GB, but with a 2kB blocksize it's 256GB. Could it be you've somehow actually got a 1kB blocksize on the partition? Where does such a strange limitation come from? Haven't a clue. I would have expected the maximum file size to be a number of blocks, which makes it seem strange that doubling the block size multiplies max file size by 16. Ext* started as a UFS clone and UFS filesize is limited to 2**63 while UFS filesystem size is limited to 1 TB. Just for ease of comparison, 16GB = 2**34 bytes = 2**24 1k blocks. 1TB = 2**40 bytes. That is much more than you claim for Ext3 I'm not doing any claiming, since I'm not an expert on the subject. I was just drawing the OP's attention to something which might be useful. Jörg -- Alan Mackenzie (Nuremberg, Germany).
Re: [gentoo-user] Ext3 FS File Size Limits
On Sun, Jun 02 2013, Alan Mackenzie wrote: Hello, Jörg. On Sun, Jun 02, 2013 at 04:06:11PM +0200, Joerg Schilling wrote: Alan Mackenzie a...@muc.de wrote: The wikipedia page on Ext3 says that with a 1kB blocksize, the maximum file size is 16GB, but with a 2kB blocksize it's 256GB. Could it be you've somehow actually got a 1kB blocksize on the partition? Where does such a strange limitation come from? Haven't a clue. I would have expected the maximum file size to be a number of blocks, which makes it seem strange that doubling the block size multiplies max file size by 16. The wonders of indirect blocks. When the blocksize doubles, a single indirect block points to twice as many direct blocks, each of which is double the size. For double indirect you get twice as many single indirect, etc. There are plenty of places to read about this. One is my lecture notes http://cs.nyu.edu/~gottlieb/courses/os/class-notes.html#inodes allan
Re: [gentoo-user] Ext3 FS File Size Limits
AM == Alan Mackenzie a...@muc.de writes: AM Haven't a clue. I would have expected the maximum file size to be a AM number of blocks, which makes it seem strange that doubling the block AM size multiplies max file size by 16. Doubling the block size means that the structure specifying which blocks are in use by a given inode is twice as long. And the structure supports indirect blocks, which contain references to the actual blocks used by the file data. So you get one doubling for the size of the blocks, and another three for the number of blocks each file can use. Most ext3 and ext4 filesystems use 4096 octet blocks. -JimC -- James Cloos cl...@jhcloos.com OpenPGP: 1024D/ED7DAEA6
