ext2fs read errors
Hello misc, I'm migrating my data from an ext3 partition (formatted under Debian 6.0, sparc64) to my new i386 OBSD system. When copying I found out that some files weren't copied correctly and returned the error: read error: Invalid argument. The files are usable when mounting the disk under i386 Debian, so it's not an cpu-architectural problem. Other programs trying to operate on these files via ext2fs also fail with the same notion, (e.g. md5). And extracting these files from a tarball also result in the same error. The ls command does succeed and shows the correct information (file-size, access-time, etc). When moving these files over via nfs the problem doesn't occur and the files are saved correctly on my ffs partition. How can I trace this problem, so I can fill in a decent bug-report? Sincerely, Martijn van Duren
Re: ext2fs read errors
On Dec 30 10:43:00, m.vandu...@jonker.nl wrote: I'm migrating my data from an ext3 partition (formatted under Debian 6.0, sparc64) to my new i386 OBSD system. You need to give more detail. You installed an i386 obsd machine, and did what? Took an ext3fs disk out of a Debian sparc64 machine, put it in the new obsd machine? How did you mount it? How exactly are you migrating it? When copying I found out that some files weren't copied correctly and returned the error: read error: Invalid argument. *Some* files. Other files copied correctly? Can you tell a pattern of which was which? The files are usable when mounting the disk under i386 Debian, so it's not an cpu-architectural problem. So, you put the same sparc64 disk into an i386 Debian machine, and did what exactly? Copied the whole thing over, with cp, without a read error? Other programs trying to operate on these files via ext2fs also fail with the same notion, (e.g. md5). And extracting these files from a tarball also result in the same error. What tarball? The ls command does succeed and shows the correct information (file-size, access-time, etc). When moving these files over via nfs the problem doesn't occur and the files are saved correctly on my ffs partition. That (or scp) is how I always copied files from one FS/OS/arch to a completely different FS/OS/arch.
Re: ext2fs read errors
Jan Stary schreef op zo 30-12-2012 om 12:24 [+0100]: On Dec 30 10:43:00, m.vandu...@jonker.nl wrote: I'm migrating my data from an ext3 partition (formatted under Debian 6.0, sparc64) to my new i386 OBSD system. You need to give more detail. You installed an i386 obsd machine, and did what? Took an ext3fs disk out of a Debian sparc64 machine, put it in the new obsd machine? How did you mount it? How exactly are you migrating it? That is correct. And I mounted it mount_ext2fs /dev/wd0i /mnt. The migration was just a simple cp command, but like I said, it even fails a simple md5 command. When copying I found out that some files weren't copied correctly and returned the error: read error: Invalid argument. *Some* files. Other files copied correctly? Can you tell a pattern of which was which? I haven't found any pattern in the files which fail and which succeed. There are no special characters in the file-names and the files can be successfully accessed via an FFS partition once copied. And the content of a file should never be an issue when trying to retrieve data. The files are usable when mounting the disk under i386 Debian, so it's not an cpu-architectural problem. So, you put the same sparc64 disk into an i386 Debian machine, and did what exactly? Copied the whole thing over, with cp, without a read error? I booted the same i386 machine with a debian live-disk and I could access the disk without any problems. After that I placed the disk back in my sparc64 box and configured NFS to transfer the data to my OBSD system. Other programs trying to operate on these files via ext2fs also fail with the same notion, (e.g. md5). And extracting these files from a tarball also result in the same error. What tarball? I also tried placing the corrupted files in a tarball under Debian to see if it could have anything to do with file-names or whatever, but once the corrupted files were reached it gave me the same read error. (this was also done on the ext3 fs) The ls command does succeed and shows the correct information (file-size, access-time, etc). When moving these files over via nfs the problem doesn't occur and the files are saved correctly on my ffs partition. That (or scp) is how I always copied files from one FS/OS/arch to a completely different FS/OS/arch. And my point isn't the migration of my data. There is a work-around so I already fixed that. My point is that there apparently is a bug in the ext2fs driver and I want to locate the source of the problem, but I don't know how (yet). I kept my old disk still available for debugging purposes, but I will want to format it to ffs someday in the future. So I want to look into this as soon as possible.
Re: ext2fs read errors
Other programs trying to operate on these files via ext2fs also fail with the same notion, (e.g. md5). And extracting these files from a tarball also result in the same error. What tarball? I also tried placing the corrupted files in a tarball under Debian to see if it could have anything to do with file-names or whatever, but once the corrupted files were reached it gave me the same read error. (this was also done on the ext3 fs) Sigh, files were reached. Meaning what? As soon as the tar'ing Debian system got to them, while creating the tarball, it failed with a read error? Do you mean to say you already have read problems on the (source) Debian system? Or as soon as the untar'ing OpenBSD system reached those files while reading the tarball, it did what? Failing to read the tarball is not the same error as failing to read the original files. You need to tell us what _exactly_ you did, and what _exactly_ happened; preferably with a script(1). When moving these files over via nfs the problem doesn't occur and the files are saved correctly on my ffs partition. That (or scp) is how I always copied files from one FS/OS/arch to a completely different FS/OS/arch. And my point isn't the migration of my data. There is a work-around so I already fixed that. That's not a workaround. You cannot take a disk holding a certain filesystem from a certain OS on a certain architecture, put it into a different machine of a different architecture, running a different OS, mount it as a different filesystem, and just expect it to work. Going through a defined protocol such as NFS of SCP is the normal way.
Re: ext2fs read errors
On Sun, Dec 30, 2012 at 12:54 PM, Martijn van Duren m.vandu...@jonker.nl wrote: Jan Stary schreef op zo 30-12-2012 om 12:24 [+0100]: On Dec 30 10:43:00, m.vandu...@jonker.nl wrote: I'm migrating my data from an ext3 partition [...] snip That is correct. And I mounted it mount_ext2fs /dev/wd0i /mnt. Why would you expect an ext3fs partition to be working properly using ext2fs tools? The man pages for the tools involved do not mention ext3fs support or its journal features. Can you reproduce the issue with an ext2fs filesystem as well? Regards, Rogier
Re: ext2fs read errors
Jan Stary schreef op zo 30-12-2012 om 13:49 [+0100]: Other programs trying to operate on these files via ext2fs also fail with the same notion, (e.g. md5). And extracting these files from a tarball also result in the same error. What tarball? I also tried placing the corrupted files in a tarball under Debian to see if it could have anything to do with file-names or whatever, but once the corrupted files were reached it gave me the same read error. (this was also done on the ext3 fs) Sigh, files were reached. Meaning what? As soon as the tar'ing Debian system got to them, while creating the tarball, it failed with a read error? Do you mean to say you already have read problems on the (source) Debian system? Or as soon as the untar'ing OpenBSD system reached those files while reading the tarball, it did what? Failing to read the tarball is not the same error as failing to read the original files. You need to tell us what _exactly_ you did, and what _exactly_ happened; preferably with a script(1). What I did was as follow: martijn@debian:~$ tar -cf /path/to/e2fs/archive.tar \ /path/to/e2fs/sanefile /path/to/e2fs/corruptfile martijn@debian:~$ md5sum /path/to/e2fs/sanefile d93b7c75f55c4de770c3c038a2fef3a6 /path/to/e2fs/sanefile martijn@debian:~$ md5sum /path/to/e2fs/corruptfile 8a8535e6c0cbd85c0cfedb30606b9d71 /path/to/e2fs/corruptfile attach disk to OBSD martijn@OBSD$ tar -xf /path/to/e2fs/archive.tar tar: Failed read on archive volume 1: Invalid argument tar: Attempting to recover from an archive read failure. martijn@OBSD$ md5 /path/to/e2fs/corruptfile md5: /path/to/e2fs/corruptfile: read error: Invalid argument martijn@OBSD$ md5 /path/to/e2fs/sanefile MD5: (/path/to/e2fs/sanefile) = d93b7c75f55c4de770c3c038a2fef3a6 Of course the archive and corruptfile works correct on my Linux systems. When moving these files over via nfs the problem doesn't occur and the files are saved correctly on my ffs partition. That (or scp) is how I always copied files from one FS/OS/arch to a completely different FS/OS/arch. And my point isn't the migration of my data. There is a work-around so I already fixed that. That's not a workaround. You cannot take a disk holding a certain filesystem from a certain OS on a certain architecture, put it into a different machine of a different architecture, running a different OS, mount it as a different filesystem, and just expect it to work. Going through a defined protocol such as NFS of SCP is the normal way. This should not be an issue (this is also my response to Rogier). Ext3 is nothing more than ext2 with extra journaling features enabled, so the disk should be able to be usable (for the majority of the files it is). It should also be architecture independent since everything is stored in little endian bite-order.[1] If a filesystem isn't a defined protocol then it shouldn't be offered as a mountable filesystem. And your response still hasn't given me an answer to my question. How can I trace where/how these read-errors occur? [1] http://disktype.sourceforge.net/doc/ch03s05.html#id894875
Re: ext2fs read errors
When moving these files over via nfs the problem doesn't occur and the files are saved correctly on my ffs partition. That (or scp) is how I always copied files from one FS/OS/arch to a completely different FS/OS/arch. And my point isn't the migration of my data. There is a work-around so I already fixed that. That's not a workaround. You cannot take a disk holding a certain filesystem from a certain OS on a certain architecture, put it into a different machine of a different architecture, running a different OS, mount it as a different filesystem, and just expect it to work. Going through a defined protocol such as NFS of SCP is the normal way. This should not be an issue (this is also my response to Rogier). Ext3 is nothing more than ext2 with extra journaling features enabled, So in particular, the ext3 inode structure is precisely the ext2 inode structure? If a filesystem isn't a defined protocol then it shouldn't be offered as a mountable filesystem. Nobody is offering ext3 as a mountable filesystem on OpenBSD.
Re: ext2fs read errors
Nobody is offering ext3 as a mountable filesystem on OpenBSD. http://www.openbsd.org/faq/faq8.html#Journaling http://www.openbsd.org/faq/faq14.html#foreignfs
Re: ext2fs read errors
Jan Stary schreef op zo 30-12-2012 om 15:36 [+0100]: When moving these files over via nfs the problem doesn't occur and the files are saved correctly on my ffs partition. That (or scp) is how I always copied files from one FS/OS/arch to a completely different FS/OS/arch. And my point isn't the migration of my data. There is a work-around so I already fixed that. That's not a workaround. You cannot take a disk holding a certain filesystem from a certain OS on a certain architecture, put it into a different machine of a different architecture, running a different OS, mount it as a different filesystem, and just expect it to work. Going through a defined protocol such as NFS of SCP is the normal way. This should not be an issue (this is also my response to Rogier). Ext3 is nothing more than ext2 with extra journaling features enabled, So in particular, the ext3 inode structure is precisely the ext2 inode structure? Yes [1] If a filesystem isn't a defined protocol then it shouldn't be offered as a mountable filesystem. Nobody is offering ext3 as a mountable filesystem on OpenBSD. And that's why I'm mounting it as an ext2 filesystem and not as a journaled ext3. So until you can point me to some (backwards-)incompatible differences between the two filesystems I'm convinced that there shouldn't be a problem and I want to find out what causes this anomaly. [1] http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html Q: What is ext3? Q: How do I convert my ext2 partition to ext3? (was: How do I use ext3?)
Re: ext2fs read errors
On Sun, 30 Dec 2012 15:36:39 +0100 Jan Stary h...@stare.cz wrote: This should not be an issue (this is also my response to Rogier). Ext3 is nothing more than ext2 with extra journaling features enabled, So in particular, the ext3 inode structure is precisely the ext2 inode structure? I know the defaults for inodes on linux have to be changed with -I 128 these days to be compatible with OpenBSD. Also maybe it's just ext4 but I thought it was ext3 too that requires file carving rather than just recovering deleted files which would suggest greater differences?
Re: ext2fs read errors
Kevin Chadwick schreef op zo 30-12-2012 om 15:37 [+]: On Sun, 30 Dec 2012 15:36:39 +0100 Jan Stary h...@stare.cz wrote: This should not be an issue (this is also my response to Rogier). Ext3 is nothing more than ext2 with extra journaling features enabled, So in particular, the ext3 inode structure is precisely the ext2 inode structure? I know the defaults for inodes on linux have to be changed with -I 128 these days to be compatible with OpenBSD. The inodesize is indeed 256 of my ext3 filesystem. So if the only accepted inodesize is 128 that could explain a thing or to. Although this isn't mentioned in mount_ext2fs(8) and the filesystem is mount- and usable for the major part, which shouldn't usually be the case with an incorrect inodesize. Most files are also of approximately the same size, which rules out different inodecounts per file. I also found an old threat[1] where they say they have a patch for accessing ext2 partitions with a different inodesize then 128, although I can't find any information of what ever happened with that patch. Also maybe it's just ext4 but I thought it was ext3 too that requires file carving rather than just recovering deleted files which would suggest greater differences? I don't know much about file carving, but what for what I understand of it, it's not possible to do file carving with ext3 and/or ext4, because it leverages on file signatures to restore a file and this is usually overwritten by the journal (please, correct me when I'm wrong). So this shouldn't change the inode layout needed by e2fs to operate on the existing files. (deleted files aren't of my interest at this point). [1] http://openbsd.7691.n7.nabble.com/ext2-with-inode-size-other-than-128b-td165287.html
Re: ext2fs read errors
On 2012 Dec 30 (Sun) at 16:26:01 +0100 (+0100), Martijn van Duren wrote: :Jan Stary schreef op zo 30-12-2012 om 15:36 [+0100]: : When moving these files over via nfs the problem doesn't occur and the : files are saved correctly on my ffs partition. : : That (or scp) is how I always copied files : from one FS/OS/arch to a completely different FS/OS/arch. : :And my point isn't the migration of my data. There is a work-around so I :already fixed that. : : That's not a workaround. You cannot take a disk holding : a certain filesystem from a certain OS on a certain architecture, : put it into a different machine of a different architecture, : running a different OS, mount it as a different filesystem, : and just expect it to work. Going through a defined protocol : such as NFS of SCP is the normal way. : : This should not be an issue (this is also my response to Rogier). Ext3 : is nothing more than ext2 with extra journaling features enabled, : : So in particular, the ext3 inode structure : is precisely the ext2 inode structure? : :Yes [1] : : : If a filesystem isn't a defined protocol then it shouldn't be offered : as a mountable filesystem. : : Nobody is offering ext3 as a mountable filesystem on OpenBSD. : : :And that's why I'm mounting it as an ext2 filesystem and not as a :journaled ext3. :So until you can point me to some (backwards-)incompatible differences :between the two filesystems I'm convinced that there shouldn't be a :problem and I want to find out what causes this anomaly. : Feel free to submit patches. We are certainly interested in better 3rd party filesystem support. -- You can get more of what you want with a kind word and a gun than you can with just a kind word. -- Bumper Sticker
Re: ext2fs read errors
Martijn van Duren schreef op zo 30-12-2012 om 17:15 [+0100]: I also found an old threat[1] where they say they have a patch for accessing ext2 partitions with a different inodesize then 128, although I can't find any information of what ever happened with that patch. On some further investigation I found that big inodesizes have indeed been implemented.[1][2][3] This explains why I can mount the filesystem and use most of the files, but doesn't answer my question where the read request go haywire or how I can actually debug this issue myself. (I don't know how to debug/trace read(2), so it would be highly appreciated if someone can explain me how to do this, or point me to the documentation that explains me how to do this.) [1]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_dinode.h#rev1.11 [2]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_inode.c#rev1.43 [3]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_vfsops.c#rev1.51
Re: ext2fs read errors
On Sun, Dec 30, 2012 at 05:46:54PM +0100, Martijn van Duren wrote: Martijn van Duren schreef op zo 30-12-2012 om 17:15 [+0100]: I also found an old threat[1] where they say they have a patch for accessing ext2 partitions with a different inodesize then 128, although I can't find any information of what ever happened with that patch. On some further investigation I found that big inodesizes have indeed been implemented.[1][2][3] This explains why I can mount the filesystem and use most of the files, but doesn't answer my question where the read request go haywire or how I can actually debug this issue myself. (I don't know how to debug/trace read(2), so it would be highly appreciated if someone can explain me how to do this, or point me to the documentation that explains me how to do this.) [1]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_dinode.h#rev1.11 [2]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_inode.c#rev1.43 [3]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_vfsops.c#rev1.51 I've a ext3 partition and use this on OpenBSD each day. I don't know the inodesize of my partition, but I used the defaults for create it approximately a year ago. Can you run a fsck pass to the partition on Linux? Maybe something is wrong. -- Juan Francisco Cantero Hurtado http://juanfra.info
Re: ext2fs read errors
Juan Francisco Cantero Hurtado schreef op zo 30-12-2012 om 19:26 [+0100]: On Sun, Dec 30, 2012 at 05:46:54PM +0100, Martijn van Duren wrote: Martijn van Duren schreef op zo 30-12-2012 om 17:15 [+0100]: I also found an old threat[1] where they say they have a patch for accessing ext2 partitions with a different inodesize then 128, although I can't find any information of what ever happened with that patch. On some further investigation I found that big inodesizes have indeed been implemented.[1][2][3] This explains why I can mount the filesystem and use most of the files, but doesn't answer my question where the read request go haywire or how I can actually debug this issue myself. (I don't know how to debug/trace read(2), so it would be highly appreciated if someone can explain me how to do this, or point me to the documentation that explains me how to do this.) [1]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_dinode.h#rev1.11 [2]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_inode.c#rev1.43 [3]http://www.openbsd.org/cgi-bin/cvsweb/src/sys/ufs/ext2fs/ext2fs_vfsops.c#rev1.51 I've a ext3 partition and use this on OpenBSD each day. I don't know the inodesize of my partition, but I used the defaults for create it approximately a year ago. Can you run a fsck pass to the partition on Linux? Maybe something is wrong. I've already did a fsck beforehand. It's completely clean. In the meantime I've made a minor test-program which reads one byte at a time. With one of my files it fails at byte 49153 and there's nothing special about the returned byte (tested by the file I've transferred via nfs). Ktrace(1) also doesn't show me anything interesting concerning this issue. Just that a read-command has failed. So I really need a way to debug filesystem drivers.
