I did some tests this weekend. I made a copy of the test directory into a second HD and I did write a small compare utility to check how many bytes are different for each file. I've see that there's always just one byte different between files, it's always the same bit wrong and always of big files, while files not matching varies between compare executions.
First compare of "pictures" directory files: files differ: byte 0000025d4862, 8a != 82 (08), file1: /disk1/pictures/photo/movie/2011/2011-03-06_14-57-43.mov, file2: /opt/pictures/photo/movie/2011/2011-03-06_14-57-43.mov files differ: byte 000011673a62, 68 != 60 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-01_10-11-31.mov, file2: /opt/pictures/photo/movie/2012/2012-02-01_10-11-31.mov files differ: byte 0000074dd4e2, 7b != 73 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov files differ: byte 000025749162, d8 != d0 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov files differ: byte 000025749a62, e9 != e1 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov files differ: byte 00000a469962, 39 != 31 (08), file1: /disk1/pictures/photo/movie/2012/2012-04-03_07-39-47.mov, file2: /opt/pictures/photo/movie/2012/2012-04-03_07-39-47.mov files differ: byte 00000b608162, 9e != 96 (08), file1: /disk1/pictures/photo/movie/2012/2012-12-30_19-24-49.mov, file2: /opt/pictures/photo/movie/2012/2012-12-30_19-24-49.mov files differ: byte 0000048301e2, 6a != 62 (08), file1: /disk1/pictures/photo/movie/2013/2013-01-13_20-57-28.mov, file2: /opt/pictures/photo/movie/2013/2013-01-13_20-57-28.mov files differ: byte 00000598a4e2, 8b != 83 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-07_06-24-19.mov, file2: /opt/pictures/photo/movie/2013/2013-02-07_06-24-19.mov files differ: byte 0000087194e2, fb != f3 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-13_06-09-45.mov, file2: /opt/pictures/photo/movie/2013/2013-02-13_06-09-45.mov files differ: byte 000014bec162, ca != c2 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-14_13-29-22.mov, file2: /opt/pictures/photo/movie/2013/2013-02-14_13-29-22.mov files differ: byte 00001b82d462, 2a != 22 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-14_13-38-16.mov, file2: /opt/pictures/photo/movie/2013/2013-02-14_13-38-16.mov files differ: byte 0000063e74e2, 58 != 50 (08), file1: /disk1/pictures/photo/movie/2013/2013-05-23_16-42-36.mov, file2: /opt/pictures/photo/movie/2013/2013-05-23_16-42-36.mov files differ: byte 00000f5b5162, 4b != 43 (08), file1: /disk1/pictures/photo/movie/2013/2013-08-18_14-43-36.mov, file2: /opt/pictures/photo/movie/2013/2013-08-18_14-43-36.mov files differ: byte 00000d82fa62, 7f != 77 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-05-37.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-05-37.mov files differ: byte 0000046134e2, ff != f7 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-24-52.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-24-52.mov files differ: byte 00001bad84e2, 3b != 33 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-24-52.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-24-52.mov Second compare: files differ: byte 0000025d4862, 8a != 82 (08), file1: /disk1/pictures/photo/movie/2011/2011-03-06_14-57-43.mov, file2: /opt/pictures/photo/movie/2011/2011-03-06_14-57-43.mov files differ: byte 000011673a62, 68 != 60 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-01_10-11-31.mov, file2: /opt/pictures/photo/movie/2012/2012-02-01_10-11-31.mov files differ: byte 0000074dd4e2, 7b != 73 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov files differ: byte 000025749162, d8 != d0 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov files differ: byte 000025749a62, e9 != e1 (08), file1: /disk1/pictures/photo/movie/2012/2012-02-26_04-30-24.mov, file2: /opt/pictures/photo/movie/2012/2012-02-26_04-30-24.mov files differ: byte 00000a469962, 39 != 31 (08), file1: /disk1/pictures/photo/movie/2012/2012-04-03_07-39-47.mov, file2: /opt/pictures/photo/movie/2012/2012-04-03_07-39-47.mov files differ: byte 00000b608162, 9e != 96 (08), file1: /disk1/pictures/photo/movie/2012/2012-12-30_19-24-49.mov, file2: /opt/pictures/photo/movie/2012/2012-12-30_19-24-49.mov files differ: byte 0000048301e2, 6a != 62 (08), file1: /disk1/pictures/photo/movie/2013/2013-01-13_20-57-28.mov, file2: /opt/pictures/photo/movie/2013/2013-01-13_20-57-28.mov files differ: byte 00000598a4e2, 8b != 83 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-07_06-24-19.mov, file2: /opt/pictures/photo/movie/2013/2013-02-07_06-24-19.mov files differ: byte 0000087194e2, fb != f3 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-13_06-09-45.mov, file2: /opt/pictures/photo/movie/2013/2013-02-13_06-09-45.mov files differ: byte 000014bec162, ca != c2 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-14_13-29-22.mov, file2: /opt/pictures/photo/movie/2013/2013-02-14_13-29-22.mov files differ: byte 00001b82d462, 2a != 22 (08), file1: /disk1/pictures/photo/movie/2013/2013-02-14_13-38-16.mov, file2: /opt/pictures/photo/movie/2013/2013-02-14_13-38-16.mov files differ: byte 0000063e74e2, 58 != 50 (08), file1: /disk1/pictures/photo/movie/2013/2013-05-23_16-42-36.mov, file2: /opt/pictures/photo/movie/2013/2013-05-23_16-42-36.mov files differ: byte 00000f5b5162, 4b != 43 (08), file1: /disk1/pictures/photo/movie/2013/2013-08-18_14-43-36.mov, file2: /opt/pictures/photo/movie/2013/2013-08-18_14-43-36.mov files differ: byte 00000d82fa62, 7f != 77 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-05-37.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-05-37.mov files differ: byte 0000046134e2, ff != f7 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-24-52.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-24-52.mov files differ: byte 00001bad84e2, 3b != 33 (08), file1: /disk1/pictures/photo/movie/2013/2013-12-30_06-24-52.mov, file2: /opt/pictures/photo/movie/2013/2013-12-30_06-24-52.mov The compare utility list the byte offset, file 1 byte value, file 2 byte value and the xor of both (all hex values). You may see that end part of the file offset is always e2 or 62 and the xor is always 8, so it's always bit 3 different. I've tested on 10.04 LTS, 14.04.1 LTS (3.13.0-43-generic) and also on CentOS 7 with similar results. If I do the compare opening the files with __O_DIRECT flag the problem is minimized, far less bit errors. I tested my memory extensively with memtest and never got an error. I also tried with btrfs file-system and I get similar errors without any btrfs checksum error. It seems to me that the problem is in the buffering between file-system and the application and it also seems to me a memory error with a bitrot under stress, despite my memory seems ok. Having always just one bit flipped is probably an HW error, not SW one. I'll do some more testing today and tomorrow on upstrem kernel BR Paolo -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1400792 Title: Silent ext4 read error To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1400792/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
