[Soekris] net6501, Linux, SSD/kernel/hardware failure?
Hi all, I'm testing my first net6501 with an OCZ Petrol SSD. Everything seemed OK, got the OS installed, then I started getting checksum mismatches from rsync, then I/O errors, and then the drive stopped working completely. The rsync errors started like this: + rsync -avP --delete --exclude /dev --exclude /proc --exclude /sys --exclude /tmp --exclude /var/tmp --exclude /var/log --exclude /var/run --exclude /var/lib/postfix 10.0.156.222::root/ /mnt/target/ media/hdb1/system-imager/image/usr/lib/locale/en_CA.utf8/LC_TIMEWARNING: media/hdb1/system-imager/image/usr/lib/locale/en_IE.utf8/LC_CTYPE failed verification -- update discarded (will try again). WARNING: media/hdb1/system-imager/image/usr/lib/locale/en_NZ.utf8/LC_CTYPE failed verification -- update discarded (will try again). WARNING: media/hdb1/system-imager/image/usr/lib/locale/en_PH.utf8/LC_CTYPE failed verification -- update discarded (will try again). Nothing in dmesg at this point. Then it went like this: rsync: readlink_stat(/mnt/target/media/hdb1/system-imager/image/usr/share/icons/Humanity/mimes/16/application-vnd.ms-powerpoint.svg) failed: Input/output error (5) rsync: recv_generator: failed to stat /mnt/target/media/hdb1/system-imager/image/usr/share/icons/Humanity/mimes/16/application-vnd.ms-powerpoint.svg: Input/output error (5) rsync: symlink /mnt/target/media/hdb1/system-imager/image/usr/share/icons/Humanity/mimes/48/application-x-tar.svg - tar.svg failed: Input/output error (5) I checked the drive for hardware errors using smartctl, and it status seemed good: root@system-imager:~# smartctl -a /dev/sda smartctl 5.40 2010-07-12 r3124 [i686-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: OCZ-PETROL Serial Number:OCZ-25G7LV2577Z5OGG8 Firmware Version: 3.12 User Capacity:128,035,676,160 bytes Device is:Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is:Wed Jan 2 05:47:24 1980 GMT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 249) Self-test routine in progress... 90% of test remaining. Total time to complete Offline data collection: ( 0) seconds. Offline data collection capabilities:(0x1d) SMART execute Offline immediate. No Auto Offline data collection support. Abort Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. No Selective Self-test supported. SMART capabilities:(0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability:(0x00) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time:( 0) minutes. Extended self-test routine recommended polling time:( 0) minutes. SMART Attributes Data Structure revision number: 18 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x 006 000 000Old_age Offline - 6 3 Spin_Up_Time0x 100 100 000Old_age Offline - 0 4 Start_Stop_Count0x 100 100 000Old_age Offline - 0 5 Reallocated_Sector_Ct 0x 100 100 000Old_age Offline - 0 9 Power_On_Hours 0x 100 100 000Old_age Offline - 5 12 Power_Cycle_Count 0x 100 100 000Old_age Offline - 8 232 Available_Reservd_Space 0x 100 100 000Old_age Offline - 160 233 Media_Wearout_Indicator 0x 100 000 000Old_age Offline - 100 SMART Error Log Version: 0 No Errors Logged So I thought filesystem corruption, and tried to unmount the drive for a fsck, which took forever, and then I started seeing errors in the system logs: [15179.138794] ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen [15179.142753] ata1.00:
Re: [Soekris] net6501, Linux, SSD/kernel/hardware failure?
This probably does not help too much, but FYI I am running Debian Wheezy on my net6501 with an OCZ Vertex-3 SSD, and no problem encountered so far... -- Philippe On 29/05/12 18:41, Chris Wilson wrote: Hi all, I'm testing my first net6501 with an OCZ Petrol SSD. Everything seemed OK, got the OS installed, then I started getting checksum mismatches from rsync, then I/O errors, and then the drive stopped working completely. snip ___ Soekris-tech mailing list Soekris-tech@lists.soekris.com http://lists.soekris.com/mailman/listinfo/soekris-tech
Re: [Soekris] net6501, Linux, SSD/kernel/hardware failure?
As a matter of general troubleshooting, I'd recommend trying the drive in a PC (or other machine) running Linux, trying the same thing with a different hard drive with the Soekris board, and swapping out cables with new in both scenarios... with those errors, seems likely that the drive controller is fried, the bus controller is fried, or the cable is bad, and these tests should isolate it... Good luck. Regards, Stephen J Alexander MPBX, LLC http://mpbx.com 832-713-6729 On Tue, May 29, 2012 at 11:41 AM, Chris Wilson chris-soek...@aptivate.orgwrote: Hi all, I'm testing my first net6501 with an OCZ Petrol SSD. Everything seemed OK, got the OS installed, then I started getting checksum mismatches from rsync, then I/O errors, and then the drive stopped working completely. The rsync errors started like this: + rsync -avP --delete --exclude /dev --exclude /proc --exclude /sys --exclude /tmp --exclude /var/tmp --exclude /var/log --exclude /var/run --exclude /var/lib/postfix 10.0.156.222::root/ /mnt/target/ media/hdb1/system-imager/image/usr/lib/locale/en_CA.utf8/LC_TIMEWARNING: media/hdb1/system-imager/image/usr/lib/locale/en_IE.utf8/LC_CTYPE failed verification -- update discarded (will try again). WARNING: media/hdb1/system-imager/image/usr/lib/locale/en_NZ.utf8/LC_CTYPE failed verification -- update discarded (will try again). WARNING: media/hdb1/system-imager/image/usr/lib/locale/en_PH.utf8/LC_CTYPE failed verification -- update discarded (will try again). Nothing in dmesg at this point. Then it went like this: rsync: readlink_stat(/mnt/target/media/hdb1/system-imager/image/usr/share/icons/Humanity/mimes/16/application-vnd.ms-powerpoint.svg) failed: Input/output error (5) rsync: recv_generator: failed to stat /mnt/target/media/hdb1/system-imager/image/usr/share/icons/Humanity/mimes/16/application-vnd.ms-powerpoint.svg: Input/output error (5) rsync: symlink /mnt/target/media/hdb1/system-imager/image/usr/share/icons/Humanity/mimes/48/application-x-tar.svg - tar.svg failed: Input/output error (5) I checked the drive for hardware errors using smartctl, and it status seemed good: root@system-imager:~# smartctl -a /dev/sda smartctl 5.40 2010-07-12 r3124 [i686-pc-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: OCZ-PETROL Serial Number:OCZ-25G7LV2577Z5OGG8 Firmware Version: 3.12 User Capacity:128,035,676,160 bytes Device is:Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is:Wed Jan 2 05:47:24 1980 GMT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 249) Self-test routine in progress... 90% of test remaining. Total time to complete Offline data collection: ( 0) seconds. Offline data collection capabilities:(0x1d) SMART execute Offline immediate. No Auto Offline data collection support. Abort Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. No Selective Self-test supported. SMART capabilities:(0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability:(0x00) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time:( 0) minutes. Extended self-test routine recommended polling time:( 0) minutes. SMART Attributes Data Structure revision number: 18 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x 006 000 000Old_age Offline - 6 3 Spin_Up_Time0x 100 100 000Old_age Offline - 0 4 Start_Stop_Count0x 100 100 000Old_age Offline - 0 5 Reallocated_Sector_Ct 0x 100 100 000Old_age Offline - 0 9
Re: [Soekris] net6501, Linux, SSD/kernel/hardware failure?
Chris Wilson [chris-soek...@aptivate.org] wrote: But after a power off (not just a hard reset) the drive is fine again! Some people are seeing issues with Debian kernels (like this one) and multiple drives appearing to fail at the same time, but working fine after a reboot: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=625922 So I suspect there's nothing wrong with the drive; it may be a firmware bug, but I was unable to apply firmware fixes using OCZ's updater for Linux. Has anyone seen anything like this? Anyone got a sense for whether it could be a hardware bug with the AHCI controller or BIOS on the net6501? The drive wasn't responding. It's likely to be the drive itself, not some failure on the controller or magic in the wire. SSDs are not the ultra-reliable machines they're made out to be, in my experience. The firmware probably crashed. And if that's what happened, it'll happen again under certain conditions -- that is until you get an update that breaks the firmware in some new and bizarre way. ___ Soekris-tech mailing list Soekris-tech@lists.soekris.com http://lists.soekris.com/mailman/listinfo/soekris-tech
