Hi Sean, No, Intel’s feedback has been… Pathetic… I have yet to receive anything more than a request to ‘sign’ a non-disclosure agreement, to obtain beta firmware. No official answer as to whether or not one can logically unlock the drives, no answer to my question whether or not Intel publish serial numbers anywhere pertaining to recalled batches and no information pertaining to whether or not firmware updates would address any known issues.
This with us being an accredited Intel Gold partner… We’ve returned the lot and ended up with 9/12 of the drives failing in the same manner. The replaced drives, which had different serial number ranges, also failed. Very frustrating is that the drives fail in a way that result in unbootable servers, unless one adds ‘rootdelay=240’ to the kernel. I would be interested to know what platform your drives were in and whether or not they were connected to a RAID module/card. PS: After much searching we’ve decided to order the NVMe conversion kit and have ordered HGST UltraStar SN200 2.5 inch SFF drives with a 3 DWPD rating. Regards David Herselman From: Sean Redmond [mailto:sean.redmo...@gmail.com] Sent: Thursday, 11 January 2018 12:45 AM To: David Herselman <d...@syrex.co> Cc: Christian Balzer <ch...@gol.com>; ceph-users@lists.ceph.com Subject: Re: [ceph-users] Many concurrent drive failures - How do I activate pgs? Hi, I have a case where 3 out to 12 of these Intel S4600 2TB model failed within a matter of days after being burn-in tested then placed into production. I am interested to know, did you every get any further feedback from the vendor on your issue? Thanks On Thu, Dec 21, 2017 at 1:38 PM, David Herselman <d...@syrex.co<mailto:d...@syrex.co>> wrote: Hi, I assume this can only be a physical manufacturing flaw or a firmware bug? Do Intel publish advisories on recalled equipment? Should others be concerned about using Intel DC S4600 SSD drives? Could this be an electrical issue on the Hot Swap Backplane or BMC firmware issue? Either way, all pure Intel... The hole is only 1.3 GB (4 MB x 339 objects) but perfectly striped through images, file systems are subsequently severely damaged. Is it possible to get Ceph to read in partial data shards? It would provide between 25-75% more yield... Is there anything wrong with how we've proceeded thus far? Would be nice to reference examples of using ceph-objectstore-tool but documentation is virtually non-existent. We used another SSD drive to simulate bringing all the SSDs back online. We carved up the drive to provide equal partitions to essentially simulate the original SSDs: # Partition a drive to provide 12 x 150GB partitions, eg: sdd 8:48 0 1.8T 0 disk |-sdd1 8:49 0 140G 0 part |-sdd2 8:50 0 140G 0 part |-sdd3 8:51 0 140G 0 part |-sdd4 8:52 0 140G 0 part |-sdd5 8:53 0 140G 0 part |-sdd6 8:54 0 140G 0 part |-sdd7 8:55 0 140G 0 part |-sdd8 8:56 0 140G 0 part |-sdd9 8:57 0 140G 0 part |-sdd10 8:58 0 140G 0 part |-sdd11 8:59 0 140G 0 part +-sdd12 8:60 0 140G 0 part Pre-requisites: ceph osd set noout; apt-get install uuid-runtime; for ID in `seq 24 35`; do UUID=`uuidgen`; OSD_SECRET=`ceph-authtool --gen-print-key`; DEVICE='/dev/sdd'$[$ID-23]; # 24-23 = /dev/sdd1, 35-23 = /dev/sdd12 echo "{\"cephx_secret\": \"$OSD_SECRET\"}" | ceph osd new $UUID $ID -i - -n client.bootstrap-osd -k /var/lib/ceph/bootstrap-osd/ceph.keyring; mkdir /var/lib/ceph/osd/ceph-$ID; mkfs.xfs $DEVICE; mount $DEVICE /var/lib/ceph/osd/ceph-$ID; ceph-authtool --create-keyring /var/lib/ceph/osd/ceph-$ID/keyring --name osd.$ID --add-key $OSD_SECRET; ceph-osd -i $ID --mkfs --osd-uuid $UUID; chown -R ceph:ceph /var/lib/ceph/osd/ceph-$ID; systemctl enable ceph-osd@$ID; systemctl start ceph-osd@$ID; done Once up we imported previous exports of empty head files in to 'real' OSDs: kvm5b: systemctl stop ceph-osd@8; ceph-objectstore-tool --op import --pgid 7.4s0 --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --file /var/lib/vz/template/ssd_recovery/osd8_7.4s0.export; chown ceph:ceph -R /var/lib/ceph/osd/ceph-8; systemctl start ceph-osd@8; kvm5f: systemctl stop ceph-osd@23; ceph-objectstore-tool --op import --pgid 7.fs0 --data-path /var/lib/ceph/osd/ceph-23 --journal-path /var/lib/ceph/osd/ceph-23/journal --file /var/lib/vz/template/ssd_recovery/osd23_7.fs0.export; chown ceph:ceph -R /var/lib/ceph/osd/ceph-23; systemctl start ceph-osd@23; Bulk import previously exported objects: cd /var/lib/vz/template/ssd_recovery; for FILE in `ls -1A osd*_*.export | grep -Pv '^osd(8|23)_'`; do OSD=`echo $FILE | perl -pe 's/^osd(\d+).*/\1/'`; PGID=`echo $FILE | perl -pe 's/^osd\d+_(.*?).export/\1/g'`; echo -e "systemctl stop ceph-osd@$OSD\t ceph-objectstore-tool --op import --pgid $PGID --data-path /var/lib/ceph/osd/ceph-$OSD --journal-path /var/lib/ceph/osd/ceph-$OSD/journal --file /var/lib/vz/template/ssd_recovery/osd"$OSD"_$PGID.export"; done | sort Sample output (this will wrap): systemctl stop ceph-osd@27 ceph-objectstore-tool --op import --pgid 7.4s3 --data-path /var/lib/ceph/osd/ceph-27 --journal-path /var/lib/ceph/osd/ceph-27/journal --file /var/lib/vz/template/ssd_recovery/osd27_7.4s3.export systemctl stop ceph-osd@27 ceph-objectstore-tool --op import --pgid 7.fs5 --data-path /var/lib/ceph/osd/ceph-27 --journal-path /var/lib/ceph/osd/ceph-27/journal --file /var/lib/vz/template/ssd_recovery/osd27_7.fs5.export systemctl stop ceph-osd@30 ceph-objectstore-tool --op import --pgid 7.fs4 --data-path /var/lib/ceph/osd/ceph-30 --journal-path /var/lib/ceph/osd/ceph-30/journal --file /var/lib/vz/template/ssd_recovery/osd30_7.fs4.export systemctl stop ceph-osd@31 ceph-objectstore-tool --op import --pgid 7.4s2 --data-path /var/lib/ceph/osd/ceph-31 --journal-path /var/lib/ceph/osd/ceph-31/journal --file /var/lib/vz/template/ssd_recovery/osd31_7.4s2.export systemctl stop ceph-osd@32 ceph-objectstore-tool --op import --pgid 7.4s4 --data-path /var/lib/ceph/osd/ceph-32 --journal-path /var/lib/ceph/osd/ceph-32/journal --file /var/lib/vz/template/ssd_recovery/osd32_7.4s4.export systemctl stop ceph-osd@32 ceph-objectstore-tool --op import --pgid 7.fs2 --data-path /var/lib/ceph/osd/ceph-32 --journal-path /var/lib/ceph/osd/ceph-32/journal --file /var/lib/vz/template/ssd_recovery/osd32_7.fs2.export systemctl stop ceph-osd@34 ceph-objectstore-tool --op import --pgid 7.4s5 --data-path /var/lib/ceph/osd/ceph-34 --journal-path /var/lib/ceph/osd/ceph-34/journal --file /var/lib/vz/template/ssd_recovery/osd34_7.4s5.export systemctl stop ceph-osd@34 ceph-objectstore-tool --op import --pgid 7.fs1 --data-path /var/lib/ceph/osd/ceph-34 --journal-path /var/lib/ceph/osd/ceph-34/journal --file /var/lib/vz/template/ssd_recovery/osd34_7.fs1.export Reset permissions and then started the OSDs: for OSD in 27 30 31 32 34; do chown -R ceph:ceph /var/lib/ceph/osd/ceph-$OSD; systemctl start ceph-osd@$OSD; done Then finally started all the OSDs... Now to hope that Intel have a way of accessing drives that are in a 'disable logical state'. The imports succeed, herewith a link to the output after running an import for placement group 7.4s2 on OSD 31: https://drive.google.com/open?id=1-Jo1jmrWrGLO2OgflacGPlEf2p32Y4hn Sample snippet: Write 1#7:fffcd2ec:::rbd_data.4.be8e9974b0dc51.0000000000002869:head# snapset 0=[]:{} Write 1#7:fffd4823:::rbd_data.4.ba24ef2ae8944a.000000000000a2b0:head# snapset 0=[]:{} Write 1#7:fffd6fb6:::benchmark_data_kvm5b_20945_object14722:head# snapset 0=[]:{} Write 1#7:ffffa069:::rbd_data.4.ba24ef2ae8944a.000000000000aea9:head# snapset 0=[]:{} Import successful Data does get written, I can tell by the size of the FileStore mount points: [root@kvm5b ssd_recovery]# df -h | grep -P 'ceph-(27|30|31|32|34)$' /dev/sdd4 140G 5.2G 135G 4% /var/lib/ceph/osd/ceph-27 /dev/sdd7 140G 14G 127G 10% /var/lib/ceph/osd/ceph-30 /dev/sdd8 140G 14G 127G 10% /var/lib/ceph/osd/ceph-31 /dev/sdd9 140G 22G 119G 16% /var/lib/ceph/osd/ceph-32 /dev/sdd11 140G 22G 119G 16% /var/lib/ceph/osd/ceph-34 How do I tell Ceph to read these object shards? PS: It's probably a good idea to reweight the OSDs to 0 before starting again. This should prevent data flowing on to them, if they are not in a different device class or other crush selection ruleset. Ie: for OSD in `seq 24 35`; do ceph osd crush reweight osd.$OSD 0; done Regards David Herselman -----Original Message----- From: David Herselman Sent: Thursday, 21 December 2017 3:49 AM To: 'Christian Balzer' <ch...@gol.com<mailto:ch...@gol.com>>; ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Subject: RE: [ceph-users] Many concurrent drive failures - How do I activate pgs? Hi Christian, Thanks for taking the time, I haven't been contacted by anyone yet but managed to get the down placement groups cleared by exporting 7.4s0 and 7.fs0 and then marking them as complete on the surviving OSDs: kvm5c: ceph-objectstore-tool --op export --pgid 7.4s0 --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --file /var/lib/vz/template/ssd_recovery/osd8_7.4s0.export; ceph-objectstore-tool --op mark-complete --data-path /var/lib/ceph/osd/ceph-8 --journal-path /var/lib/ceph/osd/ceph-8/journal --pgid 7.4s0; kvm5f: ceph-objectstore-tool --op export --pgid 7.fs0 --data-path /var/lib/ceph/osd/ceph-23 --journal-path /var/lib/ceph/osd/ceph-23/journal --file /var/lib/vz/template/ssd_recovery/osd23_7.fs0.export; ceph-objectstore-tool --op mark-complete --data-path /var/lib/ceph/osd/ceph-23 --journal-path /var/lib/ceph/osd/ceph-23/journal --pgid 7.fs0; This would presumably simply punch holes in the RBD images but at least we can copy them out of that pool and hope that Intel can somehow unlock the drives for us to then export/import objects. To answer your questions though, we have 6 near identical Intel Wildcat Pass 1U servers and have Proxmox loaded on them. Proxmox uses a Debian 9 base with the Ubuntu kernel, for which they apply cherry picked kernel patches (eg Intel NIC driver updates, vhost perf regression and mem-leak fixes, etc): kvm5a: Intel R1208WTTGSR System (serial: BQWS55091014) Intel S2600WTTR Motherboard (serial: BQWL54950385, BIOS ID: SE5C610.86B.01.01.0021.032120170601) 2 x Intel Xeon E5-2640v4 2.4GHz (HT disabled) 24 x Micron 8GB DDR4 2133MHz (24 x 18ASF1G72PZ-2G1B1) Intel AXX10GBNIA I/O Module kvm5b: Intel R1208WTTGS System (serial: BQWS53890178) Intel S2600WTT Motherboard (serial: BQWL52550359, BIOS ID: SE5C610.86B.01.01.0021.032120170601) 2 x Intel Xeon E5-2640v4 2.4GHz (HT enabled) 4 x Micron 64GB DDR4 2400MHz LR-DIMM (4 x 72ASS8G72LZ-2G3B2) Intel AXX10GBNIA I/O Module kvm5c: Intel R1208WT2GS System (serial: BQWS50490279) Intel S2600WT2 Motherboard (serial: BQWL44650203, BIOS ID: SE5C610.86B.01.01.0021.032120170601) 2 x Intel Xeon E5-2640v3 2.6GHz (HT enabled) 4 x Micron 64GB DDR4 2400MHz LR-DIMM (4 x 72ASS8G72LZ-2G3B2) Intel AXX10GBNIA I/O Module kvm5d: Intel R1208WTTGSR System (serial: BQWS62291318) Intel S2600WTTR Motherboard (serial: BQWL61855187, BIOS ID: SE5C610.86B.01.01.0021.032120170601) 2 x Intel Xeon E5-2640v4 2.4GHz (HT enabled) 4 x Micron 64GB DDR4 2400MHz LR-DIMM (4 x 72ASS8G72LZ-2G3B2) Intel AXX10GBNIA I/O Module kvm5e: Intel R1208WTTGSR System (serial: BQWS64290162) Intel S2600WTTR Motherboard (serial: BQWL63953066, BIOS ID: SE5C610.86B.01.01.0021.032120170601) 2 x Intel Xeon E5-2640v4 2.4GHz (HT enabled) 4 x Micron 64GB DDR4 2400MHz LR-DIMM (4 x 72ASS8G72LZ-2G3B2) Intel AXX10GBNIA I/O Module kvm5f: Intel R1208WTTGSR System (serial: BQWS71790632) Intel S2600WTTR Motherboard (serial: BQWL71050622, BIOS ID: SE5C610.86B.01.01.0021.032120170601) 2 x Intel Xeon E5-2640v4 2.4GHz (HT enabled) 4 x Micron 64GB DDR4 2400MHz LR-DIMM (4 x 72ASS8G72LZ-2G3B2) Intel AXX10GBNIA I/O Module Summary: * 5b has an Intel S2600WTT, 5c has an Intel S2600WT2, all others have S2600WTTR Motherboards * 5a has ECC Registered Dual Rank DDR DIMMs, all others have ECC LoadReduced-DIMMs * 5c has an Intel X540-AT2 10 GbE adapter as the on-board NICs are only 1 GbE Each system has identical discs: * 2 x 480 GB Intel SSD DC S3610 (SSDSC2BX480G4) - partitioned as software RAID1 OS volume and Ceph FileStore journals (spinners) * 4 x 2 TB Seagate discs (ST2000NX0243) - Ceph FileStore OSDs (journals in S3610 partitions) * 2 x 1.9 TB Intel SSD DC S4600 (SSDSC2KG019T7) - Ceph BlueStore OSDs (problematic) Additional information: * All drives are directly attached to the on-board AHCI SATA controllers, via the standard 2.5 inch drive chassis hot-swap bays. * We added 12 x 1.9 TB SSD DC S4600 drives last week Thursday, 2 in each system's slots 7 & 8 * Systems have been operating with existing Intel SSD DC S3610 and 2 TB Seagate discs for over a year; we added the most recent node (kvm5f) on the 23rd of November. * 6 of the 12 Intel SSD DC S4600 drives failed in less than 100 hours. * They work perfectly until they suddenly stop responding and are thereafter, even with us physically shutting down the server and powering it back up again, completely inaccessible. Intel diagnostic tool reports 'logically locked'. Drive failures appear random to me: kvm5a - bay 7 - offline - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM739208851P9DGN kvm5a - bay 8 - online - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=PHYM727602TM1P9DGN kvm5b - bay 7 - offline - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=PHYM7276031E1P9DGN kvm5b - bay 8 - online - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM7392087W1P9DGN kvm5c - bay 7 - offline - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM739200ZJ1P9DGN kvm5c - bay 8 - offline - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM7392088B1P9DGN kvm5d - bay 7 - offline - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM738604Y11P9DGN kvm5d - bay 8 - online - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=PHYM727603181P9DGN kvm5e - bay 7 - online - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM7392013B1P9DGN kvm5e - bay 8 - offline - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM7392087E1P9DGN kvm5f - bay 7 - online - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM739208721P9DGN kvm5f - bay 8 - online - Model=INTEL SSDSC2KG019T7, FwRev=SCV10100, SerialNo=BTYM739208C41P9DGN Intel SSD Data Center Tool reports: C:\isdct>isdct.exe show -intelssd - Intel SSD DC S4600 Series PHYM7276031E1P9DGN - Bootloader : Property not found DevicePath : \\\\.\\PHYSICALDRIVE1<file://.//PHYSICALDRIVE1> DeviceStatus : Selected drive is in a disable logical state. Firmware : SCV10100 FirmwareUpdateAvailable : Please contact Intel Customer Support for further assistance at the following website: http://www.intel.com/go/ssdsupport. Index : 0 ModelNumber : INTEL SSDSC2KG019T7 ProductFamily : Intel SSD DC S4600 Series SerialNumber : PHYM7276031E1P9DGN C:\isdct>isdct show -a -intelssd 0 - Intel SSD DC S4600 Series PHYM7276031E1P9DGN - AccessibleMaxAddressSupported : True AggregationThreshold : Selected drive is in a disable logical state. AggregationTime : Selected drive is in a disable logical state. ArbitrationBurst : Selected drive is in a disable logical state. BusType : 11 CoalescingDisable : Selected drive is in a disable logical state. ControllerCompatibleIDs : PCI\\VEN_8086&DEV_8C02&REV_05PCI\\VEN_8086&DEV_8C02PCI\\VEN_8086&CC_010601PCI\\VEN_8086&CC_0106PCI\\VEN_8086PCI\\CC_010601PCI\\CC_0106 ControllerDescription : @mshdc.inf,%pci\\cc_010601.devicedesc%;Standard SATA AHCI Controller ControllerID : PCI\\VEN_8086&DEV_8C02&SUBSYS_78461462&REV_05\\3&11583659&0&FA ControllerIDEMode : False ControllerManufacturer : @mshdc.inf,%ms-ahci%;Standard SATA AHCI Controller ControllerService : storahci DIPMEnabled : False DIPMSupported : False DevicePath : \\\\.\\PHYSICALDRIVE1<file://.//PHYSICALDRIVE1> DeviceStatus : Selected drive is in a disable logical state. DigitalFenceSupported : False DownloadMicrocodePossible : True DriverDescription : Standard SATA AHCI Controller DriverMajorVersion : 10 DriverManufacturer : Standard SATA AHCI Controller DriverMinorVersion : 0 DriverVersion : 10.0.16299.98 DynamicMMIOEnabled : The selected drive does not support this feature. EnduranceAnalyzer : Selected drive is in a disable logical state. ErrorString : *BAD_CONTEXT_2020 F4 Firmware : SCV10100 FirmwareUpdateAvailable : Please contact Intel Customer Support for further assistance at the following website: http://www.intel.com/go/ssdsupport. HDD : False HighPriorityWeightArbitration : Selected drive is in a disable logical state. IEEE1667Supported : False IOCompletionQueuesRequested : Selected drive is in a disable logical state. IOSubmissionQueuesRequested : Selected drive is in a disable logical state. Index : 0 Intel : True IntelGen3SATA : True IntelNVMe : False InterruptVector : Selected drive is in a disable logical state. IsDualPort : False LatencyTrackingEnabled : Selected drive is in a disable logical state. LowPriorityWeightArbitration : Selected drive is in a disable logical state. Lun : 0 MaximumLBA : 3750748847 MediumPriorityWeightArbitration : Selected drive is in a disable logical state. ModelNumber : INTEL SSDSC2KG019T7 NVMePowerState : Selected drive is in a disable logical state. NativeMaxLBA : Selected drive is in a disable logical state. OEM : Generic OpalState : Selected drive is in a disable logical state. PLITestTimeInterval : Selected drive is in a disable logical state. PNPString : SCSI\\DISK&VEN_INTEL&PROD_SSDSC2KG019T7\\4&2BE6C224&0&010000 PathID : 1 PhySpeed : Selected drive is in a disable logical state. PhysicalSectorSize : Selected drive is in a disable logical state. PhysicalSize : 1920383410176 PowerGovernorAveragePower : Selected drive is in a disable logical state. PowerGovernorBurstPower : Selected drive is in a disable logical state. PowerGovernorMode : Selected drive is in a disable logical state. Product : Youngsville ProductFamily : Intel SSD DC S4600 Series ProductProtocol : ATA ReadErrorRecoveryTimer : Selected drive is in a disable logical state. RemoteSecureEraseSupported : False SCSIPortNumber : 0 SMARTEnabled : True SMARTHealthCriticalWarningsConfiguration : Selected drive is in a disable logical state. SMARTSelfTestSupported : True SMBusAddress : Selected drive is in a disable logical state. SSCEnabled : False SanitizeBlockEraseSupported : False SanitizeCryptoScrambleSupported : True SanitizeSupported : True SataGen1 : True SataGen2 : True SataGen3 : True SataNegotiatedSpeed : Unknown SectorSize : 512 SecurityEnabled : False SecurityFrozen : False SecurityLocked : False SecuritySupported : False SerialNumber : PHYM7276031E1P9DGN TCGSupported : False TargetID : 0 TempThreshold : Selected drive is in a disable logical state. TemperatureLoggingInterval : Selected drive is in a disable logical state. TimeLimitedErrorRecovery : Selected drive is in a disable logical state. TrimSize : 4 TrimSupported : True VolatileWriteCacheEnabled : Selected drive is in a disable logical state. WWID : 3959312879584368077 WriteAtomicityDisableNormal : Selected drive is in a disable logical state. WriteCacheEnabled : True WriteCacheReorderingStateEnabled : Selected drive is in a disable logical state. WriteCacheState : Selected drive is in a disable logical state. WriteCacheSupported : True WriteErrorRecoveryTimer : Selected drive is in a disable logical state. SMART information is inaccessible, overall status is failed. Herewith the stats from a partner disc which was still working when the others failed: Device Model: INTEL SSDSC2KG019T7 Serial Number: PHYM727602TM1P9DGN LU WWN Device Id: 5 5cd2e4 14e1636bb Firmware Version: SCV10100 User Capacity: 1,920,383,410,176 bytes [1.92 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-3 T13/2161-D revision 5 SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Dec 18 19:33:51 2017 SAST SMART support is: Available - device has SMART capability. SMART support is: Enabled ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 98 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 3 170 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0 171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 1 172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 175 Program_Fail_Count_Chip 0x0033 100 100 010 Pre-fail Always - 17567121432 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 077 076 000 Old_age Always - 23 (Min/Max 17/29) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 0 194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 23 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 225 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 14195 226 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 0 227 Unknown_SSD_Attribute 0x0032 100 100 000 Old_age Always - 42 228 Power-off_Retract_Count 0x0032 100 100 000 Old_age Always - 5905 232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0 233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0 234 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 241 Total_LBAs_Written 0x0032 100 100 000 Old_age Always - 14195 242 Total_LBAs_Read 0x0032 100 100 000 Old_age Always - 10422 243 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 41906 Media wear out : 0% used LBAs written: 14195 Power on hours: <100 Power cycle count: once at the factory, once at our offices to check if there was newer firmware (there wasn't) and once when we restarted the node to see if it could then access a failed drive. Regards David Herselman -----Original Message----- From: Christian Balzer [mailto:ch...@gol.com<mailto:ch...@gol.com>] Sent: Thursday, 21 December 2017 3:24 AM To: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> Cc: David Herselman <d...@syrex.co<mailto:d...@syrex.co>> Subject: Re: [ceph-users] Many concurrent drive failures - How do I activate pgs? Hello, first off, I don't have anything to add to your conclusions of the current status, alas there are at least 2 folks here on the ML making a living from Ceph disaster recovery, so I hope you have been contacted already. Now once your data is safe or you have a moment, I and others here would probably be quite interested in some more details, see inline below. On Wed, 20 Dec 2017 22:25:23 +0000 David Herselman wrote: [snip] > > We've happily been running a 6 node cluster with 4 x FileStore HDDs per node > (journals on SSD partitions) for over a year and recently upgraded all nodes > to Debian 9, Ceph Luminous 12.2.2 and kernel 4.13.8. We ordered 12 x Intel DC > S4600 SSDs which arrived last week so we added two per node on Thursday > evening and brought them up as BlueStore OSDs. We had proactively updated our > existing pools to reference only devices classed as 'hdd', so that we could > move select images over to ssd replicated and erasure coded pools. > Could you tell us more about that cluster, as in HW, how are the SSDs connected and FW version of the controller if applicable. Kernel 4.13.8 suggests that this is a handrolled, upstream kernel. While not necessarily related I'll note that as far as Debian kernels (which are very lightly if at all patched) are concerned, nothing beyond 4.9 has been working to my satisfaction. 4.11 still worked, but 4.12 crash-reboot-looped on all my Supermicro X10 machines (quite a varied selection). The current 4.13.13 backport boots on some of those machines, but still throws errors with the EDAC devices, which works fine with 4.9. 4.14 is known to happily destroy data if used with bcache and even if one doesn't use that it should give you pause. > We were pretty diligent and downloaded Intel's Firmware Update Tool and > validated that each new drive had the latest available firmware before > installing them in the nodes. We did numerous benchmarks on Friday and > eventually moved some images over to the new storage pools. Everything was > working perfectly and extensive tests on Sunday showed excellent performance. > Sunday night one of the new SSDs died and Ceph replicated and redistributed > data accordingly, then another failed in the early hours of Monday morning > and Ceph did what it needed to. > > We had the two failed drives replaced by 11am and Ceph was up to 2/4918587 > objects degraded (0.000%) when a third drive failed. At this point we updated > the crush maps for the rbd_ssd and ec_ssd pools and set the device class to > 'hdd', to essentially evacuate everything off the SSDs. Other SSDs then > failed at 3:22pm, 4:19pm, 5:49pm and 5:50pm. We've ultimately lost half the > Intel S4600 drives, which are all completely inaccessible. Our status at > 11:42pm Monday night was: 1/1398478 objects unfound (0.000%) and 339/4633062 > objects degraded (0.007%). > The relevant logs when and how those SSDs failed would be interesting. Was the distribution of the failed SSDs random among the cluster? Are you running smartd and did it have something to say? Completely inaccessible sounds a lot like the infamous "self-bricking" of Intel SSDs when they discover something isn't right, or they don't like the color scheme of the server inside (^.^). I'm using quite a lot of Intel SSDs and had only one "fatal" incident. A DC S3700 detected that its powercap had failed, but of course kept working fine. Until a reboot was need, when it promptly bricked itself, data inaccessible, SMART reporting barely that something was there. So one wonders what caused your SSDs to get their knickers in such a twist. Are the survivors showing any unusual signs in their SMART output? Of course what your vendor/Intel will have to say will also be of interest. ^o^ Regards, Christian -- Christian Balzer Network/Systems Engineer ch...@gol.com<mailto:ch...@gol.com> Rakuten Communications _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
