Hello,
we likely faced the same issue with spindowns.
We set max spindown timers on all HDDs and disabled the tuned.service, which we
were told might change back/affect the set timers.
To correctly disable tuned:
tuned-adm stop
tuned-adm off
systemctl stop tuned.service
systemctl disable tuned.service
Our spindown problem is 'fixed'.
________________________________
Von: ceph-users <ceph-users-boun...@lists.ceph.com> im Auftrag von Florian
Engelmann <florian.engelm...@everyware.ch>
Gesendet: Montag, 3. Dezember 2018 11:02:28
An: ceph-users@lists.ceph.com
Betreff: [ceph-users] HDD spindown problem
Hello,
we are fighting a HDD spin-down problem on our production ceph cluster
since two weeks now. The problem is not ceph related but I guess this
topic is interesting to the list and to be honest I hope to find a
solution here.
We do use 6 OSD Nodes like:
OS: Suse 12 SP3
Ceph: SES 5.5 (12.2.8)
Server: Supermicro 6048R-E1CR36L
Controller: LSI 3008 (LSI3008-IT)
Disk: 12x Seagate ST8000NM0055-1RM112 8TB (SN05 Firmware (some still
SN02 and SN04)
NVMe: 1x Intel DC P3700 800GB (used for 80GB RocksDB and 2GB WAL for
each OSD (only 7 Disks are online right now - up to 9 Disks will have
there RocksDB/WAL on one NVMe SSD)
Problem:
This Ceph cluster is used for objectstorage (RadosGW) only and is mostly
used for backups to S3 (RadosGW). There is not that much activity -
mostly at night time. We do not want any HDD to spin down but they do.
We tried to disable the spindown timers by using sdparm and also with
the Seagate tool SeaChest but "something" does re-enable them:
Disable standby on all HDD:
for i in sd{c..n}; do
/root/SeaChestUtilities/Linux/Lin64/SeaChest_PowerControl_191_1183_64 -d
/dev/$i --onlySeagate --changePower --disableMode --powerMode standby ;
done
Monitor standby timer status:
while true; do for i in sd{c..n}; do echo "$(date) $i
$(/root/SeaChestUtilities/Linux/Lin64/SeaChest_PowerControl_191_1183_64
-d /dev/$i --onlySeagate --showEPCSettings -v0 | grep Stand)"; done;
sleep 1 ; done
This will show:
Mon Dec 3 10:42:54 CET 2018 sdc Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:54 CET 2018 sdd Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:54 CET 2018 sde Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:54 CET 2018 sdf Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:54 CET 2018 sdg Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:54 CET 2018 sdh Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:54 CET 2018 sdi Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:55 CET 2018 sdj Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:55 CET 2018 sdk Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:55 CET 2018 sdl Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:55 CET 2018 sdm Standby Z 0 9000
65535 120 Y Y
Mon Dec 3 10:42:55 CET 2018 sdn Standby Z 0 9000
65535 120 Y Y
So everything is fine right now. Standby timer is 0 and disabled (no *
shown) while the default value is 9000 and the saved timer is FFFF (we
saved this value so the disks have a huge time after reboots). But after
a unknown amount of time (in this case ~7 minutes) things start to get
weird:
Mon Dec 3 10:47:52 CET 2018 sdc Standby Z *3500 9000
65535 120 Y Y
[...]
65535 120 Y Y
Mon Dec 3 10:48:07 CET 2018 sdc Standby Z *3500 9000
65535 120 Y Y
Mon Dec 3 10:48:09 CET 2018 sdc Standby Z *3500 9000
65535 120 Y Y
Mon Dec 3 10:48:12 CET 2018 sdc Standby Z *4500 9000
65535 120 Y Y
Mon Dec 3 10:48:14 CET 2018 sdc Standby Z *4500 9000
65535 120 Y Y
Mon Dec 3 10:48:16 CET 2018 sdc Standby Z *4500 9000
65535 120 Y Y
Mon Dec 3 10:48:19 CET 2018 sdc Standby Z *4500 9000
65535 120 Y Y
Mon Dec 3 10:48:21 CET 2018 sdc Standby Z *4500 9000
65535 120 Y Y
Mon Dec 3 10:48:23 CET 2018 sdc Standby Z *5500 9000
65535 120 Y Y
Mon Dec 3 10:48:26 CET 2018 sdc Standby Z *5500 9000
65535 120 Y Y
Mon Dec 3 10:48:28 CET 2018 sdc Standby Z *5500 9000
65535 120 Y Y
Mon Dec 3 10:48:30 CET 2018 sdc Standby Z *5500 9000
65535 120 Y Y
Mon Dec 3 10:48:32 CET 2018 sdc Standby Z *5500 9000
65535 120 Y Y
Mon Dec 3 10:48:35 CET 2018 sdc Standby Z *5500 9000
65535 120 Y Y
Mon Dec 3 10:48:37 CET 2018 sdc Standby Z *5500 9000
65535 120 Y Y
Mon Dec 3 10:48:40 CET 2018 sdc Standby Z *5500 9000
65535 120 Y Y
Mon Dec 3 10:48:42 CET 2018 sdc Standby Z *6500 9000
65535 120 Y Y
Mon Dec 3 10:48:44 CET 2018 sdc Standby Z *6500 9000
65535 120 Y Y
Mon Dec 3 10:48:47 CET 2018 sdc Standby Z *6500 9000
65535 120 Y Y
Mon Dec 3 10:48:49 CET 2018 sdc Standby Z *6500 9000
65535 120 Y Y
Mon Dec 3 10:48:52 CET 2018 sdc Standby Z *7500 9000
65535 120 Y Y
Mon Dec 3 10:48:52 CET 2018 sde Standby Z *65535 9000
65535 120 Y Y
Mon Dec 3 10:48:54 CET 2018 sdc Standby Z *7500 9000
65535 120 Y Y
Mon Dec 3 10:48:55 CET 2018 sde Standby Z *65535 9000
65535 120 Y Y
Mon Dec 3 10:48:57 CET 2018 sdc Standby Z *7500 9000
65535 120 Y Y
Mon Dec 3 10:48:57 CET 2018 sde Standby Z *65535 9000
65535 120 Y Y
Mon Dec 3 10:48:59 CET 2018 sdc Standby Z *7500 9000
65535 120 Y Y
Mon Dec 3 10:49:00 CET 2018 sde Standby Z *65535 9000
65535 120 Y Y
Mon Dec 3 10:49:02 CET 2018 sdc Standby Z *8500 9000
65535 120 Y Y
Mon Dec 3 10:49:02 CET 2018 sde Standby Z *11500 9000
65535 120 Y Y
Mon Dec 3 10:49:04 CET 2018 sdc Standby Z *8500 9000
65535 120 Y Y
Mon Dec 3 10:49:05 CET 2018 sde Standby Z *11500 9000
65535 120 Y Y
Mon Dec 3 10:49:07 CET 2018 sdc Standby Z *8500 9000
65535 120 Y Y
Mon Dec 3 10:49:07 CET 2018 sde Standby Z *11500 9000
65535 120 Y Y
So "something" starts to re-enable those standby timers with strange
numbers. After those timers going up and down and disabled/enabled a
certain (unknown) amount of time they get "stable" at a value of 3000
and stay enabled (*):
Mon Dec 3 10:50:43 CET 2018 sde Standby Z *3000 9000
65535 120 Y Y
Mon Dec 3 10:50:45 CET 2018 sde Standby Z *3000 9000
65535 120 Y Y
3000 = 3000 / 100ms = 5 minutes. This is exactly what we measured when
we started to analyse the issue. Disks powered off (spin-down) after 5
Minutes.
We tried to add:
options mpt3sas allow_drive_spindown=0
did not help anything...
The only workaround right now is to have a cronjob in place running all
3 minutes to disable standby for all disks.
Anyone this a proper solution?
All the best,
Flo
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
