It appears I have three stuck IOs after switching my tunables to optimal. We are running 10.2.9 and the offending pool is for gnocchi (which has caused us quite a bit pain at this point). Here are the stuck IOs:
2017-09-22 09:05:40.095125 osd.2 ##########:6802/1453572 164 : cluster [WRN] 3 slow requests, 3 included below; oldest blocked for > 61440.543437 secs 2017-09-22 09:05:40.095138 osd.2 ##########:6802/1453572 165 : cluster [WRN] slow request 61440.543437 seconds old, received at 2017-09-21 16:01:39.551597: osd_op(client.268119861.0:141848 7.e637a4b3 measure [omap-get-vals 0~16] snapc 0=[] RETRY=1 ack+retry+read+balance_reads+skiprwlocks+known_if_redirected e8015) currently waiting for missing object 2017-09-22 09:05:40.095147 osd.2 ##########:6802/1453572 166 : cluster [WRN] slow request 61440.542366 seconds old, received at 2017-09-21 16:01:39.552668: osd_op(client.267462582.0:1628859 7.e637a4b3 measure [omap-get-vals 0~16] snapc 0=[] RETRY=1 ack+retry+read+balance_reads+skiprwlocks+known_if_redirected e8015) currently waiting for missing object 2017-09-22 09:05:40.095152 osd.2 ##########:6802/1453572 167 : cluster [WRN] slow request 61440.169811 seconds old, received at 2017-09-21 16:01:39.925223: osd_op(client.267488428.0:141465 7.e637a4b3 measure [omap-get-vals 0~16] snapc 0=[] RETRY=1 ack+retry+read+balance_reads+skiprwlocks+known_if_redirected e8015) currently waiting for missing object Currently all IOs aren’t causing issues, but I can’t get these IOs to clear. I have bounced the OSD multiple times, but they haven’t cleared. Any advice? Also, if anyone has pro tips on how to setup ceph for gnocchi, I’m all ears. Thanks, Matthew Stroud ________________________________ CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you.
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
