hi,
Am 31.12.2014 um 17:21 schrieb Wido den Hollander:
Hi,
Last week I upgraded a 250 OSD cluster from Dumpling 0.67.10 to Firefly
0.80.7 and after the upgrade there was a severe performance drop on the
cluster.
It started raining slow requests after the upgrade and most of them
included a 'snapc' in the request.
That lead me to investigate the RBD snapshots and I found that a rogue
process had created ~1800 snapshots spread out over 200 volumes.
One image even had 181 snapshots!
As the snapshots weren't used I removed them all and after the snapshots
were removed the performance of the cluster came back to normal level again.
I'm wondering what changed between Dumpling and Firefly which caused
this? I saw OSDs spiking to 100% disk util constantly under Firefly
where this didn't happen with Dumpling.
Did something change in the way OSDs handle RBD snapshots which causes
them to create more disk I/O?
I saw the same and addionally a slowdown in librbd too, that's why i'm
still on dumpling and won't upgrade until hammer.
Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
