After some investigation, we realized it looks like the bottleneck is in the OSDs IOPS. The time it takes to resolve every IOP seems to be too high.
We'll apply the following upgrades: * Ceph.conf modifications to allow better utilization of SSD Drives * Some extra sysctl modifications (Although in that front everything seems quite well adjusted) * Jewel (Mainly to allow use of TCMalloc 2.4 with TC 128MB) * Newer Kernel (We are using a fair old 3.14 kernel as standard in our platform) * TC Malloc 2.4 (Not clear if we'll upgrade to Ubuntu 16.04 or we'll upgrade TCMalloc on 14.04) We still have to consider: * Using Jemalloc in OSD Servers * Using Jemalloc in QEMU Any comments or suggestions are welcome :) Thanks! Saludos Cordiales, Xavier Trilla P. SiliconHosting<https://siliconhosting.com/> ¿Un Servidor Cloud con SSDs, redundado y disponible en menos de 30 segundos? ¡Pruébalo ahora en Clouding.io<https://clouding.io/>! De: ceph-users [mailto:[email protected]] En nombre de Xavier Trilla Enviado el: martes, 7 de marzo de 2017 21:50 Para: [email protected] Asunto: [ceph-users] Strange read results using FIO inside RBD QEMU VM ... Hi, We have a pure SSD based Ceph cluster (+100 OSDs with Enterprise SSDs and IT mode cards) Hammer 0.94.9 over 10G. It's really stable and we are really happy with the performance we are getting. But after a customer ran some tests, we realized about something quite strange. Our user did some tests using FIO, and the strange thing is that Write tests did work as expected, but some Read tests did not. The VM he used was artificially limited via QEMU to 3200 read and 3200 write IOPS. In the write department everything works more or less as expected. The results get close to 3200 IOPS but the read tests are the ones we don't really understand. We ran tests using different IO Engines: Sync, libaio and POSIX AIO, during the write tests the 3 of them expect quite similar -which is something I did not really expect- but on the read department there is a huge difference: Read Results (Random Read - Buffered: No - Direct: Yes - Block Size: 4KB): LibAIO - Average: 3196 IOPS POSIX AIO - Average: 878 IOPS Sync - Average: 929 IOPS Write Results (Random Read - Buffered: No - Direct: Yes - Block Size: 4KB): LibAIO - Average: 2741 IOPS POSIX AIO - Average: 2673 IOPS Sync - Average: 2795 IOPS I would expect a difference when using LibAIO or POSIX AIO, but I would expect it in both read and write results, not only during reads. So, I'm quite disoriented with this one... Does anyone have an idea about what might be going on? Thanks! Saludos Cordiales, Xavier Trilla P. Clouding.io<https://clouding.io/> ¿Un Servidor Cloud con SSDs, redundado y disponible en menos de 30 segundos? ¡Pruébalo ahora en Clouding.io<https://clouding.io/>!
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
