Hi John, I have 9 HDDs and 3 SSDs behind a SAS3008 PCI-Express Fusion-MPT SAS-3 from LSI. HDDs are HGST HUH721008AL (8TB, 7200k rpm), SSDs are Toshiba PX05SMB040 (400GB). OSDs are bluestore format, 3 HDDs have their wal and db on one SSD (DB Size 50GB, wal 10 GB). I did not change any cache settings.
I disabled cstates which improved performance slightly. Do you suggest to turn off caching on disks? Regards Felix ------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- Von: John Petrini <jpetr...@coredial.com> Datum: Freitag, 7. Juni 2019 um 15:49 An: "Stolte, Felix" <f.sto...@fz-juelich.de> Cc: Sinan Polat <si...@turka.nl>, ceph-users <ceph-users@lists.ceph.com> Betreff: Re: [ceph-users] Expected IO in luminous Ceph Cluster How's iowait look on your disks? How have you configured your disks and what are your cache settings? Did you disable cstates? On Friday, June 7, 2019, Stolte, Felix <mailto:f.sto...@fz-juelich.de> wrote: > Hi Sinan, > > thanks for the numbers. I am a little bit surprised that your SSD pool has > nearly the same stats as you SAS pool. > > Nevertheless I would expect our pools to perform like your SAS pool, at least > regarding to writes since all our write ops should be placed on our SSDs. But > since I only achieve 10% of your numbers I need to figure out my bottle neck. > For now I have no clue. According to our monitoring system network bandwith, > ram or cpu usage is even close to be saturated. > > Could someone advice me on where to look? > > Regards Felix > ------------------------------------------------------------------------------------- > ------------------------------------------------------------------------------------- > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ------------------------------------------------------------------------------------- > ------------------------------------------------------------------------------------- > > > Am 07.06.19, 13:33 schrieb "Sinan Polat" <mailto:si...@turka.nl>: > > Hi Felix, > > I have 2 Pools, a SSD only and a SAS only pool. > > SSD pool is spread over 12 OSD servers. > SAS pool is spread over 6 OSD servers. > > > See results (SSD Only Pool): > > # sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G > --file-test-mode=rndrw --file-rw-ratio=2 run > sysbench 1.0.17 (using system LuaJIT 2.0.4) > > Running the test with following options: > Number of threads: 16 > Initializing random number generator from current time > > > Extra file open flags: (none) > 128 files, 8MiB each > 1GiB total file size > Block size 16KiB > Number of IO requests: 0 > Read/Write ratio for combined random IO test: 2.00 > Periodic FSYNC enabled, calling fsync() each 1 requests. > Calling fsync() at the end of test, Enabled. > Using synchronous I/O mode > Doing random r/w test > Initializing worker threads... > > Threads started! > > > File operations: > reads/s: 508.38 > writes/s: 254.19 > fsyncs/s: 32735.14 > > Throughput: > read, MiB/s: 7.94 > written, MiB/s: 3.97 > > General statistics: > total time: 10.0103s > total number of events: 333336 > > Latency (ms): > min: 0.00 > avg: 0.48 > max: 10.18 > 95th percentile: 2.11 > sum: 159830.07 > > Threads fairness: > events (avg/stddev): 20833.5000/335.70 > execution time (avg/stddev): 9.9894/0.00 > # > > See results (SAS Only Pool): > # sysbench --file-fsync-freq=1 --threads=16 fileio --file-total-size=1G > --file-test-mode=rndrw --file-rw-ratio=2 run > sysbench 1.0.17 (using system LuaJIT 2.0.4) > > Running the test with following options: > Number of threads: 16 > Initializing random number generator from current time > > > Extra file open flags: (none) > 128 files, 8MiB each > 1GiB total file size > Block size 16KiB > Number of IO requests: 0 > Read/Write ratio for combined random IO test: 2.00 > Periodic FSYNC enabled, calling fsync() each 1 requests. > Calling fsync() at the end of test, Enabled. > Using synchronous I/O mode > Doing random r/w test > Initializing worker threads... > > Threads started! > > > File operations: > reads/s: 490.11 > writes/s: 245.10 > fsyncs/s: 31565.00 > > Throughput: > read, MiB/s: 7.66 > written, MiB/s: 3.83 > > General statistics: > total time: 10.0143s > total number of events: 321477 > > Latency (ms): > min: 0.00 > avg: 0.50 > max: 20.50 > 95th percentile: 2.30 > sum: 159830.82 > > Threads fairness: > events (avg/stddev): 20092.3125/186.66 > execution time (avg/stddev): 9.9894/0.00 > # > > > Kind regards, > Sinan Polat > > > > > Op 7 juni 2019 om 12:47 schreef "Stolte, Felix" > <mailto:f.sto...@fz-juelich.de>: > > > > > > Hi Sinan, > > > > that would be great. The numbers should differ a lot, since you have an > all > > flash pool, but it would be interesting, what we could expect from such > a > > configuration. > > > > Regards > > Felix > > > > > ------------------------------------------------------------------------------------- > > > ------------------------------------------------------------------------------------- > > Forschungszentrum Juelich GmbH > > 52425 Juelich > > Sitz der Gesellschaft: Juelich > > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > > Prof. Dr. Sebastian M. Schmidt > > > ------------------------------------------------------------------------------------- > > > ------------------------------------------------------------------------------------- > > > > > > Am 07.06.19, 12:02 schrieb "Sinan Polat" <mailto:si...@turka.nl>: > > > > Hi Felix, > > > > I can run your commands inside an OpenStack VM. Tthe storage cluster > > contains of 12 OSD servers, holding each 8x 960GB SSD. Luminous > FileStore. > > Replicated 3. > > > > Would it help you to run your command on my cluster? > > > > Sinan > > > > > Op 7 jun. 2019 om 08:52 heeft Stolte, Felix > <mailto:f.sto...@fz-juelich.de> het > > > volgende geschreven: > > > > > > I have no performance data before we migrated to bluestore. You > should > > > start a separate topic regarding your question. > > > > > > Could anyone with an more or less equally sized cluster post the > output > > > of a sysbench with the following parameters (either from inside an > > > openstack vm or a mounted rbd)? > > > > > > sysbench --file-fsync-freq=1 --threads=16 fileio > --file-total-size=1G > > > --file-test-mode=rndrw --file-rw-ratio=2 prepare > > > > > > sysbench --file-fsync-freq=1 --threads=16 fileio > --file-total-size=1G > > > --file-test-mode=rndrw --file-rw-ratio=2 run > > > > > > Thanks in advance. > > > > > > Regards > > > Felix > > > > > > > ------------------------------------------------------------------------------------- > > > > ------------------------------------------------------------------------------------- > > > Forschungszentrum Juelich GmbH > > > 52425 Juelich > > > Sitz der Gesellschaft: Juelich > > > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B > 3498 > > > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > > > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt > (Vorsitzender), > > > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > > > Prof. Dr. Sebastian M. Schmidt > > > > ------------------------------------------------------------------------------------- > > > > ------------------------------------------------------------------------------------- > > > > > > > > > Am 06.06.19, 15:09 schrieb "Marc Roos" > <mailto:m.r...@f1-outsourcing.eu>: > > > > > > > > > I am also thinking of moving the wal/db to ssd of the sata > hdd's. Did > > > > > > you do tests before and after this change, and know what the > > > difference > > > is iops? And is the advantage more or less when your sata > hdd's are > > > slower? > > > > > > > > > -----Original Message----- > > > From: Stolte, Felix [mailto:mailto:f.sto...@fz-juelich.de] > > > Sent: donderdag 6 juni 2019 10:47 > > > To: ceph-users > > > Subject: [ceph-users] Expected IO in luminous Ceph Cluster > > > > > > Hello folks, > > > > > > we are running a ceph cluster on Luminous consisting of 21 OSD > Nodes > > > with 9 8TB SATA drives and 3 Intel 3700 SSDs for Bluestore WAL > and DB > > > > > > (1:3 Ratio). OSDs have 10Gb for Public and Cluster Network. The > > > cluster > > > is running stable for over a year. We didn’t had a closer look > on IO > > > until one of our customers started to complain about a VM we > migrated > > > > > > from VMware with Netapp Storage to our Openstack Cloud with > ceph > > > storage. He sent us a sysbench report from the machine, which > I could > > > > > > reproduce on other VMs as well as on a mounted RBD on physical > > > hardware: > > > > > > sysbench --file-fsync-freq=1 --threads=16 fileio > --file-total-size=1G > > > > > > --file-test-mode=rndrw --file-rw-ratio=2 run sysbench 1.0.11 > (using > > > system LuaJIT 2.1.0-beta3) > > > > > > Running the test with following options: > > > Number of threads: 16 > > > Initializing random number generator from current time > > > > > > Extra file open flags: 0 > > > 128 files, 8MiB each > > > 1GiB total file size > > > Block size 16KiB > > > Number of IO requests: 0 > > > Read/Write ratio for combined random IO test: 2.00 Periodic > FSYNC > > > enabled, calling fsync() each 1 requests. > > > Calling fsync() at the end of test, Enabled. > > > Using synchronous I/O mode > > > Doing random r/w test > > > > > > File operations: > > > reads/s: 36.36 > > > writes/s: 18.18 > > > fsyncs/s: 2318.59 > > > > > > Throughput: > > > read, MiB/s: 0.57 > > > written, MiB/s: 0.28 > > > > > > General statistics: > > > total time: 10.0071s > > > total number of events: 23755 > > > > > > Latency (ms): > > > min: 0.01 > > > avg: 6.74 > > > max: 1112.58 > > > 95th percentile: 26.68 > > > sum: 160022.67 > > > > > > Threads fairness: > > > events (avg/stddev): 1484.6875/52.59 > > > execution time (avg/stddev): 10.0014/0.00 > > > > > > Are these numbers reasonable for a cluster of our size? > > > > > > Best regards > > > Felix > > > IT-Services > > > Telefon 02461 61-9243 > > > E-Mail: mailto:f.sto...@fz-juelich.de > > > > > > > ------------------------------------------------------------------------ > > > ------------- > > > > > > > ------------------------------------------------------------------------ > > > ------------- > > > Forschungszentrum Juelich GmbH > > > 52425 Juelich > > > Sitz der Gesellschaft: Juelich > > > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR > B 3498 > > > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen > Huthmacher > > > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt > (Vorsitzender), > > > > > > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald > Bolt, > > > Prof. > > > Dr. Sebastian M. Schmidt > > > > > > > ------------------------------------------------------------------------ > > > ------------- > > > > > > > ------------------------------------------------------------------------ > > > ------------- > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > mailto:ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > _______________________________________________ > > > ceph-users mailing list > > > mailto:ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > _______________________________________________ > ceph-users mailing list > mailto:ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- John Petrini Platforms Engineer tel:215-297-4400 https://coredial.com/ https://www.google.com/maps/place/CoreDial,+LLC/@40.140902,-75.2878857,17z/data=!3m1!4b1!4m5!3m4!1s0x89c6bc587f1cfd47:0x4c79d505f2ee580b!8m2!3d40.140902!4d-75.285697 The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
