On Tue, Jun 19, 2018 at 1:04 AM Jialin Liu <jaln...@lbl.gov> wrote: > > Hi Dan, Thanks for the follow-ups. > > I have just tried running multiple librados MPI applications from multiple > nodes, it does show increased bandwidth, > with ceph -w, I observed as high as 500MB/sec (previously only 160MB/sec ), I > think I can do finer tuning by > coordinating more concurrent applications to get the peak. (Sorry, I only > have one node having rados cli installed, so I can't follow your example to > stress the server) > >> Then you can try different replication or erasure coding settings to >> learn their impact on performance... > > > Good points. > >> >> PPS. What are those 21.8TB devices ? > > > The storage arrays are Nexsan E60 arrays having two active-active redundant > controllers, 60 3 TB disk drives. The disk drives are organized into six 8+2 > Raid 6 LUNs of 24 TB each. >
This is not the ideal Ceph hardware. Ceph is designed to use disks directly -- JBODs. All redundancy is handled at the RADOS level, so you can happily save lots of cash on your servers. I suggest reading through the various Ceph hardware recommendations that you can find via Google. I can't tell from here if this is the root cause of your performance issue -- but you should plan future clusters to use JBODs instead of expensive arrays. > >> >> PPPS. Any reason you are running jewel instead of luminous or mimic? > > > I have to ask the cluster admin, I'm not sure about it. > > I have one more questions, regarding the OSD server and OSDs, I was told that > the IO has to go through the 4 OSD servers (hosts), before touching the OSDs, > This is confusing to me, as I learned from the ceph document > http://docs.ceph.com/docs/jewel/rados/operations/monitoring-osd-pg/#monitoring-osds > the librados can talk to the OSDs directly, what am I missing here? You should have one ceph-osd process per disk (or per LUN in your case). The clients connect to the ceph-osd processes directly. -- dan > > > Best, > Jialin > NERSC/LBNL > > >> >> On Mon, Jun 18, 2018 at 3:43 PM Jialin Liu <jaln...@lbl.gov> wrote: >> > >> > Hi, To make the the problem clearer, here is the configuration of the >> > cluster: >> > >> > The 'problem' I have is the low bandwidth no matter how I increase the >> > concurrency. >> > I have tried using MPI to launch 322 processes, each calling librados to >> > create a handle and initialize the io context, and write one 80MB object. >> > I only got ~160 MB/sec, with one process, I can get ~40 MB/sec, I'm >> > wondering if the number of client-osd connection is limited by the number >> > of hosts. >> > >> > Best, >> > Jialin >> > NERSC/LBNL >> > >> > $ceph osd tree >> > >> > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >> > >> > -1 1047.59473 root default >> > >> > -2 261.89868 host ngfdv036 >> > >> > 0 21.82489 osd.0 up 1.00000 1.00000 >> > >> > 4 21.82489 osd.4 up 1.00000 1.00000 >> > >> > 8 21.82489 osd.8 up 1.00000 1.00000 >> > >> > 12 21.82489 osd.12 up 1.00000 1.00000 >> > >> > 16 21.82489 osd.16 up 1.00000 1.00000 >> > >> > 20 21.82489 osd.20 up 1.00000 1.00000 >> > >> > 24 21.82489 osd.24 up 1.00000 1.00000 >> > >> > 28 21.82489 osd.28 up 1.00000 1.00000 >> > >> > 32 21.82489 osd.32 up 1.00000 1.00000 >> > >> > 36 21.82489 osd.36 up 1.00000 1.00000 >> > >> > 40 21.82489 osd.40 up 1.00000 1.00000 >> > >> > 44 21.82489 osd.44 up 1.00000 1.00000 >> > >> > -3 261.89868 host ngfdv037 >> > >> > 1 21.82489 osd.1 up 1.00000 1.00000 >> > >> > 5 21.82489 osd.5 up 1.00000 1.00000 >> > >> > 9 21.82489 osd.9 up 1.00000 1.00000 >> > >> > 13 21.82489 osd.13 up 1.00000 1.00000 >> > >> > 17 21.82489 osd.17 up 1.00000 1.00000 >> > >> > 21 21.82489 osd.21 up 1.00000 1.00000 >> > >> > 25 21.82489 osd.25 up 1.00000 1.00000 >> > >> > 29 21.82489 osd.29 up 1.00000 1.00000 >> > >> > 33 21.82489 osd.33 up 1.00000 1.00000 >> > >> > 37 21.82489 osd.37 up 1.00000 1.00000 >> > >> > 41 21.82489 osd.41 up 1.00000 1.00000 >> > >> > 45 21.82489 osd.45 up 1.00000 1.00000 >> > >> > -4 261.89868 host ngfdv038 >> > >> > 2 21.82489 osd.2 up 1.00000 1.00000 >> > >> > 6 21.82489 osd.6 up 1.00000 1.00000 >> > >> > 10 21.82489 osd.10 up 1.00000 1.00000 >> > >> > 14 21.82489 osd.14 up 1.00000 1.00000 >> > >> > 18 21.82489 osd.18 up 1.00000 1.00000 >> > >> > 22 21.82489 osd.22 up 1.00000 1.00000 >> > >> > 26 21.82489 osd.26 up 1.00000 1.00000 >> > >> > 30 21.82489 osd.30 up 1.00000 1.00000 >> > >> > 34 21.82489 osd.34 up 1.00000 1.00000 >> > >> > 38 21.82489 osd.38 up 1.00000 1.00000 >> > >> > 42 21.82489 osd.42 up 1.00000 1.00000 >> > >> > 46 21.82489 osd.46 up 1.00000 1.00000 >> > >> > -5 261.89868 host ngfdv039 >> > >> > 3 21.82489 osd.3 up 1.00000 1.00000 >> > >> > 7 21.82489 osd.7 up 1.00000 1.00000 >> > >> > 11 21.82489 osd.11 up 1.00000 1.00000 >> > >> > 15 21.82489 osd.15 up 1.00000 1.00000 >> > >> > 19 21.82489 osd.19 up 1.00000 1.00000 >> > >> > 23 21.82489 osd.23 up 1.00000 1.00000 >> > >> > 27 21.82489 osd.27 up 1.00000 1.00000 >> > >> > 31 21.82489 osd.31 up 1.00000 1.00000 >> > >> > 35 21.82489 osd.35 up 1.00000 1.00000 >> > >> > 39 21.82489 osd.39 up 1.00000 1.00000 >> > >> > 43 21.82489 osd.43 up 1.00000 1.00000 >> > >> > 47 21.82489 osd.47 up 1.00000 1.00000 >> > >> > >> > ceph -s >> > >> > cluster 2b0e2d2b-3f63-4815-908a-b032c7f9427a >> > >> > health HEALTH_OK >> > >> > monmap e1: 2 mons at >> > {ngfdv076=128.55.xxx.xx:6789/0,ngfdv078=128.55.xxx.xx:6789/0} >> > >> > election epoch 4, quorum 0,1 ngfdv076,ngfdv078 >> > >> > osdmap e280: 48 osds: 48 up, 48 in >> > >> > flags sortbitwise,require_jewel_osds >> > >> > pgmap v117283: 3136 pgs, 11 pools, 25600 MB data, 510 objects >> > >> > 79218 MB used, 1047 TB / 1047 TB avail >> > >> > 3136 active+clean >> > >> > >> > >> > On Mon, Jun 18, 2018 at 1:06 AM Jialin Liu <jaln...@lbl.gov> wrote: >> >> >> >> Thank you Dan. I’ll try it. >> >> >> >> Best, >> >> Jialin >> >> NERSC/LBNL >> >> >> >> > On Jun 18, 2018, at 12:22 AM, Dan van der Ster <d...@vanderster.com> >> >> > wrote: >> >> > >> >> > Hi, >> >> > >> >> > One way you can see exactly what is happening when you write an object >> >> > is with --debug_ms=1. >> >> > >> >> > For example, I write a 100MB object to a test pool: rados >> >> > --debug_ms=1 -p test put 100M.dat 100M.dat >> >> > I pasted the output of this here: https://pastebin.com/Zg8rjaTV >> >> > In this case, it first gets the cluster maps from a mon, then writes >> >> > the object to osd.58, which is the primary osd for PG 119.77: >> >> > >> >> > # ceph pg 119.77 query | jq .up >> >> > [ >> >> > 58, >> >> > 49, >> >> > 31 >> >> > ] >> >> > >> >> > Otherwise I answered your questions below... >> >> > >> >> >> On Sun, Jun 17, 2018 at 8:29 PM Jialin Liu <jaln...@lbl.gov> wrote: >> >> >> >> >> >> Hello, >> >> >> >> >> >> I have a couple questions regarding the IO on OSD via librados. >> >> >> >> >> >> >> >> >> 1. How to check which osd is receiving data? >> >> >> >> >> > >> >> > See `ceph osd map`. >> >> > For my example above: >> >> > >> >> > # ceph osd map test 100M.dat >> >> > osdmap e236396 pool 'test' (119) object '100M.dat' -> pg 119.864b0b77 >> >> > (119.77) -> up ([58,49,31], p58) acting ([58,49,31], p58) >> >> > >> >> >> 2. Can the write operation return immediately to the application once >> >> >> the write to the primary OSD is done? or does it return only when the >> >> >> data is replicated twice? (size=3) >> >> > >> >> > Write returns once it is safe on *all* replicas or EC chunks. >> >> > >> >> >> 3. What is the I/O size in the lower level in librados, e.g., if I >> >> >> send a 100MB request with 1 thread, does librados send the data by a >> >> >> fixed transaction size? >> >> > >> >> > This depends on the client. The `rados` CLI example I showed you broke >> >> > the 100MB object into 4MB parts. >> >> > Most use-cases keep the objects around 4MB or 8MB. >> >> > >> >> >> 4. I have 4 OSS, 48 OSDs, will the 4 OSS become the bottleneck? from >> >> >> the ceph documentation, once the cluster map is received by the >> >> >> client, the client can talk to OSD directly, so the assumption is the >> >> >> max parallelism depends on the number of OSDs, is this correct? >> >> >> >> >> > >> >> > That's more or less correct -- the IOPS and BW capacity of the cluster >> >> > generally scales linearly with number of OSDs. >> >> > >> >> > Cheers, >> >> > Dan >> >> > CERN _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
