Hey, guys.
I'm still feeling unlucky about these experiments. Here's what I did:
1) Setthe parameters described below in ceph.conf
2) Pushthe ceph.conf to all nodes using ceph-deploy
3) Restartmonitor, mds and osd’s on all nodes
4) Ranthe test twice atleast and look at the results which came out the
second time.
5) Imounted the filesystem using mount –t ceph 10.76.38.57:/ /mnt/mycephfs
–oname=admin,secret=xxxxxxxxx
6) Ran the benchmark
I modified the following parameters and ran each testseparately
- “osd journal size” to 5 GB and 10 GB and 20 GB
- “osd client message size cap” to 0,1,1024
- “osd pool default min size” to 1 and 3
In all the above I observed similar pattern as below.
- about 5-6 Gbps throughput at the start of the test and
graduallydropping to 900 Mbps till the test completes. I also observed that
post 150-160files being written there is a wait for about 10-15 seconds before
the nextfile is written.
Test using FUSE.
I also installed FUSE on cephnode1 and mounted the fusemount using the
following command
ceph-fuse –m 10.76.38.56 /mnt/mycephfs
I saw a drastic reduction in write throughput to around 170Mbps. The system
took about 5-6 seconds before it started writing any files andwas constantly at
150 – 180 Mbps write through put when the directory was mountedusing FUSE.
Any additional thoughts? Would the problem be due to my NFS client?
Regards,
F.
From: Gregory Farnum <[email protected]>
To: Patrick Donnelly <[email protected]>
Cc: Fabiano de O. Lucchese <[email protected]>; "[email protected]"
<[email protected]>
Sent: Tuesday, July 19, 2016 5:23 PM
Subject: Re: [ceph-users] CephFS write performance
On Tue, Jul 19, 2016 at 9:39 AM, Patrick Donnelly <[email protected]> wrote:
> On Tue, Jul 19, 2016 at 10:25 AM, Fabiano de O. Lucchese
> <[email protected]> wrote:
>> I configured the cluster to replicate data twice (3 copies), so these
>> numbers fall within my expectations. So far so good, but here's comes the
>> issue: I configured CephFS and mounted a share locally on one of my servers.
>> When I write data to it, it shows abnormally high performance at the
>> beginning for about 5 seconds, stalls for about 20 seconds and then picks up
>> again. For long running tests, the observed write throughput is very close
>> to what the rados bench provided (about 640 MB/s), but for short-lived
>> tests, I get peak performances of over 5GB/s. I know that journaling is
>> expected to cause spiky performance patters like that, but not to this
>> level, which makes me think that CephFS is buffering my writes and returning
>> the control back to client before persisting them to the jounal, which looks
>> undesirable.
>
> The client is buffering the writes to RADOS which would give you the
> abnormally high initial performance until the cache needs flushed. You
> might try tweaking certain osd settings:
>
> http://docs.ceph.com/docs/hammer/rados/configuration/osd-config-ref/
>
> in particular: "osd client message size cap". Also:
I am reasonably sure you don't want to change the message size cap;
that's entirely an OSD-side throttle about how much dirty data it lets
in before it stops reading off the wire — and I don't think the client
feeds back from outgoing data. More likely it's about how much dirty
data is being absorbed by the Client before it forces writes out to
the OSDs and you want to look at
client_oc_size (default 1024*1024*200, aka 200MB)
client_oc_max_dirty (default 100MB)
client_oc_target_dirty (default 8MB)
and turn down the max dirty limits if you're finding it's too bumpy a ride.
-Greg
>
> http://docs.ceph.com/docs/hammer/rados/configuration/journal-ref/
>
> --
> Patrick Donnelly
> _______________________________________________
> ceph-users mailing list
> [email protected]
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
