Dear CephFS Gurus...
I have a question regarding ceph-fuse and its memory usage.
1./ My Ceph and CephFS setups are the following:
Ceph:
a. ceph 9.0.3
b. 32 OSDs distributed in 4 servers (8 OSD per server).
c. 'osd pool default size = 3' and 'osd pool default min size = 2'
d. All servers running Centos6.7
CephFS:
e. a single mds
f. dedicated pools for data and metadata
g. clients in different locations / sites mounting CephFS via ceph-fuse
h. All servers and clients running Centos6.7
2./ I have been running fio tests in two CephFS clients:
- Client A is in the same data center as all OSDs connected at 1 GbE
- Client B is in a different data center (in another city) also
connected at 1 GbE. However, I've seen that the connection is
problematic, and sometimes, the network performance is well bellow the
theoretical 1 Gbps limit.
- Client A has 24 GB RAM + 98 GB of SWAP and client B has 48 GB of
RAM + 98 GB of SWAP
3./ I have been running some fio write tests (with 128 threads) in both
clients, and surprisingly, the results show that the aggregated
throughput is better for client B than client A.
CLIENT A results:
# grep agg
fio128threadsALL/fio128write_ioenginelibaio_iodepth64_direct1_bs512K_20151001015558.out
WRITE: io=1024.0GB, *aggrb=114878KB/s*, minb=897KB/s, maxb=1785KB/s,
mint=4697347msec, maxt=9346754msec
CLIENT B results:
# grep agg
fio128threadsALL/fio128write_ioenginelibaio_iodepth64_direct1_bs512K_20151001015555.out
WRITE: io=1024.0GB,*aggrb=483254KB/s*, minb=3775KB/s, maxb=3782KB/s,
mint=2217808msec, maxt=2221896msec
4./ If I actually monitor the memory usage of ceph-fuse during the I/O
tests, I see that
CLIENT A: ceph-fuse does not seem to go behond 7GB of VMEM and 1 GB
of RMEM.
CLIENT B: ceph-fuse uses 11 GB of VMEM and 7 GB of RMEM.
5./ These results make me think that caching is playing a critical role
in these results.
My questions are the following:
a./ Why CLIENT B uses more memory than CLIENT A? My hint is that
there is a network bottleneck between CLIENT B and the Ceph Cluster,
and memory is more used because of that.
b/ Is the FIO write performance better in CLIENT B a consequence of
the fact that it is using more memory than client A?
c./ Is there a parameters we can set for the CEPHFS clients to limit
the amount of memory they can use?
Cheers
Goncalo
--
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW 2006
T: +61 2 93511937
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
