On Mon, May 06, 2013 at 02:36:39PM +0200, Gandalf Corvotempesta wrote:
> 2013/5/6 Andreas Friedrich <[email protected]>:
> > To enable the LD_PRELOAD mechanism for the Ceph daemons only, a little
> > generic extension in the global section of /etc/ceph/ceph.conf would
> > be helpful, e.g.:
> >
> > [global]
> > environment = LD_PRELOAD=/usr/lib64/libsdp.so.1
> >
> > The appending patch adds 5 lines in the Bobtail (0.56.6) init script.
> > The init script will then read the environment setting and - if present -
> > call the Ceph daemons with the preceding environment string.
>
> Cool! We are planning the same infrastructure with IB as networking.
> Could you share more details about this?
> - which performance are you having and on which hardware?
6x Storage-Server, each with 3x Intel-910 PCIe-SSD (OSD-xfs), Journal in 1GB
RAM-Disk per OSD
plus 1x Client-server, each Server with 10GbE, Intel-QDR as well as MLX-FDR
The performance values are still scattering ...
(1 fio client job, 128 QD, 1TB of data, 60 seconds)
e.g.
- read_4m_128, read_8m_128, randread_4m_128, randread_8m_128 approx.
2.2 GB/s on 56 GbIPoIB_CM and 56 GbSDP
- write_4m_128, write_8m_128, randwrite_4m_128, randwrite_8m_128 approx.
500 MB/s on 56 GbIPoIB_CM and 56 GbSDP,
but
900 MS/s on 40 GbIPoIB_CM
> - any issue with IB and SDP?
Well, you have to compile the complete OFED stack (ib_core, ... ib_sdp) from
one vendor.
MLX is running, the Intel status is 'wip' ... the cluster is up, but jerking
...
(OSDs are getting out/in ... timeouts) looks like a shortage on ressource ?!
> - are you able to use the newer rsockets and not SDP? It also has a
> preloader library and is still developed (SDP is deprecated)
rsockets.so is up, but stuttering ...
... and for the moment rsockets is UserLand only, in contrast to sdp.ko
Cheers,
-Dieter
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
