Re: [openstack-dev] [Manila] Ceph native driver for manila

2015-03-04 Thread Gregory Farnum
On Wed, Mar 4, 2015 at 7:03 AM, Csaba Henk  wrote:
>
>
> - Original Message -
>> From: "Danny Al-Gaaf" 
>> To: "Csaba Henk" , "OpenStack Development Mailing List 
>> (not for usage questions)"
>> 
>> Cc: ceph-de...@vger.kernel.org
>> Sent: Wednesday, March 4, 2015 3:26:52 PM
>> Subject: Re: [openstack-dev] [Manila] Ceph native driver for manila
>>
>> Am 04.03.2015 um 15:12 schrieb Csaba Henk:
>> > - Original Message -
>> >> From: "Danny Al-Gaaf"  To: "OpenStack
>> >> Development Mailing List (not for usage questions)"
>> >> , ceph-de...@vger.kernel.org
>> >> Sent: Sunday, March 1, 2015 3:07:36 PM Subject: Re:
>> >> [openstack-dev] [Manila] Ceph native driver for manila
>> > ...
>> >> For us security is very critical, as the performance is too. The
>> >> first solution via ganesha is not what we prefer (to use CephFS
>> >> via p9 and NFS would not perform that well I guess). The second
>> >> solution, to use
>> >
>> > Can you please explain that why does the Ganesha based stack
>> > involve 9p? (Maybe I miss something basic, but I don't know.)
>>
>> Sorry, seems that I mixed it up with the p9 case. But the performance
>> is may still an issue if you use NFS on top of CephFS (incl. all the
>> VM layer involved within this setup).
>>
>> For me the question with all these NFS setups is: why should I use NFS
>> on top on CephFS? What is the right to exist of CephFS in this case? I
>> would like to use CephFS directly or via filesystem passthrough.
>
> That's a good question. Or indeed, two questions:
>
> 1. Why to use NFS?
> 2. Why does the NFS export of Ceph need to involve CephFS?
>
> 1.
>
> As of "why NFS" -- it's probably a good selling point that it's
> standard filesystem export technology and the tenants can remain
> backend-unaware as long as the backend provides NFS export.
>
> We are working on the Ganesha library --
>
> https://blueprints.launchpad.net/manila/+spec/gateway-mediated-with-ganesha
>
> with the aim to make it easy to create Ganesha based drivers. So if you have
> already an FSAL, you can get at an NFS exporting driver almost for free (with 
> a
> modest amount of glue code). So you could consider making such a driver for
> Ceph, to satisfy customers who demand NFS access, even if there is a native
> driver which gets the limelight.
>
> (See commits implementing this under "Work Items" of the BP -- one is the
> actual Ganesha library and the other two show how it can be hooked in, by the
> example of the Gluster driver. At the moment flat network (share-server-less)
> drivers are supported.)
>
> 2.
>
> As of why CephFS was the technology chosen for implementing the Ceph FSAL for
> Ganesha, that's something I'd also like to know. I have the following naive
> question in mind: "Would it not have been better to implement Ceph FSAL with
> something »closer to« Ceph?", and I have three actual questions about it:
>
> - does this question make sense in this form, and if not, how to amend?
> - I'm asking the question itself, or the amended version of it.
> - If the answer is yes, is there a chance someone would create an alternative
>   Ceph FSAL on that assumed closer-to-Ceph technology?

I don't understand. What "closer-to-Ceph" technology do you want than
native use of the libcephfs library? Are you saying to use raw RADOS
to provide storage instead of CephFS?

In that case, it doesn't make a lot of sense: CephFS is how you
provide a real filesystem in the Ceph ecosystem. I suppose if you
wanted to create a lighter-weight pseudo-filesystem you could do so
(somebody is building a "RadosFS", I think from CERN?) but then it's
not interoperable with other stuff.
-Greg

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ceph] Why performance of benchmarks with small blocks is extremely small?

2014-09-30 Thread Gregory Farnum
On Sat, Sep 27, 2014 at 8:14 AM, Timur Nurlygayanov
 wrote:
> Hello all,
>
> I installed OpenStack with Glance + Ceph OSD with replication factor 2 and
> now I can see the write operations are extremly slow.
> For example, I can see only 0.04 MB/s write speed when I run rados bench
> with 512b blocks:
>
> rados bench -p test 60 write --no-cleanup -t 1 -b 512
>
>  Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0
> objects
>  Object prefix: benchmark_data_node-17.domain.tld_15862
>sec Cur ops   started  finishedavg MB/s cur MB/s   last lat
> avg lat
>  0   0 0 0  00
> -   0
>  1   183820.0400341   0.0400391
> 0.008465   0.0120985
>  2   1   169   168  0.04101110.0419922
> 0.080433   0.0118995
>  3   1   240   239  0.03889590.034668
> 0.008052   0.0125385
>  4   1   356   355  0.0433309   0.0566406
> 0.00837 0.0112662
>  5   1   472   471  0.0459919   0.0566406
> 0.008343   0.0106034
>  6   1   550   549  0.0446735   0.0380859
> 0.036639   0.0108791
>  7   1   581   580  0.0404538   0.0151367
> 0.008614   0.0120654
>
>
> My test environment configuration:
> Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores per
> node, HDDs WDC WD5003ABYX-01WERA0.
> OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate
> nodes).
> CentOS 6.5, kernel 2.6.32-431.el6.x86_64.
>
> I tested several config options for optimizations, like in
> /etc/ceph/ceph.conf:
>
> [default]
> ...
> osd_pool_default_pg_num = 1024
> osd_pool_default_pgp_num = 1024
> osd_pool_default_flag_hashpspool = true
> ...
> [osd]
> osd recovery max active = 1
> osd max backfills = 1
> filestore max sync interval = 30
> filestore min sync interval = 29
> filestore flusher = false
> filestore queue max ops = 1
> filestore op threads = 16
> osd op threads = 16
> ...
> [client]
> rbd_cache = true
> rbd_cache_writethrough_until_flush = true
>
> and in /etc/cinder/cinder.conf:
>
> [DEFAULT]
> volume_tmp_dir=/tmp
>
> but in the result performance was increased only on ~30 % and it not looks
> like huge success.
>
> Non-default mount options and TCP optimization increase the speed in about
> 1%:
>
> [root@node-17 ~]# mount | grep ceph
> /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs
> (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)
>
> [root@node-17 ~]# cat /etc/sysctl.conf
> net.core.rmem_max = 16777216
> net.core.wmem_max = 16777216
> net.ipv4.tcp_rmem = 4096 87380 16777216
> net.ipv4.tcp_wmem = 4096 65536 16777216
> net.ipv4.tcp_window_scaling = 1
> net.ipv4.tcp_timestamps = 1
> net.ipv4.tcp_sack = 1
>
>
> Do we have other ways to significantly improve CEPH storage performance?
> Any feedback and comments are welcome!

This is entirely latency dominated and OpenStack configuration changes
aren't going to be able to do much — you're getting 80 sequential ops
a second out of a system that has to do two round trips over a network
and hit two hard drives on every operation. You might want to spend
some time looking at how latency, bandwidth, and concurrency are
(often not) related. :)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev