Re: Teuthology Integration to native openstack

[Loic Dachary]

[moving to ceph-devel]

Hi,

On 27/09/2015 21:20, Bharath Krishna wrote:
> Hi,
> 
> We have an openstack deployment in place with CEPH as CINDER backend.
> 
> We would like to perform functional testing for CEPH and found teuthology as 
> recommended option.
> 
> Have successfully installed teuthology. Now to integrate it with Openstack, I 
> could see that the possible providers could be either OVH, REDHAT or 
> ENTERCLOUDSITE.
> 
> Is there any option where in we can source openstack deployment of our own 
> and test CEPH using teuthology?

The documentation mentions these providers because they have been tested. But 
there should be no blocker to run teuthology against a regular OpenStack 
provider. Should you run into troubles, please let me know and I'll help.

Cheers

> 
> If NO, please suggest on how to test CEPH in such scenarios?
> 
> Please help.
> 
> Thank you.
> Bharath Krishna
> ___
> ceph-users mailing list
> ceph-us...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature

Isn't --allhosts dangerous?

[Nathan Cutler]

Taking actions cluster-wide via a shell script seems like it could have
dangerous unexpected side effects, and indeed while looking through old
tickets, I ran across http://tracker.ceph.com/issues/9407 . . .

This is "fixed" (by getting rid of the --allhosts option to init-ceph)
in https://github.com/ceph/ceph/pull/6089

The same PR also adds getopt to make processing of command-line options
more flexible.

Opinions? Reviews?

-- 
Nathan Cutler
Software Engineer Distributed Storage
SUSE LINUX, s.r.o.
Tel.: +420 284 084 037
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: a patch to improve cephfs direct io performance

[zhucaifeng]

Hi， Yan

Your comment is insightful. It is about the loop invariant, the 
condition which

must be satisfied when iovs can be combined into one page vector. The loop
invariant is ensured in function dio_get_pagevlen.  Note that 
dio_get_pagevlen
should be called before dio_alloc_pagev, so in dio_alloc_pagev, this 
loop invariant

has been ensured indirectly.  Please see my inline comments.

I'll deliver a new patch for the newest kernel, using iov_iter KPIs.

Best Regards!



在 2015/9/28 11:17, Yan, Zheng 写道:

Hi, zhucaifeng

The patch generally looks good. But I think there is minor flaw, see
my comment below. Besides, could you write a patch for the newest
kernel (it's easier because there is iov_iter_get_pages() helper).


On Fri, Sep 25, 2015 at 11:52 AM, zhucaifeng  wrote:

Hi, all

When using cephfs, we find that cephfs direct io is very slow.
For example, installing Windows 7 takes more than 1 hour on a
virtual machine whose disk is a file in cephfs.

The cause is that when doing direct io, both ceph_sync_direct_write
and ceph_sync_read iterate iov elements one by one, and send one
osd_op request for each iov that covers about 4K~8K bytes data. The
result is lots of small requests and replies shuttled among the kernel
client and OSDs.

The fix tries to combine as many iovs as possible into one page vector, and
then send one request for this page vector. In this way, small requests and
replies are reduced; the OSD can serve a large read/write request one time.

We observe that the performance is improved by about 5~15 times, dependent
on
different application workload.

The fix is below. Some change notes are as follows:
 - function ceph_get_direct_page_vetor and ceph_put_page_vector
   are moved from net/ceph/pagevec.c to fs/ceph/file.c, and are
   renamed as dio_grab_pages and dio_free_pagev respectively.

 - function dio_get_pagevlen and dio_alloc_pagev are newly introduced.

   dio_get_pagevlen is to calculate the page vector length in bytes
   from the io vectors. Except for the first and last one, an io vector
   can be merged into a page vector iff its base and tail are page
aligned.
   for the first vector, only its tail should be page aligned; for the
last
   vector, only its head should be page aligned.

   dio_alloc_pagev allocates pages for each io vector and put these pages
   together in one page pointer array.

Best Regards!

--- /home/linux-3.10.0-229.7.2.el7/fs/ceph/file.c2015-05-16
09:05:28.0 +0800
+++ ./fs/ceph/file.c2015-09-17 10:43:44.798591846 +0800
@@ -34,6 +34,115 @@
   * need to wait for MDS acknowledgement.
   */

+/*
+ * Calculate the length sum of direct io vectors that can
+ * be combined into one page vector.
+ */
+static int
+dio_get_pagevlen(const struct iov_iter *it)
+{
+const struct iovec *iov = it->iov;
+const struct iovec *iovend = iov + it->nr_segs;
+int pagevlen;
+
+pagevlen = iov->iov_len - it->iov_offset;
+/*
+ * An iov can be page vectored when both the current tail
+ * and the next base are page aligned.
+ */
+while (PAGE_ALIGNED((iov->iov_base + iov->iov_len)) &&
+   (++iov < iovend && PAGE_ALIGNED((iov->iov_base {
+pagevlen += iov->iov_len;
+}
+dout("dio_get_pagevlen len = %d\n", pagevlen);
+return pagevlen;
+}
+
+/*
+ * Grab @num_pages from the process vm space. These pages are
+ * continuous and start from @data.
+ */
+static int
+dio_grab_pages(const void *data, int num_pages, bool write_page,
+struct page **pages)
+{
+int got = 0;
+int rc = 0;
+
+down_read(>mm->mmap_sem);
+while (got < num_pages) {
+rc = get_user_pages(current, current->mm,
+(unsigned long)data + ((unsigned long)got * PAGE_SIZE),
+num_pages - got, write_page, 0, pages + got, NULL);
+if (rc < 0)
+break;
+BUG_ON(rc == 0);
+got += rc;
+}
+up_read(>mm->mmap_sem);
+return got;
+}
+
+static void
+dio_free_pagev(struct page **pages, int num_pages, bool dirty)
+{
+int i;
+
+for (i = 0; i < num_pages; i++) {
+if (dirty)
+set_page_dirty_lock(pages[i]);
+put_page(pages[i]);
+}
+kfree(pages);
+}
+
+/*
+ * Allocate a page vector based on (@it, @pagevlen).
+ * The return value is the tuple describing a page vector,
+ * that is (@pages, @pagevlen, @page_align, @num_pages).
+ */
+static struct page **
+dio_alloc_pagev(const struct iov_iter *it, int pagevlen, bool write_page,
+int *page_align, int *num_pages)
+{
+const struct iovec *iov = it->iov;
+struct page **pages;
+int n, m, k, npages;
+int align;
+int len;
+void *data;
+
+data = iov->iov_base + it->iov_offset;
+len = iov->iov_len - it->iov_offset;
+align = ((ulong)data) & ~PAGE_MASK;
+npages = calc_pages_for((ulong)data, pagevlen);
+pages = kmalloc(sizeof(*pages) * npages, GFP_NOFS);
+if (!pages)
+   

Re: a patch to improve cephfs direct io performance

[Yan, Zheng]

Hi, zhucaifeng

The patch generally looks good. But I think there is minor flaw, see
my comment below. Besides, could you write a patch for the newest
kernel (it's easier because there is iov_iter_get_pages() helper).


On Fri, Sep 25, 2015 at 11:52 AM, zhucaifeng  wrote:
> Hi, all
>
> When using cephfs, we find that cephfs direct io is very slow.
> For example, installing Windows 7 takes more than 1 hour on a
> virtual machine whose disk is a file in cephfs.
>
> The cause is that when doing direct io, both ceph_sync_direct_write
> and ceph_sync_read iterate iov elements one by one, and send one
> osd_op request for each iov that covers about 4K~8K bytes data. The
> result is lots of small requests and replies shuttled among the kernel
> client and OSDs.
>
> The fix tries to combine as many iovs as possible into one page vector, and
> then send one request for this page vector. In this way, small requests and
> replies are reduced; the OSD can serve a large read/write request one time.
>
> We observe that the performance is improved by about 5~15 times, dependent
> on
> different application workload.
>
> The fix is below. Some change notes are as follows:
> - function ceph_get_direct_page_vetor and ceph_put_page_vector
>   are moved from net/ceph/pagevec.c to fs/ceph/file.c, and are
>   renamed as dio_grab_pages and dio_free_pagev respectively.
>
> - function dio_get_pagevlen and dio_alloc_pagev are newly introduced.
>
>   dio_get_pagevlen is to calculate the page vector length in bytes
>   from the io vectors. Except for the first and last one, an io vector
>   can be merged into a page vector iff its base and tail are page
> aligned.
>   for the first vector, only its tail should be page aligned; for the
> last
>   vector, only its head should be page aligned.
>
>   dio_alloc_pagev allocates pages for each io vector and put these pages
>   together in one page pointer array.
>
> Best Regards!
>
> --- /home/linux-3.10.0-229.7.2.el7/fs/ceph/file.c2015-05-16
> 09:05:28.0 +0800
> +++ ./fs/ceph/file.c2015-09-17 10:43:44.798591846 +0800
> @@ -34,6 +34,115 @@
>   * need to wait for MDS acknowledgement.
>   */
>
> +/*
> + * Calculate the length sum of direct io vectors that can
> + * be combined into one page vector.
> + */
> +static int
> +dio_get_pagevlen(const struct iov_iter *it)
> +{
> +const struct iovec *iov = it->iov;
> +const struct iovec *iovend = iov + it->nr_segs;
> +int pagevlen;
> +
> +pagevlen = iov->iov_len - it->iov_offset;
> +/*
> + * An iov can be page vectored when both the current tail
> + * and the next base are page aligned.
> + */
> +while (PAGE_ALIGNED((iov->iov_base + iov->iov_len)) &&
> +   (++iov < iovend && PAGE_ALIGNED((iov->iov_base {
> +pagevlen += iov->iov_len;
> +}
> +dout("dio_get_pagevlen len = %d\n", pagevlen);
> +return pagevlen;
> +}
> +
> +/*
> + * Grab @num_pages from the process vm space. These pages are
> + * continuous and start from @data.
> + */
> +static int
> +dio_grab_pages(const void *data, int num_pages, bool write_page,
> +struct page **pages)
> +{
> +int got = 0;
> +int rc = 0;
> +
> +down_read(>mm->mmap_sem);
> +while (got < num_pages) {
> +rc = get_user_pages(current, current->mm,
> +(unsigned long)data + ((unsigned long)got * PAGE_SIZE),
> +num_pages - got, write_page, 0, pages + got, NULL);
> +if (rc < 0)
> +break;
> +BUG_ON(rc == 0);
> +got += rc;
> +}
> +up_read(>mm->mmap_sem);
> +return got;
> +}
> +
> +static void
> +dio_free_pagev(struct page **pages, int num_pages, bool dirty)
> +{
> +int i;
> +
> +for (i = 0; i < num_pages; i++) {
> +if (dirty)
> +set_page_dirty_lock(pages[i]);
> +put_page(pages[i]);
> +}
> +kfree(pages);
> +}
> +
> +/*
> + * Allocate a page vector based on (@it, @pagevlen).
> + * The return value is the tuple describing a page vector,
> + * that is (@pages, @pagevlen, @page_align, @num_pages).
> + */
> +static struct page **
> +dio_alloc_pagev(const struct iov_iter *it, int pagevlen, bool write_page,
> +int *page_align, int *num_pages)
> +{
> +const struct iovec *iov = it->iov;
> +struct page **pages;
> +int n, m, k, npages;
> +int align;
> +int len;
> +void *data;
> +
> +data = iov->iov_base + it->iov_offset;
> +len = iov->iov_len - it->iov_offset;
> +align = ((ulong)data) & ~PAGE_MASK;
> +npages = calc_pages_for((ulong)data, pagevlen);
> +pages = kmalloc(sizeof(*pages) * npages, GFP_NOFS);
> +if (!pages)
> +return ERR_PTR(-ENOMEM);
> +for (n = 0; n < npages; n += m) {
> +m = calc_pages_for((ulong)data, len);
> +if (n + m > npages)
> +m = npages - n;
> +k = dio_grab_pages(data, m, write_page, pages + n);
> +if (k < m) {
> +  

Re: Teuthology Integration to native openstack

[Bharath Krishna]

Hi Dachary,

Thanks for the reply. I am following your blog http://dachary.org/?p=3767
And the README in 
https://github.com/dachary/teuthology/tree/wip-6502-openstack-v2/#openstack
-backend

I have sourced the openrc file of my Openstack deployment and verified
that clients are working fine. My Openstack deployment has Cinder
integrated with CEPH backend.

I have cloned and installed teuthology using the below steps:

$ git clone -b wip-6502-openstack-v2 http://github.com/dachary/teuthology
$ cd teuthology ; ./bootstrap install
$ source virtualenv/bin/activate


Then I tried to run a dummy suite as test and I ran into following error:

Traceback (most recent call last):
  File "/root/teuthology/virtualenv/bin/teuthology-openstack", line 9, in

load_entry_point('teuthology==0.1.0', 'console_scripts',
'teuthology-openstack')()
  File "/root/teuthology/scripts/openstack.py", line 8, in main
teuthology.openstack.main(parse_args(argv), argv)
  File "/root/teuthology/teuthology/openstack.py", line 375, in main
return TeuthologyOpenStack(ctx, teuth_config, argv).main()
  File "/root/teuthology/teuthology/openstack.py", line 181, in main
self.verify_openstack()
  File "/root/teuthology/teuthology/openstack.py", line 270, in
verify_openstack
str(providers))
Exception: ('OS_AUTH_URL=http://:5000/v2.0', " does is not a
known OpenStack provider (('cloud.ovh.net', 'ovh'), ('control.os1.phx2',
'redhat'), ('entercloudsuite.com', 'entercloudsuite'))")


Thank you.

Regards,
M Bharath Krishna

On 9/28/15, 1:47 AM, "Loic Dachary"  wrote:

>[moving to ceph-devel]
>
>Hi,
>
>On 27/09/2015 21:20, Bharath Krishna wrote:
>> Hi,
>> 
>> We have an openstack deployment in place with CEPH as CINDER backend.
>> 
>> We would like to perform functional testing for CEPH and found
>>teuthology as recommended option.
>> 
>> Have successfully installed teuthology. Now to integrate it with
>>Openstack, I could see that the possible providers could be either OVH,
>>REDHAT or ENTERCLOUDSITE.
>> 
>> Is there any option where in we can source openstack deployment of our
>>own and test CEPH using teuthology?
>
>The documentation mentions these providers because they have been tested.
>But there should be no blocker to run teuthology against a regular
>OpenStack provider. Should you run into troubles, please let me know and
>I'll help.
>
>Cheers
>
>> 
>> If NO, please suggest on how to test CEPH in such scenarios?
>> 
>> Please help.
>> 
>> Thank you.
>> Bharath Krishna
>> ___
>> ceph-users mailing list
>> ceph-us...@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
>
>-- 
>Loïc Dachary, Artisan Logiciel Libre
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Isn't --allhosts dangerous?

[Sage Weil]

On Sun, 27 Sep 2015, Nathan Cutler wrote:
> Taking actions cluster-wide via a shell script seems like it could have
> dangerous unexpected side effects, and indeed while looking through old
> tickets, I ran across http://tracker.ceph.com/issues/9407 . . .
> 
> This is "fixed" (by getting rid of the --allhosts option to init-ceph)
> in https://github.com/ceph/ceph/pull/6089
> 
> The same PR also adds getopt to make processing of command-line options
> more flexible.
> 
> Opinions? Reviews?

I certainly won't miss it, but I suspect a fair number of people use it 
(e.g., service ceph -a start).  The general strategy is to avoid adding 
any such hacks to the newer init system scripts, so as everyone moves to 
systemd this will go away.

sage
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html