Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

...@lists.ceph.com, ceph-devel ceph-devel@vger.kernel.org mailto:ceph-devel@vger.kernel.org, Milosz Tanski mil...@adfin.com mailto:mil...@adfin.com Envoyé: Vendredi 24 Avril 2015 13:37:52 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

Avril 2015 08:12:49 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops If I want to use librados API for performance testing, are there any existing benchmark tools which directly accesses librados (not through rbd or gateway

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

: Vendredi 24 Avril 2015 20:02:15 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops We haven't done any kind of real testing on jemalloc, so use at your own peril. Having said that, we've also been very interested in hearing community

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

@vger.kernel.org, Milosz Tanski mil...@adfin.com Envoyé: Lundi 27 Avril 2015 07:01:21 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Hi, also another big difference, I can reach now 180k iops with a single jemalloc

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

...@adfin.com Envoyé: Lundi 27 Avril 2015 07:01:21 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Hi, also another big difference, I can reach now 180k iops with a single jemalloc osd (data in buffer) vs

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

Avril 2015 18:34:50 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops On 04/27/2015 10:11 AM, Alexandre DERUMIER wrote: Is it possible that you were suffering from the bug during the first test but once reinstalled you

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

Tanski mil...@adfin.com Envoyé: Lundi 27 Avril 2015 07:01:21 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Hi, also another big difference, I can reach now 180k iops with a single jemalloc osd (data in buffer) vs 50k iops

RE: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

Tanski Subject: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Ok, just to make sure that I understand: tcmalloc un-tuned: ~50k IOPS once bug sets in yes, it's really random, but when hitting the bug, yes this is the worste I

RE: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

, ceph-devel ceph-devel@vger.kernel.org, Milosz Tanski mil...@adfin.com Envoyé: Lundi 27 Avril 2015 18:34:50 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops On 04/27/2015 10:11 AM, Alexandre DERUMIER wrote: Is it possible

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

...@redhat.com Cc: ceph-users ceph-us...@lists.ceph.com, ceph-devel ceph-devel@vger.kernel.org, Milosz Tanski mil...@adfin.com Envoyé: Samedi 25 Avril 2015 06:45:43 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops We haven't

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

On Fri, Apr 24, 2015 at 12:38 PM, Alexandre DERUMIER aderum...@odiso.com wrote: Hi, I have finished to rebuild ceph with jemalloc, all seem to working fine. I got a constant 300k iops for the moment, so no speed regression. I'll do more long benchmark next week. Regards, Alexandre

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

...@adfin.com Envoyé: Vendredi 24 Avril 2015 13:37:52 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Hi,Alexandre! Do not try to change the parameter vm.min_free_kbytes? 2015-04-23 19:24 GMT+03:00 Somnath Roy somnath

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

...@lists.ceph.com, ceph-devel ceph-devel@vger.kernel.org mailto:ceph-devel@vger.kernel.org, Milosz Tanski mil...@adfin.com mailto:mil...@adfin.com Envoyé: Vendredi 24 Avril 2015 13:37:52 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

...@lists.ceph.com , ceph-devel ceph-devel@vger.kernel.org , Milosz Tanski mil...@adfin.com Envoyé: Jeudi 23 Avril 2015 13:33:00 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Thanks for the testing Alexandre! If you have

RE: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

Nelson Cc: ceph-users; ceph-devel; Milosz Tanski Subject: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops If you have the means to compile the same version of ceph with jemalloc, I would be very interested to see how it does. Yes

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

-users ceph-us...@lists.ceph.com Envoyé: Mercredi 22 Avril 2015 16:34:33 Objet: RE: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops I feel it is due to tcmalloc issue I have seen similar issue in my setup after 20 days. Thanks

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

, ceph-devel ceph-devel@vger.kernel.org, Milosz Tanski mil...@adfin.com Envoyé: Jeudi 23 Avril 2015 10:00:34 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Hi, I'm hitting this bug again today. So don't seem to be numa

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

-devel@vger.kernel.org, ceph-users ceph-us...@lists.ceph.com Envoyé: Mercredi 22 Avril 2015 16:34:33 Objet: RE: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops I feel it is due to tcmalloc issue I have seen similar issue in my setup

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

: Jeudi 23 Avril 2015 13:33:00 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Thanks for the testing Alexandre! If you have the means to compile the same version of ceph with jemalloc, I would be very interested to see

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

2015 16:34:33 Objet: RE: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops I feel it is due to tcmalloc issue I have seen similar issue in my setup after 20 days. Thanks, Srinivas -Original Message- From: ceph-users

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

: aderumier aderum...@odiso.com, Milosz Tanski mil...@adfin.com Cc: ceph-devel ceph-devel@vger.kernel.org, ceph-users ceph-us...@lists.ceph.com Envoyé: Mercredi 22 Avril 2015 16:01:12 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

original - De: Milosz Tanski mil...@adfin.com À: aderumier aderum...@odiso.com Cc: ceph-devel ceph-devel@vger.kernel.org, ceph-users ceph-us...@lists.ceph.com Envoyé: Mercredi 22 Avril 2015 12:54:23 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance

RE: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

Tanski Cc: ceph-devel; ceph-users Subject: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops Hi Alexandre, We should discuss this at the perf meeting today. We knew NUMA node affinity issues were going to crop up sooner or later

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

Envoyé: Mercredi 22 Avril 2015 12:54:23 Objet: Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops On Wed, Apr 22, 2015 at 5:01 AM, Alexandre DERUMIER aderum...@odiso.com wrote: I wonder if it could be numa related, I'm using

Re: [ceph-users] strange benchmark problem : restarting osd daemon improve performance from 100k iops to 300k iops

I wonder if it could be numa related, I'm using centos 7.1, and auto numa balacning is enabled cat /proc/sys/kernel/numa_balancing = 1 Maybe osd daemon access to buffer on wrong numa node. I'll try to reproduce the problem - Mail original - De: aderumier aderum...@odiso.com À: