Re: [Gluster-devel] performance issues Manoj found in EC testing

[Pranith Kumar Karampuri]

On Tue, Jun 28, 2016 at 10:21 AM, Poornima Gurusiddaiah  wrote:

> Regards,
> Poornima
>
> --
>
> *From: *"Pranith Kumar Karampuri" 
> *To: *"Xavier Hernandez" 
> *Cc: *"Gluster Devel" 
> *Sent: *Monday, June 27, 2016 5:48:24 PM
> *Subject: *Re: [Gluster-devel] performance issues Manoj found in EC
> testing
>
>
>
> On Mon, Jun 27, 2016 at 12:42 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez > > wrote:
>>
>>> Hi Manoj,
>>>
>>> I always enable client-io-threads option for disperse volumes. It
>>> improves performance sensibly, most probably because of the problem you
>>> have detected.
>>>
>>> I don't see any other way to solve that problem.
>>>
>>
>> I agree. Updated the bug with same info.
>>
>>>
>>> I think it would be a lot better to have a true thread pool (and maybe
>>> an I/O thread pool shared by fuse, client and server xlators) in
>>> libglusterfs instead of the io-threads xlator. This would allow each xlator
>>> to decide when and what should be parallelized in a more intelligent way,
>>> since basing the decision solely on the fop type seems too simplistic to me.
>>>
>>> In the specific case of EC, there are a lot of operations to perform for
>>> a single high level fop, and not all of them require the same priority.
>>> Also some of them could be executed in parallel instead of sequentially.
>>>
>>
>> I think it is high time we actually schedule(for which release) to get
>> this in gluster. May be you should send out a doc where we can work out
>> details? I will be happy to explore options to integrate io-threads,
>> syncop/barrier with this infra based on the design may be.
>>
>
> I was just thinking why we can't reuse synctask framework. It already
> scales up/down based on the tasks. At max it uses 16 threads. Whatever we
> want to be executed in parallel we can create a synctask around it and run
> it. Would that be good enough?
>
> Yes, synctask framework can be preferred over io-threads, else it would
> mean 16 synctask threads + 16(?) io-threads for one instance of mount, this
> will blow out the gfapi clients if they have many mounts from the same
> process. Also using synctask would mean code changes in EC?
>

Yes it will need some changes but I don't think they are big changes. I
think the functions to decode/encode already exist. We just to need to move
encoding/decoding as tasks and run as synctasks.

Xavi,
  Long time back we chatted a bit about synctask code and you wanted
the scheduling to happen by kernel or something. Apart from that do you see
any other issues? At least if the tasks are synchronous i.e. nothing goes
out the wire, task scheduling = thread scheduling by kernel and it works
exactly like thread-pool you were referring to. It does multi-tasking only
if the tasks are asynchronous in nature.


>
>
>>> Xavi
>>>
>>>
>>> On 25/06/16 19:42, Manoj Pillai wrote:
>>>

 - Original Message -

> From: "Pranith Kumar Karampuri" 
> To: "Xavier Hernandez" 
> Cc: "Manoj Pillai" , "Gluster Devel" <
> gluster-devel@gluster.org>
> Sent: Thursday, June 23, 2016 8:50:44 PM
> Subject: performance issues Manoj found in EC testing
>
> hi Xavi,
>   Meet Manoj from performance team Redhat. He has been testing
> EC
> performance in his stretch clusters. He found some interesting things
> we
> would like to share with you.
>
> 1) When we perform multiple streams of big file writes(12 parallel dds
> I
> think) he found one thread to be always hot (99%CPU always). He was
> asking
> me if fuse_reader thread does any extra processing in EC compared to
> replicate. Initially I thought it would just lock and epoll threads
> will
> perform the encoding but later realized that once we have the lock and
> version details, next writes on the file would be encoded in the same
> thread that comes to EC. write-behind could play a role and make the
> writes
> come to EC in an epoll thread but we saw consistently there was just
> one
> thread that is hot. Not multiple threads. We will be able to confirm
> this
> in tomorrow's testing.
>
> 2) This is one more thing Raghavendra G found, that our current
> implementation of epoll doesn't let other epoll threads pick messages
> from
> a socket while one thread is processing one message from that socket.
> In
> EC's case that can be encoding of the write/decoding read. This will
> not
> let replies of operations on different files to be processed in
> parallel.
> He thinks this can be fixed for 3.9.
>
> Manoj will be raising a bug to gather all his findings. I just wanted
> to
> introduce him and let you know the interesting things he is finding
> before
> you see the bug :-).
> --
> Pranith
>

 Thanks, Pranith :).

 H

Re: [Gluster-devel] performance issues Manoj found in EC testing

[Poornima Gurusiddaiah]

Regards, 
Poornima 

- Original Message -

> From: "Pranith Kumar Karampuri" 
> To: "Xavier Hernandez" 
> Cc: "Gluster Devel" 
> Sent: Monday, June 27, 2016 5:48:24 PM
> Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing

> On Mon, Jun 27, 2016 at 12:42 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com > wrote:

> > On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez < xhernan...@datalab.es
> > >
> > wrote:
> 

> > > Hi Manoj,
> > 
> 

> > > I always enable client-io-threads option for disperse volumes. It
> > > improves
> > > performance sensibly, most probably because of the problem you have
> > > detected.
> > 
> 

> > > I don't see any other way to solve that problem.
> > 
> 

> > I agree. Updated the bug with same info.
> 

> > > I think it would be a lot better to have a true thread pool (and maybe an
> > > I/O
> > > thread pool shared by fuse, client and server xlators) in libglusterfs
> > > instead of the io-threads xlator. This would allow each xlator to decide
> > > when and what should be parallelized in a more intelligent way, since
> > > basing
> > > the decision solely on the fop type seems too simplistic to me.
> > 
> 

> > > In the specific case of EC, there are a lot of operations to perform for
> > > a
> > > single high level fop, and not all of them require the same priority.
> > > Also
> > > some of them could be executed in parallel instead of sequentially.
> > 
> 

> > I think it is high time we actually schedule(for which release) to get this
> > in gluster. May be you should send out a doc where we can work out details?
> > I will be happy to explore options to integrate io-threads, syncop/barrier
> > with this infra based on the design may be.
> 

> I was just thinking why we can't reuse synctask framework. It already scales
> up/down based on the tasks. At max it uses 16 threads. Whatever we want to
> be executed in parallel we can create a synctask around it and run it. Would
> that be good enough?

Yes, synctask framework can be preferred over io-threads, else it would mean 16 
synctask threads + 16(?) io-threads for one instance of mount, this will blow 
out the gfapi clients if they have many mounts from the same process. Also 
using synctask would mean code changes in EC? 

> > > Xavi
> > 
> 

> > > On 25/06/16 19:42, Manoj Pillai wrote:
> > 
> 

> > > > - Original Message -
> > > 
> > 
> 

> > > > > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > > > 
> > > 
> > 
> 
> > > > > To: "Xavier Hernandez" < xhernan...@datalab.es >
> > > > 
> > > 
> > 
> 
> > > > > Cc: "Manoj Pillai" < mpil...@redhat.com >, "Gluster Devel" <
> > > > > gluster-devel@gluster.org >
> > > > 
> > > 
> > 
> 
> > > > > Sent: Thursday, June 23, 2016 8:50:44 PM
> > > > 
> > > 
> > 
> 
> > > > > Subject: performance issues Manoj found in EC testing
> > > > 
> > > 
> > 
> 

> > > > > hi Xavi,
> > > > 
> > > 
> > 
> 
> > > > > Meet Manoj from performance team Redhat. He has been testing EC
> > > > 
> > > 
> > 
> 
> > > > > performance in his stretch clusters. He found some interesting things
> > > > > we
> > > > 
> > > 
> > 
> 
> > > > > would like to share with you.
> > > > 
> > > 
> > 
> 

> > > > > 1) When we perform multiple streams of big file writes(12 parallel
> > > > > dds
> > > > > I
> > > > 
> > > 
> > 
> 
> > > > > think) he found one thread to be always hot (99%CPU always). He was
> > > > > asking
> > > > 
> > > 
> > 
> 
> > > > > me if fuse_reader thread does any extra processing in EC compared to
> > > > 
> > > 
> > 
> 
> > > > > replicate. Initially I thought it would just lock and epoll threads
> > > > > will
> > > > 
> > > 
> > 
> 
> > > > > perform the encoding but later realized that once we have the lock
> > > > > and
> > > > 
> > > 
> > 
> 
> > > > > version details, next writes on the file would be encoded in the same
> > > > 
> > > 
> > 
> 
> > > > > thread that comes to EC. write-behind could play a role and make the
> > > > > writes
> > > > 
> > > 
> > 
> 
> > > > > come to EC in an epoll thread but we saw consistently there was just
> > > > > one
> > > > 
> > > 
> > 
> 
> > > > > thread that is hot. Not multiple threads. We will be able to confirm
> > > > > this
> > > > 
> > > 
> > 
> 
> > > > > in tomorrow's testing.
> > > > 
> > > 
> > 
> 

> > > > > 2) This is one more thing Raghavendra G found, that our current
> > > > 
> > > 
> > 
> 
> > > > > implementation of epoll doesn't let other epoll threads pick messages
> > > > > from
> > > > 
> > > 
> > 
> 
> > > > > a socket while one thread is processing one message from that socket.
> > > > > In
> > > > 
> > > 
> > 
> 
> > > > > EC's case that can be encoding of the write/decoding read. This will
> > > > > not
> > > > 
> > > 
> > 
> 
> > > > > let replies of operations on different files to be processed in
> > > > > parallel.
> > > > 
> > > 
> > 
> 
> > > > > He thinks this can be fixed for 3.9.
> > > > 
> > > 
> > 
> 

> > > > > Manoj will be raising a bug to gather all his findings

Re: [Gluster-devel] [Gluster-infra] Access to Jenkins

[Nigel Babu]

On Thu, Jun 16, 2016 at 04:58:52PM +0530, Nigel Babu wrote:
> On Thu, Jun 16, 2016 at 12:29:24PM +0200, Niels de Vos wrote:
> > On Wed, Jun 15, 2016 at 09:14:16PM +0530, Nigel Babu wrote:
> > > Hello folks,
> > >
> > > We have 45 people who have access to Jenkins UI. Pretty much everyone 
> > > will be
> > > losing this access in the next couple of weeks.
> > >
> > > At the moment, I understand that access to the UI is essential for 
> > > configuring
> > > jobs. I’m going to change this in the near future. Jenkins Job Builder[1] 
> > > will
> > > talk to Jenkins to create/update jobs. Job information will be managed as 
> > > a
> > > collection of yaml files. If you want a new job, you can give us a pull 
> > > request
> > > with the correct format. The jobs will then be updated (probably via 
> > > Jenkins).
> > > You will then no longer need access to Jenkins to create or manage jobs. 
> > > In
> > > fact, editing the jobs in the UI will make the YAML files out of sync.
> > >
> > > Before we start this process, I’d love to know if you use your Jenkins 
> > > access.
> >
> > Initially it was only possible to start regression tests by logging into
> > Jenkins and running the job manually. There were fewer patches and the
> > duration of the job was much shorter too.
> >
> > Later on, we got more slaves and longer running regression tests.
> > Someone configured the Jenkins jobs to get triggered on each change that
> > was posted (now modified to only run after Verified=+1). This caused a
> > lot of weird side-effects, including tests failing more regularly for
> > unknown reasons. Retriggering a job was only possible by clicking the
> > "retrigger" link in the Jenkins webui (now we can do so with "recheck
> > ..." in Gerrit comments).
> >
> > Sometimes regression tests cause a fatal problem on a slave. The only
> > way to get the slave back, could be through a hard reboot. The reboot-vm
> > job in Jenkins made that possible. The job prevented people from
> > requiring a Rackspace account.
> >
> > Release tarballs are created through Jenkins. Anyone doing releases
> > should have their own Jenkins account to run the job.
> >
> > I think many of the maintainers should be able to run at least the
> > release job. Configuration of the jobs can be restricted, specially if
> > Jenkins Job Builder is used. Somewhere in the Contributors Guide there
> > should be a description on how to make changes to the Jenkins jobs. I've
> > never used JBB and definitely need some assistance with it.
> >   http://gluster.readthedocs.io/en/latest/Contributors-Guide/Index/
> >
> > HTH,
> > Niels
> >
> >
> > > If you do use it, please let me know off-list what you use it for.
> > >
> > > [1]: http://docs.openstack.org/infra/system-config/jjb.html
> > >
> > > --
> > > nigelb
> > > ___
> > > Gluster-infra mailing list
> > > gluster-in...@gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-infra
>
> Hi Niels,
>
> This is more complex than I thought, I won't be doing *anything* immediately.
> I'll take some time to get configuration right so we don't accidentally end up
> in a place where nobody can get work done.
>
> I'm going to slightly change what I originally intended:
> 1. We will still go ahead with Jenkins Job Builder. It has reasonably good
>documentation, but I'll be available to help in case anyone is stuck. I 
> will
>convert the existing jobs into yaml format for jjb, so they will serve as 
> an
>example. I'll make sure our documentation has a section for how to write
>jobs. Context: JJB was written by Openstack infra team who also use Jenkins
>and Gerrit, so it shouldn't give us too much trouble.
> 2. The VM restart is an interesting problem that I didn't have insight into
>before. I'm happy to grant current developers access to have access to the
>restart job (We have to define "current developers" at some point).
>Eventually, I hope to make it less of a problem by restarting after every
>X number of jobs, restarting after a failure, or some such (we can talk
>about this later).
> 3. For release jobs, I'll similiarly create a group who will have full access
>to that job. This way I won't be blocking work while we look at better
>options.
>
> Does this solve our immediate concerns?
>
> --
> nigelb

I'm ready to make changes to Jenkins this week. I'm going to create a bot user
with access to modify jobs. This user will be used to run Jenkins Job builder.
The rest of us will all be demoted one level to being able to do everything but
modify or jobs. Everyone who currently has access will also have access to
run all jobs, so rebooting machines shouldn't be an issue.

Please let me know if this will block your work in anyway. If I don't hear
anything, I'll go ahead with this tomorrow morning.

--
nigelb
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluste

[Gluster-devel] Gluster Community Newsletter, June 2016

[Amye Scavarda]

Gluster Community Newsletter, June 2016
Announcing 3.8!

As of June 14, 3.8 is released for general use.
The 3.8 release focuses on:

containers with inclusion of Heketi
hyperconvergence
ecosystem integration
protocol improvements with NFS Ganesha
http://blog.gluster.org/2016/06/glusterfs-3-8-released/

Please note that this release also marks the end of updates for Gluster 3.5.

Upcoming Events:

Red Hat Summit, June 28-30
https://www.redhat.com/en/summit

Gluster related talks include:
Ceph and Gluster: A comparative theology
Sage Weil, Vijay Bellur
https://rh2016.smarteventscloud.com/connect/sessionDetail.ww?SESSION_ID=75196

Red Hat Gluster Storage: Strategic direction, roadmap, and integrations
Sayan Saha, Jeff Darcy
https://rh2016.smarteventscloud.com/connect/sessionDetail.ww?SESSION_ID=43750

Aplo – Gluster in containers
Luis Pabon
https://rh2016.smarteventscloud.com/connect/sessionDetail.ww?SESSION_ID=75679


Gluster Developers Summit, deadline for application closes July 31

We'll be implementing a rolling invitation list for those who need to
book flights, put in your application for invitation before July 31st
if you'd like to attend!
https://www.gluster.org/events/summit2016/


Top 5 contributors:
Nigel Babu, Pranith Kumar, Niels de Vos, Krutika Dhananjay, Kaleb S. Keithley

Any events that should be added to https://www.gluster.org/events/?
Submit a PR!


-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] performance issues Manoj found in EC testing

[Manoj Pillai]

- Original Message -
> From: "Sankarshan Mukhopadhyay" 
> To: "Gluster Devel" 
> Sent: Monday, June 27, 2016 5:54:19 PM
> Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing
> 
> On Mon, Jun 27, 2016 at 2:38 PM, Manoj Pillai  wrote:
> > Thanks, folks! As a quick update, throughput on a single client test jumped
> > from ~180 MB/s to 700+MB/s after enabling client-io-threads. Throughput is
> > now more in line with what is expected for this workload based on
> > back-of-the-envelope calculations.
> 
> Is it possible to provide additional detail about this exercise in
> terms of setup; tests executed; data sets generated?

Yes, in the bz. The "before" number is from here:
https://bugzilla.redhat.com/show_bug.cgi?id=1349953#c1

-- Manoj


> 
> --
> sankarshan mukhopadhyay
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Review request for http://review.gluster.org/#/c/14815/

[Karthik Subrahmanya]

Hi,

I have added implementation for the posix_do_futimes function, which is
not complete in the current implementation. I did the implementation by
looking into the current implementation of posix_do_utimes function.
Please review the patch and correct me if I am wrong anywhere.

Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1350406
Patch: http://review.gluster.org/#/c/14815/

Thanks & Regards,
Karthik
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] performance issues Manoj found in EC testing

[Sankarshan Mukhopadhyay]

On Mon, Jun 27, 2016 at 2:38 PM, Manoj Pillai  wrote:
> Thanks, folks! As a quick update, throughput on a single client test jumped
> from ~180 MB/s to 700+MB/s after enabling client-io-threads. Throughput is
> now more in line with what is expected for this workload based on
> back-of-the-envelope calculations.

Is it possible to provide additional detail about this exercise in
terms of setup; tests executed; data sets generated?

-- 
sankarshan mukhopadhyay

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] performance issues Manoj found in EC testing

[Pranith Kumar Karampuri]

On Mon, Jun 27, 2016 at 12:42 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez 
> wrote:
>
>> Hi Manoj,
>>
>> I always enable client-io-threads option for disperse volumes. It
>> improves performance sensibly, most probably because of the problem you
>> have detected.
>>
>> I don't see any other way to solve that problem.
>>
>
> I agree. Updated the bug with same info.
>
>
>>
>> I think it would be a lot better to have a true thread pool (and maybe an
>> I/O thread pool shared by fuse, client and server xlators) in libglusterfs
>> instead of the io-threads xlator. This would allow each xlator to decide
>> when and what should be parallelized in a more intelligent way, since
>> basing the decision solely on the fop type seems too simplistic to me.
>>
>> In the specific case of EC, there are a lot of operations to perform for
>> a single high level fop, and not all of them require the same priority.
>> Also some of them could be executed in parallel instead of sequentially.
>>
>
> I think it is high time we actually schedule(for which release) to get
> this in gluster. May be you should send out a doc where we can work out
> details? I will be happy to explore options to integrate io-threads,
> syncop/barrier with this infra based on the design may be.
>

I was just thinking why we can't reuse synctask framework. It already
scales up/down based on the tasks. At max it uses 16 threads. Whatever we
want to be executed in parallel we can create a synctask around it and run
it. Would that be good enough?


>
>
>>
>> Xavi
>>
>>
>> On 25/06/16 19:42, Manoj Pillai wrote:
>>
>>>
>>> - Original Message -
>>>
 From: "Pranith Kumar Karampuri" 
 To: "Xavier Hernandez" 
 Cc: "Manoj Pillai" , "Gluster Devel" <
 gluster-devel@gluster.org>
 Sent: Thursday, June 23, 2016 8:50:44 PM
 Subject: performance issues Manoj found in EC testing

 hi Xavi,
   Meet Manoj from performance team Redhat. He has been testing
 EC
 performance in his stretch clusters. He found some interesting things we
 would like to share with you.

 1) When we perform multiple streams of big file writes(12 parallel dds I
 think) he found one thread to be always hot (99%CPU always). He was
 asking
 me if fuse_reader thread does any extra processing in EC compared to
 replicate. Initially I thought it would just lock and epoll threads will
 perform the encoding but later realized that once we have the lock and
 version details, next writes on the file would be encoded in the same
 thread that comes to EC. write-behind could play a role and make the
 writes
 come to EC in an epoll thread but we saw consistently there was just one
 thread that is hot. Not multiple threads. We will be able to confirm
 this
 in tomorrow's testing.

 2) This is one more thing Raghavendra G found, that our current
 implementation of epoll doesn't let other epoll threads pick messages
 from
 a socket while one thread is processing one message from that socket. In
 EC's case that can be encoding of the write/decoding read. This will not
 let replies of operations on different files to be processed in
 parallel.
 He thinks this can be fixed for 3.9.

 Manoj will be raising a bug to gather all his findings. I just wanted to
 introduce him and let you know the interesting things he is finding
 before
 you see the bug :-).
 --
 Pranith

>>>
>>> Thanks, Pranith :).
>>>
>>> Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349953
>>>
>>> Comparing EC and replica-2 runs, the hot thread is seen in both cases, so
>>> I have not opened this as an EC bug. But initial impression is that
>>> performance impact for EC is particularly bad (details in the bug).
>>>
>>> -- Manoj
>>>
>>>
>
>
> --
> Pranith
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] performance issues Manoj found in EC testing

[Pranith Kumar Karampuri]

On Mon, Jun 27, 2016 at 2:38 PM, Manoj Pillai  wrote:

>
>
> - Original Message -
> > From: "Raghavendra Gowdappa" 
> > To: "Pranith Kumar Karampuri" 
> > Cc: "Gluster Devel" 
> > Sent: Monday, June 27, 2016 12:48:49 PM
> > Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing
> >
> >
> >
> > - Original Message -
> > > From: "Pranith Kumar Karampuri" 
> > > To: "Xavier Hernandez" 
> > > Cc: "Gluster Devel" 
> > > Sent: Monday, June 27, 2016 12:42:35 PM
> > > Subject: Re: [Gluster-devel] performance issues Manoj found in EC
> testing
> > >
> > >
> > >
> > > On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez <
> xhernan...@datalab.es
> > > >
> > > wrote:
> > >
> > >
> > > Hi Manoj,
> > >
> > > I always enable client-io-threads option for disperse volumes. It
> improves
> > > performance sensibly, most probably because of the problem you have
> > > detected.
> > >
> > > I don't see any other way to solve that problem.
> > >
> > > I agree. Updated the bug with same info.
> > >
> > >
> > >
> > > I think it would be a lot better to have a true thread pool (and maybe
> an
> > > I/O
> > > thread pool shared by fuse, client and server xlators) in libglusterfs
> > > instead of the io-threads xlator. This would allow each xlator to
> decide
> > > when and what should be parallelized in a more intelligent way, since
> > > basing
> > > the decision solely on the fop type seems too simplistic to me.
> > >
> > > In the specific case of EC, there are a lot of operations to perform
> for a
> > > single high level fop, and not all of them require the same priority.
> Also
> > > some of them could be executed in parallel instead of sequentially.
> > >
> > > I think it is high time we actually schedule(for which release) to get
> this
> > > in gluster. May be you should send out a doc where we can work out
> details?
> > > I will be happy to explore options to integrate io-threads,
> syncop/barrier
> > > with this infra based on the design may be.
> >
> > +1. I can volunteer too.
>
> Thanks, folks! As a quick update, throughput on a single client test jumped
> from ~180 MB/s to 700+MB/s after enabling client-io-threads. Throughput is
> now more in line with what is expected for this workload based on
> back-of-the-envelope calculations.
>
> Are there any reservations about recommending client-io-threads=on as
> "default" tuning, until the enhancement discussed above becomes reality?
>

The only thing I can think of is possible races we may have to address
after enabling this option. So I would let it bake on master for a while
with this as default may be?


> -- Manoj
>
> >
> > >
> > >
> > >
> > > Xavi
> > >
> > >
> > > On 25/06/16 19:42, Manoj Pillai wrote:
> > >
> > >
> > >
> > > - Original Message -
> > >
> > >
> > > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > > To: "Xavier Hernandez" < xhernan...@datalab.es >
> > > Cc: "Manoj Pillai" < mpil...@redhat.com >, "Gluster Devel" <
> > > gluster-devel@gluster.org >
> > > Sent: Thursday, June 23, 2016 8:50:44 PM
> > > Subject: performance issues Manoj found in EC testing
> > >
> > > hi Xavi,
> > > Meet Manoj from performance team Redhat. He has been testing EC
> > > performance in his stretch clusters. He found some interesting things
> we
> > > would like to share with you.
> > >
> > > 1) When we perform multiple streams of big file writes(12 parallel dds
> I
> > > think) he found one thread to be always hot (99%CPU always). He was
> asking
> > > me if fuse_reader thread does any extra processing in EC compared to
> > > replicate. Initially I thought it would just lock and epoll threads
> will
> > > perform the encoding but later realized that once we have the lock and
> > > version details, next writes on the file would be encoded in the same
> > > thread that comes to EC. write-behind could play a role and make the
> writes
> > > come to EC in an epoll thread but we saw consistently there was just
> one
> > > thread that is hot. Not multiple threads. We will be able to confirm
> this
> > > in tomorrow's testing.
> > >
> > > 2) This is one more thing Raghavendra G found, that our current
> > > implementation of epoll doesn't let other epoll threads pick messages
> from
> > > a socket while one thread is processing one message from that socket.
> In
> > > EC's case that can be encoding of the write/decoding read. This will
> not
> > > let replies of operations on different files to be processed in
> parallel.
> > > He thinks this can be fixed for 3.9.
> > >
> > > Manoj will be raising a bug to gather all his findings. I just wanted
> to
> > > introduce him and let you know the interesting things he is finding
> before
> > > you see the bug :-).
> > > --
> > > Pranith
> > >
> > > Thanks, Pranith :).
> > >
> > > Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349953
> > >
> > > Comparing EC and replica-2 runs, the hot thread is seen in both cases,
> so
> > > I have not opened this as an EC bug. But initial im

Re: [Gluster-devel] performance issues Manoj found in EC testing

[Manoj Pillai]

- Original Message -
> From: "Raghavendra Gowdappa" 
> To: "Pranith Kumar Karampuri" 
> Cc: "Gluster Devel" 
> Sent: Monday, June 27, 2016 12:48:49 PM
> Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing
> 
> 
> 
> - Original Message -
> > From: "Pranith Kumar Karampuri" 
> > To: "Xavier Hernandez" 
> > Cc: "Gluster Devel" 
> > Sent: Monday, June 27, 2016 12:42:35 PM
> > Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing
> > 
> > 
> > 
> > On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez < xhernan...@datalab.es
> > >
> > wrote:
> > 
> > 
> > Hi Manoj,
> > 
> > I always enable client-io-threads option for disperse volumes. It improves
> > performance sensibly, most probably because of the problem you have
> > detected.
> > 
> > I don't see any other way to solve that problem.
> > 
> > I agree. Updated the bug with same info.
> > 
> > 
> > 
> > I think it would be a lot better to have a true thread pool (and maybe an
> > I/O
> > thread pool shared by fuse, client and server xlators) in libglusterfs
> > instead of the io-threads xlator. This would allow each xlator to decide
> > when and what should be parallelized in a more intelligent way, since
> > basing
> > the decision solely on the fop type seems too simplistic to me.
> > 
> > In the specific case of EC, there are a lot of operations to perform for a
> > single high level fop, and not all of them require the same priority. Also
> > some of them could be executed in parallel instead of sequentially.
> > 
> > I think it is high time we actually schedule(for which release) to get this
> > in gluster. May be you should send out a doc where we can work out details?
> > I will be happy to explore options to integrate io-threads, syncop/barrier
> > with this infra based on the design may be.
> 
> +1. I can volunteer too.

Thanks, folks! As a quick update, throughput on a single client test jumped 
from ~180 MB/s to 700+MB/s after enabling client-io-threads. Throughput is 
now more in line with what is expected for this workload based on 
back-of-the-envelope calculations.

Are there any reservations about recommending client-io-threads=on as 
"default" tuning, until the enhancement discussed above becomes reality? 

-- Manoj

> 
> > 
> > 
> > 
> > Xavi
> > 
> > 
> > On 25/06/16 19:42, Manoj Pillai wrote:
> > 
> > 
> > 
> > - Original Message -
> > 
> > 
> > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > To: "Xavier Hernandez" < xhernan...@datalab.es >
> > Cc: "Manoj Pillai" < mpil...@redhat.com >, "Gluster Devel" <
> > gluster-devel@gluster.org >
> > Sent: Thursday, June 23, 2016 8:50:44 PM
> > Subject: performance issues Manoj found in EC testing
> > 
> > hi Xavi,
> > Meet Manoj from performance team Redhat. He has been testing EC
> > performance in his stretch clusters. He found some interesting things we
> > would like to share with you.
> > 
> > 1) When we perform multiple streams of big file writes(12 parallel dds I
> > think) he found one thread to be always hot (99%CPU always). He was asking
> > me if fuse_reader thread does any extra processing in EC compared to
> > replicate. Initially I thought it would just lock and epoll threads will
> > perform the encoding but later realized that once we have the lock and
> > version details, next writes on the file would be encoded in the same
> > thread that comes to EC. write-behind could play a role and make the writes
> > come to EC in an epoll thread but we saw consistently there was just one
> > thread that is hot. Not multiple threads. We will be able to confirm this
> > in tomorrow's testing.
> > 
> > 2) This is one more thing Raghavendra G found, that our current
> > implementation of epoll doesn't let other epoll threads pick messages from
> > a socket while one thread is processing one message from that socket. In
> > EC's case that can be encoding of the write/decoding read. This will not
> > let replies of operations on different files to be processed in parallel.
> > He thinks this can be fixed for 3.9.
> > 
> > Manoj will be raising a bug to gather all his findings. I just wanted to
> > introduce him and let you know the interesting things he is finding before
> > you see the bug :-).
> > --
> > Pranith
> > 
> > Thanks, Pranith :).
> > 
> > Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349953
> > 
> > Comparing EC and replica-2 runs, the hot thread is seen in both cases, so
> > I have not opened this as an EC bug. But initial impression is that
> > performance impact for EC is particularly bad (details in the bug).
> > 
> > -- Manoj
> > 
> > 
> > 
> > 
> > --
> > Pranith
> > 
> > ___
> > Gluster-devel mailing list
> > Gluster-devel@gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluste

[Gluster-devel] Glusterfs ls test

[Keiviw]

hi,
 I have tested the DHT to list the directory.The enviroment of this test 
is: dir1-70/subdir1-subdir96/file1-96,dirX and subdirX were directories.I 
excuted "time ls dir1" to list the subdir1-99,it toke 2.5s,while I excuted 
"time ls subdir1" to list the file1-99,it toke 0.14s.Of course,both of the two 
tests are the first time to list the dentries. 
 Here is my question, the number of dentries are the same,but the time 
spent on list the dentries which are directories is much more than the time 
spent on list dentries which are files.___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] performance issues Manoj found in EC testing

[Raghavendra Gowdappa]

- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Xavier Hernandez" 
> Cc: "Gluster Devel" 
> Sent: Monday, June 27, 2016 12:42:35 PM
> Subject: Re: [Gluster-devel] performance issues Manoj found in EC testing
> 
> 
> 
> On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez < xhernan...@datalab.es >
> wrote:
> 
> 
> Hi Manoj,
> 
> I always enable client-io-threads option for disperse volumes. It improves
> performance sensibly, most probably because of the problem you have
> detected.
> 
> I don't see any other way to solve that problem.
> 
> I agree. Updated the bug with same info.
> 
> 
> 
> I think it would be a lot better to have a true thread pool (and maybe an I/O
> thread pool shared by fuse, client and server xlators) in libglusterfs
> instead of the io-threads xlator. This would allow each xlator to decide
> when and what should be parallelized in a more intelligent way, since basing
> the decision solely on the fop type seems too simplistic to me.
> 
> In the specific case of EC, there are a lot of operations to perform for a
> single high level fop, and not all of them require the same priority. Also
> some of them could be executed in parallel instead of sequentially.
> 
> I think it is high time we actually schedule(for which release) to get this
> in gluster. May be you should send out a doc where we can work out details?
> I will be happy to explore options to integrate io-threads, syncop/barrier
> with this infra based on the design may be.

+1. I can volunteer too.

> 
> 
> 
> Xavi
> 
> 
> On 25/06/16 19:42, Manoj Pillai wrote:
> 
> 
> 
> - Original Message -
> 
> 
> From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> To: "Xavier Hernandez" < xhernan...@datalab.es >
> Cc: "Manoj Pillai" < mpil...@redhat.com >, "Gluster Devel" <
> gluster-devel@gluster.org >
> Sent: Thursday, June 23, 2016 8:50:44 PM
> Subject: performance issues Manoj found in EC testing
> 
> hi Xavi,
> Meet Manoj from performance team Redhat. He has been testing EC
> performance in his stretch clusters. He found some interesting things we
> would like to share with you.
> 
> 1) When we perform multiple streams of big file writes(12 parallel dds I
> think) he found one thread to be always hot (99%CPU always). He was asking
> me if fuse_reader thread does any extra processing in EC compared to
> replicate. Initially I thought it would just lock and epoll threads will
> perform the encoding but later realized that once we have the lock and
> version details, next writes on the file would be encoded in the same
> thread that comes to EC. write-behind could play a role and make the writes
> come to EC in an epoll thread but we saw consistently there was just one
> thread that is hot. Not multiple threads. We will be able to confirm this
> in tomorrow's testing.
> 
> 2) This is one more thing Raghavendra G found, that our current
> implementation of epoll doesn't let other epoll threads pick messages from
> a socket while one thread is processing one message from that socket. In
> EC's case that can be encoding of the write/decoding read. This will not
> let replies of operations on different files to be processed in parallel.
> He thinks this can be fixed for 3.9.
> 
> Manoj will be raising a bug to gather all his findings. I just wanted to
> introduce him and let you know the interesting things he is finding before
> you see the bug :-).
> --
> Pranith
> 
> Thanks, Pranith :).
> 
> Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349953
> 
> Comparing EC and replica-2 runs, the hot thread is seen in both cases, so
> I have not opened this as an EC bug. But initial impression is that
> performance impact for EC is particularly bad (details in the bug).
> 
> -- Manoj
> 
> 
> 
> 
> --
> Pranith
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] performance issues Manoj found in EC testing

[Pranith Kumar Karampuri]

On Mon, Jun 27, 2016 at 11:52 AM, Xavier Hernandez 
wrote:

> Hi Manoj,
>
> I always enable client-io-threads option for disperse volumes. It improves
> performance sensibly, most probably because of the problem you have
> detected.
>
> I don't see any other way to solve that problem.
>

I agree. Updated the bug with same info.


>
> I think it would be a lot better to have a true thread pool (and maybe an
> I/O thread pool shared by fuse, client and server xlators) in libglusterfs
> instead of the io-threads xlator. This would allow each xlator to decide
> when and what should be parallelized in a more intelligent way, since
> basing the decision solely on the fop type seems too simplistic to me.
>
> In the specific case of EC, there are a lot of operations to perform for a
> single high level fop, and not all of them require the same priority. Also
> some of them could be executed in parallel instead of sequentially.
>

I think it is high time we actually schedule(for which release) to get this
in gluster. May be you should send out a doc where we can work out details?
I will be happy to explore options to integrate io-threads, syncop/barrier
with this infra based on the design may be.


>
> Xavi
>
>
> On 25/06/16 19:42, Manoj Pillai wrote:
>
>>
>> - Original Message -
>>
>>> From: "Pranith Kumar Karampuri" 
>>> To: "Xavier Hernandez" 
>>> Cc: "Manoj Pillai" , "Gluster Devel" <
>>> gluster-devel@gluster.org>
>>> Sent: Thursday, June 23, 2016 8:50:44 PM
>>> Subject: performance issues Manoj found in EC testing
>>>
>>> hi Xavi,
>>>   Meet Manoj from performance team Redhat. He has been testing EC
>>> performance in his stretch clusters. He found some interesting things we
>>> would like to share with you.
>>>
>>> 1) When we perform multiple streams of big file writes(12 parallel dds I
>>> think) he found one thread to be always hot (99%CPU always). He was
>>> asking
>>> me if fuse_reader thread does any extra processing in EC compared to
>>> replicate. Initially I thought it would just lock and epoll threads will
>>> perform the encoding but later realized that once we have the lock and
>>> version details, next writes on the file would be encoded in the same
>>> thread that comes to EC. write-behind could play a role and make the
>>> writes
>>> come to EC in an epoll thread but we saw consistently there was just one
>>> thread that is hot. Not multiple threads. We will be able to confirm this
>>> in tomorrow's testing.
>>>
>>> 2) This is one more thing Raghavendra G found, that our current
>>> implementation of epoll doesn't let other epoll threads pick messages
>>> from
>>> a socket while one thread is processing one message from that socket. In
>>> EC's case that can be encoding of the write/decoding read. This will not
>>> let replies of operations on different files to be processed in parallel.
>>> He thinks this can be fixed for 3.9.
>>>
>>> Manoj will be raising a bug to gather all his findings. I just wanted to
>>> introduce him and let you know the interesting things he is finding
>>> before
>>> you see the bug :-).
>>> --
>>> Pranith
>>>
>>
>> Thanks, Pranith :).
>>
>> Here's the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1349953
>>
>> Comparing EC and replica-2 runs, the hot thread is seen in both cases, so
>> I have not opened this as an EC bug. But initial impression is that
>> performance impact for EC is particularly bad (details in the bug).
>>
>> -- Manoj
>>
>>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel