Re: [Gluster-devel] [Gluster-users] 120k context switches on GlsuterFS nodes

[Ravishankar N]

On 05/17/2017 11:07 PM, Pranith Kumar Karampuri wrote:

+ gluster-devel

On Wed, May 17, 2017 at 10:50 PM, mabi > wrote:


I don't know exactly what kind of context-switches it was but what
I know is that it is the "cs" number under "system" when you run
vmstat.

Okay, that could be due to the  syscalls themselves or pre-emptive 
multitasking in case there aren't enough cpu cores. I think the spike in 
numbers is due to more users accessing the files at the same time like 
you observed, translating into more syscalls.  You can try capturing the 
gluster volume profile info the next time it occurs and co-relate with 
the cs count. If you don't see any negative performance impact, I think 
you don't need to be bothered much by the numbers.


HTH,
Ravi



Also I use the percona linux monitoring template for cacti

(https://www.percona.com/doc/percona-monitoring-plugins/LATEST/cacti/linux-templates.html

)
which monitors context switches too. If that's of any use
interrupts where also quite high during that time with peaks up to
50k interrupts.




 Original Message 
Subject: Re: [Gluster-users] 120k context switches on GlsuterFS nodes
Local Time: May 17, 2017 2:37 AM
UTC Time: May 17, 2017 12:37 AM
From: ravishan...@redhat.com 
To: mabi >,
Gluster Users >


On 05/16/2017 11:13 PM, mabi wrote:

Today I even saw up to 400k context switches for around 30
minutes on my two nodes replica... Does anyone else have so high
context switches on their GlusterFS nodes?

I am wondering what is "normal" and if I should be worried...





 Original Message 
Subject: 120k context switches on GlsuterFS nodes
Local Time: May 11, 2017 9:18 PM
UTC Time: May 11, 2017 7:18 PM
From: m...@protonmail.ch 
To: Gluster Users 


Hi,

Today I noticed that for around 50 minutes my two GlusterFS
3.8.11 nodes had a very high amount of context switches, around
120k. Usually the average is more around 1k-2k. So I checked
what was happening and there where just more users accessing
(downloading) their files at the same time. These are
directories with typical cloud files, which means files of any
sizes ranging from a few kB to MB and a lot of course.

Now I never saw such a high number in context switches in my
entire life so I wanted to ask if this is normal or to be
expected? I do not find any signs of errors or warnings in any
log files.



What context switch are you referring to (syscalls context-switch
on the bricks?) ? How did you measure this?
-Ravi


My volume is a replicated volume on two nodes with ZFS as
filesystem behind and the volume is mounted using FUSE on the
client (the cloud server). On that cloud server the glusterfs
process was using quite a lot of system CPU but that server
(VM) only has 2 vCPUs so maybe I should increase the number of
vCPUs...

Any ideas or recommendations?



Regards,
M.




___
Gluster-users mailing list
gluster-us...@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-users




___ Gluster-users
mailing list gluster-us...@gluster.org

http://lists.gluster.org/mailman/listinfo/gluster-users
 


--
Pranith


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Release 3.11: Rc1 and release tagging dates

[Shyam]

Hi,

We have had a steady stream of backports to 3.11 and hence will tag RC1 
around May-22nd and will do the final release tagging around May-29th.


Post RC1 that gives exactly one week to get any critical bug fixes into 
the code base, so be aware of those dates, and also mark any blockers 
against the release tracking bug [1].


Thanks,
Shyam

[1] Tracker BZ for 3.11.0 blockers:
https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.11.0
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Release 3.12 and 4.0: Thoughts on scope

[Shyam]

A short reminder on dates for these releases:

3.12 is targeted around Aug-30
4.0 is targeted around Nov-30

These are tracked as milestones in github and the dates reflected 
appropriately there [1]. Further the community webpage is updated to 
point to the github pages to keep ambiguity out of the picture [2].


Thanks,
Shyam

[1] github milestone dates: 
https://github.com/gluster/glusterfs/milestones?direction=asc=due_date=open

[2] Gluster web page: https://www.gluster.org/community/roadmap/

On 05/15/2017 08:46 PM, Shyam wrote:

Hi,

Let's start a bit early on 3.12 and 4.0 roadmap items, as there have
been quite a few discussions around this in various meetups.

Here is what we are hearing (or have heard), so if you are working on
any of these items, do put up your github issue, and let us know which
release you are targeting these for.

If you are working on something that is not represented here, shout out,
and we can get that added to the list of items in the upcoming releases.

Once we have a good collection slotted into the respective releases (on
github), we can further announce the same in the users list as well.

3.12:
1. Geo-replication to cloud (ie, s3 or glacier like storage target)
2. Basic level of throttling support on server side to manage the
self-heal processes running.
3. Brick Multiplexing (Better support, more control)
4. GFID to path improvements
5. Resolve issues around disconnects and ping-timeouts
6. Halo with hybrid mode was supposed to be with 3.12
7. Procedures and code for +1 scaling the cluster?
8. Lookup-optimized turned on by default.
9. Thin client (or server side clustering) - phase 1.

4.0: (more thematic than actual features at the moment)
1. Separation of Management and Filesystem layers (aka GlusterD2 related
efforts)
2. Scaling Distribution logic
3. Better consistency with rename() and link() operations
4. Thin client || Clustering Logic on server side - Phase 2
5. Quota: re-look at optimal support
6. Improvements in debug-ability and more focus on testing coverage
based on use-cases.

Components moving out of support in possibly 4.0
- Stripe translator
- AFR with just 2 subvolume (either use Arbiter or 3 way replicate)
- Re-validate few performance translator's presence.

Thanks,
Shyam

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Weekly GlusterFS Untriaged Bugs Report

[staticanalysis]

Weekly GlusterFS Untriaged Bugs Report

1450567 / core: brick process cannot be started at the first time
1449416 / core: errno used incorrectly or misleadingly in error messages
1449861 / core: glusterd-geo-rep.c doesn't log stderr on failure to execute 
subprocesses
1450593 / core: Gluster Python scripts do not check return value of find_library
1450546 / core: Paths to some tools are hardcoded to /sbin or /usr/sbin
1449232 / coreutils: race condition between client_ctx_get and client_ctx_set
1450698 / distribute: a core generated when running regression test 
/tests/bugs/glusterd/bug-1004744.t
1451434 / distribute: Use a bitmap to store local node info instead of 
conf->local_nodeuuids[i].uuids
1450685 / doc: Document options to configure geo-replication for lower latency
1450684 / geo-replication: Geo-replication delay cannot be configured to less 
than 3 seconds
1451559 / glusterd: Brick Multiplexing: During Remove brick when glusterd of a 
node is stopped, the brick process gets disconnected from glusterd purview  and 
hence losing multiplexing feature
1449933 / glusterd: Brick Multiplexing :- resetting a brick bring down other 
bricks with same PID
1447523 / glusterd: Glusterd segmentation fault in ' _Unwind_Backtrace' while 
running peer probe
1450588 / packaging: Ubuntu builds have unsubsituted autoconf macros in source 
code
1451184 / project-infrastructure: Assign reviewers based on who touched the 
file last
1449773 / project-infrastructure: Finish the installation and freebsd10.3.rht 
and clean password in jenkins
1449895 / project-infrastructure: netbsd regression log not found
1449495 / rdma: glfsheal: crashed(segfault) with disperse volume in RDMA
1450565 / rdma: glfsheal: crashed(segfault) with disperse volume in RDMA
1449311 / read-ahead: [whql][virtio-block+glusterfs]"Disk Stress" and "Disk 
Verification" job always failed on win7-32/win2012/win2k8R2 guest
1449313 / read-ahead: [whql][virtio-block+glusterfs]"Disk Stress" and "Disk 
Verification" job always failed on win7-32/win2012/win2k8R2 guest
1451843 / replicate: gluster volume performance issue
1449167 / selfheal: After selfheal of brick file size of few files differs
1444108 / unclassified: ls: cannot access 'file': No data available.  In log 
Error -->  buf->ia_gfid is null for
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 120k context switches on GlsuterFS nodes

[Pranith Kumar Karampuri]

+ gluster-devel

On Wed, May 17, 2017 at 10:50 PM, mabi  wrote:

> I don't know exactly what kind of context-switches it was but what I know
> is that it is the "cs" number under "system" when you run vmstat.
>
> Also I use the percona linux monitoring template for cacti (
> https://www.percona.com/doc/percona-monitoring-plugins/
> LATEST/cacti/linux-templates.html) which monitors context switches too.
> If that's of any use interrupts where also quite high during that time with
> peaks up to 50k interrupts.
>
>
>
>  Original Message 
> Subject: Re: [Gluster-users] 120k context switches on GlsuterFS nodes
> Local Time: May 17, 2017 2:37 AM
> UTC Time: May 17, 2017 12:37 AM
> From: ravishan...@redhat.com
> To: mabi , Gluster Users 
>
>
> On 05/16/2017 11:13 PM, mabi wrote:
>
> Today I even saw up to 400k context switches for around 30 minutes on my
> two nodes replica... Does anyone else have so high context switches on
> their GlusterFS nodes?
>
> I am wondering what is "normal" and if I should be worried...
>
>
>
>
>  Original Message 
> Subject: 120k context switches on GlsuterFS nodes
> Local Time: May 11, 2017 9:18 PM
> UTC Time: May 11, 2017 7:18 PM
> From: m...@protonmail.ch
> To: Gluster Users  
>
> Hi,
>
> Today I noticed that for around 50 minutes my two GlusterFS 3.8.11 nodes
> had a very high amount of context switches, around 120k. Usually the
> average is more around 1k-2k. So I checked what was happening and there
> where just more users accessing (downloading) their files at the same time.
> These are directories with typical cloud files, which means files of any
> sizes ranging from a few kB to MB and a lot of course.
>
> Now I never saw such a high number in context switches in my entire life
> so I wanted to ask if this is normal or to be expected? I do not find any
> signs of errors or warnings in any log files.
>
>
> What context switch are you referring to (syscalls context-switch on the
> bricks?) ? How did you measure this?
> -Ravi
>
> My volume is a replicated volume on two nodes with ZFS as filesystem
> behind and the volume is mounted using FUSE on the client (the cloud
> server). On that cloud server the glusterfs process was using quite a lot
> of system CPU but that server (VM) only has 2 vCPUs so maybe I should
> increase the number of vCPUs...
>
> Any ideas or recommendations?
>
>
>
> Regards,
> M.
>
>
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Coverity covscan for 2017-05-17-2592fa04 (master branch)

[staticanalysis]

GlusterFS Coverity covscan results are available from
http://download.gluster.org/pub/gluster/glusterfs/static-analysis/master/glusterfs-coverity/2017-05-17-2592fa04
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t - regression failures

[Pranith Kumar Karampuri]

On Wed, May 17, 2017 at 4:34 PM, Ravishankar N 
wrote:

> On 05/17/2017 04:09 PM, Pranith Kumar Karampuri wrote:
>
> karthik, Ravi,
>  What is the plan to bring it back? Did you guys find RC for the
> failure?
>
> Are you referring to gfid-mismatch-resolution-with-fav-child-policy.t? I
> already mentioned the RCA in the patch I linked to earlier in this thread?
>

Ah! missed that. Thanks. We can also use self-heald to do the healing and
verify that heals were performed correctly. But it is fine to wait for
karthik's next patch.


>
> On Mon, May 15, 2017 at 10:52 AM, Ravishankar N 
> wrote:
>
>> On 05/12/2017 03:33 PM, Atin Mukherjee wrote:
>>
>> tests/basic/afr/add-brick-self-heal.t
>> 
>> is the 2nd in the list.
>>
>>
>> All failures (https://fstat.gluster.org/weeks/1/failure/2) are in netbsd
>> and looks like an issue with the slaves.
>> Most of the runs have this error:
>>
>> [07:29:32] Running tests in file ./tests/basic/afr/add-brick-self-heal.t
>> volume create: patchy: failed: Host nbslave70.cloud.gluster.org is not in 
>> 'Peer in Cluster' state
>>
>>
>> Not investigating the .t itself any further. -Ravi
>> ___ Gluster-devel mailing
>> list Gluster-devel@gluster.org http://lists.gluster.org/mailm
>> an/listinfo/gluster-devel
>
> --
> Pranith
>
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Change in rebalance graph

[Pranith Kumar Karampuri]

AFR had similar problem 2 years back and we started persisting afr children
and graph always gets generated with what the xattrs will be in that order
as an option. You can check afr_pending_xattrs_init() for it. On the
glusterd side we give an identifer for each of the bricks.

The main reason we did this is because we didn't want to change this logic
if new xlators get introduced in between client xlator and afr. What will
you do if you add one more xlator between readdir-ahead and dht in future?

Now that I am seeing this mail and also Facebook guys also expressed that
there should be a way for gluster to identify if it is talking to right
bricks recently when they met us. May be we should let each xlator choose
an identifier for its subvolumes and persist it on the bricks and use that
instead of depending on mgmt to do the right things.

I understand this patch is already solving your problem. I see that it is
merged as well but it is not future proof.


On Wed, May 17, 2017 at 10:03 AM, Raghavendra Gowdappa 
wrote:

> dht: Add readdir-ahead in rebalance graph if parallel-readdir is on
>
> Issue:
> The value of linkto xattr is generally the name of the dht's
> next subvol, this requires that the next subvol of dht is not
> changed for the life time of the volume. But with parallel
> readdir enabled, the readdir-ahead loaded below dht, is optional.
> The linkto xattr for first subvol, when:
> - parallel readdir is enabled : "-readdir-head-0"
> - plain distribute volume : "-client-0"
> - distribute replicate volume : "-afr-0"
>
> The value of linkto xattr is "-readdir-head-0" when
> parallel readdir is enabled, and is "-client-0" if
> its disabled. But the dht_lookup takes care of healing if it
> cannot identify which linkto subvol, the xattr points to.
>
> In dht_lookup_cbk, if linkto xattr is found to be "-client-0"
> and parallel readdir is enabled, then it cannot understand the
> value "-client-0" as it expects "-readdir-head-0".
> In that case, dht_lookup_everywhere is issued and then the linkto file
> is unlinked and recreated with the right linkto xattr. The issue is
> when parallel readdir is enabled, mount point accesses the file
> that is currently being migrated. Since rebalance process doesn't
> have parallel-readdir feature, it expects "-client-0"
> where as mount expects "-readdir-head-0". Thus at some point
> either the mount or rebalance will fail.
>
> Solution:
> Enable parallel-readdir for rebalance as well and then do not
> allow enabling/disabling parallel-readdir if rebalance is in
> progress.
>
> Change-Id: I241ab966bdd850e667f7768840540546f5289483
> BUG: 1436090
> Signed-off-by: Poornima G 
> Reviewed-on: https://review.gluster.org/17056
> Smoke: Gluster Build System 
> NetBSD-regression: NetBSD Build System 
> CentOS-regression: Gluster Build System 
>
>
> Patch: https://review.gluster.org/17056
> owners: Me and Poornima
>
> PS: readdir-ahead is loaded in rebalance graph on top of each subvolume of
> DHT.
>
> regards,
> Raghavendra
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t - regression failures

[Ravishankar N]

On 05/17/2017 04:09 PM, Pranith Kumar Karampuri wrote:

karthik, Ravi,
 What is the plan to bring it back? Did you guys find RC for the 
failure?
Are you referring to gfid-mismatch-resolution-with-fav-child-policy.t? I 
already mentioned the RCA in the patch I linked to earlier in this thread?


On Mon, May 15, 2017 at 10:52 AM, Ravishankar N 
> wrote:


On 05/12/2017 03:33 PM, Atin Mukherjee wrote:

|tests/basic/afr/add-brick-self-heal.t|


is the 2nd in the list.



All failures (https://fstat.gluster.org/weeks/1/failure/2
) are in netbsd and
looks like an issue with the slaves.
Most of the runs have this error:

[07:29:32] Running tests in file ./tests/basic/afr/add-brick-self-heal.t
volume create: patchy: failed: Hostnbslave70.cloud.gluster.org 
  is not in 'Peer in Cluster' state

Not investigating the .t itself any further. -Ravi
___ Gluster-devel
mailing list Gluster-devel@gluster.org

http://lists.gluster.org/mailman/listinfo/gluster-devel
 


--
Pranith


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Suggestion to optimize posix_getxattr call

[Pranith Kumar Karampuri]

yeah +1 from me too.

On Mon, May 15, 2017 at 2:56 PM, Mohit Agrawal  wrote:

> Hi,
>
>   I was just checking of posix_getxattr code to debug of my one problem.I
> have found scope of improvement in
>   this function to execute system call in better way.As of now
> function(posix_getxattr) is calling system
>   calls (sys_lgetxattr) two times to get the value of xattr or in case of
> list of xattr it calls multiple times
>
>  1) In first time it get the buffer size required to store the value.
>  call "size = sys_lgetxattr (real_path, key, NULL, 0);"
>  2) In second time it calls the sys_lgetxattr with buffer after allocate
> the buffer size
>  sys_lgetxattr (real_path, key, value, size);
>
>   In the same way function is calling for listxattr also.I think we can
> optimise it as like below
>   declare buffer buf[8192](size can be double to this also)
>   1) Call size = sys_lgetxattr (real_path,key,buf,8192)
>   2) if size == -1 and errono == ERANGE then we need to call systemcalls
> in old way like above otherwise
>  we don't need to do anything.
>
>   In the case of failure we need to call systemcall 3 times while buffer
> is more than our declared buffer
>   size otherwise in most of the cases we need to call it only once.
>
> Please share your input on this, appreciate your input.
>
> Regards
> Mohit Agrawal
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t - regression failures

[Pranith Kumar Karampuri]

karthik, Ravi,
 What is the plan to bring it back? Did you guys find RC for the
failure?

On Mon, May 15, 2017 at 10:52 AM, Ravishankar N 
wrote:

> On 05/12/2017 03:33 PM, Atin Mukherjee wrote:
>
> tests/basic/afr/add-brick-self-heal.t
> 
> is the 2nd in the list.
>
>
> All failures (https://fstat.gluster.org/weeks/1/failure/2) are in netbsd
> and looks like an issue with the slaves.
> Most of the runs have this error:
>
> [07:29:32] Running tests in file ./tests/basic/afr/add-brick-self-heal.t
> volume create: patchy: failed: Host nbslave70.cloud.gluster.org is not in 
> 'Peer in Cluster' state
>
>
> Not investigating the .t itself any further.
>
> -Ravi
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel