Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

2016-08-31 Thread Krutika Dhananjay 
On Wed, Aug 31, 2016 at 8:13 PM, David Gossage 
wrote:

> Just as a test I did not shut down the one VM on the cluster as finding a
> window before weekend where I can shut down all VM's and fit in a full heal
> is unlikely so wanted to see what occurs.
>
>
> kill -15 brick pid
> rm -Rf /gluster2/brick1/1
> mkdir /gluster2/brick1/1
> mkdir /rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard/fake3
> setfattr -n "user.some-name" -v "some-value" /rhev/data-center/mnt/glusterS
> D/192.168.71.10\:_glustershard
>
> getfattr -d -m . -e hex /gluster2/brick2/1
> # file: gluster2/brick2/1
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x0001
> trusted.afr.glustershard-client-0=0x0002
>

This is unusual. The last digit ought to have been 1 on account of "fake3"
being created while hte first brick is offline.

This discussion is becoming unnecessary lengthy. Mind if we discuss this
and sort it out on IRC today, at least the communication will be continuous
and in real-time. I'm kdhananjay on #gluster (Freenode). Ping me when
you're online.

-Krutika



> trusted.afr.glustershard-client-2=0x
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> getfattr -d -m . -e hex /gluster2/brick3/1
> # file: gluster2/brick3/1
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x0001
> trusted.afr.glustershard-client-0=0x0002
> trusted.gfid=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> setfattr -n trusted.afr.glustershard-client-0 -v
> 0x00010002 /gluster2/brick2/1
> setfattr -n trusted.afr.glustershard-client-0 -v
> 0x00010002 /gluster2/brick3/1
>
> getfattr -d -m . -e hex /gluster2/brick3/1/
> getfattr: Removing leading '/' from absolute path names
> # file: gluster2/brick3/1/
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x
> trusted.afr.glustershard-client-0=0x00010002
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> getfattr -d -m . -e hex /gluster2/brick2/1/
> getfattr: Removing leading '/' from absolute path names
> # file: gluster2/brick2/1/
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x
> trusted.afr.glustershard-client-0=0x00010002
> trusted.afr.glustershard-client-2=0x
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> gluster v start glustershard force
>
> gluster heal counts climbed up and down a little as it healed everything
> in visible gluster mount and .glusterfs for visible mount files then
> stalled with around 15 shards and the fake3 directory still in list
>
> getfattr -d -m . -e hex /gluster2/brick2/1/
> getfattr: Removing leading '/' from absolute path names
> # file: gluster2/brick2/1/
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x
> trusted.afr.glustershard-client-0=0x0001
> trusted.afr.glustershard-client-2=0x
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> getfattr -d -m . -e hex /gluster2/brick3/1/
> getfattr: Removing leading '/' from absolute path names
> # file: gluster2/brick3/1/
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x
> trusted.afr.glustershard-client-0=0x0001
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> getfattr -d -m . -e hex /gluster2/brick1/1/
> getfattr: Removing leading '/' from absolute path names
> # file: gluster2/brick1/1/
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
>

Re: [Gluster-users] group write permissions not being respected

2016-08-31 Thread Pat Haley 



hi Pat,
  Are you seeing this issue only after migration or even before? 
May be we should look at the gid numbers on the disk and the ones that 
are coming from client for the given user to see if they match or not?

-
This issue was not being seen before the migration.  We have copied the 
/etc/passwd and /etc/group files from the front-end machine (the client) 
to the data server, so they all match

-
Could you give stat output of the directory in question from both the 
brick and the nfs client



--
From the server for gluster:
[root@mseas-data2 ~]# stat /gdata/projects/nsf_alpha
  File: `/gdata/projects/nsf_alpha'
  Size: 4096  Blocks: 8  IO Block: 131072 directory
Device: 13h/19dInode: 13094773206281819436  Links: 13
Access: (2775/drwxrwsr-x)  Uid: (0/root)   Gid: ( 598/nsf_alpha)
Access: 2016-08-31 19:08:59.735990904 -0400
Modify: 2016-08-31 16:37:09.048997167 -0400
Change: 2016-08-31 16:37:41.315997148 -0400

From the server for first underlying brick
[root@mseas-data2 ~]# stat /mnt/brick1/projects/nsf_alpha/
  File: `/mnt/brick1/projects/nsf_alpha/'
  Size: 4096  Blocks: 8  IO Block: 4096 directory
Device: 800h/2048dInode: 185630  Links: 13
Access: (2775/drwxrwsr-x)  Uid: (0/root)   Gid: ( 598/nsf_alpha)
Access: 2016-08-31 19:08:59.669990907 -0400
Modify: 2016-08-31 16:37:09.048997167 -0400
Change: 2016-08-31 16:37:41.315997148 -0400

From the server for second underlying brick
[root@mseas-data2 ~]# stat /mnt/brick2/projects/nsf_alpha/
  File: `/mnt/brick2/projects/nsf_alpha/'
  Size: 4096  Blocks: 8  IO Block: 4096 directory
Device: 810h/2064dInode: 24085468Links: 13
Access: (2775/drwxrwsr-x)  Uid: (0/root)   Gid: ( 598/nsf_alpha)
Access: 2016-08-31 19:08:59.735990904 -0400
Modify: 2016-08-03 14:01:52.0 -0400
Change: 2016-08-31 16:37:41.315997148 -0400

From the client
[root@mseas FixOwn]# stat /gdata/projects/nsf_alpha
  File: `/gdata/projects/nsf_alpha'
  Size: 4096  Blocks: 8  IO Block: 1048576 directory
Device: 23h/35dInode: 13094773206281819436  Links: 13
Access: (2775/drwxrwsr-x)  Uid: (0/root)   Gid: ( 598/nsf_alpha)
Access: 2016-08-31 19:08:59.735990904 -0400
Modify: 2016-08-31 16:37:09.048997167 -0400
Change: 2016-08-31 16:37:41.315997148 -0400



Could you also let us know version of gluster you are using
-



[root@mseas-data2 ~]# gluster --version
glusterfs 3.7.11 built on Apr 27 2016 14:09:22


[root@mseas-data2 ~]# gluster volume info

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Options Reconfigured:
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off

[root@mseas-data2 ~]# gluster volume status
Status of volume: data-volume
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick mseas-data2:/mnt/brick1   49154 0  Y   5005
Brick mseas-data2:/mnt/brick2   49155 0  Y   5010

Task Status of Volume data-volume
--
Task : Rebalance
ID   : 892d9e3a-b38c-4971-b96a-8e4a496685ba
Status   : completed


[root@mseas-data2 ~]# gluster peer status
Number of Peers: 0



-
On Thu, Sep 1, 2016 at 2:46 AM, Pat Haley > wrote:



Hi,

Another piece of data.  There are 2 distinct volumes on the file
server

 1. a straight nfs partition
 2. a gluster volume (served over nfs)

The straight nfs partition does respect the group write
permissions, while the gluster volume does not.  Any suggestions
on how to debug this or what additional information would be
helpful would be greatly appreciated

Thanks

On 08/30/2016 06:01 PM, Pat Haley wrote:


Hi

We have just migrated our data to a new file server (more space,
old server was showing its age). We have a volume for
collaborative use, based on group membership.  In our new server,
the group write permissions

Re: [Gluster-users] group write permissions not being respected

2016-08-31 Thread Pranith Kumar Karampuri 
hi Pat,
  Are you seeing this issue only after migration or even before? May be
we should look at the gid numbers on the disk and the ones that are coming
from client for the given user to see if they match or not?

Could you give stat output of the directory in question from both the brick
and the nfs client

Could you also let us know version of gluster you are using.


On Thu, Sep 1, 2016 at 2:46 AM, Pat Haley  wrote:

>
> Hi,
>
> Another piece of data.  There are 2 distinct volumes on the file server
>
>1. a straight nfs partition
>2. a gluster volume (served over nfs)
>
> The straight nfs partition does respect the group write permissions, while
> the gluster volume does not.  Any suggestions on how to debug this or what
> additional information would be helpful would be greatly appreciated
>
> Thanks
>
> On 08/30/2016 06:01 PM, Pat Haley wrote:
>
>
> Hi
>
> We have just migrated our data to a new file server (more space, old
> server was showing its age). We have a volume for collaborative use, based
> on group membership.  In our new server, the group write permissions are
> not being respected (e.g.  the owner of a directory can still write to that
> directory but any other member of the associated group cannot, even though
> the directory clearly has group write permissions set).  This is occurring
> regardless of how many groups the user is a member of (i.e. users that are
> members of fewer then 16 groups are still affected).
>
> the relevant fstab line from the server looks like
> localhost:/data-volume /gdataglusterfs   defaults 0 0
>
> and for a client:
> mseas-data2:/gdata   /gdata  nfs defaults0 0
>
> Any help would be greatly appreciated.
>
> Thanks
>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley  Email:  pha...@mit.edu
> Center for Ocean Engineering   Phone:  (617) 253-6824
> Dept. of Mechanical EngineeringFax:(617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] group write permissions not being respected

2016-08-31 Thread Pat Haley 


Hi,

Another piece of data.  There are 2 distinct volumes on the file server

1. a straight nfs partition
2. a gluster volume (served over nfs)

The straight nfs partition does respect the group write permissions, 
while the gluster volume does not.  Any suggestions on how to debug this 
or what additional information would be helpful would be greatly appreciated


Thanks

On 08/30/2016 06:01 PM, Pat Haley wrote:


Hi

We have just migrated our data to a new file server (more space, old 
server was showing its age). We have a volume for collaborative use, 
based on group membership.  In our new server, the group write 
permissions are not being respected (e.g.  the owner of a directory 
can still write to that directory but any other member of the 
associated group cannot, even though the directory clearly has group 
write permissions set).  This is occurring regardless of how many 
groups the user is a member of (i.e. users that are members of fewer 
then 16 groups are still affected).


the relevant fstab line from the server looks like
localhost:/data-volume /gdataglusterfs   defaults 0 0

and for a client:
mseas-data2:/gdata   /gdata  nfs defaults0 0

Any help would be greatly appreciated.

Thanks



--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:  pha...@mit.edu
Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

2016-08-31 Thread David Gossage 
test server
root@ccengine2 ~]# ls -l /gluster2/brick?/1/.glusterfs/indices/xattrop/
/gluster2/brick1/1/.glusterfs/indices/xattrop/:
total 1
--. 1 root root 0 Aug 31 08:48
xattrop-542c3fdb-add6-4efa-8cde-288991935eee

/gluster2/brick2/1/.glusterfs/indices/xattrop/:
total 1
--. 2 root root 0 Aug 30 09:33 ----0001
--. 2 root root 0 Aug 30 09:33
xattrop-58f5b39b-0935-4153-b85b-4f4b2724906f

/gluster2/brick3/1/.glusterfs/indices/xattrop/:
total 1
--. 2 root root 0 Aug 30 09:40 ----0001
--. 2 root root 0 Aug 30 09:40
xattrop-6a58e1ac-dfdb-4f6e-93d3-5f02d49bf94b


do i need to do something about these entries?  2 are from yesterday
probably during some of testing one from this morning.
gluster volume heal glustershard statistics heal-count
Gathering count of entries to be healed on volume glustershard has been
successful

Brick 192.168.71.10:/gluster2/brick1/1
Number of entries: 0

Brick 192.168.71.11:/gluster2/brick2/1
Number of entries: 1

Brick 192.168.71.12:/gluster2/brick3/1
Number of entries: 1

getfattr -d -m . -e hex /gluster2/brick?/1/
getfattr: Removing leading '/' from absolute path names
# file: gluster2/brick1/1/
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

# file: gluster2/brick2/1/
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.glustershard-client-0=0x0001
trusted.afr.glustershard-client-2=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

# file: gluster2/brick3/1/
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.glustershard-client-0=0x0001
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565




*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284

On Wed, Aug 31, 2016 at 9:43 AM, David Gossage 
wrote:

> Just as a test I did not shut down the one VM on the cluster as finding a
> window before weekend where I can shut down all VM's and fit in a full heal
> is unlikely so wanted to see what occurs.
>
>
> kill -15 brick pid
> rm -Rf /gluster2/brick1/1
> mkdir /gluster2/brick1/1
> mkdir /rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard/fake3
> setfattr -n "user.some-name" -v "some-value" /rhev/data-center/mnt/glusterS
> D/192.168.71.10\:_glustershard
>
> getfattr -d -m . -e hex /gluster2/brick2/1
> # file: gluster2/brick2/1
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x0001
> trusted.afr.glustershard-client-0=0x0002
> trusted.afr.glustershard-client-2=0x
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> getfattr -d -m . -e hex /gluster2/brick3/1
> # file: gluster2/brick3/1
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x0001
> trusted.afr.glustershard-client-0=0x0002
> trusted.gfid=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> setfattr -n trusted.afr.glustershard-client-0 -v
> 0x00010002 /gluster2/brick2/1
> setfattr -n trusted.afr.glustershard-client-0 -v
> 0x00010002 /gluster2/brick3/1
>
> getfattr -d -m . -e hex /gluster2/brick3/1/
> getfattr: Removing leading '/' from absolute path names
> # file: gluster2/brick3/1/
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x
> trusted.afr.glustershard-client-0=0x00010002
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
> user.some-name=0x736f6d652d76616c7565
>
> getfattr -d -m . -e hex /gluster2/brick2/1/
> getfattr: Removing leading '/' from absolute path names
> # file: gluster2/brick2/1/
>

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

2016-08-31 Thread David Gossage 
Just as a test I did not shut down the one VM on the cluster as finding a
window before weekend where I can shut down all VM's and fit in a full heal
is unlikely so wanted to see what occurs.


kill -15 brick pid
rm -Rf /gluster2/brick1/1
mkdir /gluster2/brick1/1
mkdir /rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard/fake3
setfattr -n "user.some-name" -v "some-value" /rhev/data-center/mnt/
glusterSD/192.168.71.10\:_glustershard

getfattr -d -m . -e hex /gluster2/brick2/1
# file: gluster2/brick2/1
security.selinux=0x756e636f6e66696e65645f753a6f
626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x0001
trusted.afr.glustershard-client-0=0x0002
trusted.afr.glustershard-client-2=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

getfattr -d -m . -e hex /gluster2/brick3/1
# file: gluster2/brick3/1
security.selinux=0x756e636f6e66696e65645f753a6f
626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x0001
trusted.afr.glustershard-client-0=0x0002
trusted.gfid=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

setfattr -n trusted.afr.glustershard-client-0 -v 0x00010002
/gluster2/brick2/1
setfattr -n trusted.afr.glustershard-client-0 -v 0x00010002
/gluster2/brick3/1

getfattr -d -m . -e hex /gluster2/brick3/1/
getfattr: Removing leading '/' from absolute path names
# file: gluster2/brick3/1/
security.selinux=0x756e636f6e66696e65645f753a6f
626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.glustershard-client-0=0x00010002
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

getfattr -d -m . -e hex /gluster2/brick2/1/
getfattr: Removing leading '/' from absolute path names
# file: gluster2/brick2/1/
security.selinux=0x756e636f6e66696e65645f753a6f
626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.glustershard-client-0=0x00010002
trusted.afr.glustershard-client-2=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

gluster v start glustershard force

gluster heal counts climbed up and down a little as it healed everything in
visible gluster mount and .glusterfs for visible mount files then stalled
with around 15 shards and the fake3 directory still in list

getfattr -d -m . -e hex /gluster2/brick2/1/
getfattr: Removing leading '/' from absolute path names
# file: gluster2/brick2/1/
security.selinux=0x756e636f6e66696e65645f753a6f
626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.glustershard-client-0=0x0001
trusted.afr.glustershard-client-2=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

getfattr -d -m . -e hex /gluster2/brick3/1/
getfattr: Removing leading '/' from absolute path names
# file: gluster2/brick3/1/
security.selinux=0x756e636f6e66696e65645f753a6f
626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.glustershard-client-0=0x0001
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

getfattr -d -m . -e hex /gluster2/brick1/1/
getfattr: Removing leading '/' from absolute path names
# file: gluster2/brick1/1/
security.selinux=0x756e636f6e66696e65645f753a6f
626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x5889332e50ba441e8fa5cce3ae6f3a15
user.some-name=0x736f6d652d76616c7565

heal count stayed same for awhile then ran

gluster v heal glustershard full

heals jump up to 700 as shards actually get read in as needing heals.
 glustershd shows 3 sweeps started one per brick

It heals shards things look ok heal <> info shows 0 files but statistics
heal-info shows 1 left for brick 2 and 3. perhaps cause I didnt stop vm
running?

# file: gluster2/brick1/1/

Re: [Gluster-users] CFP for Gluster Developer Summit

2016-08-31 Thread Samikshan Bairagya 



On 08/31/2016 12:42 PM, Paul Cuzner wrote:

Sounds great!

I had to knit together different cli commands in the past for 'gstatus' to
provide a view of the cluster - so this is a cool.



Hi Paul,

Glad you found it interesting. I had checked gstatus a few months back 
and yeah, I had noticed that you had to knit up different cli commands 
together to arrive at a state representation for the cluster. This new 
command kind of does the knitting up part but it still doesn't really 
provide a global view of the cluster as gstatus currently does. This 
command provides a view of the cluster as seen locally; and the external 
application would need to collate all the data from all nodes to come up 
with a complete view of the cluster state. During the talk my plan is to 
highlight the possibilities that this command opens up while also 
pointing out all the limitations of this command.



Would it be possible to add an example of the output to the RFE BZ* 1353156
* 



Yes, I have attached a sample output to the BZ here: 
https://bugzilla.redhat.com/show_bug.cgi?id=1353156#c19


Thanks and Regards,

Samikshan


*?*
Paul C

On Wed, Aug 31, 2016 at 6:13 PM, Samikshan Bairagya 
wrote:


Hi all,

I'd like to propose the following talk for Gluster Developer Summit 2016.

Title: How an external application looking to integrate with Gluster can
use the CLI to get the state of a cluster

Theme: Experience (Developers integrating Gluster with other ecosystems)

Gluster 3.9 will have a new CLI that can be used to get the local state
representation of a cluster. This can be used by external applications
(like storage managers) to get a representation of the entire state of a
cluster. I plan to talk about this during the summit and will cover the
following:

- Introduction
- List of data points covered in the state representation
- How to consume this CLI
- Discussion on what other data points might need to be added later on.
- Demo (External application representing the state of a cluster using
data obtained from the CLI)

Thanks and Regards,

Samikshan

On 08/13/2016 01:18 AM, Vijay Bellur wrote:


Hey All,

Gluster Developer Summit 2016 is fast approaching [1] on us. We are
looking to have talks and discussions related to the following themes in
the summit:

1. Gluster.Next - focusing on features shaping the future of Gluster

2. Experience - Description of real world experience and feedback from:
   a> Devops and Users deploying Gluster in production
   b> Developers integrating Gluster with other ecosystems

3. Use cases  - focusing on key use cases that drive Gluster.today and
Gluster.Next

4. Stability & Performance - focusing on current improvements to reduce
our technical debt backlog

5. Process & infrastructure  - focusing on improving current workflow,
infrastructure to make life easier for all of us!

If you have a talk/discussion proposal that can be part of these themes,
please send out your proposal(s) by replying to this thread. Please
clearly mention the theme for which your proposal is relevant when you
do so. We will be ending the CFP by 12 midnight PDT on August 31st, 2016.

If you have other topics that do not fit in the themes listed, please
feel free to propose and we might be able to accommodate some of them as
lightening talks or something similar.

Please do reach out to me or Amye if you have any questions.

Thanks!
Vijay

[1] https://www.gluster.org/events/summit2016/
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

2016-08-31 Thread David Gossage 
On Wed, Aug 31, 2016 at 12:59 AM, Krutika Dhananjay 
wrote:

> Tried this.
>
> With me, only 'fake2' gets healed after i bring the 'empty' brick back up
> and it stops there unless I do a 'heal-full'.
>

When you say heal-full is that a command I don't see when running gluster
help or are you just abbreviating the 'gluster v heal <> full'  line?

>
> Is that what you're seeing as well?
>

Yes and no.  Right now on my prod even if running heal <> full nothing
happens.  My understanding is that a sweep should occur on each brick and
it only occurs on the down node then no shard healing occurs.


>
> -Krutika
>
> On Wed, Aug 31, 2016 at 4:43 AM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> Same issue brought up glusterd on problem node heal count still stuck at
>> 6330.
>>
>> Ran gluster v heal GUSTER1 full
>>
>> glustershd on problem node shows a sweep starting and finishing in
>> seconds.  Other 2 nodes show no activity in log.  They should start a sweep
>> too shouldn't they?
>>
>> Tried starting from scratch
>>
>> kill -15 brickpid
>> rm -Rf /brick
>> mkdir -p /brick
>> mkdir mkdir /gsmount/fake2
>> setfattr -n "user.some-name" -v "some-value" /gsmount/fake2
>>
>> Heals visible dirs instantly then stops.
>>
>> gluster v heal GLUSTER1 full
>>
>> see sweep star on problem node and end almost instantly.  no files added
>> t heal list no files healed no more logging
>>
>> [2016-08-30 23:11:31.544331] I [MSGID: 108026]
>> [afr-self-heald.c:646:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>> starting full sweep on subvol GLUSTER1-client-1
>> [2016-08-30 23:11:33.776235] I [MSGID: 108026]
>> [afr-self-heald.c:656:afr_shd_full_healer] 0-GLUSTER1-replicate-0:
>> finished full sweep on subvol GLUSTER1-client-1
>>
>> same results no matter which node you run command on.  Still stuck with
>> 6330 files showing needing healed out of 19k.  still showing in logs no
>> heals are occuring.
>>
>> Is their a way to forcibly reset any prior heal data?  Could it be stuck
>> on some past failed heal start?
>>
>>
>>
>>
>> *David Gossage*
>> *Carousel Checks Inc. | System Administrator*
>> *Office* 708.613.2284
>>
>> On Tue, Aug 30, 2016 at 10:03 AM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> On Tue, Aug 30, 2016 at 10:02 AM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
 updated test server to 3.8.3

 Brick1: 192.168.71.10:/gluster2/brick1/1
 Brick2: 192.168.71.11:/gluster2/brick2/1
 Brick3: 192.168.71.12:/gluster2/brick3/1
 Options Reconfigured:
 cluster.granular-entry-heal: on
 performance.readdir-ahead: on
 performance.read-ahead: off
 nfs.disable: on
 nfs.addr-namelookup: off
 nfs.enable-ino32: off
 cluster.background-self-heal-count: 16
 cluster.self-heal-window-size: 1024
 performance.quick-read: off
 performance.io-cache: off
 performance.stat-prefetch: off
 cluster.eager-lock: enable
 network.remote-dio: on
 cluster.quorum-type: auto
 cluster.server-quorum-type: server
 storage.owner-gid: 36
 storage.owner-uid: 36
 server.allow-insecure: on
 features.shard: on
 features.shard-block-size: 64MB
 performance.strict-o-direct: off
 cluster.locking-scheme: granular

 kill -15 brickpid
 rm -Rf /gluster2/brick3
 mkdir -p /gluster2/brick3/1
 mkdir mkdir /rhev/data-center/mnt/glusterSD/192.168.71.10
 \:_glustershard/fake2
 setfattr -n "user.some-name" -v "some-value"
 /rhev/data-center/mnt/glusterSD/192.168.71.10\:_glustershard/fake2
 gluster v start glustershard force

 at this point brick process starts and all visible files including new
 dir are made on brick
 handful of shards are in heal statistics still but no .shard directory
 created and no increase in shard count

 gluster v heal glustershard

 At this point still no increase in count or dir made no additional
 activity in logs for healing generated.  waited few minutes tailing logs to
 check if anything kicked in.

 gluster v heal glustershard full

 gluster shards added to list and heal commences.  logs show full sweep
 starting on all 3 nodes.  though this time it only shows as finishing on
 one which looks to be the one that had brick deleted.

 [2016-08-30 14:45:33.098589] I [MSGID: 108026]
 [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
 starting full sweep on subvol glustershard-client-0
 [2016-08-30 14:45:33.099492] I [MSGID: 108026]
 [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
 starting full sweep on subvol glustershard-client-1
 [2016-08-30 14:45:33.100093] I [MSGID: 108026]
 [afr-self-heald.c:646:afr_shd_full_healer] 0-glustershard-replicate-0:
 starting full sweep on subvol glustershard-client-2
 [2016-08-30 14:52:29.760213] I [MSGID: 108026]

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

2016-08-31 Thread Krutika Dhananjay 
No, sorry, it's working fine. I may have missed some step because of which
i saw that problem. /.shard is also healing fine now.

Let me know if it works for you.

-Krutika

On Wed, Aug 31, 2016 at 12:49 PM, Krutika Dhananjay 
wrote:

> OK I just hit the other issue too, where .shard doesn't get healed. :)
>
> Investigating as to why that is the case. Give me some time.
>
> -Krutika
>
> On Wed, Aug 31, 2016 at 12:39 PM, Krutika Dhananjay 
> wrote:
>
>> Just figured the steps Anuradha has provided won't work if granular entry
>> heal is on.
>> So when you bring down a brick and create fake2 under / of the volume,
>> granular entry heal feature causes
>> sh to remember only the fact that 'fake2' needs to be recreated on the
>> offline brick (because changelogs are granular).
>>
>> In this case, we would be required to indicate to self-heal-daemon that
>> the entire directory tree from '/' needs to be repaired on the brick that
>> contains no data.
>>
>> To fix this, I did the following (for users who use granular entry
>> self-healing):
>>
>> 1. Kill the last brick process in the replica (/bricks/3)
>>
>> 2. [root@server-3 ~]# rm -rf /bricks/3
>>
>> 3. [root@server-3 ~]# mkdir /bricks/3
>>
>> 4. Create a new dir on the mount point:
>> [root@client-1 ~]# mkdir /mnt/fake
>>
>> 5. Set some fake xattr on the root of the volume, and not the 'fake'
>> directory itself.
>> [root@client-1 ~]# setfattr -n "user.some-name" -v "some-value" /mnt
>>
>> 6. Make sure there's no io happening on your volume.
>>
>> 7. Check the pending xattrs on the brick directories of the two good
>> copies (on bricks 1 and 2), you should be seeing same values as the one
>> marked in red in both bricks.
>> (note that the client- xattr key will have the same last digit as
>> the index of the brick that is down, when counting from 0. So if the first
>> brick is the one that is down, it would read trusted.afr.*-client-0; if the
>> second brick is the one that is empty and down, it would read
>> trusted.afr.*-client-1 and so on).
>>
>> [root@server-1 ~]# getfattr -d -m . -e hex /bricks/1
>> # file: 1
>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
>> 23a6574635f72756e74696d655f743a733000
>> trusted.afr.dirty=0x
>> *trusted.afr.rep-client-2=0x00010001*
>> trusted.gfid=0x0001
>> trusted.glusterfs.dht=0x0001
>> trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b
>>
>> [root@server-2 ~]# getfattr -d -m . -e hex /bricks/2
>> # file: 2
>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
>> 23a6574635f72756e74696d655f743a733000
>> trusted.afr.dirty=0x
>> *trusted.afr.rep-client-2=0x000**10001*
>> trusted.gfid=0x0001
>> trusted.glusterfs.dht=0x0001
>> trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b
>>
>> 8. Flip the 8th digit in the trusted.afr.-client-2 to a 1.
>>
>> [root@server-1 ~]# setfattr -n trusted.afr.rep-client-2 -v
>> *0x000100010001* /bricks/1
>> [root@server-2 ~]# setfattr -n trusted.afr.rep-client-2 -v
>> *0x000100010001* /bricks/2
>>
>> 9. Get the xattrs again and check the xattrs are set properly now
>>
>> [root@server-1 ~]# getfattr -d -m . -e hex /bricks/1
>> # file: 1
>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
>> 23a6574635f72756e74696d655f743a733000
>> trusted.afr.dirty=0x
>> *trusted.afr.rep-client-2=0x000**100010001*
>> trusted.gfid=0x0001
>> trusted.glusterfs.dht=0x0001
>> trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b
>>
>> [root@server-2 ~]# getfattr -d -m . -e hex /bricks/2
>> # file: 2
>> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
>> 23a6574635f72756e74696d655f743a733000
>> trusted.afr.dirty=0x
>> *trusted.afr.rep-client-2=0x000**100010001*
>> trusted.gfid=0x0001
>> trusted.glusterfs.dht=0x0001
>> trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b
>>
>> 10. Force-start the volume.
>>
>> [root@server-1 ~]# gluster volume start rep force
>> volume start: rep: success
>>
>> 11. Monitor heal-info command to ensure the number of entries keeps
>> growing.
>>
>> 12. Keep monitoring with step 10 and eventually the number of entries
>> needing heal must come down to 0.
>> Also the checksums of the files on the previously empty brick should now
>> match with the copies on the other two bricks.
>>
>> Could you check if the above steps work for you, in your test environment?
>>
>> You caught a nice bug in the manual steps to follow when granular
>> entry-heal is enabled and an empty brick needs heal. Thanks for reporting
>> it. :) We will fix the documentation appropriately.

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

2016-08-31 Thread Krutika Dhananjay 
OK I just hit the other issue too, where .shard doesn't get healed. :)

Investigating as to why that is the case. Give me some time.

-Krutika

On Wed, Aug 31, 2016 at 12:39 PM, Krutika Dhananjay 
wrote:

> Just figured the steps Anuradha has provided won't work if granular entry
> heal is on.
> So when you bring down a brick and create fake2 under / of the volume,
> granular entry heal feature causes
> sh to remember only the fact that 'fake2' needs to be recreated on the
> offline brick (because changelogs are granular).
>
> In this case, we would be required to indicate to self-heal-daemon that
> the entire directory tree from '/' needs to be repaired on the brick that
> contains no data.
>
> To fix this, I did the following (for users who use granular entry
> self-healing):
>
> 1. Kill the last brick process in the replica (/bricks/3)
>
> 2. [root@server-3 ~]# rm -rf /bricks/3
>
> 3. [root@server-3 ~]# mkdir /bricks/3
>
> 4. Create a new dir on the mount point:
> [root@client-1 ~]# mkdir /mnt/fake
>
> 5. Set some fake xattr on the root of the volume, and not the 'fake'
> directory itself.
> [root@client-1 ~]# setfattr -n "user.some-name" -v "some-value" /mnt
>
> 6. Make sure there's no io happening on your volume.
>
> 7. Check the pending xattrs on the brick directories of the two good
> copies (on bricks 1 and 2), you should be seeing same values as the one
> marked in red in both bricks.
> (note that the client- xattr key will have the same last digit as the
> index of the brick that is down, when counting from 0. So if the first
> brick is the one that is down, it would read trusted.afr.*-client-0; if the
> second brick is the one that is empty and down, it would read
> trusted.afr.*-client-1 and so on).
>
> [root@server-1 ~]# getfattr -d -m . -e hex /bricks/1
> # file: 1
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a6574635f72756e74696d655f743a733000
> trusted.afr.dirty=0x
> *trusted.afr.rep-client-2=0x00010001*
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b
>
> [root@server-2 ~]# getfattr -d -m . -e hex /bricks/2
> # file: 2
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a6574635f72756e74696d655f743a733000
> trusted.afr.dirty=0x
> *trusted.afr.rep-client-2=0x000**10001*
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b
>
> 8. Flip the 8th digit in the trusted.afr.-client-2 to a 1.
>
> [root@server-1 ~]# setfattr -n trusted.afr.rep-client-2 -v
> *0x000100010001* /bricks/1
> [root@server-2 ~]# setfattr -n trusted.afr.rep-client-2 -v
> *0x000100010001* /bricks/2
>
> 9. Get the xattrs again and check the xattrs are set properly now
>
> [root@server-1 ~]# getfattr -d -m . -e hex /bricks/1
> # file: 1
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a6574635f72756e74696d655f743a733000
> trusted.afr.dirty=0x
> *trusted.afr.rep-client-2=0x000**100010001*
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b
>
> [root@server-2 ~]# getfattr -d -m . -e hex /bricks/2
> # file: 2
> security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
> 23a6574635f72756e74696d655f743a733000
> trusted.afr.dirty=0x
> *trusted.afr.rep-client-2=0x000**100010001*
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b
>
> 10. Force-start the volume.
>
> [root@server-1 ~]# gluster volume start rep force
> volume start: rep: success
>
> 11. Monitor heal-info command to ensure the number of entries keeps
> growing.
>
> 12. Keep monitoring with step 10 and eventually the number of entries
> needing heal must come down to 0.
> Also the checksums of the files on the previously empty brick should now
> match with the copies on the other two bricks.
>
> Could you check if the above steps work for you, in your test environment?
>
> You caught a nice bug in the manual steps to follow when granular
> entry-heal is enabled and an empty brick needs heal. Thanks for reporting
> it. :) We will fix the documentation appropriately.
>
> -Krutika
>
>
> On Wed, Aug 31, 2016 at 11:29 AM, Krutika Dhananjay 
> wrote:
>
>> Tried this.
>>
>> With me, only 'fake2' gets healed after i bring the 'empty' brick back up
>> and it stops there unless I do a 'heal-full'.
>>
>> Is that what you're seeing as well?
>>
>> -Krutika
>>
>> On Wed, Aug 31, 2016 at 4:43 AM, David Gossage <
>>

Re: [Gluster-users] CFP for Gluster Developer Summit

2016-08-31 Thread Paul Cuzner 
Sounds great!

I had to knit together different cli commands in the past for 'gstatus' to
provide a view of the cluster - so this is a cool.

Would it be possible to add an example of the output to the RFE BZ* 1353156
* 

*?*
Paul C

On Wed, Aug 31, 2016 at 6:13 PM, Samikshan Bairagya 
wrote:

> Hi all,
>
> I'd like to propose the following talk for Gluster Developer Summit 2016.
>
> Title: How an external application looking to integrate with Gluster can
> use the CLI to get the state of a cluster
>
> Theme: Experience (Developers integrating Gluster with other ecosystems)
>
> Gluster 3.9 will have a new CLI that can be used to get the local state
> representation of a cluster. This can be used by external applications
> (like storage managers) to get a representation of the entire state of a
> cluster. I plan to talk about this during the summit and will cover the
> following:
>
> - Introduction
> - List of data points covered in the state representation
> - How to consume this CLI
> - Discussion on what other data points might need to be added later on.
> - Demo (External application representing the state of a cluster using
> data obtained from the CLI)
>
> Thanks and Regards,
>
> Samikshan
>
> On 08/13/2016 01:18 AM, Vijay Bellur wrote:
>
>> Hey All,
>>
>> Gluster Developer Summit 2016 is fast approaching [1] on us. We are
>> looking to have talks and discussions related to the following themes in
>> the summit:
>>
>> 1. Gluster.Next - focusing on features shaping the future of Gluster
>>
>> 2. Experience - Description of real world experience and feedback from:
>>a> Devops and Users deploying Gluster in production
>>b> Developers integrating Gluster with other ecosystems
>>
>> 3. Use cases  - focusing on key use cases that drive Gluster.today and
>> Gluster.Next
>>
>> 4. Stability & Performance - focusing on current improvements to reduce
>> our technical debt backlog
>>
>> 5. Process & infrastructure  - focusing on improving current workflow,
>> infrastructure to make life easier for all of us!
>>
>> If you have a talk/discussion proposal that can be part of these themes,
>> please send out your proposal(s) by replying to this thread. Please
>> clearly mention the theme for which your proposal is relevant when you
>> do so. We will be ending the CFP by 12 midnight PDT on August 31st, 2016.
>>
>> If you have other topics that do not fit in the themes listed, please
>> feel free to propose and we might be able to accommodate some of them as
>> lightening talks or something similar.
>>
>> Please do reach out to me or Amye if you have any questions.
>>
>> Thanks!
>> Vijay
>>
>> [1] https://www.gluster.org/events/summit2016/
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow

2016-08-31 Thread Krutika Dhananjay 
Just figured the steps Anuradha has provided won't work if granular entry
heal is on.
So when you bring down a brick and create fake2 under / of the volume,
granular entry heal feature causes
sh to remember only the fact that 'fake2' needs to be recreated on the
offline brick (because changelogs are granular).

In this case, we would be required to indicate to self-heal-daemon that the
entire directory tree from '/' needs to be repaired on the brick that
contains no data.

To fix this, I did the following (for users who use granular entry
self-healing):

1. Kill the last brick process in the replica (/bricks/3)

2. [root@server-3 ~]# rm -rf /bricks/3

3. [root@server-3 ~]# mkdir /bricks/3

4. Create a new dir on the mount point:
[root@client-1 ~]# mkdir /mnt/fake

5. Set some fake xattr on the root of the volume, and not the 'fake'
directory itself.
[root@client-1 ~]# setfattr -n "user.some-name" -v "some-value" /mnt

6. Make sure there's no io happening on your volume.

7. Check the pending xattrs on the brick directories of the two good copies
(on bricks 1 and 2), you should be seeing same values as the one marked in
red in both bricks.
(note that the client- xattr key will have the same last digit as the
index of the brick that is down, when counting from 0. So if the first
brick is the one that is down, it would read trusted.afr.*-client-0; if the
second brick is the one that is empty and down, it would read
trusted.afr.*-client-1 and so on).

[root@server-1 ~]# getfattr -d -m . -e hex /bricks/1
# file: 1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
23a6574635f72756e74696d655f743a733000
trusted.afr.dirty=0x
*trusted.afr.rep-client-2=0x00010001*
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b

[root@server-2 ~]# getfattr -d -m . -e hex /bricks/2
# file: 2
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
23a6574635f72756e74696d655f743a733000
trusted.afr.dirty=0x
*trusted.afr.rep-client-2=0x000**10001*
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b

8. Flip the 8th digit in the trusted.afr.-client-2 to a 1.

[root@server-1 ~]# setfattr -n trusted.afr.rep-client-2 -v
*0x000100010001* /bricks/1
[root@server-2 ~]# setfattr -n trusted.afr.rep-client-2 -v
*0x000100010001* /bricks/2

9. Get the xattrs again and check the xattrs are set properly now

[root@server-1 ~]# getfattr -d -m . -e hex /bricks/1
# file: 1
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
23a6574635f72756e74696d655f743a733000
trusted.afr.dirty=0x
*trusted.afr.rep-client-2=0x000**100010001*
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b

[root@server-2 ~]# getfattr -d -m . -e hex /bricks/2
# file: 2
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f7
23a6574635f72756e74696d655f743a733000
trusted.afr.dirty=0x
*trusted.afr.rep-client-2=0x000**100010001*
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0xa349517bb9d44bdf96da8ea324f89e7b

10. Force-start the volume.

[root@server-1 ~]# gluster volume start rep force
volume start: rep: success

11. Monitor heal-info command to ensure the number of entries keeps growing.

12. Keep monitoring with step 10 and eventually the number of entries
needing heal must come down to 0.
Also the checksums of the files on the previously empty brick should now
match with the copies on the other two bricks.

Could you check if the above steps work for you, in your test environment?

You caught a nice bug in the manual steps to follow when granular
entry-heal is enabled and an empty brick needs heal. Thanks for reporting
it. :) We will fix the documentation appropriately.

-Krutika


On Wed, Aug 31, 2016 at 11:29 AM, Krutika Dhananjay 
wrote:

> Tried this.
>
> With me, only 'fake2' gets healed after i bring the 'empty' brick back up
> and it stops there unless I do a 'heal-full'.
>
> Is that what you're seeing as well?
>
> -Krutika
>
> On Wed, Aug 31, 2016 at 4:43 AM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> Same issue brought up glusterd on problem node heal count still stuck at
>> 6330.
>>
>> Ran gluster v heal GUSTER1 full
>>
>> glustershd on problem node shows a sweep starting and finishing in
>> seconds.  Other 2 nodes show no activity in log.  They should start a sweep
>> too shouldn't they?
>>
>> Tried starting from scratch
>>
>> kill -15 brickpid
>> rm -Rf /brick
>> mkdir -p /brick
>> mkdir mkdir

Re: [Gluster-users] CFP for Gluster Developer Summit

2016-08-31 Thread Samikshan Bairagya 

Hi all,

I'd like to propose the following talk for Gluster Developer Summit 2016.

Title: How an external application looking to integrate with Gluster can 
use the CLI to get the state of a cluster


Theme: Experience (Developers integrating Gluster with other ecosystems)

Gluster 3.9 will have a new CLI that can be used to get the local state 
representation of a cluster. This can be used by external applications 
(like storage managers) to get a representation of the entire state of a 
cluster. I plan to talk about this during the summit and will cover the 
following:


- Introduction
- List of data points covered in the state representation
- How to consume this CLI
- Discussion on what other data points might need to be added later on.
- Demo (External application representing the state of a cluster using 
data obtained from the CLI)


Thanks and Regards,

Samikshan

On 08/13/2016 01:18 AM, Vijay Bellur wrote:

Hey All,

Gluster Developer Summit 2016 is fast approaching [1] on us. We are
looking to have talks and discussions related to the following themes in
the summit:

1. Gluster.Next - focusing on features shaping the future of Gluster

2. Experience - Description of real world experience and feedback from:
   a> Devops and Users deploying Gluster in production
   b> Developers integrating Gluster with other ecosystems

3. Use cases  - focusing on key use cases that drive Gluster.today and
Gluster.Next

4. Stability & Performance - focusing on current improvements to reduce
our technical debt backlog

5. Process & infrastructure  - focusing on improving current workflow,
infrastructure to make life easier for all of us!

If you have a talk/discussion proposal that can be part of these themes,
please send out your proposal(s) by replying to this thread. Please
clearly mention the theme for which your proposal is relevant when you
do so. We will be ending the CFP by 12 midnight PDT on August 31st, 2016.

If you have other topics that do not fit in the themes listed, please
feel free to propose and we might be able to accommodate some of them as
lightening talks or something similar.

Please do reach out to me or Amye if you have any questions.

Thanks!
Vijay

[1] https://www.gluster.org/events/summit2016/
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Bug Triage meeting log

2016-08-31 Thread Ankit Raj 
Hello,

Here are the log of yesterday Gluster Community Bug Triage Meeting.

 Minutes:
https://meetbot.fedoraproject.org/gluster-meeting/2016-08-30/gluster_community_bug_triage_meeting.2016-08-30-12.00.html
 Minutes (text):
https://meetbot.fedoraproject.org/gluster-meeting/2016-08-30/gluster_community_bug_triage_meeting.2016-08-30-12.00.txt
Log:
https://meetbot.fedoraproject.org/gluster-meeting/2016-08-30/gluster_community_bug_triage_meeting.2016-08-30-12.00.log.html



Thanks, for your participation.

Regards,
Ankit Raj
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users