Re: [Gluster-users] vfs_fruit and extended attributes

[Terry McGuire]

Hello Anoop.  Thanks for helping with this!

> On Sep 22, 2017, at 00:11, Anoop C S  wrote:
> 
> On Thu, 2017-09-21 at 10:35 -0600, Terry McGuire wrote:
>> Hello list.  I’m attempting to improve how Samba shares directories on our 
>> Gluster volume to Mac
>> users by using the vfs_fruit module.
> 
> What versions of GlusterFS and Samba have you installed? And which 
> platform/distro are you using as
> Samba server?

glusterfs-3.10.5-1.el7.x86_64
samba-4.4.4-14.el7_3.x86_64

CentOS Linux release 7.3.1611 (Core)

This is a brand new service.  We’re running gluster clients, and samba, on the 
servers themselves.

> 
> Please paste the output of `gluster volume info `.

(Apologies for the length of this…)

root@mfs-01 ~]#gluster volume info mfs1
 
Volume Name: mfs1
Type: Distributed-Disperse
Volume ID: 2fa02e5d-95b4-4aaa-b16c-5de90e0b11b2
Status: Started
Snapshot Count: 0
Number of Bricks: 6 x (8 + 4) = 72
Transport-type: tcp
Bricks:
Brick1: mfs-b01:/mnt/gfs001/data
Brick2: mfs-b01:/mnt/gfs002/data
Brick3: mfs-b01:/mnt/gfs003/data
Brick4: mfs-b02:/mnt/gfs019/data
Brick5: mfs-b02:/mnt/gfs020/data
Brick6: mfs-b02:/mnt/gfs021/data
Brick7: mfs-b03:/mnt/gfs037/data
Brick8: mfs-b03:/mnt/gfs038/data
Brick9: mfs-b03:/mnt/gfs039/data
Brick10: mfs-b04:/mnt/gfs055/data
Brick11: mfs-b04:/mnt/gfs056/data
Brick12: mfs-b04:/mnt/gfs057/data
Brick13: mfs-b01:/mnt/gfs004/data
Brick14: mfs-b01:/mnt/gfs005/data
Brick15: mfs-b01:/mnt/gfs006/data
Brick16: mfs-b02:/mnt/gfs022/data
Brick17: mfs-b02:/mnt/gfs023/data
Brick18: mfs-b02:/mnt/gfs024/data
Brick19: mfs-b03:/mnt/gfs040/data
Brick20: mfs-b03:/mnt/gfs041/data
Brick21: mfs-b03:/mnt/gfs042/data
Brick22: mfs-b04:/mnt/gfs058/data
Brick23: mfs-b04:/mnt/gfs059/data
Brick24: mfs-b04:/mnt/gfs060/data
Brick25: mfs-b01:/mnt/gfs007/data
Brick26: mfs-b01:/mnt/gfs008/data
Brick27: mfs-b01:/mnt/gfs009/data
Brick28: mfs-b02:/mnt/gfs025/data
Brick29: mfs-b02:/mnt/gfs026/data
Brick30: mfs-b02:/mnt/gfs027/data
Brick31: mfs-b03:/mnt/gfs043/data
Brick32: mfs-b03:/mnt/gfs044/data
Brick33: mfs-b03:/mnt/gfs045/data
Brick34: mfs-b04:/mnt/gfs061/data
Brick35: mfs-b04:/mnt/gfs062/data
Brick36: mfs-b04:/mnt/gfs063/data
Brick37: mfs-b01:/mnt/gfs010/data
Brick38: mfs-b01:/mnt/gfs011/data
Brick39: mfs-b01:/mnt/gfs012/data
Brick40: mfs-b02:/mnt/gfs028/data
Brick41: mfs-b02:/mnt/gfs029/data
Brick42: mfs-b02:/mnt/gfs030/data
Brick43: mfs-b03:/mnt/gfs046/data
Brick44: mfs-b03:/mnt/gfs047/data
Brick45: mfs-b03:/mnt/gfs048/data
Brick46: mfs-b04:/mnt/gfs064/data
Brick47: mfs-b04:/mnt/gfs065/data
Brick48: mfs-b04:/mnt/gfs066/data
Brick49: mfs-b01:/mnt/gfs013/data
Brick50: mfs-b01:/mnt/gfs014/data
Brick51: mfs-b01:/mnt/gfs015/data
Brick52: mfs-b02:/mnt/gfs031/data
Brick53: mfs-b02:/mnt/gfs032/data
Brick54: mfs-b02:/mnt/gfs033/data
Brick55: mfs-b03:/mnt/gfs049/data
Brick56: mfs-b03:/mnt/gfs050/data
Brick57: mfs-b03:/mnt/gfs051/data
Brick58: mfs-b04:/mnt/gfs067/data
Brick59: mfs-b04:/mnt/gfs068/data
Brick60: mfs-b04:/mnt/gfs069/data
Brick61: mfs-b01:/mnt/gfs016/data
Brick62: mfs-b01:/mnt/gfs017/data
Brick63: mfs-b01:/mnt/gfs018/data
Brick64: mfs-b02:/mnt/gfs034/data
Brick65: mfs-b02:/mnt/gfs035/data
Brick66: mfs-b02:/mnt/gfs036/data
Brick67: mfs-b03:/mnt/gfs052/data
Brick68: mfs-b03:/mnt/gfs053/data
Brick69: mfs-b03:/mnt/gfs054/data
Brick70: mfs-b04:/mnt/gfs070/data
Brick71: mfs-b04:/mnt/gfs071/data
Brick72: mfs-b04:/mnt/gfs072/data
Options Reconfigured:
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
nfs.disable: on
transport.address-family: inet

> 
>> This module does wonders for speeding listings and downloads of directories 
>> with large numbers of
>> files in the Finder, but it kills uploads dead.  Finder gives an error:
>> 
>> The Finder can’t complete the operation because some data in “[filename]” 
>> can’t be read or
>> written.
>> (Error code -36)
> 
> Can you please check for any errors in brick logs under 
> /var/log/glusterfs/bricks/ while you face
> this Finder error?

There’s stuff in those logs, but nothing is added when the error occurs.

> 
>> The man page for the module indicates that the vfs_streams_xattr module must 
>> also be loaded, like
>> this:
>> 
>> vfs objects = fruit streams_xattr
> 
> Can you please share the output of `testparm -s`?

root@mfs-01 ~]#testparm -s
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[sa1]"
Processing section "[nongluster]"
Processing section "[share1]"
Processing section "[share2]"
Loaded services file OK.
Server role: ROLE_DOMAIN_MEMBER

# Global parameters
[global]
realm = .UALBERTA.CA
workgroup = STS
log file = /var/log/samba/log.%m
max log size = 50
server min protocol = SMB2
map to guest = Bad User
security = ADS
idmap config * : backend = tdb
access based share enum = Yes
force create mode = 0777
force directory mode = 

Re: [Gluster-users] Performance drop from 3.8 to 3.10

[WK]

Maybe next week we can all explore this.

I'm on 3.10.5 and I don't have any complaints. Actually we are quite 
happy with the new clusters,  but these were green field installations 
that were built and then replaced our old 3.4 stuff.


So we are still really enjoying the sharding and arbiter improvements.

I therefore don't have any baseline stats to compare any performance diffs.

I'm curious as to what changed in 3.10 that would account for any change 
in performance from 3.8 and in a similar vein what changes to expect in 
3.12.x as we are thinking about making that jump soon.


-wk



On 9/22/2017 8:07 AM, Lindsay Mathieson wrote:


On 22/09/2017 1:21 PM, Krutika Dhananjay wrote:

Could you disable cluster.eager-lock and try again?



Thanks, but didn't seem to make any difference.


Can't test anymore at the moment as down a server that hung on reboot :(




___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-infra] lists.gluster.org issues this weekend

[Atin Mukherjee]

On Fri, 22 Sep 2017 at 18:54, Ravishankar N  wrote:

> Hello,
> Are our servers still facing the overload issue? My replies to
> gluster-users ML are not getting delivered to the list.
>

Same here. Even this is true for gluster-devel as well.


> Regards,
> Ravi
>
>
> On 09/19/2017 10:03 PM, Michael Scherer wrote:
>
> Le samedi 16 septembre 2017 à 20:48 +0530, Nigel Babu a écrit :
>
> Hello folks,
>
> We have discovered that for the last few weeks our mailman server was
> used
> for a spam attack. The attacker would make use of the + feature
> offered by
> gmail and hotmail. If you send an email to 
> exam...@hotmail.com,example+...@hotmail.com, example+...@hotmail.com, it goes 
> to the same
> inbox. We were constantly hit with requests to subscribe to a few
> inboxes.
> These requests overloaded our mail server so much that it gave up. We
> detected this failure because a postmortem email togluster-in...@gluster.org 
> bounced. Any emails sent to our mailman
> server
> may have been on hold for the last 24 hours or so. They should be
> processed
> now as your email provider re-attempts.
>
> For the moment, we've banned subscribing with an email address with a
> + in
> the name. If you are already subscribed to the lists with a + in your
> email
> address, you will continue to be able to use the lists.
>
> We're looking at banning the spam IP addresses from being able to hit
> the
> web interface at all. When we have a working alternative, we will
> look at
> removing the current ban of using + in address.
>
> So we have a alternative in place, I pushed a blacklist using
> mod_security and a few DNS 
> blacklist:https://github.com/gluster/gluster.org_ansible_configuration/commit/2f4
> c1b8feeae16e1d0b7d6073822a6786ed21dde
>
>
>
>
>
> Apologies for the outage and a big shout out to Michael for taking
> time out
> of his weekend to debug and fix the issue.
>
> Well, you can thanks the airport in Prague for being less interesting
> than a spammer attacking us.
>
>
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttp://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users

-- 
- Atin (atinm)
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Backup and Restore strategy in Gluster FS 3.8.4

[Sunil Aggarwal]

Thanks a lot Ben for your suggestions. I will consider them and see how it
goes.

Regards,
Sunil

On Wed, Sep 20, 2017 at 11:15 PM, Ben Werthmann  wrote:

>
> First, please note that gluster 3.8 is EOL and that 3.8.4 is rather old in
>> the 3.8 release, 3.8.15 is the current (and probably final) release of 3.8.
>>
>> "With the release of GlusterFS-3.12, GlusterFS-3.8 (LTM) and GlusterFS-
>> 3.11 (STM) have reached EOL. Except for serious security issues no
>> further updates to these versions are forthcoming. If you find a bug please
>> see if you can reproduce it with 3.10 or 3.12 and
>> file a BZ if appropriate."
>> http://lists.gluster.org/pipermail/packaging/2017-August/000363.html
>>
>> Gluster 3.12 includes '#1428061 :
>> Halo Replication feature for AFR translator' which was introduced in 3.11.
>> See Halo Replication feature in AFR has been introduced
>>  for a
>> summary. 3.11 is EOL, best to use 3.12 (long term release) if this is of
>> interest to you.
>>
>> On Thu, Aug 24, 2017 at 4:18 AM, Sunil Aggarwal 
>> wrote:
>>
>>> Hi,
>>>
>>> What is the preferred way of taking glusterfs backup?
>>>
>>> I am using Glusterfs 3.8.4.
>>>
>>> We've configured gluster on thick provisioned LV in which 50% of the VG
>>> is kept free for the LVM snapshot.
>>>
>>
>> Gluster Volume Snapshots require each brick to have it's own thin LV.
>> Only thin LVs are supported with volume snapshots.
>>
>> Gluster snapshots do not provide a recovery path in the event of
>> catastrophic loss of a Gluster volume.
>>
>>  -  Gluster Volume Snapshots provide point-in-time recovery for a healthy
>> Gluster Volume. A restore will reset the entire volume to a previous
>> point-in-time recovery point. Granular recovery may be performed with admin
>> intervention.
>>  - User Serviceable Snapshots exposes Gluster Volume Snapshots via the
>> .snaps directory in every directory of the mounted volume. Please review
>> documentation for requirements specific to each protocol used to access the
>> gluster volume.
>>
>>
>>> is it any different then taking snapshot on a thin provisioned LV?
>>>
>>
>> If you don't want Gluster Volume Snapshots, and are just looking to
>> backup the current state of a volume, Gluster offers a number of features
>> to recover from catastrophic loss of a Gluster volume.
>>
>>  - Simple method: mount the gluster volume and run your backup utility of
>> choice
>>  - glusterfind generates a file list (full and incremental) for passing
>> to another utility to generate a backup. See:
>>   http://docs.gluster.org/en/latest/GlusterFS%20Tools/glusterfind/
>>   https://milindchangireblog.wordpress.com/2016/10/28/why-glusterfind/
>>   http://lists.gluster.org/pipermail/gluster-users/2017-August
>> /032219.html
>>  - Geo-replication provides a distributed, continuous, asynchronous, and
>> incremental replication service from one site to another over Local Area
>> Networks (LANs), Wide Area Networks (WANs), and the Internet. (taken from
>> RedHat doc)
>>
>>
>> Please see docs for additional info about each of the above:
>> https://access.redhat.com/documentation/en-us/red_hat_gluste
>> r_storage/3.2/pdf/administration_guide/Red_Hat_Gluster_
>> Storage-3.2-Administration_Guide-en-US.pdf
>>
>>
>>>
>>> --
>>> Thanks,
>>> Sunil Aggarwal
>>> 844-734-5346 <%28844%29%20734-5346>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>


-- 
Thanks,
Sunil Aggarwal
844-734-5346
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] fts_read failed

[Stefan Bergstrand]

Hi,

I have simple installation, using mostly defaults, of two mirrored
servers. One brick, one volume.
GlusterFS version is 3.12.1 (server and client). All hosts involved are
Debian 9.1.

On another host I have mounted two different directories from the
cluster using /etc/fstab:

gfs1,gfs2:/vol1/sites-available/ws0 /etc/nginx/sites-available glusterfs
defaults,_netdev 0 0

and

gfs1,gfs2:/vol1/webroots/ws0 /var/www glusterfs defaults,_netdev 0 0


I cd to /var/www/example.com/public_html/www.example.com/ and run:

chown -R auser:agroup .

and get:

chown: fts_read failed: No such file or directory


If I try the same on one of the glusterfs servers, it works fine as
expected.


However, if I run:

chown auser:agroup *
chown auser:agroup */*
chown auser:agroup */*/*
[...]

all the way to the bottom of the directory tree, it all works fine.


Also, and this is really weird: I just discovered that if I shut down
nginx, the "chown -R" works...
According to "lsof| grep /var/www" no files under that mount point are
opened.
I unmounted /etc/nginx/sites-available, and it made no difference.

How could nginx possible make chown fail like this? And only when using
"-R".

(Of course, it works on a similar host/setup apart from all files
residing on local filesystem...)

Best regards,

/Stefan Bergstrand
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Heal Info Shows Split Brain, but "file not in split brain" when attempted heal

[Brett Kelly]

Hello

I am using Glusterfs 3.10.5 on CentOS7. A replicated distributed volume 
with a dist-rep hot tier.


During data migration, we noticed the tierd.log on one of nodes was 
huge. Upon review it seemed to be stuck on a certain set of files. 
Running a "gluster vol heal VOL info" showed that those same files 
caused problems in the tier, were in split brain.


So we went to fix split brain, we used "source-brick" and chose each of 
the bricks as the master to heal from.


   gluster vol heal VOL split-brain source-brick HOSTNAME:/brick/dir

However when we ran the heal command above, It spit out that each file 
was not in split brain and that it healed 0.


We then double checked the status of the files listed in heal info 
output with :


   getfattr -n replica.split-brain-status 

And it reported that the files were not in split brain

Any insight would be greatly appreciated.

Brett


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] BUG: After stop and start wrong port is advertised

[Diego Remolina]

I've noticed this as well on the official 3.8.4 gluster packages from Red Hat

# gluster v status
Status of volume: aevmstorage
Gluster process TCP Port  RDMA Port  Online  Pid
--
Brick ae-vmstore01-rbe:/bricks/brick1/brick 49153 0  Y   2659
Brick ae-vmstore02-rbe:/bricks/brick1/brick 49152 0  Y   2651
Brick ae-vmstore03-rbe:/bricks/brick1/brick 49153 0  Y   72876
NFS Server on localhost 2049  0  Y   3389
Self-heal Daemon on localhost   N/A   N/AY   3398
NFS Server on ae-vmstore02-rbe  2049  0  Y   2675
Self-heal Daemon on ae-vmstore02-rbeN/A   N/AY   2848
NFS Server on ae-vmstore03-rbe  2049  0  Y   156988
Self-heal Daemon on ae-vmstore03-rbeN/A   N/AY   156996

Task Status of Volume aevmstorage
--
There are no active volume tasks

Deigo


On Fri, Sep 22, 2017 at 11:28 AM, Jo Goossens
 wrote:
> Hi Darrell,
>
>
>
>
>
> Thanks, for us it's really easy to reproduce atm. Each restart or stop/start
> is causing the issue atm over here.
>
>
>
> Atin will look into it on monday fortunately :)
>
>
>
> Regards
>
> Jo
>
>
>
>
>
>
>
>
> -Original message-
> From: Darrell Budic 
> Sent: Fri 22-09-2017 17:24
> Subject: Re: [Gluster-users] BUG: After stop and start wrong port is
> advertised
> To: Atin Mukherjee ;
> CC: Jo Goossens ; gluster-users@gluster.org;
> I encountered this once in the past, an additional symptom was peers were in
> disconnected state on the peers that were NOT using the wrong ports.
> Disconnected peers is how it detected it in the first place.
>
> It happened to me after rebooting, and I fixed it but wasn’t able to stop
> and gather debugging info on the time.
>
> The problem seemed to be that the volume files in
> /var/lib/glusterd/vols///bricks/\:-v0- name>-brick0 were not updated to reflect a new port # after the restart (and
> the port numbers had changed to adding and deleting volumes since last
> start). I stopped glusterd, killed any remaining glusterfsd’s, hand edited
> the files to reflect the new ports they thought they were running the bricks
> on (from vol info I think, maybe log files) and restarted glusterd, then
> everything was happy again.
>
> Hope it helps, sounds like it may be a bug to me too if others are seeing
> it.
>
>  -Darrell
>
>
>> On Sep 22, 2017, at 8:10 AM, Atin Mukherjee  wrote:
>>
>> I've already replied to your earlier email. In case you've not seen it in
>> your mailbox here it goes:
>>
>> This looks like a bug to me. For some reason glusterd's portmap is
>> referring to a stale port (IMO) where as brick is still listening to the
>> correct port. But ideally when glusterd service is restarted, all the
>> portmap in-memory is rebuilt. I'd request for the following details from you
>> to let us start analysing it:
>>
>> 1. glusterd statedump output from 192.168.140.43 . You can use kill
>> -SIGUSR2  to request for a statedump and the file will be
>> available in /var/run/gluster
>> 2. glusterd, brick logfile for 192.168.140.43:/gluster/public from
>> 192.168.140.43
>> 3. cmd_history logfile from all the nodes.
>> 4. Content of /var/lib/glusterd/vols/public/
>>
>>
>> On Thu, Sep 21, 2017 at 2:08 PM, Jo Goossens
>>  wrote:
>> Hi,
>>
>>
>>
>> We use glusterfs 3.10.5 on Debian 9.
>>
>>
>> When we stop or restart the service, e.g.: service glusterfs-server
>> restart
>>
>>
>> We see that the wrong port get's advertised afterwards. For example:
>>
>>
>> Before restart:
>>
>>
>> Status of volume: public
>> Gluster process TCP Port  RDMA Port  Online
>> Pid
>>
>> --
>> Brick 192.168.140.41:/gluster/public49153 0  Y
>> 6364
>> Brick 192.168.140.42:/gluster/public49152 0  Y
>> 1483
>> Brick 192.168.140.43:/gluster/public49152 0  Y
>> 5913
>> Self-heal Daemon on localhost   N/A   N/AY
>> 5932
>> Self-heal Daemon on 192.168.140.42  N/A   N/AY
>> 13084
>> Self-heal Daemon on 192.168.140.41  N/A   N/AY
>> 15499
>>
>> Task Status of Volume public
>>
>> --
>> There are no active volume tasks
>>
>>
>> After restart of the service on one of the nodes (192.168.140.43) the port
>> seems to have changed (but it didn't):
>>
>> root@app3:/var/log/glusterfs#  gluster volume status
>> Status of volume: public
>> Gluster process   

Re: [Gluster-users] BUG: After stop and start wrong port is advertised

[Darrell Budic]

I encountered this once in the past, an additional symptom was peers were in 
disconnected state on the peers that were NOT using the wrong ports. 
Disconnected peers is how it detected it in the first place.

It happened to me after rebooting, and I fixed it but wasn’t able to stop and 
gather debugging info on the time.

The problem seemed to be that the volume files in 
/var/lib/glusterd/vols///bricks/\:-v0--brick0 
were not updated to reflect a new port # after the restart (and the port 
numbers had changed to adding and deleting volumes since last start). I stopped 
glusterd, killed any remaining glusterfsd’s, hand edited the files to reflect 
the new ports they thought they were running the bricks on (from vol info I 
think, maybe log files) and restarted glusterd, then everything was happy again.

Hope it helps, sounds like it may be a bug to me too if others are seeing it.

  -Darrell


> On Sep 22, 2017, at 8:10 AM, Atin Mukherjee  wrote:
> 
> I've already replied to your earlier email. In case you've not seen it in 
> your mailbox here it goes:
> 
> This looks like a bug to me. For some reason glusterd's portmap is referring 
> to a stale port (IMO) where as brick is still listening to the correct port. 
> But ideally when glusterd service is restarted, all the portmap in-memory is 
> rebuilt. I'd request for the following details from you to let us start 
> analysing it:
> 
> 1. glusterd statedump output from 192.168.140.43 . You can use kill -SIGUSR2 
>  to request for a statedump and the file will be available 
> in /var/run/gluster
> 2. glusterd, brick logfile for 192.168.140.43:/gluster/public from 
> 192.168.140.43
> 3. cmd_history logfile from all the nodes.
> 4. Content of /var/lib/glusterd/vols/public/
> 
> 
> On Thu, Sep 21, 2017 at 2:08 PM, Jo Goossens  
> wrote:
> Hi,
> 
>  
>  
> We use glusterfs 3.10.5 on Debian 9.
> 
>  
> When we stop or restart the service, e.g.: service glusterfs-server restart
> 
>  
> We see that the wrong port get's advertised afterwards. For example:
> 
>  
> Before restart:
> 
>  
> Status of volume: public
> Gluster process TCP Port  RDMA Port  Online  Pid
> --
> Brick 192.168.140.41:/gluster/public49153 0  Y   6364
> Brick 192.168.140.42:/gluster/public49152 0  Y   1483
> Brick 192.168.140.43:/gluster/public49152 0  Y   5913
> Self-heal Daemon on localhost   N/A   N/AY   5932
> Self-heal Daemon on 192.168.140.42  N/A   N/AY   13084
> Self-heal Daemon on 192.168.140.41  N/A   N/AY   15499
>  
> Task Status of Volume public
> --
> There are no active volume tasks
>  
>  
> After restart of the service on one of the nodes (192.168.140.43) the port 
> seems to have changed (but it didn't):
>  
> root@app3:/var/log/glusterfs#  gluster volume status
> Status of volume: public
> Gluster process TCP Port  RDMA Port  Online  Pid
> --
> Brick 192.168.140.41:/gluster/public49153 0  Y   6364
> Brick 192.168.140.42:/gluster/public49152 0  Y   1483
> Brick 192.168.140.43:/gluster/public49154 0  Y   5913
> Self-heal Daemon on localhost   N/A   N/AY   4628
> Self-heal Daemon on 192.168.140.42  N/A   N/AY   3077
> Self-heal Daemon on 192.168.140.41  N/A   N/AY   28777
>  
> Task Status of Volume public
> --
> There are no active volume tasks
>  
>  
> However the active process is STILL the same pid AND still listening on the 
> old port
>  
> root@192.168.140.43:/var/log/glusterfs# netstat -tapn | grep gluster
> tcp0  0 0.0.0.0:49152   0.0.0.0:*   LISTEN
>   5913/glusterfsd
>  
>  
> The other nodes logs fill up with errors because they can't reach the daemon 
> anymore. They try to reach it on the "new" port instead of the old one:
>  
> [2017-09-21 08:33:25.225006] E [socket.c:2327:socket_connect_finish] 
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection 
> refused); disconnecting socket
> [2017-09-21 08:33:29.226633] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:29.227490] E [socket.c:2327:socket_connect_finish] 
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection 
> refused); disconnecting socket
> [2017-09-21 08:33:33.225849] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 
> 0-public-client-2: changing port to 49154 (from 0)
> 

Re: [Gluster-users] BUG: After stop and start wrong port is advertised

[Jo Goossens]

Hi Darrell,

 
 
Thanks, for us it's really easy to reproduce atm. Each restart or stop/start is 
causing the issue atm over here.

 
Atin will look into it on monday fortunately :)



Regards

Jo

 
 

 
-Original message-
From:Darrell Budic 
Sent:Fri 22-09-2017 17:24
Subject:Re: [Gluster-users] BUG: After stop and start wrong port is advertised
To:Atin Mukherjee ; 
CC:Jo Goossens ; gluster-users@gluster.org; 
I encountered this once in the past, an additional symptom was peers were in 
disconnected state on the peers that were NOT using the wrong ports. 
Disconnected peers is how it detected it in the first place.

It happened to me after rebooting, and I fixed it but wasn’t able to stop and 
gather debugging info on the time.

The problem seemed to be that the volume files in 
/var/lib/glusterd/vols///bricks/\:-v0--brick0 
were not updated to reflect a new port # after the restart (and the port 
numbers had changed to adding and deleting volumes since last start). I stopped 
glusterd, killed any remaining glusterfsd’s, hand edited the files to reflect 
the new ports they thought they were running the bricks on (from vol info I 
think, maybe log files) and restarted glusterd, then everything was happy again.

Hope it helps, sounds like it may be a bug to me too if others are seeing it.

  -Darrell


> On Sep 22, 2017, at 8:10 AM, Atin Mukherjee  wrote:
> 
> I've already replied to your earlier email. In case you've not seen it in 
> your mailbox here it goes:
> 
> This looks like a bug to me. For some reason glusterd's portmap is referring 
> to a stale port (IMO) where as brick is still listening to the correct port. 
> But ideally when glusterd service is restarted, all the portmap in-memory is 
> rebuilt. I'd request for the following details from you to let us start 
> analysing it:
> 
> 1. glusterd statedump output from 192.168.140.43 . You can use kill -SIGUSR2 
>  to request for a statedump and the file will be available 
> in /var/run/gluster
> 2. glusterd, brick logfile for 192.168.140.43:/gluster/public from 
> 192.168.140.43
> 3. cmd_history logfile from all the nodes.
> 4. Content of /var/lib/glusterd/vols/public/
> 
> 
> On Thu, Sep 21, 2017 at 2:08 PM, Jo Goossens  
> wrote:
> Hi,
> 
>  
>  
> We use glusterfs 3.10.5 on Debian 9.
> 
>  
> When we stop or restart the service, e.g.: service glusterfs-server restart
> 
>  
> We see that the wrong port get's advertised afterwards. For example:
> 
>  
> Before restart:
> 
>  
> Status of volume: public
> Gluster process                             TCP Port  RDMA Port  Online  Pid
> --
> Brick 192.168.140.41:/gluster/public        49153     0          Y       6364
> Brick 192.168.140.42:/gluster/public        49152     0          Y       1483
> Brick 192.168.140.43:/gluster/public        49152     0          Y       5913
> Self-heal Daemon on localhost               N/A       N/A        Y       5932
> Self-heal Daemon on 192.168.140.42          N/A       N/A        Y       13084
> Self-heal Daemon on 192.168.140.41          N/A       N/A        Y       15499
>  
> Task Status of Volume public
> --
> There are no active volume tasks
>  
>  
> After restart of the service on one of the nodes (192.168.140.43) the port 
> seems to have changed (but it didn't):
>  
> root@app3:/var/log/glusterfs#  gluster volume status
> Status of volume: public
> Gluster process                             TCP Port  RDMA Port  Online  Pid
> --
> Brick 192.168.140.41:/gluster/public        49153     0          Y       6364
> Brick 192.168.140.42:/gluster/public        49152     0          Y       1483
> Brick 192.168.140.43:/gluster/public        49154     0          Y       5913
> Self-heal Daemon on localhost               N/A       N/A        Y       4628
> Self-heal Daemon on 192.168.140.42          N/A       N/A        Y       3077
> Self-heal Daemon on 192.168.140.41          N/A       N/A        Y       28777
>  
> Task Status of Volume public
> --
> There are no active volume tasks
>  
>  
> However the active process is STILL the same pid AND still listening on the 
> old port
>  
> root@192.168.140.43:/var/log/glusterfs# netstat -tapn | grep gluster
> tcp        0      0 0.0.0.0:49152           0.0.0.0:*               LISTEN    
>   5913/glusterfsd
>  
>  
> The other nodes logs fill up with errors because they can't reach the daemon 
> anymore. They try to reach it on the "new" port instead of the old one:
>  
> [2017-09-21 08:33:25.225006] E [socket.c:2327:socket_connect_finish] 
> 0-public-client-2: connection to 

Re: [Gluster-users] Performance drop from 3.8 to 3.10

[Lindsay Mathieson]

On 22/09/2017 1:21 PM, Krutika Dhananjay wrote:

Could you disable cluster.eager-lock and try again?



Thanks, but didn't seem to make any difference.


Can't test anymore at the moment as down a server that hung on reboot :(


--
Lindsay Mathieson

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] sparse files on EC volume

[Dmitri Chebotarov]

Hello

I'm running some tests to compare performance between Gluster FUSE mount
and formated sparse files (located on the same Gluster FUSE mount).

The Gluster volume is EC (same for both tests).

I'm seeing HUGE difference and trying to figure out why.

Here is an example:

GlusterFUSE mount:

# cd /mnt/glusterfs
# rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s*

Sparse file (located on GlusterFUSE mount):

# truncate -l 100GB /mnt/glusterfs/xfs-100G.img
# mkfs.xfs /mnt/glusterfs/xfs-100G.img
# mount -o loop /mnt/glusterfs/xfs-100G.img /mnt/xfs-100G
# cd /mnt/xfs-100G
# rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s*

The same goes for working with small files (i.e. code file, make, etc) with
the same data located on FUSE mount vs formated sparse file on the same
FUSE mount.

What would explain such difference?

How does Gluster work with sparse files in general? I may move some of the
data on gluster volumes to formated sparse files..

Thank you.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] AFR: Fail lookups when quorum not met

[Niels de Vos]

On Fri, Sep 22, 2017 at 12:27:46PM +0530, Ravishankar N wrote:
> Hello,
> 
> In AFR we currently allow look-ups to pass through without taking into
> account whether the lookup is served from the good or bad brick. We always
> serve from the good brick whenever possible, but if there is none, we just
> serve the lookup from one of the bricks that we got a positive reply from.
> 
> We found a bug  [1] due to this behavior were the iatt values returned in
> the lookup call was bad and caused the client to hang. The proposed fix [2]
> was to fail look ups when we definitely know the lookup can't be trusted (by
> virtue of AFR xattrs indicating the replies we got from the up bricks are
> indeed bad).
> 
> Note that this fix is *only* for replica 3 or arbiter volumes (not replica
> 2, where there is no notion of quorum). But we want to 'harden' the fix by 
> not allowing any look ups at all if quorum is not met (or) it is met but
> there are no good copies.
> 
> Some implications of this:
> 
> -If a file ends up in data/meta data split-brain in replica 3/arbiter (rare
> occurrence), we won't be able to delete it from the mount.
> 
> -Even if the only brick that is up is the good copy, we still fail it due to
> lack of quorum.
> 
> Does any one have comments/ feedback?

I think additional improvements for correctness outweigh the two
negative side-effects that you listed.

Possibly the 2nd point could get some confusion from users. "it always
worked before" may be a reason to add a volume option for this? That is
something you can consider, but if you deem that overkill then I'm ok
with that too.

Thanks,
Niels


> 
> Thanks,
> 
> Ravi
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1467250
> 
> [2] https://review.gluster.org/#/c/17673/ (See review comments on the
> landing page if interested)
> 

> ___
> Gluster-devel mailing list
> gluster-de...@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel



signature.asc
Description: PGP signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade Gluster 3.7 to 3.12 and add 3rd replica [howto/help]

[Martin Toth]

Hi, 

thanks for suggesions. Yes "gluster peer probe node3” will be first command in 
order to discover 3rd node by Gluster.
I am running on latest 3.7.x - there is 3.7.6-1ubuntu1 installed and latest 
3.7.x according https://packages.ubuntu.com/xenial/glusterfs-server 
 is 3.7.6-1ubuntu1, so 
this should be OK.

> If you are *not* on the latest 3.7.x, you are unlikely to be able to go

Do you mean latest package from Ubuntu repository or latest package from 
Gluster PPA (3.7.20-ubuntu1~xenial1). 
Currently I am using Ubuntu repository package, but want to use PPA for upgrade 
because Ubuntu has old packages of Gluster in repo.

I do not use sharding because all bricks has same size, so it will not speedup 
healing of VMs images in case of heal operation. Volume is 3TB, how long does 
it take to heal on 2x1gbit (linux bond) connection, can you approximate ? 
I want to turn every VM off because its required for upgrading gluster 
procedure, thats why I want to add 3rd brick (3rd replica) at this time (after 
upgrade when VMs will be offline).

Martin

> On 22 Sep 2017, at 12:20, Diego Remolina  wrote:
> 
> Procedure looks good.
> 
> Remember to back up Gluster config files before update:
> 
> /etc/glusterfs
> /var/lib/glusterd
> 
> If you are *not* on the latest 3.7.x, you are unlikely to be able to go back 
> to it because PPA only keeps the latest version of each major branch, so keep 
> that in mind. With Ubuntu, every time you update, make sure to download and 
> keep a manual copy of the .Deb files. Otherwise you will have to compile the 
> packages yourself in the event you wanted to go back.
> 
> Might need before adding 3rd replica:
> gluster peer probe node3 
> 
> When you add the 3rd replica, it should start healing, and there may be an 
> issue there if the VMs are running. Your plan to not have VMs up is good 
> here. Are you using sharding? If you are not sharding, I/O in running VMs may 
> be stopped for too long while a large image is healed. If you were already 
> using sharding you should be able to add the 3rd replica when VMs are running 
> without much issue.
> 
> Once healing is completed and if you are satisfied with 3.12, then remember 
> to bump op version of Gluster.
> 
> Diego
> 
> 
> On Sep 20, 2017 19:32, "Martin Toth"  > wrote:
> Hello all fellow GlusterFriends,
> 
> I would like you to comment / correct my upgrade procedure steps on replica 2 
> volume of 3.7.x gluster.
> Than I would like to change replica 2 to replica 3 in order to correct quorum 
> issue that Infrastructure currently has.
> 
> Infrastructure setup:
> - all clients running on same nodes as servers (FUSE mounts)
> - under gluster there is ZFS pool running as raidz2 with SSD ZLOG/ZIL cache
> - all two hypervisor running as GlusterFS nodes and also Qemu compute nodes 
> (Ubuntu 16.04 LTS)
> - we are running Qemu VMs that accesses VMs disks via gfapi (Opennebula)
> - we currently run : 1x2 , Type: Replicate volume
> 
> Current Versions :
> glusterfs-* [package] 3.7.6-1ubuntu1
> qemu-*[package] 2.5+dfsg-5ubuntu10.2glusterfs3.7.14xenial1
> 
> What we need : (New versions)
> - upgrade GlusterFS to 3.12 LTM version (Ubuntu 16.06 LTS packages are EOL - 
> see https://www.gluster.org/community/release-schedule/ 
> )
>   - I want to use 
> https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12 
>  as package 
> repository for 3.12
> - upgrade Qemu (with build-in support for libgfapi) - 
> https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.12 
> 
>   - (sadly Ubuntu has packages build without libgfapi support)
> - add third node to replica setup of volume (this is probably most dangerous 
> operation)
> 
> Backup Phase
> - backup "NFS storage” - raw DATA that runs on VMs
> - stop all running VMs
> - backup all running VMs (Qcow2 images) outside of gluster
> 
> Upgrading Gluster Phase
> - killall glusterfs glusterfsd glusterd (on every server)
>   (this should stop all gluster services - server and client as it runs 
> on same nodes)
> - install new Gluster Server and Client packages from repository mentioned 
> upper (on every server) 
> - install new Monotek's qemu glusterfs package with gfapi enabled support (on 
> every server) 
> - /etc/init.d/glusterfs-server start (on every server)
> - /etc/init.d/glusterfs-server status - verify that all runs ok (on every 
> server)
>   - check :
>   - gluster volume info
>   - gluster volume status
>   - check gluster FUSE clients, if mounts working as expected
> - test if various VMs are able tu boot and run as expected (if libgfapi works 
> in Qemu)
> - reboot all nodes - do system 

Re: [Gluster-users] Upgrade Gluster 3.7 to 3.12 and add 3rd replica [howto/help]

[Diego Remolina]

Hi Martin,

> Do you mean latest package from Ubuntu repository or latest package from
> Gluster PPA (3.7.20-ubuntu1~xenial1).
> Currently I am using Ubuntu repository package, but want to use PPA for
> upgrade because Ubuntu has old packages of Gluster in repo.

When you switch to PPA, make sure to download and keep a copy of each
set of gluster deb packages, otherwise if you ever want to back out an
upgrade to an older release, you will have to download the source deb
file and build it yourself, because PPAs only keep the latest version
for binaries.

>
> I do not use sharding because all bricks has same size, so it will not
> speedup healing of VMs images in case of heal operation. Volume is 3TB, how
> long does it take to heal on 2x1gbit (linux bond) connection, can you
> approximate ?

Sharding is not so much about brick size. Sharding is about preventing
a whole large VM file being locked when it is being healed. Also
minimizes the amount of data copied because gluster only heals smaller
pieces versus a whole VM image.

Say your 100GB IMG needs to be healed, the file is locked while it
gets copied from one server to the other and the running VM may not be
able to use it while the heal is going, so your VM may in fact stop
working or have I/O errors. With sharding, VMs are cut into, well,
shards, largest shard is 512MB, then the heal process only locks the
shards being healed. So gluster only heals the shards that changed
which are much smaller and faster to copy, and do not need to lock the
whole 100GB IMG file which takes longer to copy, just the shard being
healed. Do note that if you had never used sharding, if you turn it on
it will *not* convert your older files. Also you should *never* turn
on sharding and then back off, as that will result in corrupted VM
image files. Once it is on, if you want to turn it off, stop your VMs,
then move all VM IMG files elsewhere, turn off sharding and then copy
the files back  to the volume after disabling sharding.

As for speed, I really cannot tell you as it depends on the disks,
netowr, etc. For example, I have a two node setup plus an arbiter (2
nodes with bricks, one is just the arbiter to keep quorum if one of
the brick servers goes down). I recently replaced the HDDs in one
machine as the drives hit the 5 year age mark. So I took the 12 drives
out, added 24 drives to the machine (we had unused slots),
reconfigured raid 6 and left it initializing in the background and
started the heal of 13.1TB of data. My servers are connected via
10Gbit (I am not seeing reads/writes over 112MB/s) and this process
started last Monday at 7;20PM and it is not done yet. It is missing
healing about 40GB still. Now my servers are used as a file server,
which means lots of small files which take longer to heal. I would
think your VM images will heal much faster.

> I want to turn every VM off because its required for upgrading gluster
> procedure, thats why I want to add 3rd brick (3rd replica) at this time
> (after upgrade when VMs will be offline).
>

You could even attempt an online upgrade if you try to add the new
node/brick running 3.12 to the mix before upgrading from 3.7.x on the
other nodes. However, I am not sure how that is going to work. With
such a difference in versions, it may not work well.

If you can afford the downtime to upgrade, that will be the safest option.

Diego
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] how to verify bitrot signed file manually?

[Amudhan P]

ok, from bitrot code I figured out gluster using sha256 hashing algo.


Now coming to the problem, during scrub run in my cluster some of my files
were marked as bad in few set of nodes.
I just wanted to confirm bad file. so, I have used "sha256sum" tool in
Linux to manually get file hash.

here is the result.

file-1, file-2 marked as bad by scrub and file-3 is healthy.

file-1 sha256 and bitrot signature value matches but still it's been marked
as bad.

file-2 sha256 and bitrot signature value don't match, could be a victim of
bitrot or bitflip.file is still readable without any issue and no errors
found in the drive.

file-3 sha256 and bitrot signature matches and healthy.


file-1 output from

"sha256sum" =
"71eada9352b1352aaef0f806d3d561768ce2df905ded1668f665e06eca2d0bd4"


"getfattr -m. -e hex -d "
# file: file-1
trusted.bit-rot.bad-file=0x3100
trusted.bit-rot.signature=0x01020071eada9352b1352aaef0f806d3d561768ce2df905ded1668f665e06eca2d0bd4
trusted.bit-rot.version=0x020058e4f3b40006793d
trusted.ec.config=0x080a02000200
trusted.ec.dirty=0x
trusted.ec.size=0x000718996701
trusted.ec.version=0x00038c4c00038c4d
trusted.gfid=0xf078a24134fe4f9bb953eca8c28dea9a

output scrub log:
[2017-09-02 13:02:20.311160] A [MSGID: 118023]
[bit-rot-scrub.c:244:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0: CORRUPTION
DETECTED: Object /file-1 {Brick: /media/disk16/brick16 | GFID:
f078a241-34fe-4f9b-b953-eca8c28dea9a}
[2017-09-02 13:02:20.311579] A [MSGID: 118024]
[bit-rot-scrub.c:264:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0: Marking
/file-1 [GFID: f078a241-34fe-4f9b-b953-eca8c28dea9a | Brick:
/media/disk16/brick16] as corrupted..

file-2 output from

"sha256sum" =
"c41ef9c81faed4f3e6010ea67984c3cfefd842f98ee342939151f9250972dcda"


"getfattr -m. -e hex -d "
# file: file-2
trusted.bit-rot.bad-file=0x3100
trusted.bit-rot.signature=0x0102009162cb17d4f0bee676fcb7830c5286d05b8e8940d14f3d117cb90b7b1defc129
trusted.bit-rot.version=0x020058e4f3b400019bb2
trusted.ec.config=0x080a02000200
trusted.ec.dirty=0x
trusted.ec.size=0x403433f6
trusted.ec.version=0x201a201b
trusted.gfid=0xa50012b0a632477c99232313928d239a

output scrub log:
[2017-09-02 05:18:14.003156] A [MSGID: 118023]
[bit-rot-scrub.c:244:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0: CORRUPTION
DETECTED: Object /file-2 {Brick: /media/disk13/brick13 | GFID:
a50012b0-a632-477c-9923-2313928d239a}
[2017-09-02 05:18:14.006629] A [MSGID: 118024]
[bit-rot-scrub.c:264:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0: Marking
/file-2 [GFID: a50012b0-a632-477c-9923-2313928d239a | Brick:
/media/disk13/brick13] as corrupted..


file-3 output from

"sha256sum" =
"a590735b3c8936cc7ca9835128a19c38a3f79c8fd53fddc031a9349b7e273f27"


"getfattr -m. -e hex -d "
# file: file-3
trusted.bit-rot.signature=0x010200a590735b3c8936cc7ca9835128a19c38a3f79c8fd53fddc031a9349b7e273f27
trusted.bit-rot.version=0x020058e4f3b400019bb2
trusted.ec.config=0x080a02000200
trusted.ec.dirty=0x
trusted.ec.size=0x3530fc96
trusted.ec.version=0x1a981a99
trusted.gfid=0x10d8920e42cd42cf9448b8bf3941c192



most of the bitrot bad files are in the set of new nodes and data were
uploaded using gluster 3.10.1. no drive issues are any kind of error msgs
in logs.

what could be gone wrong?

regards
Amudhan

On Thu, Sep 21, 2017 at 1:23 PM, Amudhan P  wrote:

> Hi,
>
> I have a file in my brick which was signed by bitrot and latter when
> running scrub it was marked as bad.
>
> Now, I want to verify file again manually. just to clarify my doubt
>
> how can I do this?
>
>
> regards
> Amudhan
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrade Gluster 3.7 to 3.12 and add 3rd replica [howto/help]

[Diego Remolina]

Procedure looks good.

Remember to back up Gluster config files before update:

/etc/glusterfs
/var/lib/glusterd

If you are *not* on the latest 3.7.x, you are unlikely to be able to go
back to it because PPA only keeps the latest version of each major branch,
so keep that in mind. With Ubuntu, every time you update, make sure to
download and keep a manual copy of the .Deb files. Otherwise you will have
to compile the packages yourself in the event you wanted to go back.

Might need before adding 3rd replica:
gluster peer probe node3

When you add the 3rd replica, it should start healing, and there may be an
issue there if the VMs are running. Your plan to not have VMs up is good
here. Are you using sharding? If you are not sharding, I/O in running VMs
may be stopped for too long while a large image is healed. If you were
already using sharding you should be able to add the 3rd replica when VMs
are running without much issue.

Once healing is completed and if you are satisfied with 3.12, then remember
to bump op version of Gluster.

Diego


On Sep 20, 2017 19:32, "Martin Toth"  wrote:

Hello all fellow GlusterFriends,

I would like you to comment / correct my upgrade procedure steps on replica
2 volume of 3.7.x gluster.
Than I would like to change replica 2 to replica 3 in order to correct
quorum issue that Infrastructure currently has.

*Infrastructure setup:*
- all clients running on same nodes as servers (FUSE mounts)
- under gluster there is ZFS pool running as raidz2 with SSD ZLOG/ZIL cache
- all two hypervisor running as GlusterFS nodes and also Qemu compute nodes
(Ubuntu 16.04 LTS)
- we are running Qemu VMs that accesses VMs disks via gfapi (Opennebula)
- we currently run : 1x2 , Type: Replicate volume

*Current Versions :*
glusterfs-* [package] 3.7.6-1ubuntu1
qemu-* [package] 2.5+dfsg-5ubuntu10.2glusterfs3.7.14xenial1

*What we need : (New versions)*
- upgrade GlusterFS to 3.12 LTM version (Ubuntu 16.06 LTS packages are EOL
- see https://www.gluster.org/community/release-schedule/)
- I want to use https://launchpad.net/~gluster/+archive/ubuntu/
glusterfs-3.12 as package repository for 3.12
- upgrade Qemu (with build-in support for libgfapi) -
https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.12
- (sadly Ubuntu has packages build without libgfapi support)
- add third node to replica setup of volume (this is probably most
dangerous operation)

*Backup Phase*
- backup "NFS storage” - raw DATA that runs on VMs
- stop all running VMs
- backup all running VMs (Qcow2 images) outside of gluster

*Upgrading Gluster Phase*
- killall glusterfs glusterfsd glusterd (on every server)
(this should stop all gluster services - server and client as it runs on
same nodes)
- install new Gluster Server and Client packages from repository mentioned
upper (on every server)
- install new Monotek's qemu glusterfs package with gfapi enabled support
(on every server)
- /etc/init.d/glusterfs-server start (on every server)
- /etc/init.d/glusterfs-server status - verify that all runs ok (on every
server)
- check :
- gluster volume info
- gluster volume status
- check gluster FUSE clients, if mounts working as expected
- test if various VMs are able tu boot and run as expected (if libgfapi
works in Qemu)
- reboot all nodes - do system upgrade of packages
- test and check again

*Adding third node to replica 2 setup (replica 2 => replica 3)*
(volumes will be mounted and up after upgrade and we tested VMs are able to
be served with libgfapi = upgrade of gluster sucessfuly completed)
(next we extend replica 2 to replica 3 while volumes are mounted but no
data is touched = no running VMs, only glusterfs servers and clients on
nodes)
- issue command : gluster volume add-brick volume replica 3
node3.san:/tank/gluster/brick1 (on new single node - node3)
so we change :
Bricks:
Brick1: node1.san:/tank/gluster/brick1
Brick2: node2.san:/tank/gluster/brick1
to :
Bricks:
Brick1: node1.san:/tank/gluster/brick1
Brick2: node2.san:/tank/gluster/brick1
Brick3: node3.san:/tank/gluster/brick1
- check gluster status
- (is rebalance / heal required here ?)
- start all VMs and start celebration :)

*My Questions*
- is heal and rebalance necessary in order to upgrade replica 2 to replica
3 ?
- is this upgrade procedure OK ? What more/else should I do in order to do
this upgrade correctly ?

Many thanks to all for support. Hope my little preparation howto will help
others to solve same situation.

Best Regards,
Martin

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Restrict root clients / experimental patch

[Pierre C]

On Fri, Sep 22, 2017 at 10:37 AM, Soumya Koduri  wrote:

> Hi,
>
> On 09/21/2017 07:32 PM, Pierre C wrote:
>
>> Hi All,
>>
>> I would like to use glusterfs in an environment where storage servers are
>> managed by an IT service - myself :) - and several users in the
>> organization can mount the distributed fs. The users are root on their
>> machines.
>> As far as I know about glusterfs, a root client user may impersonate any
>> uid/gid since it provides its uid/gid itself when it talks to the bricks
>> (like nfsv3).
>> The thing is, we want to enforce permissions, i.e. user X may only access
>> files shared with him even if he's root on his machine.
>> I found a draft spec about glusterfs+kerberos <
>> https://github.com/gluster/glusterfs-specs/blob/master/unde
>> r_review/Kerberos.md> but not much more so I think it's not possible
>> with glusterfs right now, correct?
>> (Also I feel that kerberos would be a bit heavy to manage)
>>
>
> I am not much familiar with how ssl is handled. But from what I understand
> from your statement above, you need to restrict root users. Isn't
> root-squash option enough for it?
>
> Option: server.root-squash
> Default Value: off
> Description: Map requests from uid/gid 0 to the anonymous uid/gid. Note
> that this does not apply to any other uids or gids that might be equally
> sensitive, such as user bin or group staff.
>
> Option: server.anonuid
> Default Value: 65534
> Description: value of the uid used for the anonymous user/nfsnobody when
> root-squash is enabled.
>
> Option: server.anongid
> Default Value: 65534
> Description: value of the gid used for the anonymous user/nfsnobody when
> root-squash is enabled.
>
>
Hi Soumya,

This is not sufficient because the glusterfs protocol lets the client
choose its uid/gid/groups when the client connects to a server. (done there
https://github.com/gluster/glusterfs/blob/master/rpc/rpc-lib/src/auth-glusterfs.c#L177
)
So the current implementation relies on the client being trustworthy. A
slightly modified client can pretend to be any user.

Do you see what I mean?

Cheers,
Pierre
eshard



>
> Thanks,
> Soumya
>
>
>> ---
>>
>> An simple hack that I found is to add custom uid/gid fields in clients'
>> ssl certificates. The bricks use the client's uid/gid specified in its
>> certificate rather than using one specified by the user. This solution has
>> no effect on performance and there's no need for a central authentication.
>> The thing that changes is the way client certificates are generated and
>> glusterfsd needs a small patch.
>>
>> I did an experimental implementation > sterfs/commit/768bf63154fdc59ba67d5788743adab8679ec5ab> of this idea.
>> Custom fields "1.2.3.4.5.6.7" and "1.2.3.4.5.6.8" are used for uid and gid.
>> I tried it with custom CA trusted by all bricks and I issued a few client
>> certificates.
>> No server configuration is needed when a new client is added, when a
>> client is revoked the a CRL > /Certificate_revocation_list> must updated and pushed to all servers.
>> By the way I didn't get glusterfs servers to accept my CRLs, do some
>> people use it?
>>
>> Notes:
>>   * groups are not handled right now and since users may change groups
>> regularly I don't think it would be a great idea to freeze them in a
>> certificate. The bricks could possibly do an ldap lookup in order to
>> retrieve and cache the groups for an uid.
>>   * Clients obviously can't modify their certificates because they are
>> signed by CA
>>
>> What do you think of this implementation, is it safe?
>> Do all client operationsuse auth_glusterfs_v2_authenticate or did I miss
>> other codepaths?
>>
>> Thanks!
>>
>> Pierre Carru
>> eshard
>>
>> PS: By the way I found the source code very clean and well organized,
>> nice job :)
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Permission for glusterfs logs.

[Niels de Vos]

On Wed, Sep 20, 2017 at 07:50:58AM -0400, Kaleb S. KEITHLEY wrote:
> On 09/18/2017 09:22 PM, ABHISHEK PALIWAL wrote:
> > Any suggestion would be appreciated...
> > 
> > On Sep 18, 2017 15:05, "ABHISHEK PALIWAL"  > > wrote:
> > 
> > Any quick suggestion.?
> > 
> > On Mon, Sep 18, 2017 at 1:50 PM, ABHISHEK PALIWAL
> > > wrote:
> > 
> > Hi Team,
> > 
> > As you can see permission for the glusterfs logs in
> > /var/log/glusterfs is 600.
> > 
> > drwxr-xr-x 3 root root  140 Jan  1 00:00 ..
> > *-rw--- 1 root root    0 Jan  3 20:21 cmd_history.log*
> > drwxr-xr-x 2 root root   40 Jan  3 20:21 bricks
> > drwxr-xr-x 3 root root  100 Jan  3 20:21 .
> > *-rw--- 1 root root 2102 Jan  3 20:21
> > etc-glusterfs-glusterd.vol.log*
> > 
> > Due to that non-root user is not able to access these logs
> > files, could you please let me know how can I change these
> > permission. So that non-root user can also access these log files.
> >
> 
> There is no "quick fix."  Gluster creates the log files with 0600 — like
> nearly everything else in /var/log.
> 
> The admin can chmod the files, but when the logs rotate the new log
> files will be 0600 again.
> 
> You'd have to patch the source and rebuild to get different permission bits.
> 
> You can probably do something with ACLs, but as above, when the logs
> rotate the new files won't have the ACLs.

Actually, if you set the 'default' ACL on the /var/log/gluster and other
directories, it gets inherited to new files that are created under
there. (The 'chmod' permissions for the directory will apply as
maximum permissions for ACLs, with chmod=755 reading files is possible.)

Something like this might work (give group 'admin' read permissions):

  # setfacl -d -m g:admin:r $(find /var/log/gluster -type d)
  # setfacl -R -m g:admin:r /var/log/gluster

Once you test this out, and are successful, you might want to add this
to the documentation on http://docs.gluster.org/ somewhere. Pull
requests can be sent to https://github.com/gluster/glusterdocs/ .

Thanks,
Niels


signature.asc
Description: PGP signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Restrict root clients / experimental patch

[Soumya Koduri]

Hi,

On 09/21/2017 07:32 PM, Pierre C wrote:

Hi All,

I would like to use glusterfs in an environment where storage servers 
are managed by an IT service - myself :) - and several users in the 
organization can mount the distributed fs. The users are root on their 
machines.
As far as I know about glusterfs, a root client user may impersonate any 
uid/gid since it provides its uid/gid itself when it talks to the bricks 
(like nfsv3).
The thing is, we want to enforce permissions, i.e. user X may only 
access files shared with him even if he's root on his machine.
I found a draft spec about glusterfs+kerberos 
 
but not much more so I think it's not possible with glusterfs right now, 
correct?

(Also I feel that kerberos would be a bit heavy to manage)


I am not much familiar with how ssl is handled. But from what I 
understand from your statement above, you need to restrict root users. 
Isn't root-squash option enough for it?


Option: server.root-squash
Default Value: off
Description: Map requests from uid/gid 0 to the anonymous uid/gid. Note 
that this does not apply to any other uids or gids that might be equally 
sensitive, such as user bin or group staff.


Option: server.anonuid
Default Value: 65534
Description: value of the uid used for the anonymous user/nfsnobody when 
root-squash is enabled.


Option: server.anongid
Default Value: 65534
Description: value of the gid used for the anonymous user/nfsnobody when 
root-squash is enabled.



Thanks,
Soumya



---

An simple hack that I found is to add custom uid/gid fields in clients' 
ssl certificates. The bricks use the client's uid/gid specified in its 
certificate rather than using one specified by the user. This solution 
has no effect on performance and there's no need for a central 
authentication.
The thing that changes is the way client certificates are generated and 
glusterfsd needs a small patch.


I did an experimental implementation 
 
of this idea. Custom fields "1.2.3.4.5.6.7" and "1.2.3.4.5.6.8" are used 
for uid and gid.
I tried it with custom CA trusted by all bricks and I issued a few 
client certificates.
No server configuration is needed when a new client is added, when a 
client is revoked the a CRL 
 must updated 
and pushed to all servers.
By the way I didn't get glusterfs servers to accept my CRLs, do some 
people use it?


Notes:
  * groups are not handled right now and since users may change groups 
regularly I don't think it would be a great idea to freeze them in a 
certificate. The bricks could possibly do an ldap lookup in order to 
retrieve and cache the groups for an uid.
  * Clients obviously can't modify their certificates because they are 
signed by CA


What do you think of this implementation, is it safe?
Do all client operationsuse auth_glusterfs_v2_authenticate or did I miss 
other codepaths?


Thanks!

Pierre Carru
eshard

PS: By the way I found the source code very clean and well organized, 
nice job :)



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Permission for glusterfs logs.

[Niels de Vos]

On Wed, Sep 20, 2017 at 04:38:51PM +0530, ABHISHEK PALIWAL wrote:
> Hi Team,
> 
> I did some modification in glusterfs code and now able to modify the
> permission of maximum of files.
> 
> But still 2 file's permission in 0600
> 
> 1. cli.log
> 2. file which contains the mounting information for "mount -t glusterfs"
> command
> 
> I will really appreciate, if some can point light on this area. Also is
> there any side effect of changing these permissions apart from other user
> can access these.

Certain actions may result in filenames being logged. It may not be
appropriate to have all users know what files other users have access
to.

In an other reply, I explained how ACLs may help with this. Most
environments will have a sysadmin group that can be allowed to read the
log files without compromising too much on the confidentiality.

Changing the source code is almost always the wrong approach. It will
make it difficult for you to update to a newer version. If changes are
needed, you probably should look into sending patches that include a
configuration or commandline option to adjust log-create permissions.

Niels


> 
> Regards,
> Abhishek
> 
> On Tue, Sep 19, 2017 at 6:52 AM, ABHISHEK PALIWAL 
> wrote:
> 
> > Any suggestion would be appreciated...
> >
> > On Sep 18, 2017 15:05, "ABHISHEK PALIWAL"  wrote:
> >
> >> Any quick suggestion.?
> >>
> >> On Mon, Sep 18, 2017 at 1:50 PM, ABHISHEK PALIWAL <
> >> abhishpali...@gmail.com> wrote:
> >>
> >>> Hi Team,
> >>>
> >>> As you can see permission for the glusterfs logs in /var/log/glusterfs
> >>> is 600.
> >>>
> >>> drwxr-xr-x 3 root root  140 Jan  1 00:00 ..
> >>> *-rw--- 1 root root0 Jan  3 20:21 cmd_history.log*
> >>> drwxr-xr-x 2 root root   40 Jan  3 20:21 bricks
> >>> drwxr-xr-x 3 root root  100 Jan  3 20:21 .
> >>> *-rw--- 1 root root 2102 Jan  3 20:21 etc-glusterfs-glusterd.vol.log*
> >>>
> >>> Due to that non-root user is not able to access these logs files, could
> >>> you please let me know how can I change these permission. So that non-root
> >>> user can also access these log files.
> >>>
> >>> Regards,
> >>> Abhishek Paliwal
> >>>
> >>
> >>
> >>
> >> --
> >>
> >>
> >>
> >>
> >> Regards
> >> Abhishek Paliwal
> >>
> >
> 
> 
> -- 
> 
> 
> 
> 
> Regards
> Abhishek Paliwal

> ___
> Gluster-devel mailing list
> gluster-de...@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel



signature.asc
Description: PGP signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] After stop and start wrong port is advertised

[Atin Mukherjee]

On Fri, Sep 22, 2017 at 2:37 AM, Jo Goossens 
wrote:

> Hi,
>
>
>
>
>
> We use glusterfs 3.10.5 on Debian 9.
>
>
>
> When we stop or restart the service, e.g.: service glusterfs-server restart
>
>
>
> We see that the wrong port get's advertised afterwards. For example:
>
>
>
> Before restart:
>
>
> Status of volume: public
> Gluster process TCP Port  RDMA Port  Online
>  Pid
> 
> --
> Brick 192.168.140.41:/gluster/public49153 0  Y
> 6364
> Brick 192.168.140.42:/gluster/public49152 0  Y
> 1483
> Brick 192.168.140.43:/gluster/public49152 0  Y
> 5913
> Self-heal Daemon on localhost   N/A   N/AY
> 5932
> Self-heal Daemon on 192.168.140.42  N/A   N/AY
> 13084
> Self-heal Daemon on 192.168.140.41  N/A   N/AY
> 15499
>
> Task Status of Volume public
> 
> --
> There are no active volume tasks
>
>
> After restart of the service on one of the nodes (192.168.140.43) the port
> seems to have changed (but it didn't):
>
> root@app3:/var/log/glusterfs#  gluster volume status
> Status of volume: public
> Gluster process TCP Port  RDMA Port  Online
>  Pid
> 
> --
> Brick 192.168.140.41:/gluster/public49153 0  Y
> 6364
> Brick 192.168.140.42:/gluster/public49152 0  Y
> 1483
> Brick 192.168.140.43:/gluster/public49154 0  Y
> 5913
> Self-heal Daemon on localhost   N/A   N/AY
> 4628
> Self-heal Daemon on 192.168.140.42  N/A   N/AY
> 3077
> Self-heal Daemon on 192.168.140.41  N/A   N/AY
> 28777
>
> Task Status of Volume public
> 
> --
> There are no active volume tasks
>
>
> However the active process is STILL the same pid AND still listening on
> the old port
>
> root@192.168.140.43:/var/log/glusterfs# netstat -tapn | grep gluster
> tcp0  0 0.0.0.0:49152   0.0.0.0:*
> LISTEN  5913/glusterfsd
>
>
> The other nodes logs fill up with errors because they can't reach the
> daemon anymore. They try to reach it on the "new" port instead of the old
> one:
>
> [2017-09-21 08:33:25.225006] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
> [2017-09-21 08:33:29.226633] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:29.227490] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
> [2017-09-21 08:33:33.225849] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:33.236395] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
> [2017-09-21 08:33:37.225095] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:37.225628] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
> [2017-09-21 08:33:41.225805] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:41.226440] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
>
> So they now try 49154 instead of the old 49152
>
> Is this also by design? We had a lot of issues because of this recently.
> We don't understand why it starts advertising a completely wrong port after
> stop/start.
>

This looks like a bug to me. For some reason glusterd's portmap is
referring to a stale port (IMO) where as brick is still listening to the
correct port. But ideally when glusterd service is restarted, all the
portmap in-memory is rebuilt. I'd request for the following details from
you to let us start analysing it:

1. glusterd statedump output from 192.168.140.43 . You can use kill
-SIGUSR2  to request for a statedump and the file will be
available in /var/run/gluster
2. glusterd, brick logfile for 192.168.140.43:/gluster/public from
192.168.140.43
3. cmd_history logfile from all the nodes.
4. Content of /var/lib/glusterd/vols/public/


>
>
>
>
>
>
> Regards
>
> Jo Goossens
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>

Re: [Gluster-users] vfs_fruit and extended attributes

[Anoop C S]

On Thu, 2017-09-21 at 10:35 -0600, Terry McGuire wrote:
> Hello list.  I’m attempting to improve how Samba shares directories on our 
> Gluster volume to Mac
> users by using the vfs_fruit module.

What versions of GlusterFS and Samba have you installed? And which 
platform/distro are you using as
Samba server?

Please paste the output of `gluster volume info `.

> This module does wonders for speeding listings and downloads of directories 
> with large numbers of
> files in the Finder, but it kills uploads dead.  Finder gives an error:
> 
> The Finder can’t complete the operation because some data in “[filename]” 
> can’t be read or
> written.
> (Error code -36)

Can you please check for any errors in brick logs under 
/var/log/glusterfs/bricks/ while you face
this Finder error?

> The man page for the module indicates that the vfs_streams_xattr module must 
> also be loaded, like
> this:
> 
> vfs objects = fruit streams_xattr

Can you please share the output of `testparm -s`?

> I’ve done this, and I’ve futzed with various combinations of options, but 
> haven’t got writes
> working.

Is that a simple copy operation? Did you verify whether the file content itself 
is written
completely? We have experienced similar error while copying large files into 
GlusterFS volumes where
the real copy operation was completed successfully and the remaining extra 
extended attribute
setting failed which resulted in a Finder error. Just wanted to check whether 
it is the same case or
not?

> This issue seems specific to Gluster, as I can share a directory that’s not 
> in the Gluster volume
> and it works fine.  I’m guessing the problem has something to do with 
> extended attributes, but,
> near as I can tell, the Gluster volume ought to do xattrs just fine, as its 
> bricks are XFS.
> 
> Anyone have any experience with this?  Anyone have vfs_fruit working with 
> Gluster?

Apart from this write issue, are you facing any other issues while accessing 
GlusterFS volumes from
Mac via Samba using vfs_fruit module? We would like to really fix any obvious 
errors which stops Mac
users working with GlusterFS volumes.

> Regards,
> Terry
> _
> Terry McGuire
> Information Services and Technology (IST)
> University of Alberta
> Edmonton, Alberta, Canada  T6G 2H1
> Phone:  780-492-9422
> 
> 
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] AFR: Fail lookups when quorum not met

[Ravishankar N]

Hello,

In AFR we currently allow look-ups to pass through without taking into 
account whether the lookup is served from the good or bad brick. We 
always serve from the good brick whenever possible, but if there is 
none, we just serve the lookup from one of the bricks that we got a 
positive reply from.


We found a bug  [1] due to this behavior were the iatt values returned 
in the lookup call was bad and caused the client to hang. The proposed 
fix [2] was to fail look ups when we definitely know the lookup can't be 
trusted (by virtue of AFR xattrs indicating the replies we got from the 
up bricks are indeed bad).


Note that this fix is *only* for replica 3 or arbiter volumes (not 
replica 2, where there is no notion of quorum). But we want to 'harden' 
the fix by  not allowing any look ups at all if quorum is not met (or) 
it is met but there are no good copies.


Some implications of this:

-If a file ends up in data/meta data split-brain in replica 3/arbiter 
(rare occurrence), we won't be able to delete it from the mount.


-Even if the only brick that is up is the good copy, we still fail it 
due to lack of quorum.


Does any one have comments/ feedback?

Thanks,

Ravi

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1467250

[2] https://review.gluster.org/#/c/17673/ (See review comments on the 
landing page if interested)


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Performance drop from 3.8 to 3.10

[Krutika Dhananjay]

Could you disable cluster.eager-lock and try again?

-Krutika

On Thu, Sep 21, 2017 at 6:31 PM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> Upgraded recently from 3.8.15 to 3.10.5 and have seen a fairly substantial
> drop in read/write perfomance
>
> env:
>
> - 3 node, replica 3 cluster
>
> - Private dedicated Network: 1Gx3, bond: balance-alb
>
> - was able to down the volume for the upgrade and reboot each node
>
> - Usage: VM Hosting (qemu)
>
> - Sharded Volume
>
> - sequential read performance in VM's has dropped from 700Mbps to 300mbs
>
> - Seq Write has dropped from 115MB/s (approx) to 110
>
> - Write IOPS have dropped from 12MB/s to 8MB/s
>
> Apart from increasing the op version I made no changes to the volume
> settings.
>
> op.version is 31004
>
> gluster v info
>
> Volume Name: datastore4
> Type: Replicate
> Volume ID: 0ba131ef-311d-4bb1-be46-596e83b2f6ce
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: vnb.proxmox.softlog:/tank/vmdata/datastore4
> Brick2: vng.proxmox.softlog:/tank/vmdata/datastore4
> Brick3: vnh.proxmox.softlog:/tank/vmdata/datastore4
> Options Reconfigured:
> transport.address-family: inet
> cluster.locking-scheme: granular
> cluster.granular-entry-heal: yes
> features.shard-block-size: 64MB
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.stat-prefetch: on
> performance.strict-write-ordering: off
> nfs.enable-ino32: off
> nfs.addr-namelookup: off
> nfs.disable: on
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> features.shard: on
> cluster.data-self-heal: on
> performance.readdir-ahead: on
> performance.low-prio-threads: 32
> user.cifs: off
> performance.flush-behind: on
> server.event-threads: 4
> client.event-threads: 4
> server.allow-insecure: on
>
>
> --
> Lindsay Mathieson
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-infra] lists.gluster.org issues this weekend

[Ravishankar N]

Hello,
Are our servers still facing the overload issue? My replies to 
gluster-users ML are not getting delivered to the list.

Regards,
Ravi

On 09/19/2017 10:03 PM, Michael Scherer wrote:

Le samedi 16 septembre 2017 à 20:48 +0530, Nigel Babu a écrit :

Hello folks,

We have discovered that for the last few weeks our mailman server was
used
for a spam attack. The attacker would make use of the + feature
offered by
gmail and hotmail. If you send an email to exam...@hotmail.com,
example+...@hotmail.com, example+...@hotmail.com, it goes to the same
inbox. We were constantly hit with requests to subscribe to a few
inboxes.
These requests overloaded our mail server so much that it gave up. We
detected this failure because a postmortem email to
gluster-in...@gluster.org bounced. Any emails sent to our mailman
server
may have been on hold for the last 24 hours or so. They should be
processed
now as your email provider re-attempts.

For the moment, we've banned subscribing with an email address with a
+ in
the name. If you are already subscribed to the lists with a + in your
email
address, you will continue to be able to use the lists.

We're looking at banning the spam IP addresses from being able to hit
the
web interface at all. When we have a working alternative, we will
look at
removing the current ban of using + in address.

So we have a alternative in place, I pushed a blacklist using
mod_security and a few DNS blacklist:
https://github.com/gluster/gluster.org_ansible_configuration/commit/2f4
c1b8feeae16e1d0b7d6073822a6786ed21dde





Apologies for the outage and a big shout out to Michael for taking
time out
of his weekend to debug and fix the issue.

Well, you can thanks the airport in Prague for being less interesting
than a spammer attacking us.



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] hostname

[Atin Mukherjee]

On Wed, Sep 20, 2017 at 6:02 PM, Alexandre Blanca  wrote:

> Hi,
>
> how to change the host name of gluster servers?
> if I modify the hostname1 in /etc/lib/glusterd/peers/uuid, the change is
> not save...
>
> gluster pool list return ipserver and not new hostname...
>

If you have used the same hostname for peer probing the cluster then here
it goes:

1. You'd need to bring down all the glusterd instances in the cluster.
2. Find out the hostname entries in /var/lib/glusterd/* and replace them
with the new hostname.
3. Bring up glusterd instances one after another.


> Thank you
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Sharding option for distributed volumes

[Pavel Kutishchev]

Hello Ji-Hyeon,

Thanks, is that option available in 3.12 gluster release? because we're 
still on 3.8 and just playing around latest version in order to have our 
solution migrated.


Thank you!


9/21/17 2:26 PM, Ji-Hyeon Gim пишет:

Hello Pavel!

In my opinion, you need to check features.shard-block-size option first.
If a file nobigger than this value, it would not be sharded :)

and then If you enablesharding after usinga volume, it may not working.
Sharding is supported in new deployments only, as there is currently no
upgrade path for this feature.


On 2017년 09월 20일 23:23, Pavel Kutishchev wrote:

Hello folks,

Would please someone advice how to use sharding option for distributed
volumes. At this moment i'm facing problem with exporting big file
which are not going to be distributed across bricks inside one volume.

Thank you in advance.



Best regards.

--

Ji-Hyeon Gim
Research Engineer, Gluesys

Address. Gluesys R Center, 5F, 11-31, Simin-daero 327beon-gil,
  Dongan-gu, Anyang-si,
  Gyeonggi-do, Korea
  (14055)
Phone.   +82-70-8787-1053
Fax. +82-31-388-3261
Mobile.  +82-10-7293-8858
E-Mail.  potato...@potatogim.net
Website. www.potatogim.net

The time I wasted today is the tomorrow the dead man was eager to see yesterday.
   - Sophocles




--
Best regards
Pavel Kutishchev
DevOPS Engineer at
Self employed.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] BUG: After stop and start wrong port is advertised

[Atin Mukherjee]

I've already replied to your earlier email. In case you've not seen it in
your mailbox here it goes:

This looks like a bug to me. For some reason glusterd's portmap is
referring to a stale port (IMO) where as brick is still listening to the
correct port. But ideally when glusterd service is restarted, all the
portmap in-memory is rebuilt. I'd request for the following details from
you to let us start analysing it:

1. glusterd statedump output from 192.168.140.43 . You can use kill
-SIGUSR2  to request for a statedump and the file will be
available in /var/run/gluster
2. glusterd, brick logfile for 192.168.140.43:/gluster/public from
192.168.140.43
3. cmd_history logfile from all the nodes.
4. Content of /var/lib/glusterd/vols/public/


On Thu, Sep 21, 2017 at 2:08 PM, Jo Goossens 
wrote:

> Hi,
>
>
>
>
>
> We use glusterfs 3.10.5 on Debian 9.
>
>
>
> When we stop or restart the service, e.g.: service glusterfs-server restart
>
>
>
> We see that the wrong port get's advertised afterwards. For example:
>
>
>
> Before restart:
>
>
> Status of volume: public
> Gluster process TCP Port  RDMA Port  Online
>  Pid
> 
> --
> Brick 192.168.140.41:/gluster/public49153 0  Y
> 6364
> Brick 192.168.140.42:/gluster/public49152 0  Y
> 1483
> Brick 192.168.140.43:/gluster/public49152 0  Y
> 5913
> Self-heal Daemon on localhost   N/A   N/AY
> 5932
> Self-heal Daemon on 192.168.140.42  N/A   N/AY
> 13084
> Self-heal Daemon on 192.168.140.41  N/A   N/AY
> 15499
>
> Task Status of Volume public
> 
> --
> There are no active volume tasks
>
>
> After restart of the service on one of the nodes (192.168.140.43) the port
> seems to have changed (but it didn't):
>
> root@app3:/var/log/glusterfs#  gluster volume status
> Status of volume: public
> Gluster process TCP Port  RDMA Port  Online
>  Pid
> 
> --
> Brick 192.168.140.41:/gluster/public49153 0  Y
> 6364
> Brick 192.168.140.42:/gluster/public49152 0  Y
> 1483
> Brick 192.168.140.43:/gluster/public49154 0  Y
> 5913
> Self-heal Daemon on localhost   N/A   N/AY
> 4628
> Self-heal Daemon on 192.168.140.42  N/A   N/AY
> 3077
> Self-heal Daemon on 192.168.140.41  N/A   N/AY
> 28777
>
> Task Status of Volume public
> 
> --
> There are no active volume tasks
>
>
> However the active process is STILL the same pid AND still listening on
> the old port
>
> root@192.168.140.43:/var/log/glusterfs# netstat -tapn | grep gluster
> tcp0  0 0.0.0.0:49152   0.0.0.0:*
> LISTEN  5913/glusterfsd
>
>
> The other nodes logs fill up with errors because they can't reach the
> daemon anymore. They try to reach it on the "new" port instead of the old
> one:
>
> [2017-09-21 08:33:25.225006] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
> [2017-09-21 08:33:29.226633] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:29.227490] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
> [2017-09-21 08:33:33.225849] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:33.236395] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
> [2017-09-21 08:33:37.225095] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:37.225628] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
> [2017-09-21 08:33:41.225805] I [rpc-clnt.c:2000:rpc_clnt_reconfig]
> 0-public-client-2: changing port to 49154 (from 0)
> [2017-09-21 08:33:41.226440] E [socket.c:2327:socket_connect_finish]
> 0-public-client-2: connection to 192.168.140.43:49154 failed (Connection
> refused); disconnecting socket
>
> So they now try 49154 instead of the old 49152
>
> Is this also by design? We had a lot of issues because of this recently.
> We don't understand why it starts advertising a completely wrong port after
> stop/start.
>
>
>
>
>
>
>
> Regards
>
> Jo Goossens
>
>
>
> ___
> Gluster-users mailing list

Re: [Gluster-users] Arbiter and geo-replication

[Ravishankar N]

On 09/22/2017 02:25 AM, Kotresh Hiremath Ravishankar wrote:
The volume layout of geo-replication slave volume could be different 
from master volume.
It's not mandatory that if the master volume is arbiter type, the 
slave also needs to be arbiter.
But if it's decided to use the arbiter both at master and slave, then 
the expansion rules is

applicable both at master and slave.

Adding Ravi for arbiter related question

On Thu, Sep 21, 2017 at 2:07 PM, Marcus > wrote:


Hi all!

Today I have a small gluster replication on 2 machines.
My plan is to scale this, I though need some feedback that how I
plan things is in the right direction.

First of all I have understood the need of an arbiter.
When I scale this, say that I just have 2 replica and 1 arbiter,
when I add another two machines can I still use the same physical
machine as the arbiter?
Or when I add additional two machines I have to add another
arbiter machine as well?




You can use the same physical machine for hosting multiple arbiter 
bricks. The general rule of brick placement is the same for any type of 
replica volume: No 2 bricks of the same replica subvolume should be on 
the same node.


Thanks,
Ravi



My second question is about geo-replication.
If I want to setup a geo-replication on above gluster cluster do I
need to have the exact "same" machines in the reo-replication?
I know that disk size should be same size on both the brick and on
the geo-replication side.
So if I have 2 replica and 1 arbiter, do I need 2 replica and 1
arbiter for the geo-replication?
Or is it sufficient for a 2 replica and 1 arbiter to use 1 replica
for the geo-replication?
What I wonder is when I scale my gluster with additional 2
machines, do I need 2 machines for geo-replication or 1 machine
for geo-replication?
So adding 2 machines means adding 4 machines in total or do I just
need 3 in total?
Is there a need for a arbiter in the geo-replication?

Many questions, but I hope that you can help me out!

Many thanks in advance!

Best regards
Marcus Pedersén


-- 


*Marcus Pedersén*
/System administrator/


*Interbull Centre*
Department of Animal Breeding & Genetics — SLU
Box 7023, SE-750 07
Uppsala, Sweden

Visiting address:
Room 55614, Ulls väg 26, Ultuna
Uppsala
Sweden

Tel: +46-(0)18-67 1962 
Interbull Logo


ISO certification logo

___
Gluster-users mailing list
Gluster-users@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-users





--
Thanks and Regards,
Kotresh H R


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] date/time on gluster-users mailman

[Jim Kinney]

For the past week or so (since the spam issue) the gluster-users mail
arrives dated 12 hours in the past. Did a fix for the spam upset the
time of the server?___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: Upgrade Gluster 3.7 to 3.12 and add 3rd replica [howto/help]

[Ravishankar N]

I had replied (reply-to-all) to this email y'day but it I don't see it 
on the list. Anyway pasting it again:



On 09/21/2017 10:03 AM, Ravishankar N wrote:


On 09/20/2017 01:45 PM, Martin Toth wrote:

*My Questions*
- is heal and rebalance necessary in order to upgrade replica 2 to 
replica 3 ?
No, `gluster volume add-brick voname replica 3 
node3.san:/tank/gluster/brick1' should automatically trigger healing 
in gluster 3.12 (actually earlier than 3.12 but I don't remember which 
release the automatic healing was added on add-brick command). Ensure 
that `gluster volume heal volname info` eventually becomes zero.
- is this upgrade procedure OK ? What more/else should I do in order 
to do this upgrade correctly ?


Looks okay in theory (I haven't tried out a 3.7 to 3.12 upgrade). It 
might be good to check there are no pending heals before you stop the 
old nodes for the upgrade.

-Ravi


-Ravi

On 09/21/2017 09:50 PM, Amye Scavarda wrote:

Just making sure this gets through.


-- Forwarded message --
From: Martin Toth 
Date: Thu, Sep 21, 2017 at 9:17 AM
Subject: Upgrade Gluster 3.7 to 3.12 and add 3rd replica [howto/help]
To: gluster-users@gluster.org
Cc: Marek Toth , a...@redhat.com


Hello all fellow GlusterFriends,

I would like you to comment / correct my upgrade procedure steps on
replica 2 volume of 3.7.x gluster.
Than I would like to change replica 2 to replica 3 in order to correct
quorum issue that Infrastructure currently has.

Infrastructure setup:
- all clients running on same nodes as servers (FUSE mounts)
- under gluster there is ZFS pool running as raidz2 with SSD ZLOG/ZIL cache
- all two hypervisor running as GlusterFS nodes and also Qemu compute
nodes (Ubuntu 16.04 LTS)
- we are running Qemu VMs that accesses VMs disks via gfapi (Opennebula)
- we currently run : 1x2 , Type: Replicate volume

Current Versions :
glusterfs-* [package] 3.7.6-1ubuntu1
qemu-* [package] 2.5+dfsg-5ubuntu10.2glusterfs3.7.14xenial1

What we need : (New versions)
- upgrade GlusterFS to 3.12 LTM version (Ubuntu 16.06 LTS packages are
EOL - see https://www.gluster.org/community/release-schedule/)
- I want to use
https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12 as
package repository for 3.12
- upgrade Qemu (with build-in support for libgfapi) -
https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.12
- (sadly Ubuntu has packages build without libgfapi support)
- add third node to replica setup of volume (this is probably most
dangerous operation)

Backup Phase
- backup "NFS storage” - raw DATA that runs on VMs
- stop all running VMs
- backup all running VMs (Qcow2 images) outside of gluster

Upgrading Gluster Phase
- killall glusterfs glusterfsd glusterd (on every server)
(this should stop all gluster services - server and client as it runs
on same nodes)
- install new Gluster Server and Client packages from repository
mentioned upper (on every server)
- install new Monotek's qemu glusterfs package with gfapi enabled
support (on every server)
- /etc/init.d/glusterfs-server start (on every server)
- /etc/init.d/glusterfs-server status - verify that all runs ok (on
every server)
- check :
- gluster volume info
- gluster volume status
- check gluster FUSE clients, if mounts working as expected
- test if various VMs are able tu boot and run as expected (if
libgfapi works in Qemu)
- reboot all nodes - do system upgrade of packages
- test and check again

Adding third node to replica 2 setup (replica 2 => replica 3)
(volumes will be mounted and up after upgrade and we tested VMs are
able to be served with libgfapi = upgrade of gluster sucessfuly
completed)
(next we extend replica 2 to replica 3 while volumes are mounted but
no data is touched = no running VMs, only glusterfs servers and
clients on nodes)
- issue command : gluster volume add-brick volume replica 3
node3.san:/tank/gluster/brick1 (on new single node - node3)
so we change :
Bricks:
Brick1: node1.san:/tank/gluster/brick1
Brick2: node2.san:/tank/gluster/brick1
to :
Bricks:
Brick1: node1.san:/tank/gluster/brick1
Brick2: node2.san:/tank/gluster/brick1
Brick3: node3.san:/tank/gluster/brick1
- check gluster status
- (is rebalance / heal required here ?)
- start all VMs and start celebration :)

My Questions
- is heal and rebalance necessary in order to upgrade replica 2 to replica 3 ?
- is this upgrade procedure OK ? What more/else should I do in order
to do this upgrade correctly ?

Many thanks to all for support. Hope my little preparation howto will
help others to solve same situation.

Best Regards,
Martin




___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] "Input/output error" on mkdir for PPC64 based client

[Nithya Balachandran]

hi Walter,

I don't see EIO errors in the log snippet and the messages look fine so
far. Can you send across the rest of the log where you see the EIO?

On 20 September 2017 at 23:57, Walter Deignan  wrote:

> I put the share into debug mode and then repeated the process from a ppc64
> client and an x86 client. Weirdly the client logs were almost identical.
>
> Here's the ppc64 gluster client log of attempting to create a folder...
>
> -
>
> [2017-09-20 13:34:23.344321] D 
> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked]
> (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xfdf60)[0x3fff9ec56fe0]
> (--> 
> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked-0x26060)[0x3fff9ebd9e20]
> (--> /usr/lib64/libgfrpc.so.0(+0x1a614)[0x3fff9ebda614] (-->
> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit-0x29300)[0x3fff9ebd69b0] (-->
> /usr/lib64/glusterfs/3.10.5/xlator/protocol/client.so(+0x182e0)[0x3fff939182e0]
> ) 0-: 10.50.80.102:49152: ping timer event already removed
> [2017-09-20 13:34:23.345149] D 
> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked]
> (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xfdf60)[0x3fff9ec56fe0]
> (--> 
> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked-0x26060)[0x3fff9ebd9e20]
> (--> /usr/lib64/libgfrpc.so.0(+0x1a614)[0x3fff9ebda614] (-->
> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit-0x29300)[0x3fff9ebd69b0] (-->
> /usr/lib64/glusterfs/3.10.5/xlator/protocol/client.so(+0x182e0)[0x3fff939182e0]
> ) 0-: 10.50.80.103:49152: ping timer event already removed
> [2017-09-20 13:34:23.345977] D 
> [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked]
> (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xfdf60)[0x3fff9ec56fe0]
> (--> 
> /usr/lib64/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked-0x26060)[0x3fff9ebd9e20]
> (--> /usr/lib64/libgfrpc.so.0(+0x1a614)[0x3fff9ebda614] (-->
> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit-0x29300)[0x3fff9ebd69b0] (-->
> /usr/lib64/glusterfs/3.10.5/xlator/protocol/client.so(+0x182e0)[0x3fff939182e0]
> ) 0-: 10.50.80.104:49152: ping timer event already removed
> [2017-09-20 13:34:23.346070] D [MSGID: 0] 
> [dht-common.c:1002:dht_revalidate_cbk]
> 0-gv0-dht: revalidate lookup of / returned with op_ret 0 [Structure needs
> cleaning]
> [2017-09-20 13:34:23.347612] D [MSGID: 0] [dht-common.c:2699:dht_lookup]
> 0-gv0-dht: Calling fresh lookup for /tempdir3 on gv0-replicate-0
> [2017-09-20 13:34:23.348013] D [MSGID: 0] 
> [client-rpc-fops.c:2936:client3_3_lookup_cbk]
> 0-stack-trace: stack-address: 0x3fff88001080, gv0-client-1 returned -1
> error: No such file or directory [No such file or directory]
> [2017-09-20 13:34:23.348013] D [MSGID: 0] 
> [client-rpc-fops.c:2936:client3_3_lookup_cbk]
> 0-stack-trace: stack-address: 0x3fff88001080, gv0-client-0 returned -1
> error: No such file or directory [No such file or directory]
> [2017-09-20 13:34:23.348083] D [MSGID: 0] 
> [client-rpc-fops.c:2936:client3_3_lookup_cbk]
> 0-stack-trace: stack-address: 0x3fff88001080, gv0-client-2 returned -1
> error: No such file or directory [No such file or directory]
> [2017-09-20 13:34:23.348132] D [MSGID: 0] [afr-common.c:2264:afr_lookup_done]
> 0-stack-trace: stack-address: 0x3fff88001080, gv0-replicate-0 returned -1
> error: No such file or directory [No such file or directory]
> [2017-09-20 13:34:23.348166] D [MSGID: 0] [dht-common.c:2284:dht_lookup_cbk]
> 0-gv0-dht: fresh_lookup returned for /tempdir3 with op_ret -1 [No such file
> or directory]
> [2017-09-20 13:34:23.348195] D [MSGID: 0] [dht-common.c:2297:dht_lookup_cbk]
> 0-gv0-dht: Entry /tempdir3 missing on subvol gv0-replicate-0
> [2017-09-20 13:34:23.348220] D [MSGID: 0] 
> [dht-common.c:2068:dht_lookup_everywhere]
> 0-gv0-dht: winding lookup call to 1 subvols
> [2017-09-20 13:34:23.348551] D [MSGID: 0] 
> [client-rpc-fops.c:2936:client3_3_lookup_cbk]
> 0-stack-trace: stack-address: 0x3fff88001080, gv0-client-1 returned -1
> error: No such file or directory [No such file or directory]
> [2017-09-20 13:34:23.348551] D [MSGID: 0] 
> [client-rpc-fops.c:2936:client3_3_lookup_cbk]
> 0-stack-trace: stack-address: 0x3fff88001080, gv0-client-0 returned -1
> error: No such file or directory [No such file or directory]
> [2017-09-20 13:34:23.348613] D [MSGID: 0] 
> [client-rpc-fops.c:2936:client3_3_lookup_cbk]
> 0-stack-trace: stack-address: 0x3fff88001080, gv0-client-2 returned -1
> error: No such file or directory [No such file or directory]
> [2017-09-20 13:34:23.348639] D [MSGID: 0] [afr-common.c:2264:afr_lookup_done]
> 0-stack-trace: stack-address: 0x3fff88001080, gv0-replicate-0 returned -1
> error: No such file or directory [No such file or directory]
> [2017-09-20 13:34:23.348665] D [MSGID: 0] 
> [dht-common.c:1870:dht_lookup_everywhere_cbk]
> 0-gv0-dht: returned with op_ret -1 and op_errno 2 (/tempdir3) from subvol
> gv0-replicate-0
> [2017-09-20 13:34:23.348697] D [MSGID: 0] 
> [dht-common.c:1535:dht_lookup_everywhere_done]
> 0-gv0-dht: STATUS: 

Re: [Gluster-users] Upgrade Gluster 3.7 to 3.12 and add 3rd replica [howto/help]

[Ravishankar N]

On 09/20/2017 01:45 PM, Martin Toth wrote:

*My Questions*
- is heal and rebalance necessary in order to upgrade replica 2 to 
replica 3 ?
No, `gluster volume add-brick voname replica 3 
node3.san:/tank/gluster/brick1' should automatically trigger healing in 
gluster 3.12 (actually earlier than 3.12 but I don't remember which 
release the automatic healing was added on add-brick command). Ensure 
that `gluster volume heal volname info` eventually becomes zero.
- is this upgrade procedure OK ? What more/else should I do in order 
to do this upgrade correctly ?


Looks okay in theory (I haven't tried out a 3.7 to 3.12 upgrade). It 
might be good to check there are no pending heals before you stop the 
old nodes for the upgrade.

-Ravi
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] BUG: After stop and start wrong port is advertised

[Jo Goossens]

Hi,

 
 
We use glusterfs 3.10.5 on Debian 9.

 
When we stop or restart the service, e.g.: service glusterfs-server restart

 
We see that the wrong port get's advertised afterwards. For example:

 
Before restart:

 
Status of volume: public
Gluster process                             TCP Port  RDMA Port  Online  Pid
--
Brick 192.168.140.41:/gluster/public        49153     0          Y       6364
Brick 192.168.140.42:/gluster/public        49152     0          Y       1483
Brick 192.168.140.43:/gluster/public        49152     0          Y       5913
Self-heal Daemon on localhost               N/A       N/A        Y       5932
Self-heal Daemon on 192.168.140.42          N/A       N/A        Y       13084
Self-heal Daemon on 192.168.140.41          N/A       N/A        Y       15499
 Task Status of Volume public
--
There are no active volume tasks
  After restart of the service on one of the nodes (192.168.140.43) the port 
seems to have changed (but it didn't):
 root@app3:/var/log/glusterfs#  gluster volume status
Status of volume: public
Gluster process                             TCP Port  RDMA Port  Online  Pid
--
Brick 192.168.140.41:/gluster/public        49153     0          Y       6364
Brick 192.168.140.42:/gluster/public        49152     0          Y       1483
Brick 192.168.140.43:/gluster/public        49154     0          Y       5913
Self-heal Daemon on localhost               N/A       N/A        Y       4628
Self-heal Daemon on 192.168.140.42          N/A       N/A        Y       3077
Self-heal Daemon on 192.168.140.41          N/A       N/A        Y       28777
 Task Status of Volume public
--
There are no active volume tasks
  However the active process is STILL the same pid AND still listening on the 
old port
 root@192.168.140.43:/var/log/glusterfs# netstat -tapn | grep gluster
tcp        0      0 0.0.0.0:49152           0.0.0.0:*               LISTEN      
5913/glusterfsd
  The other nodes logs fill up with errors because they can't reach the daemon 
anymore. They try to reach it on the "new" port instead of the old one:
 [2017-09-21 08:33:25.225006] E [socket.c:2327:socket_connect_finish] 
0-public-client-2: connection to 192.168.140.43:49154 failed (Connection 
refused); disconnecting socket
[2017-09-21 08:33:29.226633] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 
0-public-client-2: changing port to 49154 (from 0)
[2017-09-21 08:33:29.227490] E [socket.c:2327:socket_connect_finish] 
0-public-client-2: connection to 192.168.140.43:49154 failed (Connection 
refused); disconnecting socket
[2017-09-21 08:33:33.225849] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 
0-public-client-2: changing port to 49154 (from 0)
[2017-09-21 08:33:33.236395] E [socket.c:2327:socket_connect_finish] 
0-public-client-2: connection to 192.168.140.43:49154 failed (Connection 
refused); disconnecting socket
[2017-09-21 08:33:37.225095] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 
0-public-client-2: changing port to 49154 (from 0)
[2017-09-21 08:33:37.225628] E [socket.c:2327:socket_connect_finish] 
0-public-client-2: connection to 192.168.140.43:49154 failed (Connection 
refused); disconnecting socket
[2017-09-21 08:33:41.225805] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 
0-public-client-2: changing port to 49154 (from 0)
[2017-09-21 08:33:41.226440] E [socket.c:2327:socket_connect_finish] 
0-public-client-2: connection to 192.168.140.43:49154 failed (Connection 
refused); disconnecting socket
 So they now try 49154 instead of the old 49152 
 Is this also by design? We had a lot of issues because of this recently. We 
don't understand why it starts advertising a completely wrong port after 
stop/start.
 
Regards

Jo Goossens

 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] hostname

[Ashish Pandey]

Hi, 

I hope you have gluster volume running on your cluster and you just want to 
change host name of your nodes to some other host name. 
Ex: From hostname1 to hostname2 
I think changing a host name of any node is very simple using linux command. 
I think changing host name should automatically detect by the peers in peer 
list of cluster. 

Only thing is what will happen to all those volume on which we have bricks with 
this host name? How that volume is going to recognize bricks when we change the 
host name. 
That's where we have introduced "reset-brick" command. 
https://gluster.readthedocs.io/en/latest/release-notes/3.9.0/ 
Please go through above link and understand the reset-brick command and usage 
and try it. 
If you have any doubt, please let us know before disturbing anything on your 
setup which can lead to volume unavailable. 
I hope you are using gluster version 3.9.0 and above. 

Ashish 





- Original Message -

From: "Alexandre Blanca"  
To: gluster-users@gluster.org 
Sent: Wednesday, September 20, 2017 6:02:16 PM 
Subject: [Gluster-users] hostname 

Hi, 

how to change the host name of gluster servers? 
if I modify the hostname1 in /etc/lib/glusterd/peers/uuid, the change is not 
save... 
gluster pool list return ipserver and not new hostname... 
Thank you 

___ 
Gluster-users mailing list 
Gluster-users@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-users 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Permission for glusterfs logs.

[Alex K]

You could trigger a chmod on log rotation.

Alex

On Sep 21, 2017 06:45, "Kaleb S. KEITHLEY"  wrote:

> On 09/18/2017 09:22 PM, ABHISHEK PALIWAL wrote:
> > Any suggestion would be appreciated...
> >
> > On Sep 18, 2017 15:05, "ABHISHEK PALIWAL"  > > wrote:
> >
> > Any quick suggestion.?
> >
> > On Mon, Sep 18, 2017 at 1:50 PM, ABHISHEK PALIWAL
> > > wrote:
> >
> > Hi Team,
> >
> > As you can see permission for the glusterfs logs in
> > /var/log/glusterfs is 600.
> >
> > drwxr-xr-x 3 root root  140 Jan  1 00:00 ..
> > *-rw--- 1 root root0 Jan  3 20:21 cmd_history.log*
> > drwxr-xr-x 2 root root   40 Jan  3 20:21 bricks
> > drwxr-xr-x 3 root root  100 Jan  3 20:21 .
> > *-rw--- 1 root root 2102 Jan  3 20:21
> > etc-glusterfs-glusterd.vol.log*
> >
> > Due to that non-root user is not able to access these logs
> > files, could you please let me know how can I change these
> > permission. So that non-root user can also access these log
> files.
> >
>
> There is no "quick fix."  Gluster creates the log files with 0600 — like
> nearly everything else in /var/log.
>
> The admin can chmod the files, but when the logs rotate the new log
> files will be 0600 again.
>
> You'd have to patch the source and rebuild to get different permission
> bits.
>
> You can probably do something with ACLs, but as above, when the logs
> rotate the new files won't have the ACLs.
>
>
>
> --
>
> Kaleb
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users