Re: [Gluster-users] Fwd: vm paused unknown storage error one node out of 3 only

[Dan Lavu]

t;> [2016-08-11 08:14:03.684492] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk]
>>>>>> 0-glusterfs: No change in volfile, continuing
>>>>>>
>>>>>>
>>>>>>
>>>>>>> 5. Brick logs
>>>>>>>
>>>>>>  Their have been some error in brick logs I hadn't noticed
>>>>>> occurring.  I've zip'd and attached all 3 nodes logs, but from this 
>>>>>> snippet
>>>>>> on one node none of them seem to coincide with the  time window when
>>>>>> migration had issues.  f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42 shard
>>>>>> refers to an image for a different vm than one I had issues with as well.
>>>>>> Maybe gluster is trying to do some sort of make shard test before writing
>>>>>> out changes that would go to that image and that shard file?
>>>>>>
>>>>>> [2016-08-12 18:48:22.463628] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.697
>>>>>> failed [File exists]
>>>>>> [2016-08-12 18:48:24.553455] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.698
>>>>>> failed [File exists]
>>>>>> [2016-08-12 18:49:16.065502] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.738
>>>>>> failed [File exists]
>>>>>> The message "E [MSGID: 113022] [posix.c:1245:posix_mknod]
>>>>>> 0-GLUSTER1-posix: mknod on /gluster1/BRICK1/1/.shard/f9a7
>>>>>> f3c5-4c13-4020-b560-1f4f7b1e3c42.697 failed [File exists]" repeated
>>>>>> 5 times between [2016-08-12 18:48:22.463628] and [2016-08-12
>>>>>> 18:48:22.514777]
>>>>>> [2016-08-12 18:48:24.581216] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.698
>>>>>> failed [File exists]
>>>>>> The message "E [MSGID: 113022] [posix.c:1245:posix_mknod]
>>>>>> 0-GLUSTER1-posix: mknod on /gluster1/BRICK1/1/.shard/f9a7
>>>>>> f3c5-4c13-4020-b560-1f4f7b1e3c42.738 failed [File exists]" repeated
>>>>>> 5 times between [2016-08-12 18:49:16.065502] and [2016-08-12
>>>>>> 18:49:16.107746]
>>>>>> [2016-08-12 19:23:40.964678] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/83794e5d-2225-4560-8df6-7c903c8a648a.1301
>>>>>> failed [File exists]
>>>>>> [2016-08-12 20:00:33.498751] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/0e5ad95d-722d-4374-88fb-66fca0b14341.580
>>>>>> failed [File exists]
>>>>>> [2016-08-12 20:00:33.530938] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/0e5ad95d-722d-4374-88fb-66fca0b14341.580
>>>>>> failed [File exists]
>>>>>> [2016-08-13 01:47:23.338036] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.211
>>>>>> failed [File exists]
>>>>>> The message "E [MSGID: 113022] [posix.c:1245:posix_mknod]
>>>>>> 0-GLUSTER1-posix: mknod on /gluster1/BRICK1/1/.shard/1884
>>>>>> 3fb4-e31c-4fc3-b519-cc6e5e947813.211 failed [File exists]" repeated
>>>>>> 16 times between [2016-08-13 01:47:23.338036] and [2016-08-13
>>>>>> 01:47:23.380980]
>>>>>> [2016-08-13 01:48:02.224494] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard/ffbbcce0-3c4a-4fdf-b79f-a96ca3215657.211
>>>>>> failed [File exists]
>>>>>> [2016-08-13 01:48:42.266148] E [MSGID: 113022]
>>>>>> [posix.c:1245:posix_mknod] 0-GLUSTER1-posix: mknod on
>>>>>> /gluster1/BRICK1/1/.shard

Re: [Gluster-users] RDMA inline threshold?

[Dan Lavu]

Stefan,

We'll have to let somebody else chime in. I don't work on this project,
just another user, enthusiast and I've spent, still spending much time
tuning my own RDMA gluster configuration. In short, I won't have an answer
for you. If nobody can answer, I'd suggest filing a bug, that way it can be
tracked and reviewed by developers.

- Dan

On Wed, May 30, 2018 at 6:34 AM, Stefan Solbrig 
wrote:

> Dear Dan,
>
> thanks for the quick reply!
>
> I actually tried restarting all processes (and even rebooting all
> servers), but the error persists. I can also confirm that all birck
> processes are running.   My volume is a distrubute-only volume (not
> dispersed, no sharding).
>
> I also tried mounting with use_readdirp=no,  because the error seems to be
> connected to readdirp, but this option does not change anything.
>
> I found to options I might try:  (gluster volume get myvolumename  all |
> grep readdirp )
>performance.force-readdirp  true
>dht.force-readdirp  on
> Can I turn off these safely? (or what precisely do they do?)
>
> I also assured that all glusterd processes have  unlimited locked memory.
>
> Just to state it clearly:  I do _not_ see any data corruption.  Just the
> directory listings do not work (in very rare cases) with rdma transport:
> "ls"  shows only a part of the files.
> but then I do:
>  stat  /path/to/known/filename
> it succeeds, and even
>md5sum /path/to/known/filename/that/does/not/get/listed/with/ls
> yields the correct result.
>
> best wishes,
> Stefan
>
> > Am 30.05.2018 um 03:00 schrieb Dan Lavu :
> >
> > Forgot to mention, sometimes I have to do force start other volumes as
> well, its hard to determine which brick process is locked up from the logs.
> >
> >
> > Status of volume: rhev_vms_primary
> > Gluster process
> TCP Port  RDMA Port  Online  Pid
> > 
> --
> > Brick spidey.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
>  0 49157  Y   15666
> > Brick deadpool.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
>0 49156  Y   2542
> > Brick groot.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
>  0 49156  Y   2180
> > Self-heal Daemon on localhost
> N/A   N/AN   N/A  << Brick process is
> not running on any node.
> > Self-heal Daemon on spidey.ib.runlevelone.lan
>  N/A   N/AN   N/A
> > Self-heal Daemon on groot.ib.runlevelone.lan
>N/A   N/AN   N/A
> >
> > Task Status of Volume rhev_vms_primary
> > 
> --
> > There are no active volume tasks
> >
> >
> >  3081  gluster volume start rhev_vms_noshards force
> >  3082  gluster volume status
> >  3083  gluster volume start rhev_vms_primary force
> >  3084  gluster volume status
> >  3085  gluster volume start rhev_vms_primary rhev_vms
> >  3086  gluster volume start rhev_vms_primary rhev_vms force
> >
> > Status of volume: rhev_vms_primary
> > Gluster process
>TCP Port  RDMA Port  Online  Pid
> > 
> --
> > Brick spidey.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
> 0 49157  Y   15666
> > Brick deadpool.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
>   0 49156  Y   2542
> > Brick groot.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
> 0 49156  Y   2180
> > Self-heal Daemon on localhost
>N/A   N/AY   8343
> > Self-heal Daemon on spidey.ib.runlevelone.lan
> N/A   N/AY   22381
> > Self-heal Daemon on groot.ib.runlevelone.lan
>   N/A   N/AY   20633
> >
> > Finally..
> >
> > Dan
> >
> >
> >
> >
> > On Tue, May 29, 2018 at 8:47 PM, Dan Lavu  wrote:
> > Stefan,
> >
> > Sounds like a brick process is not running. I have notice some
> strangeness in my lab when using RDMA, I often have to forcibly restart the
> brick process, often as in every single time I do a major operation, add a
> new volume, remove a volume, stop a volume, etc.
> >
> > gluster volume status 
> >
> > Does any of the self heal daemons show N/A? If that's the case, try
> forcing a res

Re: [Gluster-users] shard corruption bug

[Dan Lavu]

Thanks, issue isn't related.

On Tue, May 29, 2018 at 10:24 PM, Jim Kinney  wrote:

> https://docs.gluster.org/en/latest/release-notes/3.12.6/
>
> The major issue in 3.12.6 is not present in 3.12.7. Bugzilla ID listed in
> link.
>
>
> On May 29, 2018 8:50:56 PM EDT, Dan Lavu  wrote:
>>
>> What shard corruption bug? bugzilla url? I'm running into some odd
>> behavior in my lab with shards and RHEV/KVM data, trying to figure out if
>> it's related.
>>
>> Thanks.
>>
>> On Fri, May 4, 2018 at 11:13 AM, Jim Kinney  wrote:
>>
>>> I upgraded my ovirt stack to 3.12.9, added a brick to a volume and left
>>> it to settle. No problems. I am now running replica 4 (preparing to remove
>>> a brick and host to replica 3).
>>>
>>> On Fri, 2018-05-04 at 14:24 +, Gandalf Corvotempesta wrote:
>>>
>>> Il giorno ven 4 mag 2018 alle ore 14:06 Jim Kinney 
>>> ha scritto:
>>>
>>>
>>> It stopped being an outstanding issue at 3.12.7. I think it's now fixed.
>>>
>>>
>>>
>>> So, is not possible to extend and rebalance a working cluster with sharded
>>> data ?
>>> Can someone confirm this ? Maybe the ones that hit the bug in the past
>>>
>>> --
>>>
>>> James P. Kinney III Every time you stop a school, you will have to build
>>> a jail. What you gain at one end you lose at the other. It's like feeding a
>>> dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark
>>> Twain http://heretothereideas.blogspot.com/
>>>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
> --
> Sent from my Android device with K-9 Mail. All tyopes are thumb related
> and reflect authenticity.
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] RDMA inline threshold?

[Dan Lavu]

Forgot to mention, sometimes I have to do force start other volumes as
well, its hard to determine which brick process is locked up from the logs.


Status of volume: rhev_vms_primary
Gluster process
  TCP Port  RDMA Port  Online  Pid
--
Brick spidey.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
 0 49157  Y   15666
Brick deadpool.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
   0 49156  Y   2542
Brick groot.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
 0 49156  Y   2180
Self-heal Daemon on localhost
  N/A   N/AN   N/A  << Brick process is not
running on any node.
Self-heal Daemon on spidey.ib.runlevelone.lan
   N/A   N/AN   N/A
Self-heal Daemon on groot.ib.runlevelone.lan
   N/A   N/AN   N/A

Task Status of Volume rhev_vms_primary
--
There are no active volume tasks


 3081  gluster volume start rhev_vms_noshards force
 3082  gluster volume status
 3083  gluster volume start rhev_vms_primary force
 3084  gluster volume status
 3085  gluster volume start rhev_vms_primary rhev_vms
 3086  gluster volume start rhev_vms_primary rhev_vms force

Status of volume: rhev_vms_primary
Gluster process
 TCP Port  RDMA Port  Online  Pid
--
Brick spidey.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
0 49157  Y   15666
Brick deadpool.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
  0 49156  Y   2542
Brick groot.ib.runlevelone.lan:/gluster/brick/rhev_vms_primary
0 49156  Y   2180
Self-heal Daemon on localhost
 N/A   N/AY   8343
Self-heal Daemon on spidey.ib.runlevelone.lan
  N/A   N/AY   22381
Self-heal Daemon on groot.ib.runlevelone.lan
  N/A   N/AY   20633

Finally..

Dan




On Tue, May 29, 2018 at 8:47 PM, Dan Lavu  wrote:

> Stefan,
>
> Sounds like a brick process is not running. I have notice some strangeness
> in my lab when using RDMA, I often have to forcibly restart the brick
> process, often as in every single time I do a major operation, add a new
> volume, remove a volume, stop a volume, etc.
>
> gluster volume status 
>
> Does any of the self heal daemons show N/A? If that's the case, try
> forcing a restart on the volume.
>
> gluster volume start  force
>
> This will also explain why your volumes aren't being replicated properly.
>
> On Tue, May 29, 2018 at 5:20 PM, Stefan Solbrig 
> wrote:
>
>> Dear all,
>>
>> I faced a problem with a glusterfs volume (pure distributed, _not_
>> dispersed) over RDMA transport.  One user had a directory with a large
>> number of files (50,000 files) and just doing an "ls" in this directory
>> yields a "Transport endpoint not connected" error. The effect is, that "ls"
>> only shows some files, but not all.
>>
>> The respective log file shows this error message:
>>
>> [2018-05-20 20:38:25.114978] W [MSGID: 114031]
>> [client-rpc-fops.c:2578:client3_3_readdirp_cbk] 0-glurch-client-0:
>> remote operation failed [Transport endpoint is not connected]
>> [2018-05-20 20:38:27.732796] W [MSGID: 103046]
>> [rdma.c:4089:gf_rdma_process_recv] 0-rpc-transport/rdma: peer (
>> 10.100.245.18:49153), couldn't encode or decode the msg properly or
>> write chunks were not provided for replies that were bigger than
>> RDMA_INLINE_THRESHOLD (2048)
>> [2018-05-20 20:38:27.732844] W [MSGID: 114031]
>> [client-rpc-fops.c:2578:client3_3_readdirp_cbk] 0-glurch-client-3:
>> remote operation failed [Transport endpoint is not connected]
>> [2018-05-20 20:38:27.733181] W [fuse-bridge.c:2897:fuse_readdirp_cbk]
>> 0-glusterfs-fuse: 72882828: READDIRP => -1 (Transport endpoint is not
>> connected)
>>
>> I already set the memlock limit for glusterd to unlimited, but the
>> problem persists.
>>
>> Only going from RDMA transport to TCP transport solved the problem.  (I'm
>> running the volume now in mixed mode, config.transport=tcp,rdma).  Mounting
>> with transport=rdma shows this error, mouting with transport=tcp is fine.
>>
>> however, this problem does not arise on all large directories, not on
>> all. I didn't recognize a pattern yet.
>>
>> I'm using glusterfs v3.12.6 on the servers, QDR Infiniband HCAs .
>>
>> Is this a known issue with RDMA transport?
>>
>> best wishes,
>> Stefan
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] shard corruption bug

[Dan Lavu]

What shard corruption bug? bugzilla url? I'm running into some odd behavior
in my lab with shards and RHEV/KVM data, trying to figure out if it's
related.

Thanks.

On Fri, May 4, 2018 at 11:13 AM, Jim Kinney  wrote:

> I upgraded my ovirt stack to 3.12.9, added a brick to a volume and left it
> to settle. No problems. I am now running replica 4 (preparing to remove a
> brick and host to replica 3).
>
> On Fri, 2018-05-04 at 14:24 +, Gandalf Corvotempesta wrote:
>
> Il giorno ven 4 mag 2018 alle ore 14:06 Jim Kinney 
> ha scritto:
>
>
> It stopped being an outstanding issue at 3.12.7. I think it's now fixed.
>
>
>
> So, is not possible to extend and rebalance a working cluster with sharded
> data ?
> Can someone confirm this ? Maybe the ones that hit the bug in the past
>
> --
>
> James P. Kinney III Every time you stop a school, you will have to build a
> jail. What you gain at one end you lose at the other. It's like feeding a
> dog on his own tail. It won't fatten the dog. - Speech 11/23/1900 Mark
> Twain http://heretothereideas.blogspot.com/
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] RDMA inline threshold?

[Dan Lavu]

Stefan,

Sounds like a brick process is not running. I have notice some strangeness
in my lab when using RDMA, I often have to forcibly restart the brick
process, often as in every single time I do a major operation, add a new
volume, remove a volume, stop a volume, etc.

gluster volume status 

Does any of the self heal daemons show N/A? If that's the case, try forcing
a restart on the volume.

gluster volume start  force

This will also explain why your volumes aren't being replicated properly.

On Tue, May 29, 2018 at 5:20 PM, Stefan Solbrig 
wrote:

> Dear all,
>
> I faced a problem with a glusterfs volume (pure distributed, _not_
> dispersed) over RDMA transport.  One user had a directory with a large
> number of files (50,000 files) and just doing an "ls" in this directory
> yields a "Transport endpoint not connected" error. The effect is, that "ls"
> only shows some files, but not all.
>
> The respective log file shows this error message:
>
> [2018-05-20 20:38:25.114978] W [MSGID: 114031] 
> [client-rpc-fops.c:2578:client3_3_readdirp_cbk]
> 0-glurch-client-0: remote operation failed [Transport endpoint is not
> connected]
> [2018-05-20 20:38:27.732796] W [MSGID: 103046]
> [rdma.c:4089:gf_rdma_process_recv] 0-rpc-transport/rdma: peer (
> 10.100.245.18:49153), couldn't encode or decode the msg properly or write
> chunks were not provided for replies that were bigger than
> RDMA_INLINE_THRESHOLD (2048)
> [2018-05-20 20:38:27.732844] W [MSGID: 114031] 
> [client-rpc-fops.c:2578:client3_3_readdirp_cbk]
> 0-glurch-client-3: remote operation failed [Transport endpoint is not
> connected]
> [2018-05-20 20:38:27.733181] W [fuse-bridge.c:2897:fuse_readdirp_cbk]
> 0-glusterfs-fuse: 72882828: READDIRP => -1 (Transport endpoint is not
> connected)
>
> I already set the memlock limit for glusterd to unlimited, but the problem
> persists.
>
> Only going from RDMA transport to TCP transport solved the problem.  (I'm
> running the volume now in mixed mode, config.transport=tcp,rdma).  Mounting
> with transport=rdma shows this error, mouting with transport=tcp is fine.
>
> however, this problem does not arise on all large directories, not on all.
> I didn't recognize a pattern yet.
>
> I'm using glusterfs v3.12.6 on the servers, QDR Infiniband HCAs .
>
> Is this a known issue with RDMA transport?
>
> best wishes,
> Stefan
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] rhev/gluster questions , possibly restoring from a failed node

[Dan Lavu]

In my lab, one of my RAID cards started acting up and took one of my three
gluster nodes offline (two nodes with data and an arbiter node). I'm hoping
it's simply the backplane, but during that time troubleshooting and waiting
for parts, the hypervisors was fenced. Since the firewall was replaced and
now several VMs are not starting correctly, fsck, scandisk and xfs_repair
on the problematic VMs are running, but It's been 2 days and xfs_repair
hasn't moved an inch.

When the gluster volume has an offline brick, should it be slow?
If it's suppose to be slow, does it make sense to remove the bricks and
re-add them when the failed node is back?
Is the VM unresponsiveness an issue of the deteriorated gluster volume?

Lastly, in the event that the VMs are totally gone, I need to restore from
the failed node, I was going to simply copy/scp the contents from the
gluster directory onto the running nodes through glusterfs, is there a
better way of restoring, overwriting the data on the good nodes?

Thanks,

Dan
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Fwd: vm paused unknown storage error one node out of 3 only

[Dan Lavu]

David,

I'm seeing similar behavior in my lab, but it has been caused by healing
files in the gluster cluster, though I attribute my problems to problems
with the storage fabric. See if 'gluster volume heal $VOL info' indicates
files that are being healed, and if those reduce in number, can the VM
start?

Dan

On Thu, Aug 11, 2016 at 7:52 AM, David Gossage 
wrote:

> Figure I would repost here as well.  one client out of 3 complaining of
> stale file handles on a few new VM's I migrated over. No errors on storage
> nodes just client.  Maybe just put that one in maintenance and restart
> gluster mount?
>
> *David Gossage*
> *Carousel Checks Inc. | System Administrator*
> *Office* 708.613.2284
>
> -- Forwarded message --
> From: David Gossage 
> Date: Thu, Aug 11, 2016 at 12:17 AM
> Subject: vm paused unknown storage error one node out of 3 only
> To: users 
>
>
> Out of a 3 node cluster running oVirt 3.6.6.2-1.el7.centos with a 3
> replicate gluster 3.7.14 starting a VM i just copied in on one node of the
> 3 gets the following errors.  The other 2 the vm starts fine.  All ovirt
> and gluster are centos 7 based. VM on start of the one node it tries to
> default to on its own accord immediately puts into paused for unknown
> reason.  Telling it to start on different node starts ok.  node with issue
> already has 5 VMs running fine on it same gluster storage plus the hosted
> engine on different volume.
>
> gluster nodes logs did not have any errors for volume
> nodes own gluster logs had this in log
>
> dfb8777a-7e8c-40ff-8faa-252beabba5f8 couldnt find in .glusterfs .shard or
> images/
>
> 7919f4a0-125c-4b11-b5c9-fb50cc195c43 is the gfid of the bootable drive of
> the vm
>
> [2016-08-11 04:31:39.982952] W [MSGID: 114031]
> [client-rpc-fops.c:3050:client3_3_readv_cbk] 0-GLUSTER1-client-2: remote
> operation failed [No such file or directory]
> [2016-08-11 04:31:39.983683] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote
> operation failed [No such file or directory]
> [2016-08-11 04:31:39.984182] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote
> operation failed [No such file or directory]
> [2016-08-11 04:31:39.984221] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote
> operation failed [No such file or directory]
> [2016-08-11 04:31:39.985941] W [MSGID: 108008]
> [afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable
> subvolume -1 found with event generation 3 for gfid
> dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain)
> [2016-08-11 04:31:39.986633] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote
> operation failed [No such file or directory]
> [2016-08-11 04:31:39.987644] E [MSGID: 109040]
> [dht-helper.c:1190:dht_migration_complete_check_task] 0-GLUSTER1-dht:
> (null): failed to lookup the file on GLUSTER1-dht [Stale file handle]
> [2016-08-11 04:31:39.987751] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 15152930: READ => -1 
> gfid=7919f4a0-125c-4b11-b5c9-fb50cc195c43
> fd=0x7f00a80bdb64 (Stale file handle)
> [2016-08-11 04:31:39.986567] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote
> operation failed [No such file or directory]
> [2016-08-11 04:31:39.986567] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote
> operation failed [No such file or directory]
> [2016-08-11 04:35:21.210145] W [MSGID: 108008]
> [afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable
> subvolume -1 found with event generation 3 for gfid
> dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain)
> [2016-08-11 04:35:21.210873] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-1: remote
> operation failed [No such file or directory]
> [2016-08-11 04:35:21.210888] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-2: remote
> operation failed [No such file or directory]
> [2016-08-11 04:35:21.210947] W [MSGID: 114031]
> [client-rpc-fops.c:1572:client3_3_fstat_cbk] 0-GLUSTER1-client-0: remote
> operation failed [No such file or directory]
> [2016-08-11 04:35:21.213270] E [MSGID: 109040]
> [dht-helper.c:1190:dht_migration_complete_check_task] 0-GLUSTER1-dht:
> (null): failed to lookup the file on GLUSTER1-dht [Stale file handle]
> [2016-08-11 04:35:21.213345] W [fuse-bridge.c:2227:fuse_readv_cbk]
> 0-glusterfs-fuse: 15156910: READ => -1 
> gfid=7919f4a0-125c-4b11-b5c9-fb50cc195c43
> fd=0x7f00a80bf6d0 (Stale file handle)
> [2016-08-11 04:35:21.211516] W [MSGID: 108008]
> [afr-read-txn.c:244:afr_read_txn] 0-GLUSTER1-replicate-0: Unreadable
> subvolume -1 found with event generation 3 for gfid
> dfb8777a-7e8c-40ff-8faa-252beabba5f8. (Possible split-brain)
> 

Re: [Gluster-users] How to upgrade/re-install the OS on a gluster server?

[Dan Lavu]

Thanks Joe for the confirmation, cheers, have a good weekend.

On Thu, Aug 11, 2016 at 5:17 PM, Joe Julian <j...@julianfamily.org> wrote:

> On 08/11/2016 02:05 PM, Gandalf Corvotempesta wrote:
>
> Il 11 ago 2016 7:21 PM, "Dan Lavu" <d...@redhat.com> ha scritto:
> >
> > Is it possible? Looking at everything, it just seems like I need the
> content of the bricks and whatever is in /etc/glusterd and
> /var/lib/glusterd maintaining the same hostname, IP and the same Gluster
> version?
> >
> >
>
> Why not start from scratch and let gluster to heal when you add the
> upgraded node back to the cluster?
>
> Because "start from scratch" means changing hostnames. When you've got a
> strategy for naming hosts, assigning a hostname that doesn't fit that
> strategy breaks consistency. When you're managing hosts in 6 datacenters on
> 4 contenents, consistent naming is critical.
>
> Having consistent names makes automated deployment (or redeployment) much
> easier to code when you're using mgmt, saltstack, ansible, puppet or even
> chef. This is also the same reason I use consistent naming for brick
> directories.
>
> Gluster has never written replacement tools with same hostname and path as
> a possibility meaning that replace-brick doesn't work. Without being able
> to use replace-brick, self-heal doesn't know to heal that one single brick
> so heal...full is needed. Replace-brick is being changed to allow in-place
> replacement and solve that problem. Until then, Dan's process is perfectly
> reasonable and is the process that I use (more or less, I actually just
> template glusterd.info and rely on the sync process to fill the rest of
> /var/lib/glusterd).
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] How to upgrade/re-install the OS on a gluster server?

[Dan Lavu]

Is it possible? Looking at everything, it just seems like I need the
content of the bricks and whatever is in /etc/glusterd and
/var/lib/glusterd maintaining the same hostname, IP and the same Gluster
version?

Thanks,

Dan
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster Infiniband/RDMA Help

[Dan Lavu]

I've since ordered a different switch, the same manufacturer as the HBAs.
We have decided to rebuild the lab since we were having issues with oVirt
as well. We can disregard this, unless the issue is reproducable with the
new equipment, I believe it is equipment related.

On Thu, Aug 11, 2016 at 2:53 AM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

> Added Rafi, Raghavendra who work on RDMA
>
> On Mon, Aug 8, 2016 at 7:58 AM, Dan Lavu <d...@redhat.com> wrote:
>
>> Hello,
>>
>> I'm having some major problems with Gluster and oVirt, I've been ripping
>> my hair out with this, so if anybody can provide insight, that will be
>> fantastic. I've tried both transports TCP and RDMA... both are having
>> instability problems.
>>
>> So the first thing I'm running into, intermittently, on one specific
>> node, will get spammed with the following message;
>>
>> "[2016-08-08 00:42:50.837992] E [rpc-clnt.c:357:saved_frames_unwind]
>> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fb728b0f293]
>> (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7fb7288d73d1]
>> (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb7288d74ee] (-->
>> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fb7288d8d0e]
>> (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fb7288d9528] )
>> 0-vmdata1-client-0: forced unwinding frame type(GlusterFS 3.3)
>> op(WRITE(13)) called at 2016-08-08 00:42:43.620710 (xid=0x6800b)"
>>
>> Then the infiniband device will get bounced and VMs will get stuck.
>>
>> Another problem I'm seeing, once a day, or every two days, an oVirt node
>> will hang on gluster mounts. Issuing a df to check the mounts will just
>> stall, this occurs hourly if RDMA is used. I can log into the hypervisor
>> remount the gluster volumes most of the time.
>>
>> This is on Fedora 23; Gluster 3.8.1-1, the Infiniband gear is 40Gb/s QDR
>> Qlogic, using the ib_qib module, this configuration was working with our
>> old infinihost III. I couldn't get OFED to compile so all the infiniband
>> modules are Fedora installed.
>>
>> So a volume looks like the following, (please if there is anything I need
>> to adjust, the settings was pulled from several examples)
>>
>> Volume Name: vmdata_ha
>> Type: Replicate
>> Volume ID: 325a5fda-a491-4c40-8502-f89776a3c642
>> Status: Started
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp,rdma
>> Bricks:
>> Brick1: deadpool.ib.runlevelone.lan:/gluster/vmdata_ha
>> Brick2: spidey.ib.runlevelone.lan:/gluster/vmdata_ha
>> Brick3: groot.ib.runlevelone.lan:/gluster/vmdata_ha (arbiter)
>> Options Reconfigured:
>> performance.least-prio-threads: 4
>> performance.low-prio-threads: 16
>> performance.normal-prio-threads: 24
>> performance.high-prio-threads: 24
>> cluster.self-heal-window-size: 32
>> cluster.self-heal-daemon: on
>> performance.md-cache-timeout: 1
>> performance.cache-max-file-size: 2MB
>> performance.io-thread-count: 32
>> network.ping-timeout: 5
>> performance.write-behind-window-size: 4MB
>> performance.cache-size: 256MB
>> performance.cache-refresh-timeout: 10
>> server.allow-insecure: on
>> network.remote-dio: enable
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> performance.readdir-ahead: on
>> nfs.disable: on
>> config.transport: tcp,rdma
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>>
>> Volume Name: vmdata1
>> Type: Distribute
>> Volume ID: 3afefcb3-887c-4315-b9dc-f4e890f786eb
>> Status: Started
>> Number of Bricks: 2
>> Transport-type: tcp,rdma
>> Bricks:
>> Brick1: spidey.ib.runlevelone.lan:/gluster/vmdata1
>> Brick2: deadpool.ib.runlevelone.lan:/gluster/vmdata1
>> Options Reconfigured:
>> config.transport: tcp,rdma
>> network.remote-dio: enable
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>> nfs.disable: on
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> performance.readdir-ahead: on
>> server.allow-insecure: on
>> performance.stat-prefetch: off
>> performance.cache-refresh-timeout: 10
>> performance.cache-size: 256MB
>> performance.write-behind-window-size: 4MB
>> network.ping-timeout: 5
>> performance.io-thread-count: 32
>> performance.cache-max-file-size: 2MB
>> performance.md-cache-timeout: 1
>> performance.high-prio-threads: 24
>

[Gluster-users] Gluster Infiniband/RDMA Help

[Dan Lavu]

Hello,

I'm having some major problems with Gluster and oVirt, I've been ripping my
hair out with this, so if anybody can provide insight, that will be
fantastic. I've tried both transports TCP and RDMA... both are having
instability problems.

So the first thing I'm running into, intermittently, on one specific node,
will get spammed with the following message;

"[2016-08-08 00:42:50.837992] E [rpc-clnt.c:357:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7fb728b0f293] (-->
/lib64/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7fb7288d73d1] (-->
/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb7288d74ee] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)[0x7fb7288d8d0e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_notify+0x88)[0x7fb7288d9528] )
0-vmdata1-client-0: forced unwinding frame type(GlusterFS 3.3)
op(WRITE(13)) called at 2016-08-08 00:42:43.620710 (xid=0x6800b)"

Then the infiniband device will get bounced and VMs will get stuck.

Another problem I'm seeing, once a day, or every two days, an oVirt node
will hang on gluster mounts. Issuing a df to check the mounts will just
stall, this occurs hourly if RDMA is used. I can log into the hypervisor
remount the gluster volumes most of the time.

This is on Fedora 23; Gluster 3.8.1-1, the Infiniband gear is 40Gb/s QDR
Qlogic, using the ib_qib module, this configuration was working with our
old infinihost III. I couldn't get OFED to compile so all the infiniband
modules are Fedora installed.

So a volume looks like the following, (please if there is anything I need
to adjust, the settings was pulled from several examples)

Volume Name: vmdata_ha
Type: Replicate
Volume ID: 325a5fda-a491-4c40-8502-f89776a3c642
Status: Started
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp,rdma
Bricks:
Brick1: deadpool.ib.runlevelone.lan:/gluster/vmdata_ha
Brick2: spidey.ib.runlevelone.lan:/gluster/vmdata_ha
Brick3: groot.ib.runlevelone.lan:/gluster/vmdata_ha (arbiter)
Options Reconfigured:
performance.least-prio-threads: 4
performance.low-prio-threads: 16
performance.normal-prio-threads: 24
performance.high-prio-threads: 24
cluster.self-heal-window-size: 32
cluster.self-heal-daemon: on
performance.md-cache-timeout: 1
performance.cache-max-file-size: 2MB
performance.io-thread-count: 32
network.ping-timeout: 5
performance.write-behind-window-size: 4MB
performance.cache-size: 256MB
performance.cache-refresh-timeout: 10
server.allow-insecure: on
network.remote-dio: enable
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
storage.owner-gid: 36
storage.owner-uid: 36
performance.readdir-ahead: on
nfs.disable: on
config.transport: tcp,rdma
performance.stat-prefetch: off
cluster.eager-lock: enable

Volume Name: vmdata1
Type: Distribute
Volume ID: 3afefcb3-887c-4315-b9dc-f4e890f786eb
Status: Started
Number of Bricks: 2
Transport-type: tcp,rdma
Bricks:
Brick1: spidey.ib.runlevelone.lan:/gluster/vmdata1
Brick2: deadpool.ib.runlevelone.lan:/gluster/vmdata1
Options Reconfigured:
config.transport: tcp,rdma
network.remote-dio: enable
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
nfs.disable: on
storage.owner-gid: 36
storage.owner-uid: 36
performance.readdir-ahead: on
server.allow-insecure: on
performance.stat-prefetch: off
performance.cache-refresh-timeout: 10
performance.cache-size: 256MB
performance.write-behind-window-size: 4MB
network.ping-timeout: 5
performance.io-thread-count: 32
performance.cache-max-file-size: 2MB
performance.md-cache-timeout: 1
performance.high-prio-threads: 24
performance.normal-prio-threads: 24
performance.low-prio-threads: 16
performance.least-prio-threads: 4


/etc/glusterfs/glusterd.vol
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket,tcp
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option ping-timeout 0
option event-threads 1
#option rpc-auth-allow-insecure on
option transport.socket.bind-address 0.0.0.0
#   option transport.address-family inet6
#   option base-port 49152
end-volume

I think that's a good start, thank you so much for taking the time to look
at this. You can find me on freenode, nick side_control if you want to
chat, I'm GMT -5.

Cheers,

Dan
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users