Re: [Gluster-devel] fallocate behavior in glusterfs

[Pranith Kumar Karampuri]

On Wed, Jul 3, 2019 at 10:59 PM FNU Raghavendra Manjunath 
wrote:

>
>
> On Wed, Jul 3, 2019 at 3:28 AM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Wed, Jul 3, 2019 at 10:14 AM Ravishankar N 
>> wrote:
>>
>>>
>>> On 02/07/19 8:52 PM, FNU Raghavendra Manjunath wrote:
>>>
>>>
>>> Hi All,
>>>
>>> In glusterfs, there is an issue regarding the fallocate behavior. In
>>> short, if someone does fallocate from the mount point with some size that
>>> is greater than the available size in the backend filesystem where the file
>>> is present, then fallocate can fail with a subset of the required number of
>>> blocks allocated and then failing in the backend filesystem with ENOSPC
>>> error.
>>>
>>> The behavior of fallocate in itself is simlar to how it would have been
>>> on a disk filesystem (atleast xfs where it was checked). i.e. allocates
>>> subset of the required number of blocks and then fail with ENOSPC. And the
>>> file in itself would show the number of blocks in stat to be whatever was
>>> allocated as part of fallocate. Please refer [1] where the issue is
>>> explained.
>>>
>>> Now, there is one small difference between how the behavior is between
>>> glusterfs and xfs.
>>> In xfs after fallocate fails, doing 'stat' on the file shows the number
>>> of blocks that have been allocated. Whereas in glusterfs, the number of
>>> blocks is shown as zero which makes tools like "du" show zero consumption.
>>> This difference in behavior in glusterfs is because of libglusterfs on how
>>> it handles sparse files etc for calculating number of blocks (mentioned in
>>> [1])
>>>
>>> At this point I can think of 3 things on how to handle this.
>>>
>>> 1) Except for how many blocks are shown in the stat output for the file
>>> from the mount point (on which fallocate was done), the remaining behavior
>>> of attempting to allocate the requested size and failing when the
>>> filesystem becomes full is similar to that of XFS.
>>>
>>> Hence, what is required is to come up with a solution on how
>>> libglusterfs calculate blocks for sparse files etc (without breaking any of
>>> the existing components and features). This makes the behavior similar to
>>> that of backend filesystem. This might require its own time to fix
>>> libglusterfs logic without impacting anything else.
>>>
>>> I think we should just revert the commit
>>> b1a5fa55695f497952264e35a9c8eb2bbf1ec4c3 (BZ 817343) and see if it really
>>> breaks anything (or check whatever it breaks is something that we can live
>>> with). XFS speculative preallocation is not permanent and the extra space
>>> is freed up eventually. It can be sped up via procfs tunable:
>>> http://xfs.org/index.php/XFS_FAQ#Q:_How_can_I_speed_up_or_avoid_delayed_removal_of_speculative_preallocation.3F.
>>> We could also tune the allocsize option to a low value like 4k so that
>>> glusterfs quota is not affected.
>>>
>>> FWIW, ENOSPC is not the only fallocate problem in gluster because of
>>> 'iatt->ia_block' tweaking. It also breaks the --keep-size option (i.e. the
>>> FALLOC_FL_KEEP_SIZE flag in fallocate(2)) and reports incorrect du size.
>>>
>> Regards,
>>> Ravi
>>>
>>>
>>> OR
>>>
>>> 2) Once the fallocate fails in the backend filesystem, make posix xlator
>>> in the brick truncate the file to the previous size of the file before
>>> attempting fallocate. A patch [2] has been sent for this. But there is an
>>> issue with this when there are parallel writes and fallocate operations
>>> happening on the same file. It can lead to a data loss.
>>>
>>> a) statpre is obtained ===> before fallocate is attempted, get the stat
>>> hence the size of the file b) A parrallel Write fop on the same file that
>>> extends the file is successful c) Fallocate fails d) ftruncate truncates it
>>> to size given by statpre (i.e. the previous stat and the size obtained in
>>> step a)
>>>
>>> OR
>>>
>>> 3) Make posix check for available disk size before doing fallocate. i.e.
>>> in fallocate once posix gets the number of bytes to be allocated for the
>>> file from a particular offset, it checks whether so many bytes are
>>> available or not in the disk. If not, fail the fallocate fop with ENOSPC
>>> (without att

Re: [Gluster-devel] fallocate behavior in glusterfs

[Pranith Kumar Karampuri]

On Wed, Jul 3, 2019 at 10:14 AM Ravishankar N 
wrote:

>
> On 02/07/19 8:52 PM, FNU Raghavendra Manjunath wrote:
>
>
> Hi All,
>
> In glusterfs, there is an issue regarding the fallocate behavior. In
> short, if someone does fallocate from the mount point with some size that
> is greater than the available size in the backend filesystem where the file
> is present, then fallocate can fail with a subset of the required number of
> blocks allocated and then failing in the backend filesystem with ENOSPC
> error.
>
> The behavior of fallocate in itself is simlar to how it would have been on
> a disk filesystem (atleast xfs where it was checked). i.e. allocates subset
> of the required number of blocks and then fail with ENOSPC. And the file in
> itself would show the number of blocks in stat to be whatever was allocated
> as part of fallocate. Please refer [1] where the issue is explained.
>
> Now, there is one small difference between how the behavior is between
> glusterfs and xfs.
> In xfs after fallocate fails, doing 'stat' on the file shows the number of
> blocks that have been allocated. Whereas in glusterfs, the number of blocks
> is shown as zero which makes tools like "du" show zero consumption. This
> difference in behavior in glusterfs is because of libglusterfs on how it
> handles sparse files etc for calculating number of blocks (mentioned in [1])
>
> At this point I can think of 3 things on how to handle this.
>
> 1) Except for how many blocks are shown in the stat output for the file
> from the mount point (on which fallocate was done), the remaining behavior
> of attempting to allocate the requested size and failing when the
> filesystem becomes full is similar to that of XFS.
>
> Hence, what is required is to come up with a solution on how libglusterfs
> calculate blocks for sparse files etc (without breaking any of the existing
> components and features). This makes the behavior similar to that of
> backend filesystem. This might require its own time to fix libglusterfs
> logic without impacting anything else.
>
> I think we should just revert the commit
> b1a5fa55695f497952264e35a9c8eb2bbf1ec4c3 (BZ 817343) and see if it really
> breaks anything (or check whatever it breaks is something that we can live
> with). XFS speculative preallocation is not permanent and the extra space
> is freed up eventually. It can be sped up via procfs tunable:
> http://xfs.org/index.php/XFS_FAQ#Q:_How_can_I_speed_up_or_avoid_delayed_removal_of_speculative_preallocation.3F.
> We could also tune the allocsize option to a low value like 4k so that
> glusterfs quota is not affected.
>
> FWIW, ENOSPC is not the only fallocate problem in gluster because of
> 'iatt->ia_block' tweaking. It also breaks the --keep-size option (i.e. the
> FALLOC_FL_KEEP_SIZE flag in fallocate(2)) and reports incorrect du size.
>
Regards,
> Ravi
>
>
> OR
>
> 2) Once the fallocate fails in the backend filesystem, make posix xlator
> in the brick truncate the file to the previous size of the file before
> attempting fallocate. A patch [2] has been sent for this. But there is an
> issue with this when there are parallel writes and fallocate operations
> happening on the same file. It can lead to a data loss.
>
> a) statpre is obtained ===> before fallocate is attempted, get the stat
> hence the size of the file b) A parrallel Write fop on the same file that
> extends the file is successful c) Fallocate fails d) ftruncate truncates it
> to size given by statpre (i.e. the previous stat and the size obtained in
> step a)
>
> OR
>
> 3) Make posix check for available disk size before doing fallocate. i.e.
> in fallocate once posix gets the number of bytes to be allocated for the
> file from a particular offset, it checks whether so many bytes are
> available or not in the disk. If not, fail the fallocate fop with ENOSPC
> (without attempting it on the backend filesystem).
>
> There still is a probability of a parallel write happening while this
> fallocate is happening and by the time falllocate system call is attempted
> on the disk, the available space might have been less than what was
> calculated before fallocate.
> i.e. following things can happen
>
>  a) statfs ===> get the available space of the backend filesystem
>  b) a parallel write succeeds and extends the file
>  c) fallocate is attempted assuming there is sufficient space in the
> backend
>
> While the above situation can arise, I think we are still fine. Because
> fallocate is attempted from the offset received in the fop. So,
> irrespective of whether write extended the file or not, the fallocate
> itself will be attempted for so many bytes from the offset which we found
> to be available by getting statfs information.
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1724754#c3
> [2] https://review.gluster.org/#/c/glusterfs/+/22969/
>
>
option 2) will affect performance if we have to serialize all the data
operations on the file.
option 3) can still lead to the same problem we 

Re: [Gluster-devel] making frame->root->unique more effective in debugging hung frames

[Pranith Kumar Karampuri]

On Sat, May 25, 2019 at 10:22 AM Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Fri, May 24, 2019 at 10:57 PM FNU Raghavendra Manjunath <
> rab...@redhat.com> wrote:
>
>>
>> The idea looks OK. One of the things that probably need to be considered
>> (more of an implementation detail though) is how to generate
>> frame->root->unique.
>>
>> Because, for fuse, frame->root->unique is obtained by finh->unique which
>> IIUC is got from the incoming fop from kernel itself.
>> For protocol/server IIUC frame->root->unique is got from req->xit of the
>> rpc request, which itself is obtained from transport->xid of the
>> rpc_transport_t structure (and from my understanding, the transport->xid is
>> just incremented by everytime a
>> new rpc request is created).
>>
>> Overall the suggestion looks fine though.
>>
>
> I am planning to do the same thing transport->xid does. I will send out
> the patch
>

https://review.gluster.org/c/glusterfs/+/22773


>
>> Regards,
>> Raghavendra
>>
>>
>> On Fri, May 24, 2019 at 2:27 AM Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>>> Hi,
>>> At the moment new stack doesn't populate frame->root->unique in
>>> all cases. This makes it difficult to debug hung frames by examining
>>> successive state dumps. Fuse and server xlator populate it whenever they
>>> can, but other xlators won't be able to assign one when they need to create
>>> a new frame/stack. Is it okay to change create_frame() code to always
>>> populate it with an increasing number for this purpose?
>>>  I checked both fuse and server xlator use it only in gf_log() so it
>>> doesn't seem like there is any other link between frame->root->unique and
>>> the functionality of fuse, server xlators.
>>>   Do let me know if I missed anything before sending this change.
>>>
>>> --
>>> Pranith
>>> ___
>>>
>>> Community Meeting Calendar:
>>>
>>> APAC Schedule -
>>> Every 2nd and 4th Tuesday at 11:30 AM IST
>>> Bridge: https://bluejeans.com/836554017
>>>
>>> NA/EMEA Schedule -
>>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>>> Bridge: https://bluejeans.com/486278655
>>>
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
>
> --
> Pranith
>


-- 
Pranith
___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] making frame->root->unique more effective in debugging hung frames

[Pranith Kumar Karampuri]

On Fri, May 24, 2019 at 10:57 PM FNU Raghavendra Manjunath <
rab...@redhat.com> wrote:

>
> The idea looks OK. One of the things that probably need to be considered
> (more of an implementation detail though) is how to generate
> frame->root->unique.
>
> Because, for fuse, frame->root->unique is obtained by finh->unique which
> IIUC is got from the incoming fop from kernel itself.
> For protocol/server IIUC frame->root->unique is got from req->xit of the
> rpc request, which itself is obtained from transport->xid of the
> rpc_transport_t structure (and from my understanding, the transport->xid is
> just incremented by everytime a
> new rpc request is created).
>
> Overall the suggestion looks fine though.
>

I am planning to do the same thing transport->xid does. I will send out the
patch


> Regards,
> Raghavendra
>
>
> On Fri, May 24, 2019 at 2:27 AM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>> Hi,
>> At the moment new stack doesn't populate frame->root->unique in
>> all cases. This makes it difficult to debug hung frames by examining
>> successive state dumps. Fuse and server xlator populate it whenever they
>> can, but other xlators won't be able to assign one when they need to create
>> a new frame/stack. Is it okay to change create_frame() code to always
>> populate it with an increasing number for this purpose?
>>  I checked both fuse and server xlator use it only in gf_log() so it
>> doesn't seem like there is any other link between frame->root->unique and
>> the functionality of fuse, server xlators.
>>   Do let me know if I missed anything before sending this change.
>>
>> --
>> Pranith
>> ___
>>
>> Community Meeting Calendar:
>>
>> APAC Schedule -
>> Every 2nd and 4th Tuesday at 11:30 AM IST
>> Bridge: https://bluejeans.com/836554017
>>
>> NA/EMEA Schedule -
>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>> Bridge: https://bluejeans.com/486278655
>>
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>>

-- 
Pranith
___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] making frame->root->unique more effective in debugging hung frames

[Pranith Kumar Karampuri]

Hi,
At the moment new stack doesn't populate frame->root->unique in all
cases. This makes it difficult to debug hung frames by examining successive
state dumps. Fuse and server xlator populate it whenever they can, but
other xlators won't be able to assign one when they need to create a new
frame/stack. Is it okay to change create_frame() code to always populate it
with an increasing number for this purpose?
 I checked both fuse and server xlator use it only in gf_log() so it
doesn't seem like there is any other link between frame->root->unique and
the functionality of fuse, server xlators.
  Do let me know if I missed anything before sending this change.

-- 
Pranith
___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/836554017

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/486278655

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] questions on callstubs and "link-count" in index translator

[Pranith Kumar Karampuri]

On Fri, Apr 26, 2019 at 10:55 PM Junsong Li  wrote:

> Hello list,
>
>
>
> I have a couple of questions on index translator implementation.
>
>- Why does gluster need callstub and a different worker queue (and
>thread) to process those call stubs? Is it just to lower the priority of
>fops of internal inodes?
>
> It is to make sure parallel [f]xattrops on same inode doesn't happen.
Index xlator has to maintain indices in the internal directories based on
whether or not the xattrs it tracks on the file are non-zero.  When we
allow parallel [f]xattrops, callbacks of the fops may not reach index
xlator in the order they are wound to lower xlators. This can lead to index
xlator not maintain indices correctly.


>
>-
>- What’s the purpose of “link-count” in xdata? It’s being used only in
>index_fstat and index_lookup. I see sometimes the key is assigned 0/1 after
>callback, and sometimes AFR uses it to store flag GF_XATTROP_INDEX_COUNT.
>Is the code purposely reusing the key?
>
> Thanks,
>
> Junsong
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Release 6.1: Expected tagging on April 10th

[Pranith Kumar Karampuri]

On Wed, Apr 17, 2019 at 8:53 AM Amar Tumballi Suryanarayan <
atumb...@redhat.com> wrote:

> My take is, lets disable sdfs for 6.1 (we also have issues with its
> performance anyways). We will fix it properly by 6.2 or 7.0. Continue with
> marking sdfs-sanity.t tests as bad in that case.
>

It is better to revert the patch like Atin mentioned. The patch that was
merged was intended to find the existing bugs with lk-owner before getting
merged. We thought there were no bugs when regression passed. But that is
not the case. So better to revert, fix other bugs found by this patch and
then get it merged again.


>
> -Amar
>
> On Wed, Apr 17, 2019 at 8:04 AM Atin Mukherjee 
> wrote:
>
>>
>>
>> On Wed, Apr 17, 2019 at 12:33 AM Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>>>
>>>
>>> On Tue, Apr 16, 2019 at 10:27 PM Atin Mukherjee 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Apr 16, 2019 at 9:19 PM Atin Mukherjee 
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan 
>>>>> wrote:
>>>>>
>>>>>> Status: Tagging pending
>>>>>>
>>>>>> Waiting on patches:
>>>>>> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>>>>>>   https://review.gluster.org/c/glusterfs/+/22579
>>>>>
>>>>>
>>>>> The regression doesn't pass for the mainline patch. I believe master
>>>>> is broken now. With latest master sdfs-sanity.t always fail. We either 
>>>>> need
>>>>> to fix it or mark it as bad test.
>>>>>
>>>>
>>>> commit 3883887427a7f2dc458a9773e05f7c8ce8e62301 (HEAD)
>>>> Author: Pranith Kumar K 
>>>> Date:   Mon Apr 1 11:14:56 2019 +0530
>>>>
>>>>features/locks: error-out {inode,entry}lk fops with all-zero lk-owner
>>>>
>>>>Problem:
>>>>Sometimes we find that developers forget to assign lk-owner for an
>>>>inodelk/entrylk/lk before writing code to wind these fops. locks
>>>>xlator at the moment allows this operation. This leads to multiple
>>>>threads in the same client being able to get locks on the inode
>>>>because lk-owner is same and transport is same. So isolation
>>>>with locks can't be achieved.
>>>>
>>>>Fix:
>>>>Disallow locks with lk-owner zero.
>>>>
>>>>fixes bz#1624701
>>>>Change-Id: I1c816280cffd150ebb392e3dcd4d21007cdd767f
>>>>Signed-off-by: Pranith Kumar K 
>>>>
>>>> With the above commit sdfs-sanity.t started failing. But when I looked
>>>> at the last regression vote at
>>>> https://build.gluster.org/job/centos7-regression/5568/consoleFull I
>>>> saw it voted back positive but the bell rang when I saw the overall
>>>> regression took less than 2 hours and when I opened the regression link I
>>>> saw the test actually failed but still this job voted back +1 at gerrit.
>>>>
>>>> *Deepshika* - *This is a bad CI bug we have now and have to be
>>>> addressed at earliest. Please take a look at
>>>> https://build.gluster.org/job/centos7-regression/5568/consoleFull
>>>> <https://build.gluster.org/job/centos7-regression/5568/consoleFull> and
>>>> investigate why the regression vote wasn't negative.*
>>>>
>>>> Pranith - I request you to investigate on the sdfs-sanity.t failure
>>>> because of this patch.
>>>>
>>>
>>> sdfs is supposed to serialize entry fops by doing entrylk, but all the
>>> locks are being done with all-zero lk-owner. In essence sdfs doesn't
>>> achieve its goal of mutual exclusion when conflicting operations are
>>> executed by same client because two locks on same entry with same
>>> all-zero-owner will get locks. The patch which lead to sdfs-sanity.t
>>> failure treats inodelk/entrylk/lk fops with all-zero lk-owner as Invalid
>>> request to prevent these kinds of bugs. So it exposed the bug in sdfs. I
>>> sent a fix for sdfs @ https://review.gluster.org/#/c/glusterfs/+/22582
>>>
>>
>> Since this patch hasn't passed the regression and now that I see
>> tests/bugs/replicate/bug-1386188-sbrain-fav-child.t hanging and timing out
>> in the latest nightly regression runs because of the above commit (teste

Re: [Gluster-devel] test failure reports for last 15 days

[Pranith Kumar Karampuri]

perse-0, start=0, len=0, pid=0}
>
> This is a fragment of the client log:
>
> [2019-04-11 08:22:20.381398]:++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 66 kill_brick patchy jahernan /d/backends/patchy2 ++
> [2019-04-11 08:22:20.675938] I [MSGID: 114018]
> [client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
> patchy-client-1. Client process will keep trying to connect to glusterd
> until brick's port is available
> [2019-04-11 08:22:21.674772] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (6). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:22.532646]:++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 67 kill_brick patchy jahernan /d/backends/patchy3 ++
> [2019-04-11 08:22:23.691171] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (8). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:23.710420] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
> to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
> [2019-04-11 08:22:23.791635] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:24.460529] I [MSGID: 114018]
> [client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
> patchy-client-1. Client process will keep trying to connect to glusterd
> until brick's port is available
> [2019-04-11 08:22:24.628478]:++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 68 5 online_brick_count ++
> [2019-04-11 08:22:26.097092]:++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o
> 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++
> [2019-04-11 08:22:26.333740]:++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 71 ec_test_make ++
> [2019-04-11 08:22:27.719299] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
> to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
> [2019-04-11 08:22:27.840342] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
>
> The problem happens for two things:
>
> 1. Brick 0 gets disconnected randomly (apparently), but the server side is
> not aware of these disconnections. This causes that at 08:22:24.460529, the
> client has already sent a successful INODELK request to brick 0. At this
> point the connection is broken on the client side, but server side doesn't
> get any notification, so it doesn't clear the locks.
>
2. When client reconnects at 08:22:27.719299, a new connection is created,
> and the servers does see this new connection (it creates a new client_t
> structure). Then the client sends the unlock request, which fails on brick
> 0 because locks xlators checks if the client is the same by comparing the
> pointers, but they are different because of the reconnection. So the lock
> is not unlocked and remains there, blocking all future inodelk requests.
>
> The first problem is why the client gets disconnected and the server
> doesn't get any notification. The script is stopping bricks 2 and 3 when
> this happens. Brick 0 shouldn't fail here. It seems related to the
>
> The second problem is that when we receive a new connection from a client
> we already consider connected, we don't cleanup the old connection, which
> should take care of the stale locks.
>
> The third problem is that locks xlator is checking if the client is the
> same by comparing pointers of client_t structs instead of comparing
> client_uid field, which remains the same.
>
> Adding +Raghavendra Gowdappa , +Pranith Kumar
> Karampuri , +Krutika Dhananjay 
> , +Shyam Ranganathan  and +Amar Tumballi
>  to help me identify why this is happening and
> what's the best way to solve it.
>

If server gets disconnect notification, then everything will be solved. I
think we need to find RC for that. Were you able to recreate it locally
even if it is happening once in a while that is fine.


>
> Xavi
>
>
>> Xavi
>>
>> ./tests/basic/uss.t 15  ==> happens in both brick mux and non
>>> brick mux runs, test just simply times out. Needs urgent analysis.
>>> ./

Re: [Gluster-devel] Issue with posix locks

[Pranith Kumar Karampuri]

On Sun, Mar 31, 2019 at 11:29 PM Soumya Koduri  wrote:

>
>
> On 3/29/19 11:55 PM, Xavi Hernandez wrote:
> > Hi all,
> >
> > there is one potential problem with posix locks when used in a
> > replicated or dispersed volume.
> >
> > Some background:
> >
> > Posix locks allow any process to lock a region of a file multiple times,
> > but a single unlock on a given region will release all previous locks.
> > Locked regions can be different for each lock request and they can
> > overlap. The resulting lock will cover the union of all locked regions.
> > A single unlock (the region doesn't necessarily need to match any of the
> > ranges used for locking) will create a "hole" in the currently locked
> > region, independently of how many times a lock request covered that
> region.
> >
> > For this reason, the locks xlator simply combines the locked regions
> > that are requested, but it doesn't track each individual lock range.
> >
> > Under normal circumstances this works fine. But there are some cases
> > where this behavior is not sufficient. For example, suppose we have a
> > replica 3 volume with quorum = 2. Given the special nature of posix
> > locks, AFR sends the lock request sequentially to each one of the
> > bricks, to avoid that conflicting lock requests from other clients could
> > require to unlock an already locked region on the client that has not
> > got enough successful locks (i.e. quorum). An unlock here not only would
> > cancel the current lock request. It would also cancel any previously
> > acquired lock.
> >
>
> I may not have fully understood, please correct me. AFAIU, lk xlator
> merges locks only if both the lk-owner and the client opaque matches.
>
> In the case which you have mentioned above, considering clientA acquired
> locks on majority of quorum (say nodeA and nodeB) and clientB on nodeC
> alone- clientB now has to unlock/cancel the lock it acquired on nodeC.
>
> You are saying the it could pose a problem if there were already
> successful locks taken by clientB for the same region which would get
> unlocked by this particular unlock request..right?
>
> Assuming the previous locks acquired by clientB are shared (otherwise
> clientA wouldn't have got granted lock for the same region on nodeA &
> nodeB), they would still hold true on nodeA & nodeB  as the unlock
> request was sent to only nodeC. Since the majority of quorum nodes still
> hold the locks by clientB, this isn't serious issue IMO.
>
> I haven't looked into heal part but would like to understand if this is
> really an issue in normal scenarios as well.
>

This is how I understood the code. Consider the following case:
Nodes A, B, C have locks with start and end offsets: 5-15 from mount-1 and
lock-range 2-3 from mount-2.
If mount-1 requests nonblocking lock with lock-range 1-7 and in parallel
lets say mount-2 issued unlock of 2-3 as well.

nodeA got unlock from mount-2 with range 2-3 then lock from mount-1 with
range 1-7, so the lock is granted and merged to give 1-15
nodeB got lock from mount-1 with range 1-7 before unlock of 2-3 which leads
to EAGAIN which will trigger unlocks on granted lock in mount-1 which will
end up doing unlock of 1-7 on nodeA leading to lock-range 8-15 instead of
the original 5-15 on nodeA. Whereas nodeB and nodeC will have range 5-15.

Let me know if my understanding is wrong.


> Thanks,
> Soumya
>
> > However, when something goes wrong (a brick dies during a lock request,
> > or there's a network partition or some other weird situation), it could
> > happen that even using sequential locking, only one brick succeeds the
> > lock request. In this case, AFR should cancel the previous lock (and it
> > does), but this also cancels any previously acquired lock on that
> > region, which is not good.
> >
> > A similar thing can happen if we try to recover (heal) posix locks that
> > were active after a brick has been disconnected (for any reason) and
> > then reconnected.
> >
> > To fix all these situations we need to change the way posix locks are
> > managed by locks xlator. One possibility would be to embed the lock
> > request inside an inode transaction using inodelk. Since inodelks do not
> > suffer this problem, the follwing posix lock could be sent safely.
> > However this implies an additional network request, which could cause
> > some performance impact. Eager-locking could minimize the impact in some
> > cases. However this approach won't work for lock recovery after a
> > disconnect.
> >
> > Another possibility is to send a special partial posix lock request
> > which won't be immediately merged with already existing locks once
> > granted. An additional confirmation request of the partial posix lock
> > will be required to fully grant the current lock and merge it with the
> > existing ones. This requires a new network request, which will add
> > latency, and makes everything more complex since there would be more
> > combinations of states in which something could fail.
> >
> > So I 

Re: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks

[Pranith Kumar Karampuri]

On Wed, Mar 27, 2019 at 8:38 PM Xavi Hernandez  wrote:

> On Wed, Mar 27, 2019 at 2:20 PM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez 
>> wrote:
>>
>>> On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri <
>>> pkara...@redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez 
>>>> wrote:
>>>>
>>>>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa <
>>>>> rgowd...@redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Raghavendra,
>>>>>>>
>>>>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa <
>>>>>>> rgowd...@redhat.com> wrote:
>>>>>>>
>>>>>>>> All,
>>>>>>>>
>>>>>>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount
>>>>>>>> through which those locks are held disconnects from bricks/server. This
>>>>>>>> helps Glusterfs to not run into a stale lock problem later (For eg., if
>>>>>>>> application unlocks while the connection was still down). However, this
>>>>>>>> means the lock is no longer exclusive as other applications/clients can
>>>>>>>> acquire the same lock. To communicate that locks are no longer valid, 
>>>>>>>> we
>>>>>>>> are planning to mark the fd (which has POSIX locks) bad on a 
>>>>>>>> disconnect so
>>>>>>>> that any future operations on that fd will fail, forcing the 
>>>>>>>> application to
>>>>>>>> re-open the fd and re-acquire locks it needs [1].
>>>>>>>>
>>>>>>>
>>>>>>> Wouldn't it be better to retake the locks when the brick is
>>>>>>> reconnected if the lock is still in use ?
>>>>>>>
>>>>>>
>>>>>> There is also  a possibility that clients may never reconnect. That's
>>>>>> the primary reason why bricks assume the worst (client will not 
>>>>>> reconnect)
>>>>>> and cleanup the locks.
>>>>>>
>>>>>
>>>>> True, so it's fine to cleanup the locks. I'm not saying that locks
>>>>> shouldn't be released on disconnect. The assumption is that if the client
>>>>> has really died, it will also disconnect from other bricks, who will
>>>>> release the locks. So, eventually, another client will have enough quorum
>>>>> to attempt a lock that will succeed. In other words, if a client gets
>>>>> disconnected from too many bricks simultaneously (loses Quorum), then that
>>>>> client can be considered as bad and can return errors to the application.
>>>>> This should also cause to release the locks on the remaining connected
>>>>> bricks.
>>>>>
>>>>> On the other hand, if the disconnection is very short and the client
>>>>> has not died, it will keep enough locked files (it has quorum) to avoid
>>>>> other clients to successfully acquire a lock. In this case, if the brick 
>>>>> is
>>>>> reconnected, all existing locks should be reacquired to recover the
>>>>> original state before the disconnection.
>>>>>
>>>>>
>>>>>>
>>>>>>> BTW, the referenced bug is not public. Should we open another bug to
>>>>>>> track this ?
>>>>>>>
>>>>>>
>>>>>> I've just opened up the comment to give enough context. I'll open a
>>>>>> bug upstream too.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Note that with AFR/replicate in picture we can prevent errors to
>>>>>>>> application as long as Quorum number of children "never ever" lost
>>>>>>>> connection with bricks after locks have been acquired. I am using the 
>>>>>>>> term
>>>>>>>> "never

Re: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks

[Pranith Kumar Karampuri]

On Wed, Mar 27, 2019 at 6:38 PM Xavi Hernandez  wrote:

> On Wed, Mar 27, 2019 at 1:13 PM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez 
>> wrote:
>>
>>> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa <
>>> rgowd...@redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez 
>>>> wrote:
>>>>
>>>>> Hi Raghavendra,
>>>>>
>>>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa <
>>>>> rgowd...@redhat.com> wrote:
>>>>>
>>>>>> All,
>>>>>>
>>>>>> Glusterfs cleans up POSIX locks held on an fd when the client/mount
>>>>>> through which those locks are held disconnects from bricks/server. This
>>>>>> helps Glusterfs to not run into a stale lock problem later (For eg., if
>>>>>> application unlocks while the connection was still down). However, this
>>>>>> means the lock is no longer exclusive as other applications/clients can
>>>>>> acquire the same lock. To communicate that locks are no longer valid, we
>>>>>> are planning to mark the fd (which has POSIX locks) bad on a disconnect 
>>>>>> so
>>>>>> that any future operations on that fd will fail, forcing the application 
>>>>>> to
>>>>>> re-open the fd and re-acquire locks it needs [1].
>>>>>>
>>>>>
>>>>> Wouldn't it be better to retake the locks when the brick is
>>>>> reconnected if the lock is still in use ?
>>>>>
>>>>
>>>> There is also  a possibility that clients may never reconnect. That's
>>>> the primary reason why bricks assume the worst (client will not reconnect)
>>>> and cleanup the locks.
>>>>
>>>
>>> True, so it's fine to cleanup the locks. I'm not saying that locks
>>> shouldn't be released on disconnect. The assumption is that if the client
>>> has really died, it will also disconnect from other bricks, who will
>>> release the locks. So, eventually, another client will have enough quorum
>>> to attempt a lock that will succeed. In other words, if a client gets
>>> disconnected from too many bricks simultaneously (loses Quorum), then that
>>> client can be considered as bad and can return errors to the application.
>>> This should also cause to release the locks on the remaining connected
>>> bricks.
>>>
>>> On the other hand, if the disconnection is very short and the client has
>>> not died, it will keep enough locked files (it has quorum) to avoid other
>>> clients to successfully acquire a lock. In this case, if the brick is
>>> reconnected, all existing locks should be reacquired to recover the
>>> original state before the disconnection.
>>>
>>>
>>>>
>>>>> BTW, the referenced bug is not public. Should we open another bug to
>>>>> track this ?
>>>>>
>>>>
>>>> I've just opened up the comment to give enough context. I'll open a bug
>>>> upstream too.
>>>>
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Note that with AFR/replicate in picture we can prevent errors to
>>>>>> application as long as Quorum number of children "never ever" lost
>>>>>> connection with bricks after locks have been acquired. I am using the 
>>>>>> term
>>>>>> "never ever" as locks are not healed back after re-connection and hence
>>>>>> first disconnect would've marked the fd bad and the fd remains so even
>>>>>> after re-connection happens. So, its not just Quorum number of children
>>>>>> "currently online", but Quorum number of children "never having
>>>>>> disconnected with bricks after locks are acquired".
>>>>>>
>>>>>
>>>>> I think this requisite is not feasible. In a distributed file system,
>>>>> sooner or later all bricks will be disconnected. It could be because of
>>>>> failures or because an upgrade is done, but it will happen.
>>>>>
>>>>> The difference here is how long are fd's kept open. If applications
>>>>> open and close files frequently enough 

Re: [Gluster-devel] [Gluster-users] POSIX locks and disconnections between clients and bricks

[Pranith Kumar Karampuri]

On Wed, Mar 27, 2019 at 5:13 PM Xavi Hernandez  wrote:

> On Wed, Mar 27, 2019 at 11:52 AM Raghavendra Gowdappa 
> wrote:
>
>>
>>
>> On Wed, Mar 27, 2019 at 12:56 PM Xavi Hernandez 
>> wrote:
>>
>>> Hi Raghavendra,
>>>
>>> On Wed, Mar 27, 2019 at 2:49 AM Raghavendra Gowdappa <
>>> rgowd...@redhat.com> wrote:
>>>
 All,

 Glusterfs cleans up POSIX locks held on an fd when the client/mount
 through which those locks are held disconnects from bricks/server. This
 helps Glusterfs to not run into a stale lock problem later (For eg., if
 application unlocks while the connection was still down). However, this
 means the lock is no longer exclusive as other applications/clients can
 acquire the same lock. To communicate that locks are no longer valid, we
 are planning to mark the fd (which has POSIX locks) bad on a disconnect so
 that any future operations on that fd will fail, forcing the application to
 re-open the fd and re-acquire locks it needs [1].

>>>
>>> Wouldn't it be better to retake the locks when the brick is reconnected
>>> if the lock is still in use ?
>>>
>>
>> There is also  a possibility that clients may never reconnect. That's the
>> primary reason why bricks assume the worst (client will not reconnect) and
>> cleanup the locks.
>>
>
> True, so it's fine to cleanup the locks. I'm not saying that locks
> shouldn't be released on disconnect. The assumption is that if the client
> has really died, it will also disconnect from other bricks, who will
> release the locks. So, eventually, another client will have enough quorum
> to attempt a lock that will succeed. In other words, if a client gets
> disconnected from too many bricks simultaneously (loses Quorum), then that
> client can be considered as bad and can return errors to the application.
> This should also cause to release the locks on the remaining connected
> bricks.
>
> On the other hand, if the disconnection is very short and the client has
> not died, it will keep enough locked files (it has quorum) to avoid other
> clients to successfully acquire a lock. In this case, if the brick is
> reconnected, all existing locks should be reacquired to recover the
> original state before the disconnection.
>
>
>>
>>> BTW, the referenced bug is not public. Should we open another bug to
>>> track this ?
>>>
>>
>> I've just opened up the comment to give enough context. I'll open a bug
>> upstream too.
>>
>>
>>>
>>>

 Note that with AFR/replicate in picture we can prevent errors to
 application as long as Quorum number of children "never ever" lost
 connection with bricks after locks have been acquired. I am using the term
 "never ever" as locks are not healed back after re-connection and hence
 first disconnect would've marked the fd bad and the fd remains so even
 after re-connection happens. So, its not just Quorum number of children
 "currently online", but Quorum number of children "never having
 disconnected with bricks after locks are acquired".

>>>
>>> I think this requisite is not feasible. In a distributed file system,
>>> sooner or later all bricks will be disconnected. It could be because of
>>> failures or because an upgrade is done, but it will happen.
>>>
>>> The difference here is how long are fd's kept open. If applications open
>>> and close files frequently enough (i.e. the fd is not kept open more time
>>> than it takes to have more than Quorum bricks disconnected) then there's no
>>> problem. The problem can only appear on applications that open files for a
>>> long time and also use posix locks. In this case, the only good solution I
>>> see is to retake the locks on brick reconnection.
>>>
>>
>> Agree. But lock-healing should be done only by HA layers like AFR/EC as
>> only they know whether there are enough online bricks to have prevented any
>> conflicting lock. Protocol/client itself doesn't have enough information to
>> do that. If its a plain distribute, I don't see a way to heal locks without
>> loosing the property of exclusivity of locks.
>>
>
> Lock-healing of locks acquired while a brick was disconnected need to be
> handled by AFR/EC. However, locks already present at the moment of
> disconnection could be recovered by client xlator itself as long as the
> file has not been closed (which client xlator already knows).
>

What if another client (say mount-2) took locks at the time of disconnect
from mount-1 and modified the file and unlocked? client xlator doing the
heal may not be a good idea.


>
> Xavi
>
>
>> What I proposed is a short term solution. mid to long term solution
>> should be lock healing feature implemented in AFR/EC. In fact I had this
>> conversation with +Karampuri, Pranith  before
>> posting this msg to ML.
>>
>>
>>>
 However, this use case is not affected if the application don't acquire
 any POSIX locks. So, I am interested in knowing
 * whether your use cases use POSIX locks?
 * Is it feasible 

Re: [Gluster-devel] The state of lock heal and inodelk/entrylk heal ?

[Pranith Kumar Karampuri]

On Thu, Mar 21, 2019 at 11:50 AM Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Thu, Mar 21, 2019 at 9:15 AM Kinglong Mee 
> wrote:
>
>> Hello folks,
>>
>> Lock self healing (recovery or replay) is added at
>> https://review.gluster.org/#/c/glusterfs/+/2766/
>>
>> But it is removed at
>> https://review.gluster.org/#/c/glusterfs/+/12363/
>>
>> I found some information about it at
>> https://anoopcs.fedorapeople.org/Lock%20recovery%20in%20GlusterFS.txt
>>
>> I download the glusterfs source but cannot find any code about lock heal.
>>
>> I wanna know the state of lock heal, and inodelk/entrylk heal.
>>
>
> hi,
>  At the moment lock heal doesn't happen. It is an open item that needs
> to be fixed. It is a problem that is gaining interest recently and we are
> thinking of solving this problem. Did you get a chance to think about this
> problem and do you have any solutions?
>

I saw your question first @
https://review.gluster.org/#/c/glusterfs/+/22377/, let us continue the
conversation here so that everyone can get involved :-).


>
>
>>
>> Can someone show me some information about it?
>>
>> thanks,
>> Kinglong Mee
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
> --
> Pranith
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] The state of lock heal and inodelk/entrylk heal ?

[Pranith Kumar Karampuri]

On Thu, Mar 21, 2019 at 9:15 AM Kinglong Mee  wrote:

> Hello folks,
>
> Lock self healing (recovery or replay) is added at
> https://review.gluster.org/#/c/glusterfs/+/2766/
>
> But it is removed at
> https://review.gluster.org/#/c/glusterfs/+/12363/
>
> I found some information about it at
> https://anoopcs.fedorapeople.org/Lock%20recovery%20in%20GlusterFS.txt
>
> I download the glusterfs source but cannot find any code about lock heal.
>
> I wanna know the state of lock heal, and inodelk/entrylk heal.
>

hi,
 At the moment lock heal doesn't happen. It is an open item that needs
to be fixed. It is a problem that is gaining interest recently and we are
thinking of solving this problem. Did you get a chance to think about this
problem and do you have any solutions?


>
> Can someone show me some information about it?
>
> thanks,
> Kinglong Mee
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] POC- Distributed regression testing framework

[Pranith Kumar Karampuri]

On Thu, Oct 4, 2018 at 2:15 PM Xavi Hernandez  wrote:

> On Thu, Oct 4, 2018 at 9:47 AM Amar Tumballi  wrote:
>
>>
>>
>> On Thu, Oct 4, 2018 at 12:54 PM Xavi Hernandez 
>> wrote:
>>
>>> On Wed, Oct 3, 2018 at 11:57 AM Deepshikha Khandelwal <
>>> dkhan...@redhat.com> wrote:
>>>
 Hello folks,

 Distributed-regression job[1] is now a part of Gluster's
 nightly-master build pipeline. The following are the issues we have
 resolved since we started working on this:

 1) Collecting gluster logs from servers.
 2) Tests failed due to infra-related issues have been fixed.
 3) Time taken to run regression testing reduced to ~50-60 minutes.

 To get time down to 40 minutes needs your help!

 Currently, there is a test that is failing:

 tests/bugs/glusterd/optimized-basic-testcases-in-cluster.t

 This needs fixing first.

 There's a test that takes 14 minutes to complete -
 `tests/bugs/index/bug-1559004-EMLINK-handling.t`. A single test taking
 14 minutes is not something we can distribute. Can we look at how we
 can speed this up[2]? When this test fails, it is re-attempted,
 further increasing the time. This happens in the regular
 centos7-regression job as well.

>>>
>>> I made a change [1] to reduce the amount of time this tests needs. With
>>> this change the test completes in about 90 seconds. It would need some
>>> reviews from maintainers though.
>>>
>>> Do you want I send a patch with this change alone ?
>>>
>>> Xavi
>>>
>>> [1]
>>> https://review.gluster.org/#/c/glusterfs/+/19254/22/tests/bugs/index/bug-1559004-EMLINK-handling.t
>>>
>>>
>>
>> Yes please! It would be useful! We can merge it sooner that way!
>>
>
> Patch: https://review.gluster.org/21341
>

Merged!


>
>
>>
>> -Amar
>>
>>
>>>
 If you see any other issues, please file a bug[3].

 [1]: https://build.gluster.org/job/distributed-regression
 [2]: https://build.gluster.org/job/distributed-regression/264/console
 [3]:
 https://bugzilla.redhat.com/enter_bug.cgi?product=glusterfs=project-infrastructure

 Thanks,
 Deepshikha Khandelwal
 On Tue, Jun 26, 2018 at 9:02 AM Nigel Babu  wrote:
 >
 >
 >
 > On Mon, Jun 25, 2018 at 7:28 PM Amar Tumballi 
 wrote:
 >>
 >>
 >>
 >>> There are currently a few known issues:
 >>> * Not collecting the entire logs (/var/log/glusterfs) from servers.
 >>
 >>
 >> If I look at the activities involved with regression failures, this
 can wait.
 >
 >
 > Well, we can't debug the current failures without having the logs. So
 this has to be fixed first.
 >
 >>
 >>
 >>>
 >>> * A few tests fail due to infra-related issues like geo-rep tests.
 >>
 >>
 >> Please open bugs for this, so we can track them, and take it to
 closure.
 >
 >
 > These are failing due to infra reasons. Most likely subtle
 differences in the setup of these nodes vs our normal nodes. We'll only be
 able to debug them once we get the logs. I know the geo-rep ones are easy
 to fix. The playbook for setting up geo-rep correctly just didn't make it
 over to the playbook used for these images.
 >
 >>
 >>
 >>>
 >>> * Takes ~80 minutes with 7 distributed servers (targetting 60
 minutes)
 >>
 >>
 >> Time can change with more tests added, and also please plan to have
 number of server as 1 to n.
 >
 >
 > While the n is configurable, however it will be fixed to a single
 digit number for now. We will need to place *some* limitation somewhere or
 else we'll end up not being able to control our cloud bills.
 >
 >>
 >>
 >>>
 >>> * We've only tested plain regressions. ASAN and Valgrind are
 currently untested.
 >>
 >>
 >> Great to have it running not 'per patch', but as nightly, or weekly
 to start with.
 >
 >
 > This is currently not targeted until we phase out current regressions.
 >
 >>>
 >>>
 >>> Before bringing it into production, we'll run this job nightly and
 >>> watch it for a month to debug the other failures.
 >>>
 >>
 >> I would say, bring it to production sooner, say 2 weeks, and also
 plan to have the current regression as is with a special command like 'run
 regression in-one-machine' in gerrit (or something similar) with voting
 rights, so we can fall back to this method if something is broken in
 parallel testing.
 >>
 >> I have seen that regardless of amount of time we put some scripts in
 testing, the day we move to production, some thing would be broken. So, let
 that happen earlier than later, so it would help next release branching
 out. Don't want to be stuck for branching due to infra failures.
 >
 >
 > Having two regression jobs that can vote is going to cause more
 confusion than it's 

Re: [Gluster-devel] index_lookup segfault in glusterfsd brick process

[Pranith Kumar Karampuri]

On Wed, Oct 3, 2018 at 11:20 PM 김경표  wrote:

> Hello folks.
>
> Few days ago I found my EC(4+2) volume was degraded.
> I am using 3.12.13-1.el7.x86_64.
> One brick was down, below is bricklog
> I am suspicious loc->inode bug in index.c (see attached picture)
> In GDB, loc->inode is null
>
>> inode_find (loc->inode->table, loc->gfid);
>>
>
I see that loc->inode is coming from resolve_gfid() where the following
should have been executed.
  0 resolve_loc->inode = server_inode_new
(state->itable,

  1resolve_loc->gfid);

As per the log:
"[2018-09-29 13:22:36.536579] W [inode.c:680:inode_new]
(-->/usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xd048)
[0x7f9bd2494048]
-->/usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xc14d)
[0x7f9bd249314d] -->/lib64/libglusterfs.so.0(inode_new+0x8a) [0x
7f9be70900ba] ) 0-gluvol02-05-server: inode not found"

it indicates that inode-table is NULL. Is there a possibility to upload the
core somewhere for us to take a closer look?


> Thansk for Gluster Community!!!
>
> - kpkim
>
> --
> [2018-09-29 13:22:36.536532] W [inode.c:942:inode_find]
> (-->/usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xd01c)
> [0x7f9bd249401c]
> -->/usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xc638)
> [0x7f9bd2493638] -->/lib64/libglusterfs.so.0(inode_find+0x92) [
> 0x7f9be7090a82] ) 0-gluvol02-05-server: table not found
> [2018-09-29 13:22:36.536579] W [inode.c:680:inode_new]
> (-->/usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xd048)
> [0x7f9bd2494048]
> -->/usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xc14d)
> [0x7f9bd249314d] -->/lib64/libglusterfs.so.0(inode_new+0x8a) [0x
> 7f9be70900ba] ) 0-gluvol02-05-server: inode not found
> [2018-09-29 13:22:36.537568] W [inode.c:2305:inode_is_linked]
> (-->/usr/lib64/glusterfs/3.12.13/xlator/features/quota.so(+0x4fc6)
> [0x7f9bd2b1cfc6]
> -->/usr/lib64/glusterfs/3.12.13/xlator/features/index.so(+0x4bb9)
> [0x7f9bd2d43bb9] -->/lib64/libglusterfs.so.0(inode_is_linke
> d+0x8a) [0x7f9be70927ea] ) 0-gluvol02-05-index: inode not found
> pending frames:
> frame : type(0) op(18)
> frame : type(0) op(18)
> frame : type(0) op(28)
> --snip --
> frame : type(0) op(28)
> frame : type(0) op(28)
> frame : type(0) op(18)
> patchset: git://git.gluster.org/glusterfs.git
> signal received: 11
> time of crash:
> 2018-09-29 13:22:36
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.12.13
> /lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f9be70804c0]
> /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f9be708a3f4]
> /lib64/libc.so.6(+0x362f0)[0x7f9be56e02f0]
>
> /usr/lib64/glusterfs/3.12.13/xlator/features/index.so(+0x4bc4)[0x7f9bd2d43bc4]
>
> /usr/lib64/glusterfs/3.12.13/xlator/features/quota.so(+0x4fc6)[0x7f9bd2b1cfc6]
>
> /usr/lib64/glusterfs/3.12.13/xlator/debug/io-stats.so(+0x4e53)[0x7f9bd28eee53]
> /lib64/libglusterfs.so.0(default_lookup+0xbd)[0x7f9be70fddfd]
>
> /usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xc342)[0x7f9bd2493342]
>
> /usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xd048)[0x7f9bd2494048]
>
> /usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xd2c0)[0x7f9bd24942c0]
>
> /usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xc89e)[0x7f9bd249389e]
>
> /usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0xd354)[0x7f9bd2494354]
>
> /usr/lib64/glusterfs/3.12.13/xlator/protocol/server.so(+0x2f829)[0x7f9bd24b6829]
> /lib64/libgfrpc.so.0(rpcsvc_request_handler+0x96)[0x7f9be6e42246]
> /lib64/libpthread.so.0(+0x7e25)[0x7f9be5edfe25]
> /lib64/libc.so.6(clone+0x6d)[0x7f9be57a8bad]
>

> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Crash in glusterfs!!!

[Pranith Kumar Karampuri]

On Tue, Sep 25, 2018 at 2:17 PM ABHISHEK PALIWAL 
wrote:

> I don't have the step to reproduce, but its a race condition where it
> seems cleanup_and_exit() is accessing the data structure which are not yet
> initialised (as gluster is in starting phase), due to SIGTERM/SIGINT is
> sent in between.
>

But the crash happened inside exit() code for which will be in libc which
doesn't access any data structures in glusterfs.


>
> Regards,
> Abhishek
>
> On Mon, Sep 24, 2018 at 9:11 PM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Mon, Sep 24, 2018 at 5:16 PM ABHISHEK PALIWAL 
>> wrote:
>>
>>> Hi Pranith,
>>>
>>> As we know this problem is getting triggered at startup of the glusterd
>>> process when it received the SIGTERM.
>>>
>>> I think there is a problem in glusterfs code, if at startup someone sent
>>> the SIGTERM the exit handler should not be crash instead it should with
>>> some information.
>>>
>>> Could please let me know the possibility to fix it from glusterfs side?
>>>
>>
>> I am not as confident as you about the RC you provided. If you could give
>> the steps to re-create, I will be happy to confirm that the RC is correct
>> and then I will send out the fix.
>>
>>
>>>
>>> Regards,
>>> Abhishek
>>>
>>> On Mon, Sep 24, 2018 at 3:12 PM Pranith Kumar Karampuri <
>>> pkara...@redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Sep 24, 2018 at 2:09 PM ABHISHEK PALIWAL <
>>>> abhishpali...@gmail.com> wrote:
>>>>
>>>>> Could you please let me know about the bug in libc which you are
>>>>> talking.
>>>>>
>>>>
>>>> No, I mean, if you give the steps to reproduce, we will be able to pin
>>>> point if the issue is with libc or glusterfs.
>>>>
>>>>
>>>>>
>>>>> On Mon, Sep 24, 2018 at 2:01 PM Pranith Kumar Karampuri <
>>>>> pkara...@redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 24, 2018 at 1:57 PM ABHISHEK PALIWAL <
>>>>>> abhishpali...@gmail.com> wrote:
>>>>>>
>>>>>>> If you see the source code in cleanup_and_exit() we are getting the
>>>>>>> SIGSEGV crash when 'exit(0)' is triggered.
>>>>>>>
>>>>>>
>>>>>> yes, that is what I was mentioning earlier. It is crashing in libc.
>>>>>> So either there is a bug in libc (glusterfs actually found 1 bug so far 
>>>>>> in
>>>>>> libc, so I wouldn't rule out that possibility) or there is something that
>>>>>> is happening in glusterfs which is leading to the problem.
>>>>>> Valgrind/address-sanitizer would help find where the problem could be in
>>>>>> some cases, so before reaching out libc developers, it is better to 
>>>>>> figure
>>>>>> out where the problem is. Do you have steps to recreate it?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 24, 2018 at 1:41 PM Pranith Kumar Karampuri <
>>>>>>> pkara...@redhat.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 24, 2018 at 1:36 PM ABHISHEK PALIWAL <
>>>>>>>> abhishpali...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Sanju,
>>>>>>>>>
>>>>>>>>> Do you have any update on this?
>>>>>>>>>
>>>>>>>>
>>>>>>>> This seems to happen while the process is dying, in libc. I am not
>>>>>>>> completely sure if there is anything glusterfs is contributing to it 
>>>>>>>> from
>>>>>>>> the bt at the moment. Do you have any steps to re-create this problem? 
>>>>>>>> It
>>>>>>>> is probably better to run the steps with valgrind/address-sanitizer 
>>>>>>>> and see
>>>>>>>> if it points to the problem in glusterfs.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Abh

Re: [Gluster-devel] [Gluster-users] Crash in glusterfs!!!

[Pranith Kumar Karampuri]

On Mon, Sep 24, 2018 at 5:16 PM ABHISHEK PALIWAL 
wrote:

> Hi Pranith,
>
> As we know this problem is getting triggered at startup of the glusterd
> process when it received the SIGTERM.
>
> I think there is a problem in glusterfs code, if at startup someone sent
> the SIGTERM the exit handler should not be crash instead it should with
> some information.
>
> Could please let me know the possibility to fix it from glusterfs side?
>

I am not as confident as you about the RC you provided. If you could give
the steps to re-create, I will be happy to confirm that the RC is correct
and then I will send out the fix.


>
> Regards,
> Abhishek
>
> On Mon, Sep 24, 2018 at 3:12 PM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Mon, Sep 24, 2018 at 2:09 PM ABHISHEK PALIWAL 
>> wrote:
>>
>>> Could you please let me know about the bug in libc which you are talking.
>>>
>>
>> No, I mean, if you give the steps to reproduce, we will be able to pin
>> point if the issue is with libc or glusterfs.
>>
>>
>>>
>>> On Mon, Sep 24, 2018 at 2:01 PM Pranith Kumar Karampuri <
>>> pkara...@redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Sep 24, 2018 at 1:57 PM ABHISHEK PALIWAL <
>>>> abhishpali...@gmail.com> wrote:
>>>>
>>>>> If you see the source code in cleanup_and_exit() we are getting the
>>>>> SIGSEGV crash when 'exit(0)' is triggered.
>>>>>
>>>>
>>>> yes, that is what I was mentioning earlier. It is crashing in libc. So
>>>> either there is a bug in libc (glusterfs actually found 1 bug so far in
>>>> libc, so I wouldn't rule out that possibility) or there is something that
>>>> is happening in glusterfs which is leading to the problem.
>>>> Valgrind/address-sanitizer would help find where the problem could be in
>>>> some cases, so before reaching out libc developers, it is better to figure
>>>> out where the problem is. Do you have steps to recreate it?
>>>>
>>>>
>>>>>
>>>>> On Mon, Sep 24, 2018 at 1:41 PM Pranith Kumar Karampuri <
>>>>> pkara...@redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 24, 2018 at 1:36 PM ABHISHEK PALIWAL <
>>>>>> abhishpali...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Sanju,
>>>>>>>
>>>>>>> Do you have any update on this?
>>>>>>>
>>>>>>
>>>>>> This seems to happen while the process is dying, in libc. I am not
>>>>>> completely sure if there is anything glusterfs is contributing to it from
>>>>>> the bt at the moment. Do you have any steps to re-create this problem? It
>>>>>> is probably better to run the steps with valgrind/address-sanitizer and 
>>>>>> see
>>>>>> if it points to the problem in glusterfs.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Abhishek
>>>>>>>
>>>>>>> On Fri, Sep 21, 2018 at 4:07 PM ABHISHEK PALIWAL <
>>>>>>> abhishpali...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Sanju,
>>>>>>>>
>>>>>>>> Output of 't a a bt full'
>>>>>>>>
>>>>>>>> (gdb) t a a bt full
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thread 7 (LWP 1743):
>>>>>>>>
>>>>>>>> #0  0x3fffa3ea7e88 in __lll_lock_wait (futex=0x0, private=0) at
>>>>>>>> lowlevellock.c:43
>>>>>>>>
>>>>>>>> r4 = 128
>>>>>>>>
>>>>>>>> r7 = 0
>>>>>>>>
>>>>>>>> arg2 = 128
>>>>>>>>
>>>>>>>> r5 = 2
>>>>>>>>
>>>>>>>> r8 = 1
>>>>>>>>
>>>>>>>> r0 = 221
>>>>>>>>
>>>>>>>> r3 = 0
>>>>>>>>
>>>>>>>> r6 = 0
>>>>>>>>
>>>&

Re: [Gluster-devel] fedora smoke failure on 3.12

[Pranith Kumar Karampuri]

Thanks a lot!

On Wed, Sep 5, 2018 at 4:55 PM Anoop C S  wrote:

> On Wed, 2018-09-05 at 16:08 +0530, Anoop C S wrote:
> > On Wed, 2018-09-05 at 15:44 +0530, Pranith Kumar Karampuri wrote:
> > > It also failed on 4.1
> https://build.gluster.org/job/fedora-smoke/1665/console
> > >
> > > Looks like quite a few changes need to be ported for them to pass?
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1625489 was opened to track
> this change in behaviour.
>
> This is fixed now. You can recheck to verify that fedora-smoke is
> skipped(while voting) for release
> branches.
>
> > > On Wed, Sep 5, 2018 at 3:41 PM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
> > > > https://build.gluster.org/job/fedora-smoke/1668/console
> > > >
> > > > I think it is happening because of missing tirpc changes in
> configure.ac? There are a series
> > > > of
> > > > patches for libtirpc starting with
> https://review.gluster.org/c/glusterfs/+/19235, I am not
> > > > very
> > > > good at reading configure.ac except for the straightforward ones. I
> would need help making
> > > > sure
> > > > 3.12 works fine. Could one of you help please?
> > > >
> > > > --
> > > > Pranith
> > >
> > >
>
>

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] fedora smoke failure on 3.12

[Pranith Kumar Karampuri]

It also failed on 4.1
https://build.gluster.org/job/fedora-smoke/1665/console

Looks like quite a few changes need to be ported for them to pass?

On Wed, Sep 5, 2018 at 3:41 PM Pranith Kumar Karampuri 
wrote:

> https://build.gluster.org/job/fedora-smoke/1668/console
>
> I think it is happening because of missing tirpc changes in configure.ac?
> There are a series of patches for libtirpc starting with
> https://review.gluster.org/c/glusterfs/+/19235, I am not very good at
> reading configure.ac except for the straightforward ones. I would need
> help making sure 3.12 works fine. Could one of you help please?
>
> --
> Pranith
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] fedora smoke failure on 3.12

[Pranith Kumar Karampuri]

https://build.gluster.org/job/fedora-smoke/1668/console

I think it is happening because of missing tirpc changes in configure.ac?
There are a series of patches for libtirpc starting with
https://review.gluster.org/c/glusterfs/+/19235, I am not very good at
reading configure.ac except for the straightforward ones. I would need help
making sure 3.12 works fine. Could one of you help please?

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] remaining entry in gluster volume heal info command even after reboot

[Pranith Kumar Karampuri]

Looks like the test case is a bit involved and also has modifications
directly on the brick. Could you let us know if there is any reason to
touch the brick directly?

On Wed, Sep 5, 2018 at 2:53 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.z...@nokia-sbell.com> wrote:

> I will try to reproduce (reboot + ftest)and tell you later, but in
> following steps you can also simulate this issue locally, at least the
> remaining entry happened and the entry heal quit also because of all thee
> empty heald_sinks.(not sure if this is exactly the same from reboot+ftest
> reproduced one but I guess it should be the same)
>
>
>
>
>
> 1>   Stop client quorum by command “gluster v set 
> cluster.quorum-type none”
>
> 2>   Isolate sn-0 from sn-1 and sn-2
>
> iptables -I OUTPUT -d sn-1.local -j DROP
>
> iptables -I OUTPUT -d sn-2.local -j DROP
>
> iptables -I INPUT -s sn-2.local -j DROP
>
> iptables -I INPUT -s sn-1.local -j DROP
>
> 3>   Touch /mnt/export/testdir/common.txt on sn-0
>
> 4>   Touch /mnt/export/testidir/common.txt on sn-1
>
> 5>   On sn-1 node,Delete all  /mnt/bricks/export/brick/testdir/common.txt
> metadata until getfattr returns empty,
>
> setfattr -x  trusted.afr.dirty
> /mnt/bricks/export/brick/testdir/common.txt
>
> setfattr -x  trusted.afr.export-client-0
> /mnt/bricks/export/brick/testdir/common.txt
>
> setfattr -x  trusted.gfid  /mnt/bricks/export/brick/testdir/common.txt
>
> setfattr -x  trusted.gfid2path.53be37be7f01389d
> /mnt/bricks/export/brick/testdir/common.txt
>
> then getfattr returns empty
>
> [root@sn-1:/home/robot]
>
> # getfattr -m . -d -e hex  /mnt/bricks/export/brick/testdir/common.txt
>
> [root@sn-1:/home/robot]
>
> 6>   then delete the corresponding entry(common.txt) in
> /mnt/bricks/export/brick/.glusterfs/indices/xattrop/
>
> [root@sn-1:/home/robot]
>
> # rm -rf
> /mnt/bricks/export/brick/.glusterfs/indices/xattrop/d0d237f7-0c43-4828-8720-dfb3792fe5fb
>
> 7>Restore network on sn-0 node.
>
> iptables -D OUTPUT -d  sn-1.local  -j DROP
>
>   iptables -D OUTPUT -d sn-2.local  -j DROP
>
>iptables -D INPUT -s sn-1.local -j DROP
>
> iptables -D INPUT -s sn-2.local  -j DROP
>
> 7>   Do touch /mnt/export/testdir/common.txt on sn-0 node
>
> 8>   Gluster v heal export info will show following and keep for long time
>
> # gluster v heal export info
>
> Brick sn-0.local:/mnt/bricks/export/brick
>
> /testdir
>
> Status: Connected
>
> Number of entries: 1
>
>
>
> Brick sn-1.local:/mnt/bricks/export/brick
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick sn-2.local:/mnt/bricks/export/brick
>
> /testdir
>
> Status: Connected
>
> Number of entries: 1
>
>
>
>
>
>
>
> *From:* Pranith Kumar Karampuri 
> *Sent:* Wednesday, September 05, 2018 4:56 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
> *Cc:* Gluster Devel ; Ravishankar N <
> ravishan...@redhat.com>
> *Subject:* Re: remaining entry in gluster volume heal info command even
> after reboot
>
>
>
>
>
> On Wed, Sep 5, 2018 at 1:27 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.z...@nokia-sbell.com> wrote:
>
> Hi glusterfs experts:
>
>Good day!
>
>Recently when I do some test on my gluster env, I found that there
> are some remaining entries in command “gluster v heal mstate info” *even
> after reboot*.
>
> fstest_035ffc492ec43551a64087f9280ffe3e is a folder in /mnt/mstate
> and in this folder only one file(fstest_458bb82d8884ed5c9dadec4ed93bec4e)
> exists.
>
> When I dbg by gdb on sn-0 I found that:
>
> Parent dir changelog(fstest_035ffc492ec43551a64087f9280ffe3e) says need
> to heal entry, but the changelog/gfid/filetype of the only entry in parent
> dir shows there is nothing to be healed, so glustershd does nothing every
> round of heal. And this entry will remain.
>
> My gdb shows that each round of heal on sn-0 , it exits in function
> __afr_selfheal_entry (if (AFR_COUNT(healed_sinks, priv->child_count) ==
> 0)), because in this case all three healed_sinks are zero.
>
>
>
> What is the return value of this function in gdb?
>
>
>
>Have you any idea how to solve this issue from glusterfs pov?
>  Thanks!
>
>
>
> [test steps]
>
>Reboot three sn nodes( sn-0, sn-1, sn-2(arbiter)) sequentially, and
> on another node (with glusterfs clients) run fstest.
>
>
>
> [problem description]
>
>
>
> Remaining entries in “gluster v heal mstate info” command even after
> reboot sn-0 many times, the entries are still there!
&

Re: [Gluster-devel] remaining entry in gluster volume heal info command even after reboot

[Pranith Kumar Karampuri]

On Wed, Sep 5, 2018 at 1:27 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.z...@nokia-sbell.com> wrote:

> Hi glusterfs experts:
>
>Good day!
>
>Recently when I do some test on my gluster env, I found that there
> are some remaining entries in command “gluster v heal mstate info” *even
> after reboot*.
>
> fstest_035ffc492ec43551a64087f9280ffe3e is a folder in /mnt/mstate
> and in this folder only one file(fstest_458bb82d8884ed5c9dadec4ed93bec4e)
> exists.
>
> When I dbg by gdb on sn-0 I found that:
>
> Parent dir changelog(fstest_035ffc492ec43551a64087f9280ffe3e) says need
> to heal entry, but the changelog/gfid/filetype of the only entry in parent
> dir shows there is nothing to be healed, so glustershd does nothing every
> round of heal. And this entry will remain.
>
> My gdb shows that each round of heal on sn-0 , it exits in function
> __afr_selfheal_entry (if (AFR_COUNT(healed_sinks, priv->child_count) ==
> 0)), because in this case all three healed_sinks are zero.
>

What is the return value of this function in gdb?


>Have you any idea how to solve this issue from glusterfs pov?
>  Thanks!
>
>
>
> [test steps]
>
>Reboot three sn nodes( sn-0, sn-1, sn-2(arbiter)) sequentially, and
> on another node (with glusterfs clients) run fstest.
>
>
>
> [problem description]
>
>
>
> Remaining entries in “gluster v heal mstate info” command even after
> reboot sn-0 many times, the entries are still there!
>
>
>
> [root@sn-0:/home/robot]
>
> #  gluster v heal mstate info
>
> Brick sn-0.local:/mnt/bricks/mstate/brick
>
> /fstest_035ffc492ec43551a64087f9280ffe3e
>
> Status: Connected
>
> Number of entries: 1
>
>
>
> Brick sn-1.local:/mnt/bricks/mstate/brick
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick sn-2.local:/mnt/bricks/mstate/brick
>
> /fstest_035ffc492ec43551a64087f9280ffe3e
>
> Status: Connected
>
> Number of entries: 1
>
>
>
>
>
>
>
> some
> env informations///
>
> # gluster v info mstate
>
> Volume Name: mstate
>
> Type: Replicate
>
> Volume ID: 1d896674-17a2-4ae7-aa7c-c6e22013df99
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 1 x (2 + 1) = 3
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: sn-0.local:/mnt/bricks/mstate/brick
>
> Brick2: sn-1.local:/mnt/bricks/mstate/brick
>
> Brick3: sn-2.local:/mnt/bricks/mstate/brick (arbiter)
>
> Options Reconfigured:
>
> performance.client-io-threads: off
>
> nfs.disable: on
>
> transport.address-family: inet
>
> cluster.server-quorum-type: none
>
> cluster.quorum-reads: no
>
> cluster.favorite-child-policy: mtime
>
> cluster.consistent-metadata: on
>
> network.ping-timeout: 42
>
> cluster.quorum-type: auto
>
> server.allow-insecure: on
>
> cluster.server-quorum-ratio: 51%
>
> [root@sn-1:/home/robot]
>
>
>
>
>
> [root@sn-2:/root]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> trusted.afr.dirty=0x
>
> trusted.afr.mstate-client-0=0x00010003
>
> trusted.gfid=0xa0975560eaef4cb299467101de00446a
>
> trusted.glusterfs.dht=0x0001
>
>
>
> [root@sn-2:/root]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> getfattr: Removing leading '/' from absolute path names
>
> # file:
> mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> trusted.afr.mstate-client-0=0x
>
> trusted.gfid=0x9fc20f587f094182816390f056f7370f
>
>
> trusted.gfid2path.864159d77373ad5f=0x61303937353536302d656165662d346362322d393934362d3731303164653030343436612f6673746573745f3435386262383264383838346564356339646164656334656439336265633465
>
>
>
> # cd /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e
>
> [root@sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> # ls
>
> fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> [root@sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> # stat fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
>   File: fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
>   Size: 0   Blocks: 8  IO Block: 4096   fifo
>
> Device: fd31h/64817d Inode: 22086   Links: 2
>
> Access: (0644/prw-r--r--)  Uid: (0/root)   Gid: (0/root)
>
> Access: 2018-08-30 04:33:17.552870661 +0300
>
> Modify: 2018-08-30 04:33:17.552870661 +0300
>
> Change: 2018-08-30 04:33:17.553870661 +0300
>
> Birth: -
>
> [root@sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> [root@sn-2:/root]
>
> # exit
>
> logout
>
> Connection to sn-2.local closed.
>
> [root@sn-0:/home/robot]
>
> # getfattr -m . -d -e hex
> 

Re: [Gluster-devel] remaining entry in gluster volume heal info command even after reboot

[Pranith Kumar Karampuri]

Which version of gluster is this?

On Wed, Sep 5, 2018 at 1:27 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.z...@nokia-sbell.com> wrote:

> Hi glusterfs experts:
>
>Good day!
>
>Recently when I do some test on my gluster env, I found that there
> are some remaining entries in command “gluster v heal mstate info” *even
> after reboot*.
>
> fstest_035ffc492ec43551a64087f9280ffe3e is a folder in /mnt/mstate
> and in this folder only one file(fstest_458bb82d8884ed5c9dadec4ed93bec4e)
> exists.
>
> When I dbg by gdb on sn-0 I found that:
>
> Parent dir changelog(fstest_035ffc492ec43551a64087f9280ffe3e) says need
> to heal entry, but the changelog/gfid/filetype of the only entry in parent
> dir shows there is nothing to be healed, so glustershd does nothing every
> round of heal. And this entry will remain.
>
> My gdb shows that each round of heal on sn-0 , it exits in function
> __afr_selfheal_entry (if (AFR_COUNT(healed_sinks, priv->child_count) ==
> 0)), because in this case all three healed_sinks are zero.
>
>Have you any idea how to solve this issue from glusterfs pov?
>  Thanks!
>
>
>
> [test steps]
>
>Reboot three sn nodes( sn-0, sn-1, sn-2(arbiter)) sequentially, and
> on another node (with glusterfs clients) run fstest.
>
>
>
> [problem description]
>
>
>
> Remaining entries in “gluster v heal mstate info” command even after
> reboot sn-0 many times, the entries are still there!
>
>
>
> [root@sn-0:/home/robot]
>
> #  gluster v heal mstate info
>
> Brick sn-0.local:/mnt/bricks/mstate/brick
>
> /fstest_035ffc492ec43551a64087f9280ffe3e
>
> Status: Connected
>
> Number of entries: 1
>
>
>
> Brick sn-1.local:/mnt/bricks/mstate/brick
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick sn-2.local:/mnt/bricks/mstate/brick
>
> /fstest_035ffc492ec43551a64087f9280ffe3e
>
> Status: Connected
>
> Number of entries: 1
>
>
>
>
>
>
>
> some
> env informations///
>
> # gluster v info mstate
>
> Volume Name: mstate
>
> Type: Replicate
>
> Volume ID: 1d896674-17a2-4ae7-aa7c-c6e22013df99
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 1 x (2 + 1) = 3
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: sn-0.local:/mnt/bricks/mstate/brick
>
> Brick2: sn-1.local:/mnt/bricks/mstate/brick
>
> Brick3: sn-2.local:/mnt/bricks/mstate/brick (arbiter)
>
> Options Reconfigured:
>
> performance.client-io-threads: off
>
> nfs.disable: on
>
> transport.address-family: inet
>
> cluster.server-quorum-type: none
>
> cluster.quorum-reads: no
>
> cluster.favorite-child-policy: mtime
>
> cluster.consistent-metadata: on
>
> network.ping-timeout: 42
>
> cluster.quorum-type: auto
>
> server.allow-insecure: on
>
> cluster.server-quorum-ratio: 51%
>
> [root@sn-1:/home/robot]
>
>
>
>
>
> [root@sn-2:/root]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> trusted.afr.dirty=0x
>
> trusted.afr.mstate-client-0=0x00010003
>
> trusted.gfid=0xa0975560eaef4cb299467101de00446a
>
> trusted.glusterfs.dht=0x0001
>
>
>
> [root@sn-2:/root]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> getfattr: Removing leading '/' from absolute path names
>
> # file:
> mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> trusted.afr.mstate-client-0=0x
>
> trusted.gfid=0x9fc20f587f094182816390f056f7370f
>
>
> trusted.gfid2path.864159d77373ad5f=0x61303937353536302d656165662d346362322d393934362d3731303164653030343436612f6673746573745f3435386262383264383838346564356339646164656334656439336265633465
>
>
>
> # cd /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e
>
> [root@sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> # ls
>
> fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> [root@sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> # stat fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
>   File: fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
>   Size: 0   Blocks: 8  IO Block: 4096   fifo
>
> Device: fd31h/64817d Inode: 22086   Links: 2
>
> Access: (0644/prw-r--r--)  Uid: (0/root)   Gid: (0/root)
>
> Access: 2018-08-30 04:33:17.552870661 +0300
>
> Modify: 2018-08-30 04:33:17.552870661 +0300
>
> Change: 2018-08-30 04:33:17.553870661 +0300
>
> Birth: -
>
> [root@sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> [root@sn-2:/root]
>
> # exit
>
> logout
>
> Connection to sn-2.local closed.
>
> [root@sn-0:/home/robot]
>
> # getfattr -m . -d -e hex
> 

Re: [Gluster-devel] [Gluster-Maintainers] Master branch lock down for stabilization (unlocking the same)

[Pranith Kumar Karampuri]

On Mon, Aug 13, 2018 at 10:55 PM Shyam Ranganathan 
wrote:

> On 08/13/2018 02:20 AM, Pranith Kumar Karampuri wrote:
> > - At the end of 2 weeks, reassess master and nightly test status, and
> > see if we need another drive towards stabilizing master by locking
> down
> > the same and focusing only on test and code stability around the
> same.
> >
> >
> > When will there be a discussion about coming up with guidelines to
> > prevent lock down in future?
>
> A thread for the same is started in the maintainers list.
>

Could you point me to the thread please? I am only finding a thread with
subject "Lock down period merge process"

>
> > I think it is better to lock-down specific components by removing commit
> > access for the respective owners for those components when a test in a
> > particular component starts to fail.
>
> Also I suggest we move this to the maintainers thread, to keep the noise
> levels across lists in check.
>
> Thanks,
> Shyam
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Master branch lock down for stabilization (unlocking the same)

[Pranith Kumar Karampuri]

On Mon, Aug 13, 2018 at 6:05 AM Shyam Ranganathan 
wrote:

> Hi,
>
> So we have had master locked down for a week to ensure we only get fixes
> for failing tests in order to stabilize the code base, partly for
> release-5 branching as well.
>
> As of this weekend, we (Atin and myself) have been looking at the
> pass/fail rates on the tests, and whether we are discovering newer
> failures of more of the same.
>
> Our runs with patch sets 10->11->12 is looking better than where we
> started, and we have a list of tests that we need to still fix.
>
> But there are other issues and fixes that are needed in the code that
> are lagging behind due to the lock down. The plan going forward is as
> follows,
>
> - Unlock master, and ensure that we do not start seeing newer failures
> as we merge other patches in, if so raise them on the lists and as bugs
> and let's work towards ensuring these are addressed. *Maintainers*
> please pay special attention when merging patches.
>
> - Address the current pending set of tests that have been identified as
> failing, over the course of the next 2 weeks. *Contributors* continue
> the focus here, so that we do not have to end up with another drive
> towards the same in 2 weeks.
>
> - At the end of 2 weeks, reassess master and nightly test status, and
> see if we need another drive towards stabilizing master by locking down
> the same and focusing only on test and code stability around the same.
>

When will there be a discussion about coming up with guidelines to prevent
lock down in future?

I think it is better to lock-down specific components by removing commit
access for the respective owners for those components when a test in a
particular component starts to fail.


>
> Atin and Shyam
> ___
> maintainers mailing list
> maintain...@gluster.org
> https://lists.gluster.org/mailman/listinfo/maintainers
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Master branch lock down status (Wed, August 08th)

[Pranith Kumar Karampuri]

On Thu, Aug 9, 2018 at 4:02 PM Pranith Kumar Karampuri 
wrote:

>
>
> On Thu, Aug 9, 2018 at 6:34 AM Shyam Ranganathan 
> wrote:
>
>> Today's patch set 7 [1], included fixes provided till last evening IST,
>> and its runs can be seen here [2] (yay! we can link to comments in
>> gerrit now).
>>
>> New failures: (added to the spreadsheet)
>> ./tests/bugs/protocol/bug-808400-repl.t (core dumped)
>> ./tests/bugs/quick-read/bug-846240.t
>>
>> Older tests that had not recurred, but failed today: (moved up in the
>> spreadsheet)
>> ./tests/bugs/replicate/bug-1433571-undo-pending-only-on-up-bricks.t
>> ./tests/bugs/index/bug-1559004-EMLINK-handling.t
>>
>
> The above test is timing out. I had to increase the timeout while adding
> the .t so that creation of maximum number of links that will max-out in
> ext4. Will re-check if it is the same issue and get back.
>

This test is timing out with lcov. I bumped up timeout to 30 minutes @
https://review.gluster.org/#/c/glusterfs/+/20699, I am not happy that this
test takes so long, but without this it is difficult to find regression on
ext4 which has limits on number of hardlinks in a directory(It took us
almost one year after we introduced regression to find this problem when we
did introduce regression last time). If there is a way of running this .t
once per day and before each release. I will be happy to make it part of
that. Let me know.


>
>
>>
>> Other issues;
>> Test ./tests/basic/ec/ec-5-2.t core dumped again
>> Few geo-rep failures, Kotresh should have more logs to look at with
>> these runs
>> Test ./tests/bugs/glusterd/quorum-validation.t dumped core again
>>
>> Atin/Amar, we may need to merge some of the patches that have proven to
>> be holding up and fixing issues today, so that we do not leave
>> everything to the last. Check and move them along or lmk.
>>
>> Shyam
>>
>> [1] Patch set 7: https://review.gluster.org/c/glusterfs/+/20637/7
>> [2] Runs against patch set 7 and its status (incomplete as some runs
>> have not completed):
>>
>> https://review.gluster.org/c/glusterfs/+/20637/7#message-37bc68ce6f2157f2947da6fd03b361ab1b0d1a77
>> (also updated in the spreadsheet)
>>
>> On 08/07/2018 07:37 PM, Shyam Ranganathan wrote:
>> > Deserves a new beginning, threads on the other mail have gone deep
>> enough.
>> >
>> > NOTE: (5) below needs your attention, rest is just process and data on
>> > how to find failures.
>> >
>> > 1) We are running the tests using the patch [2].
>> >
>> > 2) Run details are extracted into a separate sheet in [3] named "Run
>> > Failures" use a search to find a failing test and the corresponding run
>> > that it failed in.
>> >
>> > 3) Patches that are fixing issues can be found here [1], if you think
>> > you have a patch out there, that is not in this list, shout out.
>> >
>> > 4) If you own up a test case failure, update the spreadsheet [3] with
>> > your name against the test, and also update other details as needed (as
>> > comments, as edit rights to the sheet are restricted).
>> >
>> > 5) Current test failures
>> > We still have the following tests failing and some without any RCA or
>> > attention, (If something is incorrect, write back).
>> >
>> > ./tests/bugs/replicate/bug-1290965-detect-bitrotten-objects.t (needs
>> > attention)
>> > ./tests/00-geo-rep/georep-basic-dr-tarssh.t (Kotresh)
>> > ./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
>> > (Atin)
>> > ./tests/bugs/ec/bug-1236065.t (Ashish)
>> > ./tests/00-geo-rep/georep-basic-dr-rsync.t (Kotresh)
>> > ./tests/basic/ec/ec-1468261.t (needs attention)
>> > ./tests/basic/afr/add-brick-self-heal.t (needs attention)
>> > ./tests/basic/afr/granular-esh/replace-brick.t (needs attention)
>> > ./tests/bugs/core/multiplex-limit-issue-151.t (needs attention)
>> > ./tests/bugs/glusterd/validating-server-quorum.t (Atin)
>> > ./tests/bugs/replicate/bug-1363721.t (Ravi)
>> >
>> > Here are some newer failures, but mostly one-off failures except cores
>> > in ec-5-2.t. All of the following need attention as these are new.
>> >
>> > ./tests/00-geo-rep/00-georep-verify-setup.t
>> > ./tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t
>> > ./tests/basic/stats-dump.t
>> > ./tests/bugs/bug-1110262.t
>> >
>> ./tests/bugs/glusterd/mgmt-handshake-and-volume-sync-post-glusterd-restart.t
>> > ./te

Re: [Gluster-devel] [Gluster-Maintainers] Master branch lock down status (Thu, August 09th)

[Pranith Kumar Karampuri]

On Fri, Aug 10, 2018 at 6:34 AM Shyam Ranganathan 
wrote:

> Today's test results are updated in the spreadsheet in sheet named "Run
> patch set 8".
>
> I took in patch https://review.gluster.org/c/glusterfs/+/20685 which
> caused quite a few failures, so not updating new failures as issue yet.
>
> Please look at the failures for tests that were retried and passed, as
> the logs for the initial runs should be preserved from this run onward.
>
> Otherwise nothing else to report on the run status, if you are averse to
> spreadsheets look at this comment in gerrit [1].
>
> Shyam
>
> [1] Patch set 8 run status:
>
> https://review.gluster.org/c/glusterfs/+/20637/8#message-54de30fa384fd02b0426d9db6d07fad4eeefcf08
> On 08/07/2018 07:37 PM, Shyam Ranganathan wrote:
> > Deserves a new beginning, threads on the other mail have gone deep
> enough.
> >
> > NOTE: (5) below needs your attention, rest is just process and data on
> > how to find failures.
> >
> > 1) We are running the tests using the patch [2].
> >
> > 2) Run details are extracted into a separate sheet in [3] named "Run
> > Failures" use a search to find a failing test and the corresponding run
> > that it failed in.
> >
> > 3) Patches that are fixing issues can be found here [1], if you think
> > you have a patch out there, that is not in this list, shout out.
> >
> > 4) If you own up a test case failure, update the spreadsheet [3] with
> > your name against the test, and also update other details as needed (as
> > comments, as edit rights to the sheet are restricted).
> >
> > 5) Current test failures
> > We still have the following tests failing and some without any RCA or
> > attention, (If something is incorrect, write back).
> >
> > ./tests/bugs/replicate/bug-1290965-detect-bitrotten-objects.t (needs
> > attention)
> > ./tests/00-geo-rep/georep-basic-dr-tarssh.t (Kotresh)
> > ./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
> > (Atin)
> > ./tests/bugs/ec/bug-1236065.t (Ashish)
> > ./tests/00-geo-rep/georep-basic-dr-rsync.t (Kotresh)
> > ./tests/basic/ec/ec-1468261.t (needs attention)
> > ./tests/basic/afr/add-brick-self-heal.t (needs attention)
> > ./tests/basic/afr/granular-esh/replace-brick.t (needs attention)
> > ./tests/bugs/core/multiplex-limit-issue-151.t (needs attention)
> > ./tests/bugs/glusterd/validating-server-quorum.t (Atin)
> > ./tests/bugs/replicate/bug-1363721.t (Ravi)
> >
> > Here are some newer failures, but mostly one-off failures except cores
> > in ec-5-2.t. All of the following need attention as these are new.
> >
> > ./tests/00-geo-rep/00-georep-verify-setup.t
> > ./tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t
> > ./tests/basic/stats-dump.t
> > ./tests/bugs/bug-1110262.t
> >
> ./tests/bugs/glusterd/mgmt-handshake-and-volume-sync-post-glusterd-restart.t
> > ./tests/basic/ec/ec-data-heal.t
> > ./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t
>

Sent https://review.gluster.org/c/glusterfs/+/20697 for the test above.


> >
> ./tests/bugs/snapshot/bug-1482023-snpashot-issue-with-other-processes-accessing-mounted-path.t
> > ./tests/basic/ec/ec-5-2.t
> >
> > 6) Tests that are addressed or are not occurring anymore are,
> >
> > ./tests/bugs/glusterd/rebalance-operations-in-single-node.t
> > ./tests/bugs/index/bug-1559004-EMLINK-handling.t
> > ./tests/bugs/replicate/bug-1386188-sbrain-fav-child.t
> > ./tests/bugs/replicate/bug-1433571-undo-pending-only-on-up-bricks.t
> > ./tests/bitrot/bug-1373520.t
> > ./tests/bugs/distribute/bug-1117851.t
> > ./tests/bugs/glusterd/quorum-validation.t
> > ./tests/bugs/distribute/bug-1042725.t
> >
> ./tests/bugs/replicate/bug-1586020-mark-dirty-for-entry-txn-on-quorum-failure.t
> > ./tests/bugs/quota/bug-1293601.t
> > ./tests/bugs/bug-1368312.t
> > ./tests/bugs/distribute/bug-1122443.t
> > ./tests/bugs/core/bug-1432542-mpx-restart-crash.t
> >
> > Shyam (and Atin)
> >
> > On 08/05/2018 06:24 PM, Shyam Ranganathan wrote:
> >> Health on master as of the last nightly run [4] is still the same.
> >>
> >> Potential patches that rectify the situation (as in [1]) are bunched in
> >> a patch [2] that Atin and myself have put through several regressions
> >> (mux, normal and line coverage) and these have also not passed.
> >>
> >> Till we rectify the situation we are locking down master branch commit
> >> rights to the following people, Amar, Atin, Shyam, Vijay.
> >>
> >> The intention is to stabilize master and not add more patches that my
> >> destabilize it.
> >>
> >> Test cases that are tracked as failures and need action are present here
> >> [3].
> >>
> >> @Nigel, request you to apply the commit rights change as you see this
> >> mail and let the list know regarding the same as well.
> >>
> >> Thanks,
> >> Shyam
> >>
> >> [1] Patches that address regression failures:
> >> https://review.gluster.org/#/q/starredby:srangana%2540redhat.com
> >>
> >> [2] Bunched up patch against which regressions were run:
> >> https://review.gluster.org/#/c/20637
> >>
> 

Re: [Gluster-devel] Spurious smoke failure in build rpms

[Pranith Kumar Karampuri]

On Thu, Aug 9, 2018 at 4:19 PM Nigel Babu  wrote:

> Infra issue. Please file a bug.
>

https://bugzilla.redhat.com/show_bug.cgi?id=1614631

Thanks!


> On Thu, Aug 9, 2018 at 3:57 PM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>> https://build.gluster.org/job/devrpm-el7/10441/console
>>
>> *10:12:42* Wrote: 
>> /home/jenkins/root/workspace/devrpm-el7/extras/LinuxRPM/rpmbuild/SRPMS/glusterfs-4.2dev-0.240.git4657137.el7.src.rpm*10:12:42*
>>  mv rpmbuild/SRPMS/* .*10:12:44* INFO: mock.py version 1.4.11 starting 
>> (python version = 2.7.5)...*10:12:44* Start: init plugins*10:12:44* INFO: 
>> selinux disabled*10:12:44* Finish: init plugins*10:12:44* Start: 
>> run*10:12:44* INFO: Start(glusterfs-4.2dev-0.240.git4657137.el7.src.rpm)  
>> Config(epel-7-x86_64)*10:12:44* Start: clean chroot*10:12:44* ERROR: 
>> Exception(glusterfs-4.2dev-0.240.git4657137.el7.src.rpm) 
>> Config(epel-7-x86_64) 0 minutes 0 seconds
>>
>>
>> I am not sure why it is saying exception for the src.rpm and failing,
>> does anyone know?
>>
>>
>> --
>> Pranith
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> --
> nigelb
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] ./tests/basic/afr/metadata-self-heal.t core dumped

[Pranith Kumar Karampuri]

On Fri, Aug 10, 2018 at 8:54 AM Raghavendra Gowdappa 
wrote:

> All,
>
> Details can be found at:
> https://build.gluster.org/job/centos7-regression/2190/console
>
> Process that core dumped: glfs_shdheal
>
> Note that the patch on which this regression failures is on readdir-ahead
> which is not loaded in xlator graph of self heal daemon.
>
> From bt,
>
> *23:53:24* __FUNCTION__ = "syncop_getxattr"*23:53:24* #8  
> 0x7f5af8738aef in syncop_gfid_to_path_hard (itable=0x7f5ae401ce50, 
> subvol=0x7f5ae40079e0, gfid=0x7f5adc00b4e8 "", inode=0x0, 
> path_p=0x7f5acbffebe8, hard_resolve=false) at 
> /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop-utils.c:585*23:53:24*
>  ret = 0*23:53:24* path = 0x0*23:53:24* loc = {path = 
> 0x0, name = 0x0, inode = 0x7f5ac00028a8, parent = 0x0, gfid = '\000'  15 times>, pargfid = '\000' }*23:53:24* xattr = 
> 0x0*23:53:24* #9  0x7f5af8738c28 in syncop_gfid_to_path 
> (itable=0x7f5ae401ce50, subvol=0x7f5ae40079e0, gfid=0x7f5adc00b4e8 "", 
> path_p=0x7f5acbffebe8) at 
> /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop-utils.c:636*23:53:24*
>  No locals.
> *23:53:24* #10 0x7f5aeaad65e1 in afr_shd_selfheal (healer=0x7f5ae401d490, 
> child=0, gfid=0x7f5adc00b4e8 "") at 
> /home/jenkins/root/workspace/centos7-regression/xlators/cluster/afr/src/afr-self-heald.c:331*23:53:24*
>  ret = 0*23:53:24* eh = 0x0*23:53:24* priv = 
> 0x7f5ae401c780*23:53:24* shd = 0x7f5ae401c8e8*23:53:24* 
> shd_event = 0x0*23:53:24* path = 0x0*23:53:24* subvol = 
> 0x7f5ae40079e0*23:53:24* this = 0x7f5ae400d540*23:53:24* 
> crawl_event = 0x7f5ae401d4a0*23:53:24* #11 0x7f5aeaad6de5 in 
> afr_shd_full_heal (subvol=0x7f5ae40079e0, entry=0x7f5adc00b440, 
> parent=0x7f5acbffee20, data=0x7f5ae401d490) at 
> /home/jenkins/root/workspace/centos7-regression/xlators/cluster/afr/src/afr-self-heald.c:541*23:53:24*
>  healer = 0x7f5ae401d490*23:53:24* this = 
> 0x7f5ae400d540*23:53:24* priv = 0x7f5ae401c780*23:53:24* #12 
> 0x7f5af8737b2f in syncop_ftw (subvol=0x7f5ae40079e0, loc=0x7f5acbffee20, 
> pid=-6, data=0x7f5ae401d490, fn=0x7f5aeaad6d40 ) at 
> /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/syncop-utils.c:123*23:53:24*
>  child_loc = {path = 0x0, name = 0x0, inode = 0x0, parent = 0x0, gfid 
> = '\000' , pargfid = '\000' }*23:53:24*   
>   fd = 0x7f5ac0001398
>
>
> Assert for a non-null gfid failed in client_pre_getxattr_v2. From bt, it
> looks like a NULL gfid was passed to afr_shd_full.
>

Most probably it is because of the change in gf_link_inode_from_dirent() in
your patch. Why did you make that change? Wondering if we need to change
afr/ec code accordingly.


>
> *23:53:24* __PRETTY_FUNCTION__ = "client_pre_getxattr_v2"*23:53:24* 
> #5  0x7f5aeada8f2a in client4_0_getxattr (frame=0x7f5ac0008198, 
> this=0x7f5ae40079e0, data=0x7f5acbffdcc0) at 
> /home/jenkins/root/workspace/centos7-regression/xlators/protocol/client/src/client-rpc-fops_v2.c:4287*23:53:24*
>  conf = 0x7f5ae40293e0*23:53:24* args = 
> 0x7f5acbffdcc0*23:53:24* req = {gfid = '\000' , 
> namelen = 0, name = 0x0, xdata = {xdr_size = 0, count = 0, pairs = {pairs_len 
> = 0, pairs_val = 0x0}}}*23:53:24* dict = 0x0*23:53:24* ret = 
> 0*23:53:24* op_ret = -1*23:53:24* op_errno = 116*23:53:24*
>  local = 0x7f5ac00082a8*23:53:24* __FUNCTION__ = 
> "client4_0_getxattr"*23:53:24* __PRETTY_FUNCTION__ = 
> "client4_0_getxattr"
>
>
> regards,
> Raghavendra
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Master branch lock down status (Wed, August 08th)

[Pranith Kumar Karampuri]

On Thu, Aug 9, 2018 at 6:34 AM Shyam Ranganathan 
wrote:

> Today's patch set 7 [1], included fixes provided till last evening IST,
> and its runs can be seen here [2] (yay! we can link to comments in
> gerrit now).
>
> New failures: (added to the spreadsheet)
> ./tests/bugs/protocol/bug-808400-repl.t (core dumped)
> ./tests/bugs/quick-read/bug-846240.t
>
> Older tests that had not recurred, but failed today: (moved up in the
> spreadsheet)
> ./tests/bugs/replicate/bug-1433571-undo-pending-only-on-up-bricks.t
> ./tests/bugs/index/bug-1559004-EMLINK-handling.t
>

The above test is timing out. I had to increase the timeout while adding
the .t so that creation of maximum number of links that will max-out in
ext4. Will re-check if it is the same issue and get back.


>
> Other issues;
> Test ./tests/basic/ec/ec-5-2.t core dumped again
> Few geo-rep failures, Kotresh should have more logs to look at with
> these runs
> Test ./tests/bugs/glusterd/quorum-validation.t dumped core again
>
> Atin/Amar, we may need to merge some of the patches that have proven to
> be holding up and fixing issues today, so that we do not leave
> everything to the last. Check and move them along or lmk.
>
> Shyam
>
> [1] Patch set 7: https://review.gluster.org/c/glusterfs/+/20637/7
> [2] Runs against patch set 7 and its status (incomplete as some runs
> have not completed):
>
> https://review.gluster.org/c/glusterfs/+/20637/7#message-37bc68ce6f2157f2947da6fd03b361ab1b0d1a77
> (also updated in the spreadsheet)
>
> On 08/07/2018 07:37 PM, Shyam Ranganathan wrote:
> > Deserves a new beginning, threads on the other mail have gone deep
> enough.
> >
> > NOTE: (5) below needs your attention, rest is just process and data on
> > how to find failures.
> >
> > 1) We are running the tests using the patch [2].
> >
> > 2) Run details are extracted into a separate sheet in [3] named "Run
> > Failures" use a search to find a failing test and the corresponding run
> > that it failed in.
> >
> > 3) Patches that are fixing issues can be found here [1], if you think
> > you have a patch out there, that is not in this list, shout out.
> >
> > 4) If you own up a test case failure, update the spreadsheet [3] with
> > your name against the test, and also update other details as needed (as
> > comments, as edit rights to the sheet are restricted).
> >
> > 5) Current test failures
> > We still have the following tests failing and some without any RCA or
> > attention, (If something is incorrect, write back).
> >
> > ./tests/bugs/replicate/bug-1290965-detect-bitrotten-objects.t (needs
> > attention)
> > ./tests/00-geo-rep/georep-basic-dr-tarssh.t (Kotresh)
> > ./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
> > (Atin)
> > ./tests/bugs/ec/bug-1236065.t (Ashish)
> > ./tests/00-geo-rep/georep-basic-dr-rsync.t (Kotresh)
> > ./tests/basic/ec/ec-1468261.t (needs attention)
> > ./tests/basic/afr/add-brick-self-heal.t (needs attention)
> > ./tests/basic/afr/granular-esh/replace-brick.t (needs attention)
> > ./tests/bugs/core/multiplex-limit-issue-151.t (needs attention)
> > ./tests/bugs/glusterd/validating-server-quorum.t (Atin)
> > ./tests/bugs/replicate/bug-1363721.t (Ravi)
> >
> > Here are some newer failures, but mostly one-off failures except cores
> > in ec-5-2.t. All of the following need attention as these are new.
> >
> > ./tests/00-geo-rep/00-georep-verify-setup.t
> > ./tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t
> > ./tests/basic/stats-dump.t
> > ./tests/bugs/bug-1110262.t
> >
> ./tests/bugs/glusterd/mgmt-handshake-and-volume-sync-post-glusterd-restart.t
> > ./tests/basic/ec/ec-data-heal.t
> > ./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t
> >
> ./tests/bugs/snapshot/bug-1482023-snpashot-issue-with-other-processes-accessing-mounted-path.t
> > ./tests/basic/ec/ec-5-2.t
> >
> > 6) Tests that are addressed or are not occurring anymore are,
> >
> > ./tests/bugs/glusterd/rebalance-operations-in-single-node.t
> > ./tests/bugs/index/bug-1559004-EMLINK-handling.t
> > ./tests/bugs/replicate/bug-1386188-sbrain-fav-child.t
> > ./tests/bugs/replicate/bug-1433571-undo-pending-only-on-up-bricks.t
> > ./tests/bitrot/bug-1373520.t
> > ./tests/bugs/distribute/bug-1117851.t
> > ./tests/bugs/glusterd/quorum-validation.t
> > ./tests/bugs/distribute/bug-1042725.t
> >
> ./tests/bugs/replicate/bug-1586020-mark-dirty-for-entry-txn-on-quorum-failure.t
> > ./tests/bugs/quota/bug-1293601.t
> > ./tests/bugs/bug-1368312.t
> > ./tests/bugs/distribute/bug-1122443.t
> > ./tests/bugs/core/bug-1432542-mpx-restart-crash.t
> >
> > Shyam (and Atin)
> >
> > On 08/05/2018 06:24 PM, Shyam Ranganathan wrote:
> >> Health on master as of the last nightly run [4] is still the same.
> >>
> >> Potential patches that rectify the situation (as in [1]) are bunched in
> >> a patch [2] that Atin and myself have put through several regressions
> >> (mux, normal and line coverage) and these have also not passed.
> >>
> >> Till we rectify the 

[Gluster-devel] Spurious smoke failure in build rpms

[Pranith Kumar Karampuri]

https://build.gluster.org/job/devrpm-el7/10441/console

*10:12:42* Wrote:
/home/jenkins/root/workspace/devrpm-el7/extras/LinuxRPM/rpmbuild/SRPMS/glusterfs-4.2dev-0.240.git4657137.el7.src.rpm*10:12:42*
mv rpmbuild/SRPMS/* .*10:12:44* INFO: mock.py version 1.4.11 starting
(python version = 2.7.5)...*10:12:44* Start: init plugins*10:12:44*
INFO: selinux disabled*10:12:44* Finish: init plugins*10:12:44* Start:
run*10:12:44* INFO:
Start(glusterfs-4.2dev-0.240.git4657137.el7.src.rpm)
Config(epel-7-x86_64)*10:12:44* Start: clean chroot*10:12:44* ERROR:
Exception(glusterfs-4.2dev-0.240.git4657137.el7.src.rpm)
Config(epel-7-x86_64) 0 minutes 0 seconds


I am not sure why it is saying exception for the src.rpm and failing, does
anyone know?


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Master branch lock down status

[Pranith Kumar Karampuri]

On Wed, Aug 8, 2018 at 5:08 AM Shyam Ranganathan 
wrote:

> Deserves a new beginning, threads on the other mail have gone deep enough.
>
> NOTE: (5) below needs your attention, rest is just process and data on
> how to find failures.
>
> 1) We are running the tests using the patch [2].
>
> 2) Run details are extracted into a separate sheet in [3] named "Run
> Failures" use a search to find a failing test and the corresponding run
> that it failed in.
>
> 3) Patches that are fixing issues can be found here [1], if you think
> you have a patch out there, that is not in this list, shout out.
>
> 4) If you own up a test case failure, update the spreadsheet [3] with
> your name against the test, and also update other details as needed (as
> comments, as edit rights to the sheet are restricted).
>
> 5) Current test failures
> We still have the following tests failing and some without any RCA or
> attention, (If something is incorrect, write back).
>
> ./tests/bugs/replicate/bug-1290965-detect-bitrotten-objects.t (needs
> attention)
> ./tests/00-geo-rep/georep-basic-dr-tarssh.t (Kotresh)
> ./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
> (Atin)
> ./tests/bugs/ec/bug-1236065.t (Ashish)
> ./tests/00-geo-rep/georep-basic-dr-rsync.t (Kotresh)
> ./tests/basic/ec/ec-1468261.t (needs attention)
>

Sent a fix for above @ https://review.gluster.org/#/c/glusterfs/+/20685


> ./tests/basic/afr/add-brick-self-heal.t (needs attention)
> ./tests/basic/afr/granular-esh/replace-brick.t (needs attention)
> ./tests/bugs/core/multiplex-limit-issue-151.t (needs attention)
> ./tests/bugs/glusterd/validating-server-quorum.t (Atin)
> ./tests/bugs/replicate/bug-1363721.t (Ravi)
>
> Here are some newer failures, but mostly one-off failures except cores
> in ec-5-2.t. All of the following need attention as these are new.
>
> ./tests/00-geo-rep/00-georep-verify-setup.t
> ./tests/basic/afr/gfid-mismatch-resolution-with-fav-child-policy.t
> ./tests/basic/stats-dump.t
> ./tests/bugs/bug-1110262.t
>
> ./tests/bugs/glusterd/mgmt-handshake-and-volume-sync-post-glusterd-restart.t
> ./tests/basic/ec/ec-data-heal.t
> ./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t
>
> ./tests/bugs/snapshot/bug-1482023-snpashot-issue-with-other-processes-accessing-mounted-path.t
> ./tests/basic/ec/ec-5-2.t
>
> 6) Tests that are addressed or are not occurring anymore are,
>
> ./tests/bugs/glusterd/rebalance-operations-in-single-node.t
> ./tests/bugs/index/bug-1559004-EMLINK-handling.t
> ./tests/bugs/replicate/bug-1386188-sbrain-fav-child.t
> ./tests/bugs/replicate/bug-1433571-undo-pending-only-on-up-bricks.t
> ./tests/bitrot/bug-1373520.t
> ./tests/bugs/distribute/bug-1117851.t
> ./tests/bugs/glusterd/quorum-validation.t
> ./tests/bugs/distribute/bug-1042725.t
>
> ./tests/bugs/replicate/bug-1586020-mark-dirty-for-entry-txn-on-quorum-failure.t
> ./tests/bugs/quota/bug-1293601.t
> ./tests/bugs/bug-1368312.t
> ./tests/bugs/distribute/bug-1122443.t
> ./tests/bugs/core/bug-1432542-mpx-restart-crash.t
>
> Shyam (and Atin)
>
> On 08/05/2018 06:24 PM, Shyam Ranganathan wrote:
> > Health on master as of the last nightly run [4] is still the same.
> >
> > Potential patches that rectify the situation (as in [1]) are bunched in
> > a patch [2] that Atin and myself have put through several regressions
> > (mux, normal and line coverage) and these have also not passed.
> >
> > Till we rectify the situation we are locking down master branch commit
> > rights to the following people, Amar, Atin, Shyam, Vijay.
> >
> > The intention is to stabilize master and not add more patches that my
> > destabilize it.
> >
> > Test cases that are tracked as failures and need action are present here
> > [3].
> >
> > @Nigel, request you to apply the commit rights change as you see this
> > mail and let the list know regarding the same as well.
> >
> > Thanks,
> > Shyam
> >
> > [1] Patches that address regression failures:
> > https://review.gluster.org/#/q/starredby:srangana%2540redhat.com
> >
> > [2] Bunched up patch against which regressions were run:
> > https://review.gluster.org/#/c/20637
> >
> > [3] Failing tests list:
> >
> https://docs.google.com/spreadsheets/d/1IF9GhpKah4bto19RQLr0y_Kkw26E_-crKALHSaSjZMQ/edit?usp=sharing
> >
> > [4] Nightly run dashboard: https://build.gluster.org/job/nightly-master/
> ___
> maintainers mailing list
> maintain...@gluster.org
> https://lists.gluster.org/mailman/listinfo/maintainers
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 5: Master branch health report (Week of 30th July)

[Pranith Kumar Karampuri]

On Thu, Aug 2, 2018 at 10:03 PM Pranith Kumar Karampuri 
wrote:

>
>
> On Thu, Aug 2, 2018 at 7:19 PM Atin Mukherjee  wrote:
>
>> New addition - tests/basic/volume.t - failed twice atleast with shd core.
>>
>> One such ref -
>> https://build.gluster.org/job/centos7-regression/2058/console
>>
>
> I will take a look.
>

The crash is happening inside libc and there are no line numbers to debug
further. Is there anyway to get symbols, line numbers even for that? We can
find hints as to what could be going wrong. Let me try to re-create it on
the machines I have in the meanwhile.

(gdb) bt
#0  0x7feae916bb4f in _IO_cleanup () from ./lib64/libc.so.6
#1  0x7feae9127b8b in __run_exit_handlers () from ./lib64/libc.so.6
#2  0x7feae9127c27 in exit () from ./lib64/libc.so.6
#3  0x00408ba5 in cleanup_and_exit (signum=15) at
/home/jenkins/root/workspace/centos7-regression/glusterfsd/src/glusterfsd.c:1570
#4  0x0040a75f in glusterfs_sigwaiter (arg=0x7ffe6faa7540) at
/home/jenkins/root/workspace/centos7-regression/glusterfsd/src/glusterfsd.c:2332
#5  0x7feae9b27e25 in start_thread () from ./lib64/libpthread.so.0
#6  0x7feae91ecbad in clone () from ./lib64/libc.so.6


>
>>
>>
>> On Thu, Aug 2, 2018 at 6:28 PM Sankarshan Mukhopadhyay <
>> sankarshan.mukhopadh...@gmail.com> wrote:
>>
>>> On Thu, Aug 2, 2018 at 5:48 PM, Kotresh Hiremath Ravishankar
>>>  wrote:
>>> > I am facing different issue in softserve machines. The fuse mount
>>> itself is
>>> > failing.
>>> > I tried day before yesterday to debug geo-rep failures. I discussed
>>> with
>>> > Raghu,
>>> > but could not root cause it. So none of the tests were passing. It
>>> happened
>>> > on
>>> > both machine instances I tried.
>>> >
>>>
>>> Ugh! -infra team should have an issue to work with and resolve this.
>>>
>>>
>>> --
>>> sankarshan mukhopadhyay
>>> <https://about.me/sankarshan.mukhopadhyay>
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> --
> Pranith
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 5: Master branch health report (Week of 30th July)

[Pranith Kumar Karampuri]

On Thu, Aug 2, 2018 at 7:19 PM Atin Mukherjee  wrote:

> New addition - tests/basic/volume.t - failed twice atleast with shd core.
>
> One such ref -
> https://build.gluster.org/job/centos7-regression/2058/console
>

I will take a look.


>
>
> On Thu, Aug 2, 2018 at 6:28 PM Sankarshan Mukhopadhyay <
> sankarshan.mukhopadh...@gmail.com> wrote:
>
>> On Thu, Aug 2, 2018 at 5:48 PM, Kotresh Hiremath Ravishankar
>>  wrote:
>> > I am facing different issue in softserve machines. The fuse mount
>> itself is
>> > failing.
>> > I tried day before yesterday to debug geo-rep failures. I discussed with
>> > Raghu,
>> > but could not root cause it. So none of the tests were passing. It
>> happened
>> > on
>> > both machine instances I tried.
>> >
>>
>> Ugh! -infra team should have an issue to work with and resolve this.
>>
>>
>> --
>> sankarshan mukhopadhyay
>> 
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] How long should metrics collection on a cluster take?

[Pranith Kumar Karampuri]

On Thu, Jul 26, 2018 at 8:30 PM, John Strunk  wrote:

> It is configurable. Use the default as a notion of scale... 5s may become
> 30s; It won't be 5m.
> Also remember, this is the maximum, not minimum. A change to a watched
> kube resource will cause an immediate reconcile. The periodic, timer-based
> loop is just a fallback to catch state changes not represented in the kube
> API.
>

Cool, got it. Let us wait if anyone sees any objections to the solution
proposed.

Request everyone to comment if they see any issues with
https://github.com/gluster/glusterd2/issues/1069
I think EC/AFR/Quota components will definitely be affected with this
approach. CCing them.
Please feel free to CC anyone who works on commands that require a mount to
give status.


>
> On Thu, Jul 26, 2018 at 12:57 AM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Thu, Jul 26, 2018 at 9:59 AM, Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>>>
>>>
>>> On Wed, Jul 25, 2018 at 10:48 PM, John Strunk 
>>> wrote:
>>>
>>>> I have not put together a list. Perhaps the following will help w/ the
>>>> context though...
>>>>
>>>> The "reconcile loop" of the operator will take the cluster CRs and
>>>> reconcile them against the actual cluster config. At the 20k foot level,
>>>> this amounts to something like determining there should be 8 gluster pods
>>>> running, and making the appropriate changes if that doesn't match reality.
>>>> In practical terms, the construction of this reconciliation loop can be
>>>> thought of as a set (array) of 3-tuples: [{should_act() -> bool, can_act ->
>>>> bool, action() -> ok, error}, {..., ..., ...}, ...]
>>>>
>>>> Each capability of the operator would be expressed as one of these
>>>> tuples.
>>>> should_act() : true if the action() should be taken
>>>> can_act() : true if the prerequisites for taking the action are met
>>>> action() : make the change. Only run if should && can.
>>>> (note that I believe should_act() and can_act() should not be separate
>>>> in the implementation, for reasons I'll not go into here)
>>>>
>>>> An example action might be "upgrade the container image for pod X". The
>>>> associated should_act would be triggered if the "image=" of the pod doesn't
>>>> match the desired "image=" in the operator CRs. The can_act evaluation
>>>> would be verifying that it's ok to do this... Thinking from the top of my
>>>> head:
>>>> - All volumes w/ a brick on this pod should be fully healed
>>>> - Sufficient cluster nodes should be up such that quorum is not lost
>>>> when this node goes down (does this matter?)
>>>> - The proposed image is compatible with the current version of the CSI
>>>> driver(s), the operator, and other gluster pods
>>>> - Probably some other stuff
>>>> The action() would update the "image=" in the Deployment to trigger the
>>>> rollout
>>>>
>>>> The idea is that queries would be made, both to the kube API and the
>>>> gluster cluster to verify the necessary preconditions for an action prior
>>>> to that action being invoked. There would obviously be commonality among
>>>> the preconditions for various actions, so the results should be fetched
>>>> exactly once per reconcile cycle. Also note, 1 cycle == at most 1 action()
>>>> due to the action changing the state of the system.
>>>>
>>>> Given that we haven't designed (or even listed) all the potential
>>>> action()s, I can't give you a list of everything to query. I guarantee
>>>> we'll need to know the up/down status, heal counts, and free capacity for
>>>> each brick and node.
>>>>
>>>
>>> Thanks for the detailed explanation. This helps. One question though, is
>>> 5 seconds a hard limit or is there a possibility to configure it?
>>>
>>
>> I put together an idea for reducing the mgmt operation latency involving
>> mounts at https://github.com/gluster/glusterd2/issues/1069, comments
>> welcome.
>> @john Still want to know if there exists  a way to find if the hard limit
>> can be configured...
>>
>>
>>>
>>>
>>>>
>>>> -John
>>>>
>>>> On Wed, Jul 25, 2018 at 11:56 AM Pranith Kumar Karampuri <
>>>> 

Re: [Gluster-devel] How long should metrics collection on a cluster take?

[Pranith Kumar Karampuri]

On Thu, Jul 26, 2018 at 9:59 AM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Wed, Jul 25, 2018 at 10:48 PM, John Strunk  wrote:
>
>> I have not put together a list. Perhaps the following will help w/ the
>> context though...
>>
>> The "reconcile loop" of the operator will take the cluster CRs and
>> reconcile them against the actual cluster config. At the 20k foot level,
>> this amounts to something like determining there should be 8 gluster pods
>> running, and making the appropriate changes if that doesn't match reality.
>> In practical terms, the construction of this reconciliation loop can be
>> thought of as a set (array) of 3-tuples: [{should_act() -> bool, can_act ->
>> bool, action() -> ok, error}, {..., ..., ...}, ...]
>>
>> Each capability of the operator would be expressed as one of these tuples.
>> should_act() : true if the action() should be taken
>> can_act() : true if the prerequisites for taking the action are met
>> action() : make the change. Only run if should && can.
>> (note that I believe should_act() and can_act() should not be separate in
>> the implementation, for reasons I'll not go into here)
>>
>> An example action might be "upgrade the container image for pod X". The
>> associated should_act would be triggered if the "image=" of the pod doesn't
>> match the desired "image=" in the operator CRs. The can_act evaluation
>> would be verifying that it's ok to do this... Thinking from the top of my
>> head:
>> - All volumes w/ a brick on this pod should be fully healed
>> - Sufficient cluster nodes should be up such that quorum is not lost when
>> this node goes down (does this matter?)
>> - The proposed image is compatible with the current version of the CSI
>> driver(s), the operator, and other gluster pods
>> - Probably some other stuff
>> The action() would update the "image=" in the Deployment to trigger the
>> rollout
>>
>> The idea is that queries would be made, both to the kube API and the
>> gluster cluster to verify the necessary preconditions for an action prior
>> to that action being invoked. There would obviously be commonality among
>> the preconditions for various actions, so the results should be fetched
>> exactly once per reconcile cycle. Also note, 1 cycle == at most 1 action()
>> due to the action changing the state of the system.
>>
>> Given that we haven't designed (or even listed) all the potential
>> action()s, I can't give you a list of everything to query. I guarantee
>> we'll need to know the up/down status, heal counts, and free capacity for
>> each brick and node.
>>
>
> Thanks for the detailed explanation. This helps. One question though, is 5
> seconds a hard limit or is there a possibility to configure it?
>

I put together an idea for reducing the mgmt operation latency involving
mounts at https://github.com/gluster/glusterd2/issues/1069, comments
welcome.
@john Still want to know if there exists  a way to find if the hard limit
can be configured...


>
>
>>
>> -John
>>
>> On Wed, Jul 25, 2018 at 11:56 AM Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>>>
>>>
>>> On Wed, Jul 25, 2018 at 8:17 PM, John Strunk  wrote:
>>>
>>>> To add an additional data point... The operator will need to regularly
>>>> reconcile the true state of the gluster cluster with the desired state
>>>> stored in kubernetes. This task will be required frequently (i.e.,
>>>> operator-framework defaults to every 5s even if there are no config
>>>> changes).
>>>> The actual amount of data we will need to query from the cluster is
>>>> currently TBD and likely significantly affected by Heketi/GD1 vs. GD2
>>>> choice.
>>>>
>>>
>>> Do we have any partial list of data we will gather? Just want to
>>> understand what this might entail already...
>>>
>>>
>>>>
>>>> -John
>>>>
>>>>
>>>> On Wed, Jul 25, 2018 at 5:45 AM Pranith Kumar Karampuri <
>>>> pkara...@redhat.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Jul 24, 2018 at 10:10 PM, Sankarshan Mukhopadhyay <
>>>>> sankarshan.mukhopadh...@gmail.com> wrote:
>>>>>
>>>>>> On Tue, Jul 24, 2018 at 9:48 PM, Pranith Kumar Karampuri
>>>>>>  wrote:
>>>>>> > hi,
>>>>>> >   Quite a fe

Re: [Gluster-devel] How long should metrics collection on a cluster take?

[Pranith Kumar Karampuri]

On Wed, Jul 25, 2018 at 10:48 PM, John Strunk  wrote:

> I have not put together a list. Perhaps the following will help w/ the
> context though...
>
> The "reconcile loop" of the operator will take the cluster CRs and
> reconcile them against the actual cluster config. At the 20k foot level,
> this amounts to something like determining there should be 8 gluster pods
> running, and making the appropriate changes if that doesn't match reality.
> In practical terms, the construction of this reconciliation loop can be
> thought of as a set (array) of 3-tuples: [{should_act() -> bool, can_act ->
> bool, action() -> ok, error}, {..., ..., ...}, ...]
>
> Each capability of the operator would be expressed as one of these tuples.
> should_act() : true if the action() should be taken
> can_act() : true if the prerequisites for taking the action are met
> action() : make the change. Only run if should && can.
> (note that I believe should_act() and can_act() should not be separate in
> the implementation, for reasons I'll not go into here)
>
> An example action might be "upgrade the container image for pod X". The
> associated should_act would be triggered if the "image=" of the pod doesn't
> match the desired "image=" in the operator CRs. The can_act evaluation
> would be verifying that it's ok to do this... Thinking from the top of my
> head:
> - All volumes w/ a brick on this pod should be fully healed
> - Sufficient cluster nodes should be up such that quorum is not lost when
> this node goes down (does this matter?)
> - The proposed image is compatible with the current version of the CSI
> driver(s), the operator, and other gluster pods
> - Probably some other stuff
> The action() would update the "image=" in the Deployment to trigger the
> rollout
>
> The idea is that queries would be made, both to the kube API and the
> gluster cluster to verify the necessary preconditions for an action prior
> to that action being invoked. There would obviously be commonality among
> the preconditions for various actions, so the results should be fetched
> exactly once per reconcile cycle. Also note, 1 cycle == at most 1 action()
> due to the action changing the state of the system.
>
> Given that we haven't designed (or even listed) all the potential
> action()s, I can't give you a list of everything to query. I guarantee
> we'll need to know the up/down status, heal counts, and free capacity for
> each brick and node.
>

Thanks for the detailed explanation. This helps. One question though, is 5
seconds a hard limit or is there a possibility to configure it?


>
> -John
>
> On Wed, Jul 25, 2018 at 11:56 AM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Wed, Jul 25, 2018 at 8:17 PM, John Strunk  wrote:
>>
>>> To add an additional data point... The operator will need to regularly
>>> reconcile the true state of the gluster cluster with the desired state
>>> stored in kubernetes. This task will be required frequently (i.e.,
>>> operator-framework defaults to every 5s even if there are no config
>>> changes).
>>> The actual amount of data we will need to query from the cluster is
>>> currently TBD and likely significantly affected by Heketi/GD1 vs. GD2
>>> choice.
>>>
>>
>> Do we have any partial list of data we will gather? Just want to
>> understand what this might entail already...
>>
>>
>>>
>>> -John
>>>
>>>
>>> On Wed, Jul 25, 2018 at 5:45 AM Pranith Kumar Karampuri <
>>> pkara...@redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Jul 24, 2018 at 10:10 PM, Sankarshan Mukhopadhyay <
>>>> sankarshan.mukhopadh...@gmail.com> wrote:
>>>>
>>>>> On Tue, Jul 24, 2018 at 9:48 PM, Pranith Kumar Karampuri
>>>>>  wrote:
>>>>> > hi,
>>>>> >   Quite a few commands to monitor gluster at the moment take
>>>>> almost a
>>>>> > second to give output.
>>>>>
>>>>> Is this at the (most) minimum recommended cluster size?
>>>>>
>>>>
>>>> Yes, with a single volume with 3 bricks i.e. 3 nodes in cluster.
>>>>
>>>>
>>>>>
>>>>> > Some categories of these commands:
>>>>> > 1) Any command that needs to do some sort of mount/glfs_init.
>>>>> >  Examples: 1) heal info family of commands 2) statfs to find
>>>>> > space-availability etc (On my laptop replica 3 volume with

Re: [Gluster-devel] How long should metrics collection on a cluster take?

[Pranith Kumar Karampuri]

On Wed, Jul 25, 2018 at 8:17 PM, John Strunk  wrote:

> To add an additional data point... The operator will need to regularly
> reconcile the true state of the gluster cluster with the desired state
> stored in kubernetes. This task will be required frequently (i.e.,
> operator-framework defaults to every 5s even if there are no config
> changes).
> The actual amount of data we will need to query from the cluster is
> currently TBD and likely significantly affected by Heketi/GD1 vs. GD2
> choice.
>

Do we have any partial list of data we will gather? Just want to understand
what this might entail already...


>
> -John
>
>
> On Wed, Jul 25, 2018 at 5:45 AM Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Tue, Jul 24, 2018 at 10:10 PM, Sankarshan Mukhopadhyay <
>> sankarshan.mukhopadh...@gmail.com> wrote:
>>
>>> On Tue, Jul 24, 2018 at 9:48 PM, Pranith Kumar Karampuri
>>>  wrote:
>>> > hi,
>>> >   Quite a few commands to monitor gluster at the moment take
>>> almost a
>>> > second to give output.
>>>
>>> Is this at the (most) minimum recommended cluster size?
>>>
>>
>> Yes, with a single volume with 3 bricks i.e. 3 nodes in cluster.
>>
>>
>>>
>>> > Some categories of these commands:
>>> > 1) Any command that needs to do some sort of mount/glfs_init.
>>> >  Examples: 1) heal info family of commands 2) statfs to find
>>> > space-availability etc (On my laptop replica 3 volume with all local
>>> bricks,
>>> > glfs_init takes 0.3 seconds on average)
>>> > 2) glusterd commands that need to wait for the previous command to
>>> unlock.
>>> > If the previous command is something related to lvm snapshot which
>>> takes
>>> > quite a few seconds, it would be even more time consuming.
>>> >
>>> > Nowadays container workloads have hundreds of volumes if not
>>> thousands. If
>>> > we want to serve any monitoring solution at this scale (I have seen
>>> > customers use upto 600 volumes at a time, it will only get bigger) and
>>> lets
>>> > say collecting metrics per volume takes 2 seconds per volume(Let us
>>> take the
>>> > worst example which has all major features enabled like
>>> > snapshot/geo-rep/quota etc etc), that will mean that it will take 20
>>> minutes
>>> > to collect metrics of the cluster with 600 volumes. What are the ways
>>> in
>>> > which we can make this number more manageable? I was initially
>>> thinking may
>>> > be it is possible to get gd2 to execute commands in parallel on
>>> different
>>> > volumes, so potentially we could get this done in ~2 seconds. But
>>> quite a
>>> > few of the metrics need a mount or equivalent of a mount(glfs_init) to
>>> > collect different information like statfs, number of pending heals,
>>> quota
>>> > usage etc. This may lead to high memory usage as the size of the
>>> mounts tend
>>> > to be high.
>>> >
>>>
>>> I am not sure if starting from the "worst example" (it certainly is
>>> not) is a good place to start from.
>>
>>
>> I didn't understand your statement. Are you saying 600 volumes is a worst
>> example?
>>
>>
>>> That said, for any environment
>>> with that number of disposable volumes, what kind of metrics do
>>> actually make any sense/impact?
>>>
>>
>> Same metrics you track for long running volumes. It is just that the way
>> the metrics
>> are interpreted will be different. On a long running volume, you would
>> look at the metrics
>> and try to find why is the volume not giving performance as expected in
>> the last 1 hour. Where as
>> in this case, you would look at metrics and find the reason why volumes
>> that were
>> created and deleted in the last hour didn't give performance as expected.
>>
>>
>>>
>>> > I wanted to seek suggestions from others on how to come to a conclusion
>>> > about which path to take and what problems to solve.
>>> >
>>> > I will be happy to raise github issues based on our conclusions on
>>> this mail
>>> > thread.
>>> >
>>> > --
>>> > Pranith
>>> >
>>>
>>>
>>>
>>>
>>>
>>> --
>>> sankarshan mukhopadhyay
>>> <https://about.me/sankarshan.mukhopadhyay>
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>
>>
>>
>> --
>> Pranith
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] How long should metrics collection on a cluster take?

[Pranith Kumar Karampuri]

On Tue, Jul 24, 2018 at 10:10 PM, Sankarshan Mukhopadhyay <
sankarshan.mukhopadh...@gmail.com> wrote:

> On Tue, Jul 24, 2018 at 9:48 PM, Pranith Kumar Karampuri
>  wrote:
> > hi,
> >   Quite a few commands to monitor gluster at the moment take almost a
> > second to give output.
>
> Is this at the (most) minimum recommended cluster size?
>

Yes, with a single volume with 3 bricks i.e. 3 nodes in cluster.


>
> > Some categories of these commands:
> > 1) Any command that needs to do some sort of mount/glfs_init.
> >  Examples: 1) heal info family of commands 2) statfs to find
> > space-availability etc (On my laptop replica 3 volume with all local
> bricks,
> > glfs_init takes 0.3 seconds on average)
> > 2) glusterd commands that need to wait for the previous command to
> unlock.
> > If the previous command is something related to lvm snapshot which takes
> > quite a few seconds, it would be even more time consuming.
> >
> > Nowadays container workloads have hundreds of volumes if not thousands.
> If
> > we want to serve any monitoring solution at this scale (I have seen
> > customers use upto 600 volumes at a time, it will only get bigger) and
> lets
> > say collecting metrics per volume takes 2 seconds per volume(Let us take
> the
> > worst example which has all major features enabled like
> > snapshot/geo-rep/quota etc etc), that will mean that it will take 20
> minutes
> > to collect metrics of the cluster with 600 volumes. What are the ways in
> > which we can make this number more manageable? I was initially thinking
> may
> > be it is possible to get gd2 to execute commands in parallel on different
> > volumes, so potentially we could get this done in ~2 seconds. But quite a
> > few of the metrics need a mount or equivalent of a mount(glfs_init) to
> > collect different information like statfs, number of pending heals, quota
> > usage etc. This may lead to high memory usage as the size of the mounts
> tend
> > to be high.
> >
>
> I am not sure if starting from the "worst example" (it certainly is
> not) is a good place to start from.


I didn't understand your statement. Are you saying 600 volumes is a worst
example?


> That said, for any environment
> with that number of disposable volumes, what kind of metrics do
> actually make any sense/impact?
>

Same metrics you track for long running volumes. It is just that the way
the metrics
are interpreted will be different. On a long running volume, you would look
at the metrics
and try to find why is the volume not giving performance as expected in the
last 1 hour. Where as
in this case, you would look at metrics and find the reason why volumes
that were
created and deleted in the last hour didn't give performance as expected.


>
> > I wanted to seek suggestions from others on how to come to a conclusion
> > about which path to take and what problems to solve.
> >
> > I will be happy to raise github issues based on our conclusions on this
> mail
> > thread.
> >
> > --
> > Pranith
> >
>
>
>
>
>
> --
> sankarshan mukhopadhyay
> <https://about.me/sankarshan.mukhopadhyay>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] How long should metrics collection on a cluster take?

[Pranith Kumar Karampuri]

hi,
  Quite a few commands to monitor gluster at the moment take almost a
second to give output.
Some categories of these commands:
1) Any command that needs to do some sort of mount/glfs_init.
 Examples: 1) heal info family of commands 2) statfs to find
space-availability etc (On my laptop replica 3 volume with all local
bricks, glfs_init takes 0.3 seconds on average)
2) glusterd commands that need to wait for the previous command to unlock.
If the previous command is something related to lvm snapshot which takes
quite a few seconds, it would be even more time consuming.

Nowadays container workloads have hundreds of volumes if not thousands. If
we want to serve any monitoring solution at this scale (I have seen
customers use upto 600 volumes at a time, it will only get bigger) and lets
say collecting metrics per volume takes 2 seconds per volume(Let us take
the worst example which has all major features enabled like
snapshot/geo-rep/quota etc etc), that will mean that it will take 20
minutes to collect metrics of the cluster with 600 volumes. What are the
ways in which we can make this number more manageable? I was initially
thinking may be it is possible to get gd2 to execute commands in parallel
on different volumes, so potentially we could get this done in ~2 seconds.
But quite a few of the metrics need a mount or equivalent of a
mount(glfs_init) to collect different information like statfs, number of
pending heals, quota usage etc. This may lead to high memory usage as the
size of the mounts tend to be high.

I wanted to seek suggestions from others on how to come to a conclusion
about which path to take and what problems to solve.

I will be happy to raise github issues based on our conclusions on this
mail thread.

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] The ctime of fstat is not correct which lead to "tar" utility error

[Pranith Kumar Karampuri]

+Ravi

On Thu, Jul 19, 2018 at 2:29 PM, Lian, George (NSB - CN/Hangzhou) <
george.l...@nokia-sbell.com> wrote:

> Hi, Gluster Experts,
>
>
>
> In glusterfs version 3.12.3, There seems a “fstat” issue for ctime after
> we use fsync,
>
> We have a demo execute binary which write some data and then do fsync for
> this file, it named as “tt”,
>
> Then run tar command right after “tt” command, it will always error with
> “tar: /mnt/test/file1.txt: file changed as we read it”
>
>
>
> The command output is list as the below, the source code and volume info
> configuration attached FYI,
>
> This issue will be 100% reproducible! (/mnt/test is mountpoint of
> glusterfs volume “test” , which the volume info is attached in mail)
>
> --
>
> ./tt;tar -czvf /tmp/abc.gz /mnt/test/file1.txt
>
> mtime:1531247107.27200
>
> ctime:1531247107.27200
>
> tar: Removing leading `/' from member names
>
> /mnt/test/file1.txt
>
> tar: /mnt/test/file1.txt: file changed as we read it
>
> --
>
>
>
> After my investigation, the xattrop for changelog is later than the fsync
> response , this is mean:
>
> In function  “afr_fsync_cbk” will call afr_delayed_changelog_wake_resume
> (this, local->fd, stub);
>
>
>
> In our case, it always a pending changelog , so glusterfs save the
> metadata information to stub, and handle pending changelog first,
>
> But the changelog will also change the ctime, from the packet captured by
> tcpdump, the response packet of xattrop will not include the metadata
> information,  and the wake_resume also not handle this metadata changed
> case.
>
>
>
> So in this case, the metadata in mdc_cache is not right, and when cache is
> valid, the application will get WRONG metadata!
>
>
>
> For verify my guess, if I change the configuration for this volume
>
> “gluster v set test md-cache-timeout 0” or
>
> “gluster v set export stat-prefetch off”
>
> This issue will be GONE!
>
>
>
>
>
> And I restore the configuration to default, which mean stat-prefetch is on
> and md-cache-timeout is 1 second,
>
> I try invalidate the md-cache in source code as the below in function
> mdc_fync_cbk on md-cache.c
>
> The issue also will be GONE!
>
>
>
> So GLusterFS Experts,
>
> Could you please verify this issue, and share your comments on my
> investigation?
>
> And your finally solutions is highly appreciated!
>
>
>
> changes in function “mdc_fsync_cbk”
>
> int
>
> mdc_fsync_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
>
>int32_t op_ret, int32_t op_errno,
>
>struct iatt *prebuf, struct iatt *postbuf, dict_t *xdata)
>
> {
>
> mdc_local_t  *local = NULL;
>
>
>
> local = frame->local;
>
>
>
> if (op_ret != 0)
>
> goto out;
>
>
>
> if (!local)
>
> goto out;
>
>
>
> mdc_inode_iatt_set_validate(this, local->fd->inode, prebuf,
> postbuf,
>
>  _gf_true);
>
> /* new added for ctime issue*/
>
> mdc_inode_iatt_invalidate(this, local->fd->inode);
>
>
> /* new added end*/
>
> out:
>
> MDC_STACK_UNWIND (fsync, frame, op_ret, op_errno, prebuf, postbuf,
>
>   xdata);
>
>
>
> return 0;
>
> }
>
> 
> -
>
> Best Regards,
>
> George
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [features/locks] Fetching lock info in lookup

[Pranith Kumar Karampuri]

On Thu, Jun 21, 2018 at 7:14 AM, Raghavendra Gowdappa 
wrote:

>
>
> On Thu, Jun 21, 2018 at 6:55 AM, Raghavendra Gowdappa  > wrote:
>
>>
>>
>> On Wed, Jun 20, 2018 at 9:09 PM, Xavi Hernandez 
>> wrote:
>>
>>> On Wed, Jun 20, 2018 at 4:29 PM Raghavendra Gowdappa <
>>> rgowd...@redhat.com> wrote:
>>>
 Krutika,

 This patch doesn't seem to be getting counts per domain, like number of
 inodelks or entrylks acquired in a domain "xyz". Am I right? If per domain
 stats are not available, passing interested domains in xdata_req would be
 needed. Any suggestions on that?

>>>
>>> We have GLUSTERFS_INODELK_DOM_COUNT. Its data should be a domain name
>>> for which we want to know the number of inodelks (the count is returned
>>> into GLUSTERFS_INODELK_COUNT though).
>>>
>>> It only exists for inodelk. If you need it for entrylk, it would need to
>>> be implemented.
>>>
>>
>> Yes. Realised that after going through the patch a bit more deeply.
>> Thanks. I'll implement a domain based entrylk count.
>>
>
> I think I need to have a dynamic key for responses. Otherwise its
> difficult to support requests on multiple domain in the same call.
> Embedding the domain name in key helps us to keep per domain results
> separate. Also needed is ways to send multiple domains in requests. If
> EC/AFR is already using it, there is high chance of overwriting previously
> set requests for different domains. Currently this is not consumed in
> lookup path by EC/AFR/Shard (DHT is interested for this information in
> lookup path) and hence not a pressing problem. But, we cannot rely on that.
>
> what do you think is a better interface among following alternatives?
>
> In request path,
>
> 1. Separate keys with domain name embedded - For eg.,
> glusterfs.inodelk.xyz.count. Value is ignored.
> 2. A single key like GLUSTERFS_INODELK_DOM_COUNT. Value is a string of
> interested domains separated by a delimiter (which character to use as
> delimiter?)
>
> In response path,
> 1. Separate keys with domain name embedded - For eg.,
> glusterfs.inodelk.xyz.count. Value is the total number of locks (granted +
> blocked).
> 2. A single key like GLUSTERFS_INODELK_DOM_COUNT. Value is a string of
> interested domains and lock count separated by a delimiter (which character
> to use as delimiter?)
>
> I personally prefer the domain name embedded in key approach as it avoids
> the string parsing by consumers. Any other approaches you can think of?
>

Only first option gives backward compatibility. So better to go with that.
In Ovirt and gluster-block cases new clients can operate at the time old
servers are in the cluster.


>
> As of now response returned is number of (granted + blocked) locks. For
> consumers using write-locks granted locks is always 1
>

This may not be true, if the granted locks are on specified ranges. So
depends on the ranges xlator sends inodelks I guess.


> and hence blocked locks can be inferred. But read-locks consumers this is
> not possible as there can be more than one read-lock consumers. For the
> requirement in DHT, we don't need the exact number. Instead we need the
> information about are there any granted locks, which can be given by the
> existing implementation. So, I am not changing that.
>

Try and change the existing macros/functions to achieve this so that all
fops get this functionality...


>
>
>
>>
>>> Xavi
>>>
>>>
 regards,
 Raghavendra

 On Wed, Jun 20, 2018 at 12:58 PM, Raghavendra Gowdappa <
 rgowd...@redhat.com> wrote:

>
>
> On Wed, Jun 20, 2018 at 12:06 PM, Krutika Dhananjay <
> kdhan...@redhat.com> wrote:
>
>> We do already have a way to get inodelk and entrylk count from a
>> bunch of fops, introduced in http://review.gluster.org/10880.
>> Can you check if you can make use of this feature?
>>
>
> Thanks Krutika. Yes, this feature meets DHT's requirement. We might
> need a GLUSTERFS_PARENT_INODELK, but that can be easily added along the
> lines of other counts. If necessary I'll send a patch to implement
> GLUSTERFS_PARENT_INODELK.
>
>
>> -Krutika
>>
>>
>> On Wed, Jun 20, 2018 at 9:17 AM, Amar Tumballi 
>> wrote:
>>
>>>
>>>
>>> On Wed, Jun 20, 2018 at 9:06 AM, Raghavendra Gowdappa <
>>> rgowd...@redhat.com> wrote:
>>>
 All,

 We've a requirement in DHT [1] to query the number of locks granted
 on an inode in lookup fop. I am planning to use xdata_req in lookup to 
 pass
 down the relevant arguments for this query. I am proposing following
 signature:

 In lookup request path following key value pairs will be passed in
 xdata_req:
 * "glusterfs.lock.type"
 - values can be "glusterfs.posix", "glusterfs.inodelk",
 "glusterfs.entrylk"
 * If the value of "glusterfs.lock.type" is "glusterfs.entrylk",
 then basename is passed 

Re: [Gluster-devel] Split brain after replacing a brick

[Pranith Kumar Karampuri]

Hey Manu,
   Long time! Sorry for the delay.

On Sat, Mar 31, 2018 at 8:12 PM, Emmanuel Dreyfus  wrote:

> Hello
>
> After doing a replace-brick and a full heal, I am left with:
>
> Brick bidon:/export/wd0e
> Status: Connected
> Number of entries: 0
>
> Brick baril:/export/wd0e
> Status: Connected
> Number of entries: 0
>
> Brick bidon:/export/wd1e
> Status: Connected
> Number of entries: 0
>
> Brick baril:/export/wd1e
> Status: Connected
> Number of entries: 0
>
> Brick bidon:/export/wd2e
> Status: Connected
> Number of entries: 0
>
> Brick baril:/export/wd2e
> /.attribute
> Status: Connected
> Number of entries: 1
>
> Brick bidon:/export/wd3e_tmp
> Status: Connected
> Number of entries: 0
>
> Brick baril:/export/wd3e
> / - Is in split-brain
>
> Status: Connected
> Number of entries: 1
>
>
> I guess the baril:/export/wd3e split -brain for / fits /.attribute on
> baril:/export/wd2e? How can I check?
>
> .attribute is the hidden directory where NetBSD stores extended
> attributes. It should be ignored by healing. Is there a bug to fix here?
>

Could you give the extended attributes of that directory on all the bricks
to figure out the kind of split-brain?


>
> --
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz
> m...@netbsd.org
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Any feature allow to add lock on a file between different apps?

[Pranith Kumar Karampuri]

You can use posix-locks i.e. fnctl based advisory locks on glusterfs just
like any other fs.

On Wed, Apr 4, 2018 at 8:30 AM, Lei Gong  wrote:

> Hello there，
>
>
>
> I want to know if there is a feature allow user to add lock on a file when
> their app is modifying that file, so that other apps could use the file
> when its unlocked.
>
>
>
> Thanks
>
>
>
> Lei
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 4.1: LTM release targeted for end of May

[Pranith Kumar Karampuri]

On Wed, Mar 14, 2018 at 8:27 PM, Amye Scavarda <a...@redhat.com> wrote:

> Responding on the architects question:
>
> On Tue, Mar 13, 2018 at 9:57 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>>
>>
>> On Tue, Mar 13, 2018 at 4:26 PM, Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>>>
>>>
>>> On Tue, Mar 13, 2018 at 1:51 PM, Amar Tumballi <atumb...@redhat.com>
>>> wrote:
>>>
>>>> >>
>>>>> >> Further, as we hit end of March, we would make it mandatory for
>>>>> features
>>>>> >> to have required spec and doc labels, before the code is merged, so
>>>>> >> factor in efforts for the same if not already done.
>>>>> >
>>>>> >
>>>>> > Could you explain the point above further? Is it just the label or
>>>>> the
>>>>> > spec/doc
>>>>> > that we need merged before the patch is merged?
>>>>> >
>>>>>
>>>>> I'll hazard a guess that the intent of the label is to indicate
>>>>> availability of the doc. "Completeness" of code is being defined as
>>>>> including specifications and documentation.
>>>>>
>>>>>
>>>> I believe this has originated from maintainers meeting agreements [1] .
>>>> The proposal to make a spec and documentation mandatory was submitted 3
>>>> months back and is documented, and submitted for comment @
>>>> https://docs.google.com/document/d/1AFkZmRRDXRxs21GnGauieI
>>>> yiIiRZ-nTEW8CPi7Gbp3g/edit?usp=sharing
>>>>
>>>>
>>> Thanks! This clears almost all the doubts I had :).
>>>
>>
>> The document above refers to Architects - "Now Architects are approved
>> to revert a patch which violates by either not having github issue nor
>> bug-id, or uses a bug-id to get the feature in etc."
>>
>> Who are they? What are their responsibilities?
>>
>
> In our last reboot of the maintainers readme file, we expanded the
> architects role:
> https://github.com/gluster/glusterfs/blob/master/MAINTAINERS
> General Project Architects
> --
> M: Jeff Darcy <j...@pl.atyp.us>
> M: Vijay Bellur <vbel...@redhat.com>
> P: Amar Tumballi <ama...@redhat.com>
> P: Pranith Karampuri <pkara...@redhat.com>
> P: Raghavendra Gowdappa <rgowd...@redhat.com>
> P: Shyamsundar Ranganathan <srang...@redhat.com>
> P: Niels de Vos <nde...@redhat.com>
> P: Xavier Hernandez <xhernan...@datalab.es>
> What should we work to make more clear about this?
>

Wow, embarrassing, I am an architect who doesn't know his responsibilities.
Could you let me know where I could find them?


> - amye
>
>
>>
>>
>>>
>>>
>>>> The idea is, if the code is going to be released, it should have
>>>> relevant documentation for users to use it, otherwise, it doesn't matter if
>>>> the feature is present or not. If the feature is 'default', and there is no
>>>> documentation required, just mention it, so the flags can be given. Also,
>>>> if there is no general agreement about the design, it doesn't make sense to
>>>> merge a feature and then someone has to redo things.
>>>>
>>>> For any experimental code, which we want to publish for other
>>>> developers to test, who doesn't need documentation, we have 'experimental'
>>>> branch, which should be used for validation.
>>>>
>>>
>>>>  [1] - http://lists.gluster.org/pipermail/gluster-devel/2017-Dece
>>>> mber/054070.html
>>>>
>>>
>>>
>>>
>>> --
>>> Pranith
>>>
>>
>>
>>
>> --
>> Pranith
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> Amye Scavarda | a...@redhat.com | Gluster Community Lead
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 4.1: LTM release targeted for end of May

[Pranith Kumar Karampuri]

On Tue, Mar 13, 2018 at 4:26 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Tue, Mar 13, 2018 at 1:51 PM, Amar Tumballi <atumb...@redhat.com>
> wrote:
>
>> >>
>>> >> Further, as we hit end of March, we would make it mandatory for
>>> features
>>> >> to have required spec and doc labels, before the code is merged, so
>>> >> factor in efforts for the same if not already done.
>>> >
>>> >
>>> > Could you explain the point above further? Is it just the label or the
>>> > spec/doc
>>> > that we need merged before the patch is merged?
>>> >
>>>
>>> I'll hazard a guess that the intent of the label is to indicate
>>> availability of the doc. "Completeness" of code is being defined as
>>> including specifications and documentation.
>>>
>>>
>> I believe this has originated from maintainers meeting agreements [1] .
>> The proposal to make a spec and documentation mandatory was submitted 3
>> months back and is documented, and submitted for comment @
>> https://docs.google.com/document/d/1AFkZmRRDXRxs21GnGauieI
>> yiIiRZ-nTEW8CPi7Gbp3g/edit?usp=sharing
>>
>>
> Thanks! This clears almost all the doubts I had :).
>

The document above refers to Architects - "Now Architects are approved to
revert a patch which violates by either not having github issue nor bug-id,
or uses a bug-id to get the feature in etc."

Who are they? What are their responsibilities?


>
>
>> The idea is, if the code is going to be released, it should have relevant
>> documentation for users to use it, otherwise, it doesn't matter if the
>> feature is present or not. If the feature is 'default', and there is no
>> documentation required, just mention it, so the flags can be given. Also,
>> if there is no general agreement about the design, it doesn't make sense to
>> merge a feature and then someone has to redo things.
>>
>> For any experimental code, which we want to publish for other developers
>> to test, who doesn't need documentation, we have 'experimental' branch,
>> which should be used for validation.
>>
>
>>  [1] - http://lists.gluster.org/pipermail/gluster-devel/2017-Dece
>> mber/054070.html
>>
>
>
>
> --
> Pranith
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 4.1: LTM release targeted for end of May

[Pranith Kumar Karampuri]

On Tue, Mar 13, 2018 at 7:07 AM, Shyam Ranganathan 
wrote:

> Hi,
>
> As we wind down on 4.0 activities (waiting on docs to hit the site, and
> packages to be available in CentOS repositories before announcing the
> release), it is time to start preparing for the 4.1 release.
>
> 4.1 is where we have GD2 fully functional and shipping with migration
> tools to aid Glusterd to GlusterD2 migrations.
>
> Other than the above, this is a call out for features that are in the
> works for 4.1. Please *post* the github issues to the *devel lists* that
> you would like as a part of 4.1, and also mention the current state of
> development.
>
> Further, as we hit end of March, we would make it mandatory for features
> to have required spec and doc labels, before the code is merged, so
> factor in efforts for the same if not already done.
>

Could you explain the point above further? Is it just the label or the
spec/doc
that we need merged before the patch is merged?


>
> Current 4.1 project release lane is empty! I cleaned it up, because I
> want to hear from all as to what content to add, than add things marked
> with the 4.1 milestone by default.
>
> Thanks,
> Shyam
> P.S: Also any volunteers to shadow/participate/run 4.1 as a release owner?
> ___
> maintainers mailing list
> maintain...@gluster.org
> http://lists.gluster.org/mailman/listinfo/maintainers
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] tests/bugs/core/bug-1432542-mpx-restart-crash.t generated core

[Pranith Kumar Karampuri]

Thanks Atin!

On Mon, Mar 12, 2018 at 9:49 AM, Atin Mukherjee <amukh...@redhat.com> wrote:

> Mohit is aware of this issue and currently working on a patch.
>
> On Mon, Mar 12, 2018 at 9:47 AM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>> hi,
>>  In https://build.gluster.org/job/centos7-regression/274/consoleFull,
>> the test in $SUBJECT generated core. It seems to be segfaulting in quota.
>> But I didn't take a closer look.
>>
>> --
>> Pranith
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] tests/bugs/core/bug-1432542-mpx-restart-crash.t generated core

[Pranith Kumar Karampuri]

hi,
 In https://build.gluster.org/job/centos7-regression/274/consoleFull,
the test in $SUBJECT generated core. It seems to be segfaulting in quota.
But I didn't take a closer look.

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Removal of use-compound-fops option in afr

[Pranith Kumar Karampuri]

On Mon, Mar 5, 2018 at 7:10 PM, Shyam Ranganathan <srang...@redhat.com>
wrote:

> On 03/04/2018 10:15 PM, Pranith Kumar Karampuri wrote:
> > Shyam,
> > Do let me know if there is anything that needs to be done on the
> > process front.
>
> I see that use-compound-fops is now deprecated, which is something that
> users need to be aware of.
>
> So when preparing release-notes, for when this change will hit a release
> (or are you intending to backport this?), we need to call out the same.
> The process for such a change is to look for github issues across the
> commits, and will get missed if this is a bug.
>
> At times bugs are fine, but again in this case, the bug reads eager lock
> changes, and the commit is to remove compounding. So users again may not
> be aware of the change looking at the bug list.
>
> So overall, from a process (and a release) standpoint, I would have
> preferred a github issue (or a bug, which for this change seems not to
> be the case), that calls out that this feature is to be
> deprecated/disabled.
>

Patch is yet to be merged. I am waiting for any further comments on this
thread.
I will change the patch to use the following github issue. I also added
some history
and reasoning as to why I am of the opinion to remove it now on the issue.

https://github.com/gluster/glusterfs/issues/414

I will wait till tomorrow morning IST before merging the patch in case
there are no
further comments.


> HTH.
>
> >
> > On Mon, Mar 5, 2018 at 8:18 AM, Pranith Kumar Karampuri
> > <pkara...@redhat.com <mailto:pkara...@redhat.com>> wrote:
> >
> > hi,
> >   We found that compound fops is not giving better performance
> > in replicate and I am thinking of removing that code. Sent the patch
> > at https://review.gluster.org/19655 <https://review.gluster.org/
> 19655>
> >
> > --
> > Pranith
> >
> >
> >
> >
> > --
> > Pranith
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Removal of use-compound-fops option in afr

[Pranith Kumar Karampuri]

On Mon, Mar 5, 2018 at 9:19 AM, Amar Tumballi  wrote:

> Pranith,
>
>
>
>>   We found that compound fops is not giving better performance in
>> replicate and I am thinking of removing that code. Sent the patch at
>> https://review.gluster.org/19655
>>
>>
> If I understand it right, as of now AFR is the only component which uses
> Compound FOP. If it stops using that code, should we maintain the compound
> fop codebase at all in other places, like protocol, volgen (decompounder
> etc?)
>

Infra was also supposed to be used by gfapi when compound fops is
introduced. So I think it is not a bad idea to keep it until at least there
is a decision about it. It will be similar to loading feature modules like
quota even when quota is not used on the volume.


>
> Because, if AFR as a module decides compound fop is not useful, and other
> method is better, it is completely a decision of AFR maintainers. I don't
> see a concern there.
>
> Only if it had an option and if people are using it, then warning them
> upfront about this change is better.
>

By default it is off, so I am not expecting it to be in wide use. Even if
they are using it, I don't see any dramatic improvement in their workload
performance based on the numbers we got. This is going to be affective in
4.1 release.

I also added gluster-users, so that it is communicated to wider audience.

>
> Regards,
> Amar
>
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Removal of use-compound-fops option in afr

[Pranith Kumar Karampuri]

Shyam,
Do let me know if there is anything that needs to be done on the
process front.

On Mon, Mar 5, 2018 at 8:18 AM, Pranith Kumar Karampuri <pkara...@redhat.com
> wrote:

> hi,
>   We found that compound fops is not giving better performance in
> replicate and I am thinking of removing that code. Sent the patch at
> https://review.gluster.org/19655
>
> --
> Pranith
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Removal of use-compound-fops option in afr

[Pranith Kumar Karampuri]

hi,
  We found that compound fops is not giving better performance in
replicate and I am thinking of removing that code. Sent the patch at
https://review.gluster.org/19655

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 4.0: RC1 tagged

[Pranith Kumar Karampuri]

I found the following memory leak present in 3.13, 4.0 and master:
https://bugzilla.redhat.com/show_bug.cgi?id=1550078

I will clone/port to 4.0 as soon as the patch is merged.

On Wed, Feb 28, 2018 at 5:55 PM, Javier Romero  wrote:

> Hi all,
>
> Have tested on CentOS Linux release 7.4.1708 (Core) with Kernel
> 3.10.0-693.17.1.el7.x86_64
>
> This package works ok
> http://cbs.centos.org/kojifiles/work/tasks/1548/311548/centos-release-
> gluster40-0.9-1.el7.centos.x86_64.rpm
>
> # yum install http://cbs.centos.org/kojifiles/work/tasks/1548/
> 311548/centos-release-gluster40-0.9-1.el7.centos.x86_64.rpm
> # yum install glusterfs-server
> # systemctl start glusterd
> # systemctl status glusterd
> ● glusterd.service - GlusterFS, a clustered file-system server
>Loaded: loaded (/usr/lib/systemd/system/glusterd.service; disabled;
> vendor preset: disabled)
>Active: active (running) since Wed 2018-02-28 09:18:46 -03; 53s ago
>  Main PID: 2070 (glusterd)
>CGroup: /system.slice/glusterd.service
>└─2070 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
> INFO
>
> Feb 28 09:18:46 centos-7 systemd[1]: Starting GlusterFS, a clustered
> file-system server...
> Feb 28 09:18:46 centos-7 systemd[1]: Started GlusterFS, a clustered
> file-system server.
>
>
>
>
> This one fails http://cbs.centos.org/kojifiles/work/tasks/1548/
> 311548/centos-release-gluster40-0.9-1.el7.centos.x86_64.rpm
>
> # yum install -y
> https://buildlogs.centos.org/centos/7/storage/x86_64/
> gluster-4.0/glusterfs-4.0.0-0.1.rc1.el7.x86_64.rpm
> Loaded plugins: fastestmirror, langpacks
> glusterfs-4.0.0-0.1.rc1.el7.x86_64.rpm
>   | 571 kB  00:00:00
> Examining /var/tmp/yum-root-BIfa9_/glusterfs-4.0.0-0.1.rc1.el7.x86_64.rpm:
> glusterfs-4.0.0-0.1.rc1.el7.x86_64
> Marking /var/tmp/yum-root-BIfa9_/glusterfs-4.0.0-0.1.rc1.el7.x86_64.rpm
> as an update to glusterfs-3.12.6-1.el7.x86_64
> Resolving Dependencies
> --> Running transaction check
> ---> Package glusterfs.x86_64 0:3.12.6-1.el7 will be updated
> --> Processing Dependency: glusterfs = 3.12.6-1.el7 for package:
> glusterfs-server-3.12.6-1.el7.x86_64
> base
>   | 3.6 kB  00:00:00
> centos-gluster312
>   | 2.9 kB  00:00:00
> extras
>   | 3.4 kB  00:00:00
> purpleidea-vagrant-libvirt
>   | 3.0 kB  00:00:00
> updates
>   | 3.4 kB  00:00:00
> centos-gluster312/7/x86_64/primary_db
>   |  87 kB  00:00:00
> Loading mirror speeds from cached hostfile
>  * base: centos.xfree.com.ar
>  * extras: centos.xfree.com.ar
>  * updates: centos.xfree.com.ar
> --> Processing Dependency: glusterfs = 3.12.6-1.el7 for package:
> glusterfs-api-3.12.6-1.el7.x86_64
> --> Processing Dependency: glusterfs = 3.12.6-1.el7 for package:
> glusterfs-fuse-3.12.6-1.el7.x86_64
> ---> Package glusterfs.x86_64 0:4.0.0-0.1.rc1.el7 will be an update
> --> Processing Dependency: glusterfs-libs = 4.0.0-0.1.rc1.el7 for
> package: glusterfs-4.0.0-0.1.rc1.el7.x86_64
> --> Finished Dependency Resolution
> Error: Package: glusterfs-server-3.12.6-1.el7.x86_64
> (@centos-gluster312-test)
>Requires: glusterfs = 3.12.6-1.el7
>Removing: glusterfs-3.12.6-1.el7.x86_64
> (@centos-gluster312-test)
>glusterfs = 3.12.6-1.el7
>Updated By: glusterfs-4.0.0-0.1.rc1.el7.x86_64
> (/glusterfs-4.0.0-0.1.rc1.el7.x86_64)
>glusterfs = 4.0.0-0.1.rc1.el7
>Available: glusterfs-3.8.4-18.4.el7.centos.x86_64 (base)
>glusterfs = 3.8.4-18.4.el7.centos
>Available: glusterfs-3.12.0-1.el7.x86_64 (centos-gluster312)
>glusterfs = 3.12.0-1.el7
>Available: glusterfs-3.12.1-1.el7.x86_64 (centos-gluster312)
>glusterfs = 3.12.1-1.el7
>Available: glusterfs-3.12.1-2.el7.x86_64 (centos-gluster312)
>glusterfs = 3.12.1-2.el7
>Available: glusterfs-3.12.3-1.el7.x86_64 (centos-gluster312)
>glusterfs = 3.12.3-1.el7
>Available: glusterfs-3.12.4-1.el7.x86_64 (centos-gluster312)
>glusterfs = 3.12.4-1.el7
>Available: glusterfs-3.12.5-2.el7.x86_64 (centos-gluster312)
>glusterfs = 3.12.5-2.el7
> Error: Package: glusterfs-api-3.12.6-1.el7.x86_64
> (@centos-gluster312-test)
>Requires: glusterfs = 3.12.6-1.el7
>Removing: glusterfs-3.12.6-1.el7.x86_64
> (@centos-gluster312-test)
>glusterfs = 3.12.6-1.el7
>Updated By: glusterfs-4.0.0-0.1.rc1.el7.x86_64
> (/glusterfs-4.0.0-0.1.rc1.el7.x86_64)
>glusterfs = 4.0.0-0.1.rc1.el7
>Available: glusterfs-3.8.4-18.4.el7.centos.x86_64 (base)
>glusterfs = 3.8.4-18.4.el7.centos
>

Re: [Gluster-devel] [Gluster-Maintainers] Release 4.0: Release notes (please read and contribute)

[Pranith Kumar Karampuri]

On Tue, Jan 30, 2018 at 3:40 AM, Shyam Ranganathan 
wrote:

> Hi,
>
> I have posted an initial draft version of the release notes here [1].
>
> I would like to *suggest* the following contributors to help improve and
> finish the release notes by 06th Feb, 2017. As you read this mail, if
> you feel you cannot contribute, do let us know, so that we can find the
> appropriate contributors for the same.
>
> NOTE: Please use the release tracker to post patches that modify the
> release notes, the bug ID is *1539842* (see [2]).
>
> 1) Aravinda/Kotresh: Geo-replication section in the release notes
>
> 2) Kaushal/Aravinda/ppai: GD2 section in the release notes
>
> 3) Du/Poornima/Pranith: Performance section in the release notes
>

https://review.gluster.org/19535 is posted for EC changes.


>
> 4) Amar: monitoring section in the release notes
>
> Following are individual call outs for certain features:
>
> 1) "Ability to force permissions while creating files/directories on a
> volume" - Niels
>
> 2) "Replace MD5 usage to enable FIPS support" - Ravi, Amar
>
> 3) "Dentry fop serializer xlator on brick stack" - Du
>
> 4) "Add option to disable nftw() based deletes when purging the landfill
> directory" - Amar
>
> 5) "Enhancements for directory listing in readdirp" - Nithya
>
> 6) "xlators should not provide init(), fini() and others directly, but
> have class_methods" - Amar
>
> 7) "New on-wire protocol (XDR) needed to support iattx and cleaner
> dictionary structure" - Amar
>
> 8) "The protocol xlators should prevent sending binary values in a dict
> over the networks" - Amar
>
> 9) "Translator to handle 'global' options" - Amar
>
> Thanks,
> Shyam
>
> [1] github link to draft release notes:
> https://github.com/gluster/glusterfs/blob/release-4.0/
> doc/release-notes/4.0.0.md
>
> [2] Initial gerrit patch for the release notes:
> https://review.gluster.org/#/c/19370/
> ___
> maintainers mailing list
> maintain...@gluster.org
> http://lists.gluster.org/mailman/listinfo/maintainers
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Glusterfs and Structured data

[Pranith Kumar Karampuri]

On Thu, Feb 8, 2018 at 12:05 PM, Raghavendra G 
wrote:

>
>
> On Tue, Feb 6, 2018 at 8:15 PM, Vijay Bellur  wrote:
>
>>
>>
>> On Sun, Feb 4, 2018 at 3:39 AM, Raghavendra Gowdappa > > wrote:
>>
>>> All,
>>>
>>> One of our users pointed out to the documentation that glusterfs is not
>>> good for storing "Structured data" [1], while discussing an issue [2].
>>
>>
>>
>> As far as I remember, the content around structured data in the Install
>> Guide is from a FAQ that was being circulated in Gluster, Inc. indicating
>> the startup's market positioning. Most of that was based on not wanting to
>> get into performance based comparisons of storage systems that are
>> frequently seen in the structured data space.
>>
>>
>>> Does any of you have more context on the feasibility of storing
>>> "structured data" on Glusterfs? Is one of the reasons for such a suggestion
>>> "staleness of metadata" as encountered in bugs like [3]?
>>>
>>
>>
>> There are challenges that distributed storage systems face when exposed
>> to applications that were written for a local filesystem interface. We have
>> encountered problems with applications like tar [4] that are not in the
>> realm of "Structured data". If we look at the common theme across all these
>> problems, it is related to metadata & read after write consistency issues
>> with the default translator stack that gets exposed on the client side.
>> While the default stack is optimal for other scenarios, it does seem that a
>> category of applications needing strict metadata consistency is not well
>> served by that. We have observed that disabling a few performance
>> translators and tuning cache timeouts for VFS/FUSE have helped to overcome
>> some of them. The WIP effort on timestamp consistency across the translator
>> stack, patches that have been merged as a result of the bugs that you
>> mention & other fixes for outstanding issues should certainly help in
>> catering to these workloads better with the file interface.
>>
>> There are deployments that I have come across where glusterfs is used for
>> storing structured data. gluster-block  & qemu-libgfapi overcome the
>> metadata consistency problem by exposing a file as a block device & by
>> disabling most of the performance translators in the default stack.
>> Workloads that have been deemed problematic with the file interface for the
>> reasons alluded above, function well with the block interface.
>>
>
> I agree that gluster-block due to its usage of a subset of glusterfs fops
> (mostly reads/writes I guess), runs into less number of consistency issues.
> However, as you've mentioned we seem to disable perf xlator stack in our
> tests/use-cases till now. Note that perf xlator stack is one of worst
> offenders as far as the metadata consistency is concerned (relatively less
> scenarios of data inconsistency). So, I wonder,
> * what would be the scenario if we enable perf xlator stack for
> gluster-block?
> * Is performance on gluster-block satisfactory so that we don't need these
> xlators?
>   - Or is it that these xlators are not useful for the workload usually
> run on gluster-block (For random read/write workload, read/write caching
> xlators offer less or no advantage)?
>

Yes. They are not useful. Block/VM files are opened with O_DIRECT, so we
don't enable caching at any layer in glusterfs. md-cache could be useful
for serving fstat from glusterfs. But apart from that I don't see any other
xlator contributing much.


>   - Or theoretically the workload is ought to benefit from perf xlators,
> but we don't see them in our results (there are open bugs to this effect)?
>
> I am asking these questions to ascertain priority on fixing perf xlators
> for (meta)data inconsistencies. If we offer a different solution for these
> workloads, the need for fixing these issues will be less.
>

My personal opinion is that both block and fs should work correctly. i.e.
caching xlators shouldn't lead to inconsistency issues. It would be better
if we are in a position where we choose a workload on block vs fs based on
their performance for that workload and nothing else. Block/VM usecases
change the workload of the application for glusterfs, so for small file
operations the kind of performance you see on block can never be achieved
by glusterfs with the current architecture/design.


>
> I feel that we have come a long way from the time the install guide was
>> written and an update for removing the "staleness of content" might be in
>> order there :-).
>>
>> Regards,
>> Vijay
>>
>> [4] https://bugzilla.redhat.com/show_bug.cgi?id=1058526
>>
>>
>>>
>>> [1] http://docs.gluster.org/en/latest/Install-Guide/Overview/
>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1512691
>>> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1390050
>>>
>>> regards,
>>> Raghavendra
>>> ___
>>> Gluster-devel mailing list
>>> 

Re: [Gluster-devel] Glusterfs and Structured data

[Pranith Kumar Karampuri]

On Sun, Feb 4, 2018 at 5:09 PM, Raghavendra Gowdappa 
wrote:

> All,
>
> One of our users pointed out to the documentation that glusterfs is not
> good for storing "Structured data" [1], while discussing an issue [2]. Does
> any of you have more context on the feasibility of storing "structured
> data" on Glusterfs? Is one of the reasons for such a suggestion "staleness
> of metadata" as encountered in bugs like [3]?\
>

I think the default configuration of glusterfs leads to unwanted behaviour
with structured data workloads. Based on my experience with customers I
handled, structured data usecase needs stronger read after write
consistency guarantees from multiple clients which I don't think we got a
chance to qualify. People who get it working generally disable perf
xlators. Then there are some issues that happen because of distributed
nature of glusterfs (For example: ctime).  So technically we can get
glusterfs to work for structured data workloads. We will need to find and
fix issues by trying out that workload. Performance qualification on that
will also be useful to analyse.


> [1] http://docs.gluster.org/en/latest/Install-Guide/Overview/
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1512691
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1390050
>
> regards,
> Raghavendra
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

+Ravi, +Raghavendra G

On 25 Jan 2018 8:49 am, "Pranith Kumar Karampuri" <pkara...@redhat.com>
wrote:

>
>
> On 25 Jan 2018 8:43 am, "Lian, George (NSB - CN/Hangzhou)" <
> george.l...@nokia-sbell.com> wrote:
>
> Hi,
>
> I suppose the zero filled attr is for performance consider to NFS, but for
> fuse, it will lead issue such like hard LINK FOP,
> So I suggest could we add 2 attr field in the endof "struct iatt {", such
> like ia_fuse_nlink, ia_fuse_ctime,
> And in function gf_zero_fill_stat , saved the ia_nlink, ia_ctime to
> ia_fuse_nlink,ia_fuse_ctime before set its to zero,
> And restore it to valued nlink and ctime in function gf_fuse_stat2attr,
> So that kernel could get the correct nlink and ctime.
>
> Is it a considerable solution? Any risk?
>
> Please share your comments, thanks in advance!
>
>
> Adding csaba for helping with this.
>
>
> Best Regards,
> George
>
> -Original Message-
> From: gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
> gluster.org] On Behalf Of Niels de Vos
> Sent: Wednesday, January 24, 2018 7:43 PM
> To: Pranith Kumar Karampuri <pkara...@redhat.com>
> Cc: Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>; Zhou,
> Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>; Li, Deqian
> (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>; Gluster-devel@gluster.org;
> Sun, Ping (NSB - CN/Hangzhou) <ping@nokia-sbell.com>
> Subject: Re: [Gluster-devel] a link issue maybe introduced in a bug fix "
> Don't let NFS cache stat after writes"
>
> On Wed, Jan 24, 2018 at 02:24:06PM +0530, Pranith Kumar Karampuri wrote:
> > hi,
> >In the same commit you mentioned earlier, there was this code
> > earlier:
> > -/* Returns 1 if the stat seems to be filled with zeroes. */ -int
> > -nfs_zero_filled_stat (struct iatt *buf) -{
> > -if (!buf)
> > -return 1;
> > -
> > -/* Do not use st_dev because it is transformed to store the
> xlator
> > id
> > - * in place of the device number. Do not use st_ino because by
> > this time
> > - * we've already mapped the root ino to 1 so it is not
> guaranteed
> > to be
> > - * 0.
> > - */
> > -if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))
> > -return 1;
> > -
> > -return 0;
> > -}
> > -
> > -
> >
> > I moved this to a common library function that can be used in afr as
> well.
> > Why was it there in NFS? +Niels for answering that question.
>
> Sorry, I dont know why that was done. It was introduced with the initial
> gNFS implementation, long before I started to work with Gluster. The only
> reference I have is this from
> xlators/nfs/server/src/nfs3-helpers.c:nfs3_stat_to_post_op_attr()
>
>  371 /* Some performance translators return zero-filled stats when
> they
>  372  * do not have up-to-date attributes. Need to handle this by
> not
>  373  * returning these zeroed out attrs.
>  374  */
>
> This may not be true for the current situation anymore.
>
> HTH,
> Niels
>
>
> >
> > If I give you a patch which will assert the error condition, would it
> > be possible for you to figure out the first xlator which is unwinding
> > the iatt with nlink count as zero but ctime as non-zero?
> >
> > On Wed, Jan 24, 2018 at 1:03 PM, Lian, George (NSB - CN/Hangzhou) <
> > george.l...@nokia-sbell.com> wrote:
> >
> > > Hi,  Pranith Kumar,
> > >
> > >
> > >
> > > Can you tell me while need set buf->ia_nlink to “0”in function
> > > gf_zero_fill_stat(), which API or Application will care it?
> > >
> > > If I remove this line and also update corresponding in function
> > > gf_is_zero_filled_stat,
> > >
> > > The issue seems gone, but I can’t confirm will lead to other issues.
> > >
> > >
> > >
> > > So could you please double check it and give your comments?
> > >
> > >
> > >
> > > My change is as the below:
> > >
> > >
> > >
> > > gf_boolean_t
> > >
> > > gf_is_zero_filled_stat (struct iatt *buf)
> > >
> > > {
> > >
> > > if (!buf)
> > >
> > > return 1;
> > >
> > >
> > >
> > > /* Do not use st_dev because it is transformed to store the
> > > xlator id
> > >

Re: [Gluster-devel] [FAILED][master] tests/basic/afr/durability-off.t

[Pranith Kumar Karampuri]

On Thu, Jan 25, 2018 at 3:09 PM, Milind Changire 
wrote:

> could AFR engineers check why tests/basic/afr/durability-off.t fails in
> brick-mux mode;
>

Issue seems to be something with connections to the bricks at the time of
mount.

*09:30:04* dd: opening `/mnt/glusterfs/0/a.txt': Transport endpoint is
not connected*09:30:10* ./tests/basic/afr/durability-off.t ..


> here's the job URL: https://build.gluster.org/job/centos6-regression/8654/
> console
>
> --
> Milind
>
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

On 25 Jan 2018 8:43 am, "Lian, George (NSB - CN/Hangzhou)" <
george.l...@nokia-sbell.com> wrote:

Hi,

I suppose the zero filled attr is for performance consider to NFS, but for
fuse, it will lead issue such like hard LINK FOP,
So I suggest could we add 2 attr field in the endof "struct iatt {", such
like ia_fuse_nlink, ia_fuse_ctime,
And in function gf_zero_fill_stat , saved the ia_nlink, ia_ctime to
ia_fuse_nlink,ia_fuse_ctime before set its to zero,
And restore it to valued nlink and ctime in function gf_fuse_stat2attr,
So that kernel could get the correct nlink and ctime.

Is it a considerable solution? Any risk?

Please share your comments, thanks in advance!


Adding csaba for helping with this.


Best Regards,
George

-Original Message-
From: gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
gluster.org] On Behalf Of Niels de Vos
Sent: Wednesday, January 24, 2018 7:43 PM
To: Pranith Kumar Karampuri <pkara...@redhat.com>
Cc: Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>; Zhou,
Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>; Li, Deqian (NSB
- CN/Hangzhou) <deqian...@nokia-sbell.com>; Gluster-devel@gluster.org; Sun,
Ping (NSB - CN/Hangzhou) <ping@nokia-sbell.com>
Subject: Re: [Gluster-devel] a link issue maybe introduced in a bug fix "
Don't let NFS cache stat after writes"

On Wed, Jan 24, 2018 at 02:24:06PM +0530, Pranith Kumar Karampuri wrote:
> hi,
>In the same commit you mentioned earlier, there was this code
> earlier:
> -/* Returns 1 if the stat seems to be filled with zeroes. */ -int
> -nfs_zero_filled_stat (struct iatt *buf) -{
> -if (!buf)
> -return 1;
> -
> -/* Do not use st_dev because it is transformed to store the
xlator
> id
> - * in place of the device number. Do not use st_ino because by
> this time
> - * we've already mapped the root ino to 1 so it is not guaranteed
> to be
> - * 0.
> - */
> -if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))
> -return 1;
> -
> -return 0;
> -}
> -
> -
>
> I moved this to a common library function that can be used in afr as well.
> Why was it there in NFS? +Niels for answering that question.

Sorry, I dont know why that was done. It was introduced with the initial
gNFS implementation, long before I started to work with Gluster. The only
reference I have is this from
xlators/nfs/server/src/nfs3-helpers.c:nfs3_stat_to_post_op_attr()

 371 /* Some performance translators return zero-filled stats when
they
 372  * do not have up-to-date attributes. Need to handle this by
not
 373  * returning these zeroed out attrs.
 374  */

This may not be true for the current situation anymore.

HTH,
Niels


>
> If I give you a patch which will assert the error condition, would it
> be possible for you to figure out the first xlator which is unwinding
> the iatt with nlink count as zero but ctime as non-zero?
>
> On Wed, Jan 24, 2018 at 1:03 PM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
> > Hi,  Pranith Kumar,
> >
> >
> >
> > Can you tell me while need set buf->ia_nlink to “0”in function
> > gf_zero_fill_stat(), which API or Application will care it?
> >
> > If I remove this line and also update corresponding in function
> > gf_is_zero_filled_stat,
> >
> > The issue seems gone, but I can’t confirm will lead to other issues.
> >
> >
> >
> > So could you please double check it and give your comments?
> >
> >
> >
> > My change is as the below:
> >
> >
> >
> > gf_boolean_t
> >
> > gf_is_zero_filled_stat (struct iatt *buf)
> >
> > {
> >
> > if (!buf)
> >
> > return 1;
> >
> >
> >
> > /* Do not use st_dev because it is transformed to store the
> > xlator id
> >
> >  * in place of the device number. Do not use st_ino because
> > by this time
> >
> >  * we've already mapped the root ino to 1 so it is not
> > guaranteed to be
> >
> >  * 0.
> >
> >  */
> >
> > //if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))
> >
> > if (buf->ia_ctime == )
> >
> > return 1;
> >
> >
> >
> > return 0;
> >
> > }
> >
> >
> >
> > void
> >
> > gf_zero_fill_stat (struct iatt *buf)
> >
> > {
> >
> > //   buf->ia_nlink = 0;
> >
> > buf->ia_cti

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

On Wed, Jan 24, 2018 at 2:24 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

> hi,
>In the same commit you mentioned earlier, there was this code
> earlier:
> -/* Returns 1 if the stat seems to be filled with zeroes. */
> -int
> -nfs_zero_filled_stat (struct iatt *buf)
> -{
> -if (!buf)
> -return 1;
> -
> -/* Do not use st_dev because it is transformed to store the
> xlator id
> - * in place of the device number. Do not use st_ino because by
> this time
> - * we've already mapped the root ino to 1 so it is not guaranteed
> to be
> - * 0.
> - */
> -if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))
> -return 1;
> -
> -return 0;
> -}
> -
> -
>
> I moved this to a common library function that can be used in afr as well.
> Why was it there in NFS? +Niels for answering that question.
>
> If I give you a patch which will assert the error condition, would it be
> possible for you to figure out the first xlator which is unwinding the iatt
> with nlink count as zero but ctime as non-zero?
>

Hey,
  I went through the code, I think gf_fuse_stat2attr() function could
be sending both ctime and nlink count as zero to the kernel like you were
mentioning. So I guess we should wait for Niels' answer about the need for
marking nlink count as zero. We may need to patch fuse code differently and
mark entry, attr valid seconds/nseconds to zero, so that we get a lookup on
the entry.


> On Wed, Jan 24, 2018 at 1:03 PM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
>> Hi,  Pranith Kumar,
>>
>>
>>
>> Can you tell me while need set buf->ia_nlink to “0”in function
>> gf_zero_fill_stat(), which API or Application will care it?
>>
>> If I remove this line and also update corresponding in function
>> gf_is_zero_filled_stat,
>>
>> The issue seems gone, but I can’t confirm will lead to other issues.
>>
>>
>>
>> So could you please double check it and give your comments?
>>
>>
>>
>> My change is as the below:
>>
>>
>>
>> gf_boolean_t
>>
>> gf_is_zero_filled_stat (struct iatt *buf)
>>
>> {
>>
>> if (!buf)
>>
>> return 1;
>>
>>
>>
>> /* Do not use st_dev because it is transformed to store the
>> xlator id
>>
>>  * in place of the device number. Do not use st_ino because by
>> this time
>>
>>  * we've already mapped the root ino to 1 so it is not guaranteed
>> to be
>>
>>  * 0.
>>
>>  */
>>
>> //if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))
>>
>> if (buf->ia_ctime == )
>>
>> return 1;
>>
>>
>>
>> return 0;
>>
>> }
>>
>>
>>
>> void
>>
>> gf_zero_fill_stat (struct iatt *buf)
>>
>> {
>>
>> //   buf->ia_nlink = 0;
>>
>> buf->ia_ctime = 0;
>>
>> }
>>
>>
>>
>> Thanks & Best Regards
>>
>> George
>>
>> *From:* Lian, George (NSB - CN/Hangzhou)
>> *Sent:* Friday, January 19, 2018 10:03 AM
>> *To:* Pranith Kumar Karampuri <pkara...@redhat.com>; Zhou, Cynthia (NSB
>> - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>
>> *Cc:* Li, Deqian (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>;
>> Gluster-devel@gluster.org; Sun, Ping (NSB - CN/Hangzhou) <
>> ping@nokia-sbell.com>
>>
>> *Subject:* RE: [Gluster-devel] a link issue maybe introduced in a bug
>> fix " Don't let NFS cache stat after writes"
>>
>>
>>
>> Hi,
>>
>> >>> Cool, this works for me too. Send me a mail off-list once you are
>> available and we can figure out a way to get into a call and work on this.
>>
>>
>>
>> Have you reproduced the issue per the step I listed in
>> https://bugzilla.redhat.com/show_bug.cgi?id=1531457 and last mail?
>>
>>
>>
>> If not, I would like you could try it yourself , which the difference
>> between yours and mine is just create only 2 bricks instead of 6 bricks.
>>
>>
>>
>> And Cynthia could have a session with you if you needed when I am not
>> available in next Monday and Tuesday.
>>
>>
>>
>> Thanks & Best Regards,
>>
>> George
>>
>>
>>
>> *From:* gluster-dev

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

hi,
   In the same commit you mentioned earlier, there was this code
earlier:
-/* Returns 1 if the stat seems to be filled with zeroes. */
-int
-nfs_zero_filled_stat (struct iatt *buf)
-{
-if (!buf)
-return 1;
-
-/* Do not use st_dev because it is transformed to store the xlator
id
- * in place of the device number. Do not use st_ino because by
this time
- * we've already mapped the root ino to 1 so it is not guaranteed
to be
- * 0.
- */
-if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))
-return 1;
-
-return 0;
-}
-
-

I moved this to a common library function that can be used in afr as well.
Why was it there in NFS? +Niels for answering that question.

If I give you a patch which will assert the error condition, would it be
possible for you to figure out the first xlator which is unwinding the iatt
with nlink count as zero but ctime as non-zero?

On Wed, Jan 24, 2018 at 1:03 PM, Lian, George (NSB - CN/Hangzhou) <
george.l...@nokia-sbell.com> wrote:

> Hi,  Pranith Kumar,
>
>
>
> Can you tell me while need set buf->ia_nlink to “0”in function
> gf_zero_fill_stat(), which API or Application will care it?
>
> If I remove this line and also update corresponding in function
> gf_is_zero_filled_stat,
>
> The issue seems gone, but I can’t confirm will lead to other issues.
>
>
>
> So could you please double check it and give your comments?
>
>
>
> My change is as the below:
>
>
>
> gf_boolean_t
>
> gf_is_zero_filled_stat (struct iatt *buf)
>
> {
>
> if (!buf)
>
> return 1;
>
>
>
> /* Do not use st_dev because it is transformed to store the xlator
> id
>
>  * in place of the device number. Do not use st_ino because by
> this time
>
>  * we've already mapped the root ino to 1 so it is not guaranteed
> to be
>
>  * 0.
>
>  */
>
> //if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))
>
> if (buf->ia_ctime == )
>
> return 1;
>
>
>
> return 0;
>
> }
>
>
>
> void
>
> gf_zero_fill_stat (struct iatt *buf)
>
> {
>
> //   buf->ia_nlink = 0;
>
> buf->ia_ctime = 0;
>
> }
>
>
>
> Thanks & Best Regards
>
> George
>
> *From:* Lian, George (NSB - CN/Hangzhou)
> *Sent:* Friday, January 19, 2018 10:03 AM
> *To:* Pranith Kumar Karampuri <pkara...@redhat.com>; Zhou, Cynthia (NSB -
> CN/Hangzhou) <cynthia.z...@nokia-sbell.com>
> *Cc:* Li, Deqian (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>;
> Gluster-devel@gluster.org; Sun, Ping (NSB - CN/Hangzhou) <
> ping@nokia-sbell.com>
>
> *Subject:* RE: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
> Hi,
>
> >>> Cool, this works for me too. Send me a mail off-list once you are
> available and we can figure out a way to get into a call and work on this.
>
>
>
> Have you reproduced the issue per the step I listed in
> https://bugzilla.redhat.com/show_bug.cgi?id=1531457 and last mail?
>
>
>
> If not, I would like you could try it yourself , which the difference
> between yours and mine is just create only 2 bricks instead of 6 bricks.
>
>
>
> And Cynthia could have a session with you if you needed when I am not
> available in next Monday and Tuesday.
>
>
>
> Thanks & Best Regards,
>
> George
>
>
>
> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
> gluster.org <gluster-devel-boun...@gluster.org>] *On Behalf Of *Pranith
> Kumar Karampuri
> *Sent:* Thursday, January 18, 2018 6:03 PM
> *To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
> Li, Deqian (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>;
> Gluster-devel@gluster.org; Sun, Ping (NSB - CN/Hangzhou) <
> ping@nokia-sbell.com>
> *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
>
>
>
>
> On Thu, Jan 18, 2018 at 12:17 PM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
> Hi,
>
> >>>I actually tried it with replica-2 and replica-3 and then distributed
> replica-2 before replying to the earlier mail. We can have a debugging
> session if you are okay with it.
>
>
>
> It is fine if you can’t reproduce the issue in your ENV.
>
> And I has attached the deta

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

If ctime is zero, no xlator should consider it as a good iatt. The fact
that this is happening means some xlator is not doing proper checks in
code. We need to find what that xlator is and fix it. Internet in our new
office is not working so I'm not able to have call today with you guys.
What I would do is to put logs in lookup, link, fstat, stat calls to see if
anyone unwound iatt with ia_nlink count as zero but ctime as nonzero.

On 24 Jan 2018 1:03 pm, "Lian, George (NSB - CN/Hangzhou)" <
george.l...@nokia-sbell.com> wrote:

Hi,  Pranith Kumar,



Can you tell me while need set buf->ia_nlink to “0”in function
gf_zero_fill_stat(), which API or Application will care it?

If I remove this line and also update corresponding in function
gf_is_zero_filled_stat,

The issue seems gone, but I can’t confirm will lead to other issues.



So could you please double check it and give your comments?



My change is as the below:



gf_boolean_t

gf_is_zero_filled_stat (struct iatt *buf)

{

if (!buf)

return 1;



/* Do not use st_dev because it is transformed to store the xlator
id

 * in place of the device number. Do not use st_ino because by this
time

 * we've already mapped the root ino to 1 so it is not guaranteed
to be

 * 0.

 */

//if ((buf->ia_nlink == 0) && (buf->ia_ctime == 0))

if (buf->ia_ctime == )

return 1;



return 0;

}



void

gf_zero_fill_stat (struct iatt *buf)

{

//   buf->ia_nlink = 0;

buf->ia_ctime = 0;

}



Thanks & Best Regards

George

*From:* Lian, George (NSB - CN/Hangzhou)
*Sent:* Friday, January 19, 2018 10:03 AM
*To:* Pranith Kumar Karampuri <pkara...@redhat.com>; Zhou, Cynthia (NSB -
CN/Hangzhou) <cynthia.z...@nokia-sbell.com>
*Cc:* Li, Deqian (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>;
Gluster-devel@gluster.org; Sun, Ping (NSB - CN/Hangzhou) <
ping@nokia-sbell.com>

*Subject:* RE: [Gluster-devel] a link issue maybe introduced in a bug fix "
Don't let NFS cache stat after writes"



Hi,

>>> Cool, this works for me too. Send me a mail off-list once you are
available and we can figure out a way to get into a call and work on this.



Have you reproduced the issue per the step I listed in
https://bugzilla.redhat.com/show_bug.cgi?id=1531457 and last mail?



If not, I would like you could try it yourself , which the difference
between yours and mine is just create only 2 bricks instead of 6 bricks.



And Cynthia could have a session with you if you needed when I am not
available in next Monday and Tuesday.



Thanks & Best Regards,

George



*From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
gluster.org <gluster-devel-boun...@gluster.org>] *On Behalf Of *Pranith
Kumar Karampuri
*Sent:* Thursday, January 18, 2018 6:03 PM
*To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
*Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>; Li,
Deqian (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>;
Gluster-devel@gluster.org; Sun, Ping (NSB - CN/Hangzhou) <
ping@nokia-sbell.com>
*Subject:* Re: [Gluster-devel] a link issue maybe introduced in a bug fix "
Don't let NFS cache stat after writes"







On Thu, Jan 18, 2018 at 12:17 PM, Lian, George (NSB - CN/Hangzhou) <
george.l...@nokia-sbell.com> wrote:

Hi,

>>>I actually tried it with replica-2 and replica-3 and then distributed
replica-2 before replying to the earlier mail. We can have a debugging
session if you are okay with it.



It is fine if you can’t reproduce the issue in your ENV.

And I has attached the detail reproduce log in the Bugzilla FYI



But I am sorry I maybe OOO at Monday and Tuesday next week, so debug
session will be fine to me at next Wednesday.



Cool, this works for me too. Send me a mail off-list once you are available
and we can figure out a way to get into a call and work on this.







Paste the detail reproduce log FYI here:

*root@ubuntu:~# gluster peer probe ubuntu*

*peer probe: success. Probe on localhost not needed*

*root@ubuntu:~# gluster v create test replica 2 ubuntu:/home/gfs/b1
ubuntu:/home/gfs/b2 force*

*volume create: test: success: please start the volume to access data*

*root@ubuntu:~# gluster v start test*

*volume start: test: success*

*root@ubuntu:~# gluster v info test*



*Volume Name: test*

*Type: Replicate*

*Volume ID: fef5fca3-81d9-46d3-8847-74cde6f701a5*

*Status: Started*

*Snapshot Count: 0*

*Number of Bricks: 1 x 2 = 2*

*Transport-type: tcp*

*Bricks:*

*Brick1: ubuntu:/home/gfs/b1*

*Brick2: ubuntu:/home/gfs/b2*

*Options Reconfigured:*

*transport.address-family: inet*

*nfs.disable: on*

*performance.client-io-threads: off*

*root@ubuntu:~# gluster v status*

*Status of volume: test*

*Gluster process TCP Port  RDMA Por

Re: [Gluster-devel] Release 3.13.2: Planned for 19th of Jan, 2018

[Pranith Kumar Karampuri]

On Fri, Jan 19, 2018 at 6:19 AM, Shyam Ranganathan 
wrote:

> On 01/18/2018 07:34 PM, Ravishankar N wrote:
> >
> >
> > On 01/18/2018 11:53 PM, Shyam Ranganathan wrote:
> >> On 01/02/2018 11:08 AM, Shyam Ranganathan wrote:
> >>> Hi,
> >>>
> >>> As release 3.13.1 is announced, here is are the needed details for
> >>> 3.13.2
> >>>
> >>> Release date: 19th Jan, 2018 (20th is a Saturday)
> >> Heads up, this is tomorrow.
> >>
> >>> Tracker bug for blockers:
> >>> https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.13.2
> >> The one blocker bug has had its patch merged, so I am assuming there are
> >> no more that should block this release.
> >>
> >> As usual, shout out in case something needs attention.
> >
> > Hi Shyam,
> >
> > 1. There is one patch https://review.gluster.org/#/c/19218/ which
> > introduces full locks for afr writevs. We're introducing this as a
> > GD_OP_VERSION_3_13_2 option. Please wait for it to be merged on 3.13
> > branch today. Karthik, please back port the patch.
>
> Do we need this behind an option, if existing behavior causes split
> brains? Or is the option being added for workloads that do not have
> multiple clients or clients writing to non-overlapping regions (and thus
> need not suffer a penalty in performance maybe? But they should not
> anyway as a single client and AFR eager locks should ensure this is done
> only once for the lifetime of the file being accesses, right?)
>
> Basically I would like to keep options out it possible in backports, as
> that changes the gluster op-version and involves other upgrade steps to
> be sure users can use this option etc. Which means more reading and
> execution of upgrade steps for our users. Hence the concern!
>
> >
> > 2. I'm also backporting https://review.gluster.org/#/c/18571/. Please
> > consider merging it too today if it is ready.
>

Let's take this one in 3.13.3, I think we need to test a few more cases
that I missed
at the time of review.


>
> This should be fine.
>
> >
> > We will attach the relevant BZs to the tracker bug.
> >
> > Thanks
> > Ravi
> >>
> >>> Shyam
> >>> ___
> >>> Gluster-devel mailing list
> >>> Gluster-devel@gluster.org
> >>> http://lists.gluster.org/mailman/listinfo/gluster-devel
> >>>
> >> ___
> >> Gluster-devel mailing list
> >> Gluster-devel@gluster.org
> >> http://lists.gluster.org/mailman/listinfo/gluster-devel
> >
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] cluster/dht: restrict migration of opened files

[Pranith Kumar Karampuri]

On Tue, Jan 16, 2018 at 2:52 PM, Raghavendra Gowdappa 
wrote:

> All,
>
> Patch [1] prevents migration of opened files during rebalance operation.
> If patch [1] affects you, please voice out your concerns. [1] is a stop-gap
> fix for the problem discussed in issues [2][3]
>

What is the impact on VM and gluster-block usecases after this patch? Will
it rebalance any data in these usecases?


>
> [1] https://review.gluster.org/#/c/19202/
> [2] https://github.com/gluster/glusterfs/issues/308
> [3] https://github.com/gluster/glusterfs/issues/347
>
> regards,
> Raghavendra
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

On Thu, Jan 18, 2018 at 12:17 PM, Lian, George (NSB - CN/Hangzhou) <
george.l...@nokia-sbell.com> wrote:

> Hi,
>
> >>>I actually tried it with replica-2 and replica-3 and then distributed
> replica-2 before replying to the earlier mail. We can have a debugging
> session if you are okay with it.
>
>
>
> It is fine if you can’t reproduce the issue in your ENV.
>
> And I has attached the detail reproduce log in the Bugzilla FYI
>
>
>
> But I am sorry I maybe OOO at Monday and Tuesday next week, so debug
> session will be fine to me at next Wednesday.
>

Cool, this works for me too. Send me a mail off-list once you are available
and we can figure out a way to get into a call and work on this.


>
>
>
>
> Paste the detail reproduce log FYI here:
>
> *root@ubuntu:~# gluster peer probe ubuntu*
>
> *peer probe: success. Probe on localhost not needed*
>
> *root@ubuntu:~# gluster v create test replica 2 ubuntu:/home/gfs/b1
> ubuntu:/home/gfs/b2 force*
>
> *volume create: test: success: please start the volume to access data*
>
> *root@ubuntu:~# gluster v start test*
>
> *volume start: test: success*
>
> *root@ubuntu:~# gluster v info test*
>
>
>
> *Volume Name: test*
>
> *Type: Replicate*
>
> *Volume ID: fef5fca3-81d9-46d3-8847-74cde6f701a5*
>
> *Status: Started*
>
> *Snapshot Count: 0*
>
> *Number of Bricks: 1 x 2 = 2*
>
> *Transport-type: tcp*
>
> *Bricks:*
>
> *Brick1: ubuntu:/home/gfs/b1*
>
> *Brick2: ubuntu:/home/gfs/b2*
>
> *Options Reconfigured:*
>
> *transport.address-family: inet*
>
> *nfs.disable: on*
>
> *performance.client-io-threads: off*
>
> *root@ubuntu:~# gluster v status*
>
> *Status of volume: test*
>
> *Gluster process TCP Port  RDMA Port  Online
> Pid*
>
>
> *--*
>
> *Brick ubuntu:/home/gfs/b1   49152 0  Y
> 7798*
>
> *Brick ubuntu:/home/gfs/b2   49153 0  Y
> 7818*
>
> *Self-heal Daemon on localhost   N/A   N/AY
> 7839*
>
>
>
> *Task Status of Volume test*
>
>
> *--*
>
> *There are no active volume tasks*
>
>
>
>
>
> *root@ubuntu:~# gluster v set test cluster.consistent-metadata on*
>
> *volume set: success*
>
>
>
> *root@ubuntu:~# ls /mnt/test*
>
> *ls: cannot access '/mnt/test': No such file or directory*
>
> *root@ubuntu:~# mkdir -p /mnt/test*
>
> *root@ubuntu:~# mount -t glusterfs ubuntu:/test /mnt/test*
>
>
>
> *root@ubuntu:~# cd /mnt/test*
>
> *root@ubuntu:/mnt/test# echo "abc">aaa*
>
> *root@ubuntu:/mnt/test# cp aaa bbb;link bbb ccc*
>
>
>
> *root@ubuntu:/mnt/test# kill -9 7818*
>
> *root@ubuntu:/mnt/test# cp aaa ddd;link ddd eee*
>
> *link: cannot create link 'eee' to 'ddd': No such file or directory*
>
>
>
>
>
> Best Regards,
>
> George
>
>
>
> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
> gluster.org] *On Behalf Of *Pranith Kumar Karampuri
> *Sent:* Thursday, January 18, 2018 2:40 PM
>
> *To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
> Gluster-devel@gluster.org; Li, Deqian (NSB - CN/Hangzhou) <
> deqian...@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <
> ping@nokia-sbell.com>
> *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
>
>
>
>
> On Thu, Jan 18, 2018 at 6:33 AM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
> Hi,
>
> I suppose the brick numbers in your testing is six, and you just shut down
> the 3 process.
>
> When I reproduce the issue, I only create a replicate volume with 2
> bricks, only let ONE brick working and set cluster.consistent-metadata on,
>
> With this 2 test condition, the issue could 100% reproducible.
>
>
>
> Hi,
>
>   I actually tried it with replica-2 and replica-3 and then
> distributed replica-2 before replying to the earlier mail. We can have a
> debugging session if you are okay with it.
>
> I am in the middle of a customer issue myself(That is the reason for this
> delay :-( ) and thinking of wrapping it up early next week. Would that be
> fine with you?
>
>
>
>
>
>
>
>
>
> 16:44:28 :) ⚡ gluster v status
>
> Status of volum

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

On Thu, Jan 18, 2018 at 6:33 AM, Lian, George (NSB - CN/Hangzhou) <
george.l...@nokia-sbell.com> wrote:

> Hi,
>
> I suppose the brick numbers in your testing is six, and you just shut down
> the 3 process.
>
> When I reproduce the issue, I only create a replicate volume with 2
> bricks, only let ONE brick working and set cluster.consistent-metadata on,
>
> With this 2 test condition, the issue could 100% reproducible.
>

Hi,
  I actually tried it with replica-2 and replica-3 and then distributed
replica-2 before replying to the earlier mail. We can have a debugging
session if you are okay with it.
I am in the middle of a customer issue myself(That is the reason for this
delay :-( ) and thinking of wrapping it up early next week. Would that be
fine with you?


>
>
>
>
>
>
> 16:44:28 :) ⚡ gluster v status
>
> Status of volume: r2
>
> Gluster process TCP Port  RDMA Port  Online
> Pid
>
> 
> --
>
> Brick localhost.localdomain:/home/gfs/r2_0  49152 0  Y
> 5309
>
> Brick localhost.localdomain:/home/gfs/r2_1  49154 0  Y
> 5330
>
> Brick localhost.localdomain:/home/gfs/r2_2  49156 0  Y
> 5351
>
> Brick localhost.localdomain:/home/gfs/r2_3  49158 0  Y
> 5372
>
> Brick localhost.localdomain:/home/gfs/r2_4  49159 0  Y
> 5393
>
> Brick localhost.localdomain:/home/gfs/r2_5  49160 0  Y
> 5414
>
> Self-heal Daemon on localhost   N/A   N/AY
> 5436
>
>
>
> Task Status of Volume r2
>
> 
> --
>
> There are no active volume tasks
>
>
>
> root@dhcp35-190 - ~
>
> 16:44:38 :) ⚡ kill -9 5309 5351 5393
>
>
>
> Best Regards,
>
> George
>
> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
> gluster.org] *On Behalf Of *Pranith Kumar Karampuri
> *Sent:* Wednesday, January 17, 2018 7:27 PM
> *To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
> *Cc:* Li, Deqian (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>;
> Gluster-devel@gluster.org; Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.z...@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <
> ping@nokia-sbell.com>
>
> *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
>
>
>
>
> On Mon, Jan 15, 2018 at 1:55 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>
>
>
>
> On Mon, Jan 15, 2018 at 8:46 AM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
> Hi,
>
>
>
> Have you reproduced this issue? If yes, could you please confirm whether
> it is an issue or not?
>
>
>
> Hi,
>
>I tried recreating this on my laptop and on both master and 3.12
> and I am not able to recreate the issue :-(.
>
> Here is the execution log: https://paste.fedoraproject.org/paste/-
> csXUKrwsbrZAVW1KzggQQ
>
> Since I was doing this on my laptop, I changed shutting down of the
> replica to killing the brick process to simulate this test.
>
> Let me know if I missed something.
>
>
>
>
>
> Sorry, I am held up with some issue at work, so I think I will get some
> time day after tomorrow to look at this. In the mean time I am adding more
> people who know about afr to see if they get a chance to work on this
> before me.
>
>
>
>
>
> And if it is an issue,  do you have any solution for this issue?
>
>
>
> Thanks & Best Regards,
>
> George
>
>
>
> *From:* Lian, George (NSB - CN/Hangzhou)
> *Sent:* Thursday, January 11, 2018 2:01 PM
> *To:* Pranith Kumar Karampuri <pkara...@redhat.com>
> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
> Gluster-devel@gluster.org; Li, Deqian (NSB - CN/Hangzhou) <
> deqian...@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <
> ping@nokia-sbell.com>
> *Subject:* RE: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
> Hi,
>
>
>
> Please see detail test step on https://bugzilla.redhat.com/
> show_bug.cgi?id=1531457
>
>
>
> How reproducible:
>
>
>
>
>
> Steps to Reproduce:
>
> 1.create a volume name "test" with replicated
>
> 2.set volume option cluster.consistent-metadata with on:
>
>   gluster v set test cluster.consistent-metadata on
>
> 3. mount volume test

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

On Mon, Jan 15, 2018 at 1:55 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Mon, Jan 15, 2018 at 8:46 AM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
>> Hi,
>>
>>
>>
>> Have you reproduced this issue? If yes, could you please confirm whether
>> it is an issue or not?
>>
>
Hi,
   I tried recreating this on my laptop and on both master and 3.12 and
I am not able to recreate the issue :-(.
Here is the execution log:
https://paste.fedoraproject.org/paste/-csXUKrwsbrZAVW1KzggQQ
Since I was doing this on my laptop, I changed shutting down of the replica
to killing the brick process to simulate this test.
Let me know if I missed something.


> Sorry, I am held up with some issue at work, so I think I will get some
> time day after tomorrow to look at this. In the mean time I am adding more
> people who know about afr to see if they get a chance to work on this
> before me.
>
>
>>
>>
>> And if it is an issue,  do you have any solution for this issue?
>>
>>
>>
>> Thanks & Best Regards,
>>
>> George
>>
>>
>>
>> *From:* Lian, George (NSB - CN/Hangzhou)
>> *Sent:* Thursday, January 11, 2018 2:01 PM
>> *To:* Pranith Kumar Karampuri <pkara...@redhat.com>
>> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
>> Gluster-devel@gluster.org; Li, Deqian (NSB - CN/Hangzhou) <
>> deqian...@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <
>> ping@nokia-sbell.com>
>> *Subject:* RE: [Gluster-devel] a link issue maybe introduced in a bug
>> fix " Don't let NFS cache stat after writes"
>>
>>
>>
>> Hi,
>>
>>
>>
>> Please see detail test step on https://bugzilla.redhat.com/sh
>> ow_bug.cgi?id=1531457
>>
>>
>>
>> How reproducible:
>>
>>
>>
>>
>>
>> Steps to Reproduce:
>>
>> 1.create a volume name "test" with replicated
>>
>> 2.set volume option cluster.consistent-metadata with on:
>>
>>   gluster v set test cluster.consistent-metadata on
>>
>> 3. mount volume test on client on /mnt/test
>>
>> 4. create a file aaa size more than 1 byte
>>
>>echo "1234567890" >/mnt/test/aaa
>>
>> 5. shutdown a replicat node, let's say sn-1, only let sn-0 worked
>>
>> 6. cp /mnt/test/aaa /mnt/test/bbb; link /mnt/test/bbb /mnt/test/ccc
>>
>>
>>
>>
>>
>> BRs
>>
>> George
>>
>>
>>
>> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
>> gluster.org <gluster-devel-boun...@gluster.org>] *On Behalf Of *Pranith
>> Kumar Karampuri
>> *Sent:* Thursday, January 11, 2018 12:39 PM
>> *To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
>> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
>> Gluster-devel@gluster.org; Li, Deqian (NSB - CN/Hangzhou) <
>> deqian...@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <
>> ping@nokia-sbell.com>
>> *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a bug
>> fix " Don't let NFS cache stat after writes"
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Jan 11, 2018 at 6:35 AM, Lian, George (NSB - CN/Hangzhou) <
>> george.l...@nokia-sbell.com> wrote:
>>
>> Hi,
>>
>> >>> In which protocol are you seeing this issue? Fuse/NFS/SMB?
>>
>> It is fuse, within mountpoint by “mount -t glusterfs  …“ command.
>>
>>
>>
>> Could you let me know the test you did so that I can try to re-create and
>> see what exactly is going on?
>>
>> Configuration of the volume and the steps to re-create the issue you are
>> seeing would be helpful in debugging the issue further.
>>
>>
>>
>>
>>
>> Thanks & Best Regards,
>>
>> George
>>
>>
>>
>> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
>> gluster.org] *On Behalf Of *Pranith Kumar Karampuri
>> *Sent:* Wednesday, January 10, 2018 8:08 PM
>> *To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
>> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
>> Zhong, Hua (NSB - CN/Hangzhou) <hua.zh...@nokia-sbell.com>; Li, Deqian
>> (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>;
>> Gluster-devel@gluster.org; Sun, Ping (NSB - CN/Hangzhou) <
>&g

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

On Mon, Jan 15, 2018 at 8:46 AM, Lian, George (NSB - CN/Hangzhou) <
george.l...@nokia-sbell.com> wrote:

> Hi,
>
>
>
> Have you reproduced this issue? If yes, could you please confirm whether
> it is an issue or not?
>

Sorry, I am held up with some issue at work, so I think I will get some
time day after tomorrow to look at this. In the mean time I am adding more
people who know about afr to see if they get a chance to work on this
before me.


>
>
> And if it is an issue,  do you have any solution for this issue?
>
>
>
> Thanks & Best Regards,
>
> George
>
>
>
> *From:* Lian, George (NSB - CN/Hangzhou)
> *Sent:* Thursday, January 11, 2018 2:01 PM
> *To:* Pranith Kumar Karampuri <pkara...@redhat.com>
> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
> Gluster-devel@gluster.org; Li, Deqian (NSB - CN/Hangzhou) <
> deqian...@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <
> ping@nokia-sbell.com>
> *Subject:* RE: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
> Hi,
>
>
>
> Please see detail test step on https://bugzilla.redhat.com/
> show_bug.cgi?id=1531457
>
>
>
> How reproducible:
>
>
>
>
>
> Steps to Reproduce:
>
> 1.create a volume name "test" with replicated
>
> 2.set volume option cluster.consistent-metadata with on:
>
>   gluster v set test cluster.consistent-metadata on
>
> 3. mount volume test on client on /mnt/test
>
> 4. create a file aaa size more than 1 byte
>
>echo "1234567890" >/mnt/test/aaa
>
> 5. shutdown a replicat node, let's say sn-1, only let sn-0 worked
>
> 6. cp /mnt/test/aaa /mnt/test/bbb; link /mnt/test/bbb /mnt/test/ccc
>
>
>
>
>
> BRs
>
> George
>
>
>
> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
> gluster.org <gluster-devel-boun...@gluster.org>] *On Behalf Of *Pranith
> Kumar Karampuri
> *Sent:* Thursday, January 11, 2018 12:39 PM
> *To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
> Gluster-devel@gluster.org; Li, Deqian (NSB - CN/Hangzhou) <
> deqian...@nokia-sbell.com>; Sun, Ping (NSB - CN/Hangzhou) <
> ping@nokia-sbell.com>
> *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
>
>
>
>
> On Thu, Jan 11, 2018 at 6:35 AM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
> Hi,
>
> >>> In which protocol are you seeing this issue? Fuse/NFS/SMB?
>
> It is fuse, within mountpoint by “mount -t glusterfs  …“ command.
>
>
>
> Could you let me know the test you did so that I can try to re-create and
> see what exactly is going on?
>
> Configuration of the volume and the steps to re-create the issue you are
> seeing would be helpful in debugging the issue further.
>
>
>
>
>
> Thanks & Best Regards,
>
> George
>
>
>
> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
> gluster.org] *On Behalf Of *Pranith Kumar Karampuri
> *Sent:* Wednesday, January 10, 2018 8:08 PM
> *To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
> Zhong, Hua (NSB - CN/Hangzhou) <hua.zh...@nokia-sbell.com>; Li, Deqian
> (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>; Gluster-devel@gluster.org;
> Sun, Ping (NSB - CN/Hangzhou) <ping@nokia-sbell.com>
> *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
>
>
>
>
> On Wed, Jan 10, 2018 at 11:09 AM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
> Hi, Pranith Kumar,
>
>
>
> I has create a bug on Bugzilla https://bugzilla.redhat.com/
> show_bug.cgi?id=1531457
>
> After my investigation for this link issue, I suppose your changes on
> afr-dir-write.c with issue " Don't let NFS cache stat after writes" , your
> fix is like:
>
> --
>
>if (afr_txn_nothing_failed (frame, this)) {
>
> /*if it did pre-op, it will do post-op changing
> ctime*/
>
> if (priv->consistent_metadata &&
>
> afr_needs_changelog_update (local))
>
> afr_zero_fill_stat

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

On Thu, Jan 11, 2018 at 6:35 AM, Lian, George (NSB - CN/Hangzhou) <
george.l...@nokia-sbell.com> wrote:

> Hi,
>
> >>> In which protocol are you seeing this issue? Fuse/NFS/SMB?
>
> It is fuse, within mountpoint by “mount -t glusterfs  …“ command.
>

Could you let me know the test you did so that I can try to re-create and
see what exactly is going on?
Configuration of the volume and the steps to re-create the issue you are
seeing would be helpful in debugging the issue further.


>
>
> Thanks & Best Regards,
>
> George
>
>
>
> *From:* gluster-devel-boun...@gluster.org [mailto:gluster-devel-bounces@
> gluster.org] *On Behalf Of *Pranith Kumar Karampuri
> *Sent:* Wednesday, January 10, 2018 8:08 PM
> *To:* Lian, George (NSB - CN/Hangzhou) <george.l...@nokia-sbell.com>
> *Cc:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.z...@nokia-sbell.com>;
> Zhong, Hua (NSB - CN/Hangzhou) <hua.zh...@nokia-sbell.com>; Li, Deqian
> (NSB - CN/Hangzhou) <deqian...@nokia-sbell.com>; Gluster-devel@gluster.org;
> Sun, Ping (NSB - CN/Hangzhou) <ping@nokia-sbell.com>
> *Subject:* Re: [Gluster-devel] a link issue maybe introduced in a bug fix
> " Don't let NFS cache stat after writes"
>
>
>
>
>
>
>
> On Wed, Jan 10, 2018 at 11:09 AM, Lian, George (NSB - CN/Hangzhou) <
> george.l...@nokia-sbell.com> wrote:
>
> Hi, Pranith Kumar,
>
>
>
> I has create a bug on Bugzilla https://bugzilla.redhat.com/
> show_bug.cgi?id=1531457
>
> After my investigation for this link issue, I suppose your changes on
> afr-dir-write.c with issue " Don't let NFS cache stat after writes" , your
> fix is like:
>
> --
>
>if (afr_txn_nothing_failed (frame, this)) {
>
> /*if it did pre-op, it will do post-op changing
> ctime*/
>
> if (priv->consistent_metadata &&
>
> afr_needs_changelog_update (local))
>
> afr_zero_fill_stat (local);
>
> local->transaction.unwind (frame, this);
>
> }
>
> In the above fix, it set the ia_nlink to ‘0’ if option
> consistent-metadata is set to “on”.
>
> And hard link a file with which just created will lead to an error, and
> the error is caused in kernel function “vfs_link”:
>
> if (inode->i_nlink == 0 && !(inode->i_state & I_LINKABLE))
>
>  error =  -ENOENT;
>
>
>
> could you please have a check and give some comments here?
>
>
>
> When stat is "zero filled", understanding is that the higher layer
> protocol doesn't send stat value to the kernel and a separate lookup is
> sent by the kernel to get the latest stat value. In which protocol are you
> seeing this issue? Fuse/NFS/SMB?
>
>
>
>
>
> Thanks & Best Regards,
>
> George
>
>
>
>
> --
>
> Pranith
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] a link issue maybe introduced in a bug fix " Don't let NFS cache stat after writes"

[Pranith Kumar Karampuri]

On Wed, Jan 10, 2018 at 11:09 AM, Lian, George (NSB - CN/Hangzhou) <
george.l...@nokia-sbell.com> wrote:

> Hi, Pranith Kumar,
>
>
>
> I has create a bug on Bugzilla https://bugzilla.redhat.com/
> show_bug.cgi?id=1531457
>
> After my investigation for this link issue, I suppose your changes on
> afr-dir-write.c with issue " Don't let NFS cache stat after writes" , your
> fix is like:
>
> --
>
>if (afr_txn_nothing_failed (frame, this)) {
>
> /*if it did pre-op, it will do post-op changing
> ctime*/
>
> if (priv->consistent_metadata &&
>
> afr_needs_changelog_update (local))
>
> afr_zero_fill_stat (local);
>
> local->transaction.unwind (frame, this);
>
> }
>
> In the above fix, it set the ia_nlink to ‘0’ if option consistent-metadata
> is set to “on”.
>
> And hard link a file with which just created will lead to an error, and
> the error is caused in kernel function “vfs_link”:
>
> if (inode->i_nlink == 0 && !(inode->i_state & I_LINKABLE))
>
>  error =  -ENOENT;
>
>
>
> could you please have a check and give some comments here?
>

When stat is "zero filled", understanding is that the higher layer protocol
doesn't send stat value to the kernel and a separate lookup is sent by the
kernel to get the latest stat value. In which protocol are you seeing this
issue? Fuse/NFS/SMB?


>
>
> Thanks & Best Regards,
>
> George
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] reflink support for glusterfs and gluster-block using it for taking snapshots

[Pranith Kumar Karampuri]

On Thu, Nov 9, 2017 at 8:26 PM, Niels de Vos <nde...@redhat.com> wrote:

> On Tue, Nov 07, 2017 at 05:59:32PM +0530, Pranith Kumar Karampuri wrote:
> > On Tue, Nov 7, 2017 at 5:16 PM, Niels de Vos <nde...@redhat.com> wrote:
> >
> > > On Tue, Nov 07, 2017 at 07:43:17AM +0530, Pranith Kumar Karampuri
> wrote:
> > > > hi,
> > > >  I just created a github issue for reflink support
> > > > <https://github.com/gluster/glusterfs/issues/349>(#349) in
> glusterfs. We
> > > > are intending to use this feature to do block snapshots in
> gluster-block.
> > > >
> > > > Please let us know your comments on the github issue. I have added
> the
> > > > changes we may need for xlators I know a little bit about. Please
> help in
> > > > identifying gaps in implementing this FOP.
> > >
> > > For gluster-block it may be easier to have snapshot support in
> > > tcmu-runner instead? The qcow2 format would be ideal, and is in use by
> > > different Virtual Machine approaches running on Gluster already. There
> > > even is an upstream issue open for it:
> > >   https://github.com/open-iscsi/tcmu-runner/issues/32
> > >
> > > Contributing towards this might be quicker than implementing file
> > > snapshot support in Gluster?
> > >
> >
> > We tried that route by talking Fam Zheng, but the solution won't be
> > delivered in the timelines we are looking for.
> > So we went with this approach.
>
> Ok, I am not sure if adding support for reflink in Gluster has an
> immediate benefit. It surely would be a great feature to have, but I do
> not know if it will land in enterprise kernels soon.
>

https://github.com/gluster/glusterfs/issues/377 should address this concern
until reflink becomes mainstream.


>
> That said, I opened a GitHub issue to get reliable snapshot support in
> gluster-block. It describes an idea of the interface that could be
> implemented now already, without relying on reflink.
>
>   https://github.com/gluster/gluster-block/issues/42
>
> Obviously there is a need to sync/freeze data through tcmu-runner. This
> might require implementing a fsfreeze(8) like command in targetcli and
> tcmu-runner.
>
> Comments most welcome!
> Niels
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Need help figuring out the reason for test failure

[Pranith Kumar Karampuri]

On Tue, Nov 28, 2017 at 9:07 AM, Nigel Babu <nig...@redhat.com> wrote:

> Pranith,
>
> Our logging has changed slightly. Please read my email titled "Changes in
> handling logs from (centos) regressions and smoke" to gluster-devel and
> gluster-infra.
>

Thanks for the pointer. Able to get the logs now!


>
> On Tue, Nov 28, 2017 at 8:06 AM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>> One of my patches(https://review.gluster.org/18857) is consistently
>> leading to a failure for the test:
>>
>> tests/bugs/core/bug-1432542-mpx-restart-crash.t
>>
>> https://build.gluster.org/job/centos6-regression/7676/consoleFull
>>
>> Jeff/Atin,
>> Do you know anything about these kinds of failures for this test?
>>
>> Nigel,
>>Unfortunately I am not able to look at the logs because the logs
>> location is not given properly (at least for me :-) )
>>
>> *11:41:14* 
>> filename="${ARCHIVED_LOGS}/glusterfs-logs-${UNIQUE_ID}.tgz"*11:41:14* sudo 
>> -E tar -czf "${ARCHIVE_BASE}/${filename}" /var/log/glusterfs 
>> /var/log/messages*;*11:41:14* echo "Logs archived in 
>> http://${SERVER}/${filename};
>>
>>
>> Could you help me find what the location could be?
>> --
>> Pranith
>>
>
>
>
> --
> nigelb
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Need help figuring out the reason for test failure

[Pranith Kumar Karampuri]

One of my patches(https://review.gluster.org/18857) is consistently leading
to a failure for the test:

tests/bugs/core/bug-1432542-mpx-restart-crash.t

https://build.gluster.org/job/centos6-regression/7676/consoleFull

Jeff/Atin,
Do you know anything about these kinds of failures for this test?

Nigel,
   Unfortunately I am not able to look at the logs because the logs
location is not given properly (at least for me :-) )

*11:41:14* filename="${ARCHIVED_LOGS}/glusterfs-logs-${UNIQUE_ID}.tgz"*11:41:14*
sudo -E tar -czf "${ARCHIVE_BASE}/${filename}" /var/log/glusterfs
/var/log/messages*;*11:41:14* echo "Logs archived in
http://${SERVER}/${filename};


Could you help me find what the location could be?
-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Tie-breaker logic for blocking inodelks/entrylks

[Pranith Kumar Karampuri]

I got a +1 from Ashish and Xavi for the idea. I will wait for people from
other time zones to comment on the same and will start implementing the
solution tomorrow morning IST. I am kind of in a hurry to get this done,
preferably by end of this week.

Thanks in advance.

On Thu, Nov 9, 2017 at 8:56 AM, Pranith Kumar Karampuri <pkara...@redhat.com
> wrote:

> This github issue <https://github.com/gluster/glusterfs/issues/354> talks
> about the logic for implementing tie-breaker logic for blocking
> inodelks/entrylks. Your comments are welcome on the issue.
>
> --
> Pranith
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Tie-breaker logic for blocking inodelks/entrylks

[Pranith Kumar Karampuri]

This github issue  talks
about the logic for implementing tie-breaker logic for blocking
inodelks/entrylks. Your comments are welcome on the issue.

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Thin-arbiter design to solve stretch cluster usecase

[Pranith Kumar Karampuri]

hi,
 I created a new issue 
which shares a link of design document for this feature. Please add
comments to the design document itself rather than github issue, so that
all comments are in one place.

Thanks in advance.

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] reflink support for glusterfs and gluster-block using it for taking snapshots

[Pranith Kumar Karampuri]

On Tue, Nov 7, 2017 at 5:16 PM, Niels de Vos <nde...@redhat.com> wrote:

> On Tue, Nov 07, 2017 at 07:43:17AM +0530, Pranith Kumar Karampuri wrote:
> > hi,
> >  I just created a github issue for reflink support
> > <https://github.com/gluster/glusterfs/issues/349>(#349) in glusterfs. We
> > are intending to use this feature to do block snapshots in gluster-block.
> >
> > Please let us know your comments on the github issue. I have added the
> > changes we may need for xlators I know a little bit about. Please help in
> > identifying gaps in implementing this FOP.
>
> For gluster-block it may be easier to have snapshot support in
> tcmu-runner instead? The qcow2 format would be ideal, and is in use by
> different Virtual Machine approaches running on Gluster already. There
> even is an upstream issue open for it:
>   https://github.com/open-iscsi/tcmu-runner/issues/32
>
> Contributing towards this might be quicker than implementing file
> snapshot support in Gluster?
>

We tried that route by talking Fam Zheng, but the solution won't be
delivered in the timelines we are looking for.
So we went with this approach.


>
> Niels
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] reflink support for glusterfs and gluster-block using it for taking snapshots

[Pranith Kumar Karampuri]

hi,
 I just created a github issue for reflink support
(#349) in glusterfs. We
are intending to use this feature to do block snapshots in gluster-block.

Please let us know your comments on the github issue. I have added the
changes we may need for xlators I know a little bit about. Please help in
identifying gaps in implementing this FOP.

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] tests/basic/pump.t - what is it used for?

[Pranith Kumar Karampuri]

It was used for testing pump xlator functionality. When replace-brick is
done on a distribute volume, it would lead to pump xlator migrating data to
the destination brick from source. I guess we can delete this test. I don't
think we support pump xlator anymore.

On Fri, Sep 8, 2017 at 10:02 AM, Atin Mukherjee  wrote:

> Pranith,
>
> I see you're the author of the test in $Subj. Now while I was working on a
> patch https://review.gluster.org/#/c/18226/ to disallow replace brick
> operations on dist only volumes the patch failed the regression on this
> test as the test actually uses replace brick on a distribute only volume
> which IMO is wrong as then this would always end up in to data loss
> situation. I'd need some context here to understand the expectation of this
> test before doing any modifications.
>
> ~Atin
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Need inputs on patch #17985

[Pranith Kumar Karampuri]

Raghavendra,
As Ashish mentioned, there aren't any known problems if upper xlators
don't send lookups in EC at the moment.

On Wed, Aug 23, 2017 at 9:07 AM, Ashish Pandey <aspan...@redhat.com> wrote:

> Raghvendra,
>
> I have provided my comment on this patch.
> I think EC will not have any issue with this approach.
> However, I would welcome comments from Xavi and Pranith too for any side
> effects which I may not be able to foresee.
>
> Ashish
>
> --
> *From: *"Raghavendra Gowdappa" <rgowd...@redhat.com>
> *To: *"Ashish Pandey" <aspan...@redhat.com>
> *Cc: *"Pranith Kumar Karampuri" <pkara...@redhat.com>, "Xavier Hernandez"
> <xhernan...@datalab.es>, "Gluster Devel" <gluster-devel@gluster.org>
> *Sent: *Wednesday, August 23, 2017 8:29:48 AM
> *Subject: *Need inputs on patch #17985
>
>
> Hi Ashish,
>
> Following are the blockers for making a decision on whether patch [1] can
> be merged or not:
> * Evaluation of dentry operations (like rename etc) in dht
> * Whether EC works fine if a non-lookup fop (like open(dir), stat, chmod
> etc) hits EC without a single lookup performed on file/inode
>
> Can you please comment on the patch? I'll take care of dht part.
>
> [1] https://review.gluster.org/#/c/17985/
>
> regards,
> Raghavendra
>
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Changing the relative order of read-ahead and open-behind

[Pranith Kumar Karampuri]

On Mon, Jul 24, 2017 at 5:11 PM, Raghavendra G 
wrote:

>
>
> On Fri, Jul 21, 2017 at 6:39 PM, Vijay Bellur  wrote:
>
>>
>> On Fri, Jul 21, 2017 at 3:26 AM, Raghavendra Gowdappa <
>> rgowd...@redhat.com> wrote:
>>
>>> Hi all,
>>>
>>> We've a bug [1], due to which read-ahead is completely disabled when the
>>> workload is read-only. One of the easy fix was to make read-ahead as an
>>> ancestor of open-behind in xlator graph (Currently its a descendant). A
>>> patch has been sent out by Rafi to do the same. As noted in one of the
>>> comments, one flip side of this solution is that small files (which are
>>> eligible to be cached by quick read) are cached twice - once each in
>>> read-ahead and quick-read - wasting up precious memory. However, there are
>>> no other simpler solutions for this issue. If you've concerns on the
>>> approach followed by [2] or have other suggestions please voice them out.
>>> Otherwise, I am planning to merge [2] for lack of better alternatives.
>>>
>>
>>
>> Since the maximum size of files cached by quick-read is 64KB, can we have
>> read-ahead kick in for offsets greater than 64KB?
>>
>
> I got your point. We can enable read-ahead only for files whose size is
> greater than the size eligible for caching quick-read. IOW, read-ahead gets
> disabled if file size is less than 64KB. Thanks for the suggestion.
>

I added a comment on the patch to move the xlators in reverse to the way
the patch is currently doing. Milind I think implemented it. Will that lead
to any problem?


>
>
>>
>> Thanks,
>> Vijay
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> Raghavendra G
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 3.12: Status of features (Require responses!)

[Pranith Kumar Karampuri]

On Sat, Jul 22, 2017 at 1:36 AM, Shyam  wrote:

> Hi,
>
> Prepare for a lengthy mail, but needed for the 3.12 release branching, so
> here is a key to aid the impatient,
>
> Key:
> 1) If you asked for an exception to a feature (meaning delayed backport to
> 3.12 branch post branching for the release) see "Section 1"
>   - Handy list of nick's that maybe interested in this:
> - @pranithk, @sunilheggodu, @aspandey, @amarts, @kalebskeithley,
> @kshlm (IPv6), @jdarcy (Halo Hybrid)
>
> 2) If you have/had a feature targeted for 3.12 and have some code posted
> against the same, look at "Section 2" AND we want to hear back from you!
>   - Handy list of nick's that should be interested in this:
> - @csabahenk, @nixpanic, @aravindavk, @amarts, @kotreshhr,
> @soumyakoduri
>
> 3) If you have/had a feature targeted for 3.12 and have posted no code
> against the same yet, see "Section 3", your feature is being dropped from
> the release.
>   - Handy list of nick's that maybe interested in this:
> - @sanoj-unnikrishnan, @aravindavk, @kotreshhr, @amarts, @jdarcy,
> @avra (people who filed the issue)
>
> 4) Finally, if you do not have any features for the release pending,
> please help others out reviewing what is still pending, here [1] is a quick
> link to those reviews.
>
> Sections:
>
> **Section 1:**
> Exceptions granted to the following features: (Total: 8)
> Reasons:
>   - Called out in the mail sent for noting exceptions and feature status
> for 3.12
>   - Awaiting final changes/decision from a few Facebook patches
>
> Issue list:
> - Implement an xlator to delay fops
>   - https://github.com/gluster/glusterfs/issues/257
>
> - Implement parallel writes feature on EC volume
>   - https://github.com/gluster/glusterfs/issues/251


Looks like it will take one more week after this, what do you suggest?


>
>
> - DISCARD support with EC
>   - https://github.com/gluster/glusterfs/issues/254
>
> - Cache last stripe of an EC volume while write is going on
>   - https://github.com/gluster/glusterfs/issues/256
>

Looks like it will take 2 more week after this, what do you suggest?


> - gfid-path by default
>   - https://github.com/gluster/glusterfs/issues/139
>
> - allow users to enable used of localtime instead of UTC for log entries
>   - https://github.com/gluster/glusterfs/issues/272
>
> - Halo translator: Hybrid mode
>   - https://github.com/gluster/glusterfs/issues/217
>
> - [RFE] Improve IPv6 support in GlusterFS
>   - https://github.com/gluster/glusterfs/issues/192
>
> **Section 2:**
> Issues needing some further clarity: (Total: 6)
> Reason:
>   - There are issues here, for which code is already merged (or submitted)
> and issue is still open. This is the right state for an issue to be in this
> stage of the release, as documentation or release-notes would possibly be
> still pending, which will finally close the issue (or rather mark it fixed)
>   - BUT, without a call out from the contributors that required code is
> already merged in, it is difficult to assess if the issue should qualify
> for the release
>
> Issue list:
> - [RFE] libfuse rebase to latest?
>   - https://github.com/gluster/glusterfs/issues/153
>   - @csabahenk is this all done?
>
> - Decide what to do with glfs_ipc() in libgfapi
>   - https://github.com/gluster/glusterfs/issues/269
>   - @nixpanic I assume there is more than just test case disabling for
> this, is this expected to happen by 3.12?
>
> - Structured log format support for gf_log and gf_msg
>   - https://github.com/gluster/glusterfs/issues/240
>   - @aravindavk this looks done, anything code wise pending here?
>
> - xxhash: Add xxhash library
>   - https://github.com/gluster/glusterfs/issues/253
>   - @kotreshhr this looks done, anything code wise pending here?
>
> - posix: provide option to handle 'statfs()' properly when more than 1
> brick is exported from 1 node
>   - https://github.com/gluster/glusterfs/issues/241
>   - @amarts patch is still awaiting reviews, should this be tracked as an
> exception?
>
> - gfapi to support leases and lock-owner
>   - https://github.com/gluster/glusterfs/issues/213
>   - @soumyakoduri I do not see work progressing on the patches provided,
> should this be dropped from 3.12?
>
> **Section 3:**
> Issues moved out of the 3.12 Milestone: (Total: 8)
> Reasons:
>   - No commits visible against the github issue
>   - No commits against 'master' branch visible on the github issue
>
> Further changes:
>   - No new milestone assigned, IOW not moved to 4.0 by default, hence
> contributors working on these features would need to rekindle conversations
> on including the same in 4.0 on the ML or on the issue itself.
>
> Issue List:
> - [RFE] Syslog alert when Geo-rep worker is faulty for a configurable time
>   https://github.com/gluster/glusterfs/issues/226
>
> - [RFE] Geo-rep: Sync metadata operations as part of sync itself
> (rsync/tar-ssh)
>   https://github.com/gluster/glusterfs/issues/222
>
> - 

Re: [Gluster-devel] Error while mounting gluster volume

[Pranith Kumar Karampuri]

The following generally means it is not able to connect to any of the
glusterds in the cluster.

[1970-01-02 10:54:04.420406] E [glusterfsd-mgmt.c:1818:mgmt_rpc_notify]
0-glusterfsd-mgmt: failed to connect with remote-host: 128.224.95.140
(Success)
[1970-01-02 10:54:04.420422] I [MSGID: 101190]
[event-epoll.c:632:event_dispatch_epoll_worker]
0-epoll: Started thread with index 1
[1970-01-02 10:54:04.420429] I [glusterfsd-mgmt.c:1824:mgmt_rpc_notify]
0-glusterfsd-mgmt: Exhausted all volfile servers


On Thu, Jul 20, 2017 at 4:01 PM, ABHISHEK PALIWAL 
wrote:

> Hi Team,
>
> While mounting the gluster volume using 'mount -t glusterfs' command it is
> getting failed.
>
> When we checked the log file getting the below logs
>
> [1970-01-02 10:54:04.420065] E [MSGID: 101187] 
> [event-epoll.c:391:event_register_epoll]
> 0-epoll: failed to add fd(=7) to epoll fd(=0) [Invalid argument]
> [1970-01-02 10:54:04.420140] W [socket.c:3095:socket_connect] 0-: failed
> to register the event
> [1970-01-02 10:54:04.420406] E [glusterfsd-mgmt.c:1818:mgmt_rpc_notify]
> 0-glusterfsd-mgmt: failed to connect with remote-host: 128.224.95.140
> (Success)
> [1970-01-02 10:54:04.420422] I [MSGID: 101190] 
> [event-epoll.c:632:event_dispatch_epoll_worker]
> 0-epoll: Started thread with index 1
> [1970-01-02 10:54:04.420429] I [glusterfsd-mgmt.c:1824:mgmt_rpc_notify]
> 0-glusterfsd-mgmt: Exhausted all volfile servers
> [1970-01-02 10:54:04.420480] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=2
> [1970-01-02 10:54:04.420511] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=3
> [1970-01-02 10:54:04.420534] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=4
> [1970-01-02 10:54:04.420556] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=5
> [1970-01-02 10:54:04.420566] W [glusterfsd.c:1238:cleanup_and_exit]
> (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify-0xb18c) [0x3fff8155e1e4]
> -->/usr/sbin/glusterfs() [0x100103a0] 
> -->/usr/sbin/glusterfs(cleanup_and_exit-0x1beac)
> [0x100097ac] ) 0-: received signum (1), shutting down
> [1970-01-02 10:54:04.420579] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=6
> [1970-01-02 10:54:04.420606] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=7
> [1970-01-02 10:54:04.420635] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=8
> [1970-01-02 10:54:04.420664] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=9
> [1970-01-02 10:54:04.420695] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=10
> [1970-01-02 10:54:04.420722] E [MSGID: 101063] 
> [event-epoll.c:550:event_dispatch_epoll_handler]
> 0-epoll: stale fd found on idx=0, gen=0, events=0, slot->gen=11
>
> for more logs we have attached the mount log file as well.
>
> Could you please help us in to identify the root cause?
>
> --
>
> Regards
> Abhishek Paliwal
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] GlusterFS v3.12 - Nearing deadline for branch out

[Pranith Kumar Karampuri]

hi,
   Status of the following features targeted for 3.12:
1) Need a way to resolve split-brain (#135) : Mostly will be merged in a
day.
2) Halo Hybrid mode (#217): Unfortunately didn't get time to follow up on
this, so will not make it to the release.
3) Implement heal throttling (#255): Won't be making it to 3.12
4) Delay generator xlator (#257): I can definitely get this in by End of
next week, otherwise we can do this for next release.
5) Parallel writes in EC (#251): This seems like a stretch for this weekend
but definitely completable by end of next week. I am hopeful Xavi will have
some cycles to complete the final reviews. Otherwise we may have to push
this out.
6) Discard support for EC (#254): Doable for this weekend IMO, also depends
on what Xavi thinks...
7) Last stripe caching (#256): We are targetting this instead of heal
throttling (#255). This is not added to 3.12 tracker. I can add this if we
can wait till next week. This also depends on Xavi's schedule.

Also added Xavi for his inputs.


On Wed, Jul 5, 2017 at 9:07 PM, Shyam  wrote:

> Further to this,
>
> 1) I cleared up the projects lane [1] and also issues marked for 3.12 [2]
>   - I did this optimistically, moving everything to 3.12 (both from a
> projects and a milestones perspective), so if something is not making it,
> drop a note, and we can clear up the tags accordingly.
>
> 2) Reviews posted and open against the issues in [1] can be viewed here [3]
>
>   - Request maintainers and contributors to take a look at these and
> accelerate the reviews, to meet the feature cut-off deadline
>
>   - Request feature owners to ensure that their patches are listed in the
> link [3]
>
> 3) Finally, we need a status of open issues to understand how we can help.
> Requesting all feature owners to post the same (as Amar has requested).
>
> Thanks,
> Shyam
>
> [1] Project lane: https://github.com/gluster/glusterfs/projects/1
> [2] Issues with 3.12 milestone: https://github.com/gluster/glu
> sterfs/milestone/4
> [3] Reviews needing attetion: https://review.gluster.org/#/q
> /starredby:srangana%2540redhat.com
>
> "Releases are made better together"
>
>
> On 07/05/2017 03:18 AM, Amar Tumballi wrote:
>
>> All,
>>
>> We are around 10 working days remaining for branching out for 3.12
>> release, after which, we will have just 15 more days open for 'critical'
>> features to get in, for which there should be more detailed proposals.
>>
>> If you have few things planned out, but haven't taken it to completion
>> yet, OR you have sent some patches, but not yet reviewed, start whining
>> now, and get these in.
>>
>> Thanks,
>> Amar
>>
>> --
>> Amar Tumballi (amarts)
>>
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #162

[Pranith Kumar Karampuri]

Ah! sorry, I need to keep this in mind next time when I review.

On Mon, Jul 17, 2017 at 6:30 PM, Atin Mukherjee  wrote:

> kill $pid0
>
> kill $pid1
>
> EXPECT_WITHIN $CHILD_UP_TIMEOUT "4" ec_child_up_count $V0 0
>
> *21:03:33* not ok 17 Got "0" instead of "4", LINENUM:40*21:03:33* FAILED 
> COMMAND: 4 ec_child_up_count patchy 0
>
> The failure is quite obvious with brick multiplexing as all the brick 
> instances go down with kill signal. We need to use kill_brick utility here to 
> ensure
> the test is compatible with brick multiplexing.
>
>
>
> On Sun, Jul 16, 2017 at 1:22 PM, Atin Mukherjee 
> wrote:
>
>> EC dev - tests/basic/ec/ec-1468261.t is failing frequently now with
>> brick multiplex enabled. Please have a look.
>>
>> -- Forwarded message -
>> From: 
>> Date: Sun, 16 Jul 2017 at 06:32
>> Subject: [Gluster-Maintainers] Build failed in Jenkins:
>> regression-test-with-multiplex #162
>> To: , , ,
>> 
>>
>>
>> See > x/162/display/redirect>
>>
>> --
>> [...truncated 742.19 KB...]
>> ./tests/bugs/glusterd/bug-1070734.t  -  13 second
>> ./tests/bugs/distribute/bug-961615.t  -  13 second
>> ./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t
>> -  13 second
>> ./tests/bitrot/bug-1207627-bitrot-scrub-status.t  -  13 second
>> ./tests/basic/rpc-coverage.t  -  13 second
>> ./tests/basic/quota_aux_mount.t  -  13 second
>> ./tests/basic/inode-quota-enforcing.t  -  13 second
>> ./tests/basic/glusterd/arbiter-volume-probe.t  -  13 second
>> ./tests/performance/open-behind.t  -  12 second
>> ./tests/bugs/replicate/bug-986905.t  -  12 second
>> ./tests/bugs/replicate/bug-1325792.t  -  12 second
>> ./tests/bugs/quota/bug-1250582-volume-reset-should-not-
>> remove-quota-quota-deem-statfs.t  -  12 second
>> ./tests/bugs/nfs/bug-904065.t  -  12 second
>> ./tests/bugs/glusterfs/bug-902610.t  -  12 second
>> ./tests/bugs/glusterfs/bug-872923.t  -  12 second
>> ./tests/bugs/glusterd/bug-955588.t  -  12 second
>> ./tests/bugs/glusterd/bug-949930.t  -  12 second
>> ./tests/bugs/glusterd/bug-1121584-brick-existing-validation-
>> for-remove-brick-status-stop.t  -  12 second
>> ./tests/bugs/fuse/bug-985074.t  -  12 second
>> ./tests/bugs/distribute/bug-1247563.t  -  12 second
>> ./tests/bugs/distribute/bug-1088231.t  -  12 second
>> ./tests/bugs/cli/bug-1030580.t  -  12 second
>> ./tests/basic/volume-status.t  -  12 second
>> ./tests/basic/stats-dump.t  -  12 second
>> ./tests/basic/pump.t  -  12 second
>> ./tests/basic/ec/ec-root-heal.t  -  12 second
>> ./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t  -  11
>> second
>> ./tests/bugs/replicate/bug-1250170-fsync.t  -  11 second
>> ./tests/bugs/replicate/bug-1132102.t  -  11 second
>> ./tests/bugs/posix/bug-1360679.t  -  11 second
>> ./tests/bugs/posix/bug-1122028.t  -  11 second
>> ./tests/bugs/nfs/bug-1157223-symlink-mounting.t  -  11 second
>> ./tests/bugs/md-cache/bug-1211863.t  -  11 second
>> ./tests/bugs/glusterd/bug-948729/bug-948729-mode-script.t  -  11 second
>> ./tests/bugs/glusterd/bug-948729/bug-948729-force.t  -  11 second
>> ./tests/bugs/glusterd/bug-889630.t  -  11 second
>> ./tests/bugs/glusterd/bug-1242875-do-not-pass-volinfo-quota.t  -  11
>> second
>> ./tests/bugs/glusterd/bug-1046308.t  -  11 second
>> ./tests/bugs/ec/bug-1179050.t  -  11 second
>> ./tests/bugs/distribute/bug-1122443.t  -  11 second
>> ./tests/bugs/distribute/bug-1086228.t  -  11 second
>> ./tests/bugs/cli/bug-1087487.t  -  11 second
>> ./tests/bugs/cli/bug-1022905.t  -  11 second
>> ./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t  -
>> 11 second
>> ./tests/basic/quota-nfs.t  -  11 second
>> ./tests/basic/md-cache/bug-1317785.t  -  11 second
>> ./tests/basic/gfapi/upcall-cache-invalidate.t  -  11 second
>> ./tests/basic/fop-sampling.t  -  11 second
>> ./tests/basic/ec/ec-read-policy.t  -  11 second
>> ./tests/features/ssl-authz.t  -  10 second
>> ./tests/bugs/upcall/bug-1458127.t  -  10 second
>> ./tests/bugs/upcall/bug-1227204.t  -  10 second
>> ./tests/bugs/transport/bug-873367.t  -  10 second
>> ./tests/bugs/tier/bug-1205545-CTR-and-trash-integration.t  -  10 second
>> ./tests/bugs/snapshot/bug-1260848.t  -  10 second
>> ./tests/bugs/quota/bug-1287996.t  -  10 second
>> ./tests/bugs/quota/bug-1243798.t  -  10 second
>> ./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t  -  10 second
>> ./tests/bugs/io-cache/bug-read-hang.t  -  10 second
>> ./tests/bugs/glusterd/bug-948729/bug-948729.t  -  10 second
>> ./tests/bugs/glusterd/bug-1179175-uss-option-validation.t  -  10 second
>> ./tests/bugs/glusterd/bug-1091935-brick-order-check-from-cli-to-glusterd.t
>> -  10 second
>> ./tests/bugs/glusterd/bug-1022055.t  -  10 second
>> ./tests/bugs/ec/bug-1227869.t  -  10 second
>> 

Re: [Gluster-devel] Regarding positioning of nl-cache in gluster client stack

[Pranith Kumar Karampuri]

On Mon, Jul 17, 2017 at 11:31 AM, Krutika Dhananjay 
wrote:

> Hi Poornima and Pranith,
>
> I see that currently glusterd loads nl-cache between stat-prefetch and
> open-behind on the client stack. Were there any specific considerations for
> selecting this position for nl-cache?
>
> I was interested to see the performance impact of loading this translator
> between shard and DHT in the VM store use case stack in terms of reducing
> the number of lookups shard would have to do to figure out if a shard is
> already created or not, since shard does its own management of .shard and
> the files under it.
>
> So with this background, do you see any issues with loading nl-cache above
> DHT in the client stack?
>

Nothing I can think of at the moment.


>
> -Krutika
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Replicated volume, one slow brick

[Pranith Kumar Karampuri]

Adding gluster-devel

Raghavendra,
 I remember we discussing about handling these kinds of errors by
ping-timer expiry? I may have missed the final decision on how this was
decided to be handled. So asking you again ;-)

On Thu, Jul 13, 2017 at 2:14 PM, Øyvind Krosby  wrote:

> I have been trying to figure out how glusterfs-fuse client will handle it
> when 1 of 3 bricks in a 3-way replica is slower than the others.
>
> It looks like a glusterfs-fuse client will send requests to all 3 bricks
> when accessing a file. But what happens when one of the bricks is not
> responding in time?
>
> We saw an issue when we added external load to the raid volume where the
> brick was located. The disk became 100% busy, and as a result the
> glusterfs-clients hang when they access the volume.
>
> Is there a way to avoid this, and make the clients ask the other two
> bricks for the data when one brick is too slow?
>
> Thanks
>
> Øyvind Krosby
> SRE, Zedge.net
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Forgot how to download NetBSD logfiles

[Pranith Kumar Karampuri]

On Fri, Jul 14, 2017 at 9:22 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

> hi,
>I am not able to download the "Logs archived in
> http://nbslave74.cloud.gluster.org/archives/logs/
> glusterfs-logs-20170713080233.tgz"
>
> I get the following error:
> archives/logs/glusterfs-logs-20170713080233.tgz
>
This item has not been found

 This was the error :-)

>
> We had to modify something in the URL to get the correct link but now
> either it changed or I don't remember what the modification used to be.
>
> The change I made affects all platforms, so I want to find out why trash.t
> fails consistently on NetBSD with this change.
>
> --
> Pranith
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Forgot how to download NetBSD logfiles

[Pranith Kumar Karampuri]

hi,
   I am not able to download the "Logs archived in
http://nbslave74.cloud.gluster.org/archives/logs/glusterfs-logs-20170713080233.tgz
"

I get the following error:
archives/logs/glusterfs-logs-20170713080233.tgz

We had to modify something in the URL to get the correct link but now
either it changed or I don't remember what the modification used to be.

The change I made affects all platforms, so I want to find out why trash.t
fails consistently on NetBSD with this change.

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] crash in tests/bugs/core/bug-1432542-mpx-restart-crash.t

[Pranith Kumar Karampuri]

I just observed that
https://build.gluster.org/job/centos6-regression/5433/consoleFull failed
because of this .t failure.

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] gfid and volume-id extended attributes lost

[Pranith Kumar Karampuri]

Ram,
  I sent https://review.gluster.org/17765 to fix the possibility in
bulk removexattr. But I am not sure if this is indeed the reason for this
issue.

On Mon, Jul 10, 2017 at 6:30 PM, Ankireddypalle Reddy <are...@commvault.com>
wrote:

> Thanks for the swift turn around. Will try this out and let you know.
>
>
>
> Thanks and Regards,
>
> Ram
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Monday, July 10, 2017 8:31 AM
> *To:* Sanoj Unnikrishnan
> *Cc:* Ankireddypalle Reddy; Gluster Devel (gluster-devel@gluster.org);
> gluster-us...@gluster.org
>
> *Subject:* Re: [Gluster-devel] gfid and volume-id extended attributes lost
>
>
>
> Ram,
>
>   If you see it again, you can use this. I am going to send out a
> patch for the code path which can lead to removal of gfid/volume-id
> tomorrow.
>
>
>
> On Mon, Jul 10, 2017 at 5:19 PM, Sanoj Unnikrishnan <sunni...@redhat.com>
> wrote:
>
> Please use the systemtap script(https://paste.fedoraproject.org/paste/
> EGDa0ErwX0LV3y-gBYpfNA) to check which process is invoking remove xattr
> calls.
> It prints the pid, tid and arguments of all removexattr calls.
>
> I have checked for these fops at the protocol/client and posix translators.
>
>
> To run the script ..
>
> 1) install systemtap and dependencies.
> 2) install glusterfs-debuginfo
>
> 3) change the path of the translator in the systemtap script to
> appropriate values for your system
>
> (change "/usr/lib64/glusterfs/3.12dev/xlator/protocol/client.so" and
> "/usr/lib64/glusterfs/3.12dev/xlator/storage/posix.so")
>
> 4) run the script as follows
>
> #stap -v fop_trace.stp
>
> The o/p would look like these .. additionally arguments will also be
> dumped if glusterfs-debuginfo is also installed (i had not done it here.)
> pid-958: 0 glusterfsd(3893):->posix_setxattr
> pid-958:47 glusterfsd(3893):<-posix_setxattr
> pid-966: 0 glusterfsd(5033):->posix_setxattr
> pid-966:57 glusterfsd(5033):<-posix_setxattr
> pid-1423: 0 glusterfs(1431):->client_setxattr
> pid-1423:37 glusterfs(1431):<-client_setxattr
> pid-1423: 0 glusterfs(1431):->client_setxattr
> pid-1423:41 glusterfs(1431):<-client_setxattr
>
> Regards,
>
> Sanoj
>
>
>
>
>
>
>
> On Mon, Jul 10, 2017 at 2:56 PM, Sanoj Unnikrishnan <sunni...@redhat.com>
> wrote:
>
> @ pranith , yes . we can get the pid on all removexattr call and also
> print the backtrace of the glusterfsd process when trigerring removing
> xattr.
>
> I will write the script and reply back.
>
>
>
> On Sat, Jul 8, 2017 at 7:06 AM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
> Ram,
>
>As per the code, self-heal was the only candidate which *can* do
> it. Could you check logs of self-heal daemon and the mount to check if
> there are any metadata heals on root?
>
> +Sanoj
>
> Sanoj,
>
>Is there any systemtap script we can use to detect which process is
> removing these xattrs?
>
>
>
> On Sat, Jul 8, 2017 at 2:58 AM, Ankireddypalle Reddy <are...@commvault.com>
> wrote:
>
> We lost the attributes on all the bricks on servers glusterfs2 and
> glusterfs3 again.
>
>
>
> [root@glusterfs2 Log_Files]# gluster volume info
>
>
>
> Volume Name: StoragePool
>
> Type: Distributed-Disperse
>
> Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
>
> Status: Started
>
> Number of Bricks: 20 x (2 + 1) = 60
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: glusterfs1sds:/ws/disk1/ws_brick
>
> Brick2: glusterfs2sds:/ws/disk1/ws_brick
>
> Brick3: glusterfs3sds:/ws/disk1/ws_brick
>
> Brick4: glusterfs1sds:/ws/disk2/ws_brick
>
> Brick5: glusterfs2sds:/ws/disk2/ws_brick
>
> Brick6: glusterfs3sds:/ws/disk2/ws_brick
>
> Brick7: glusterfs1sds:/ws/disk3/ws_brick
>
> Brick8: glusterfs2sds:/ws/disk3/ws_brick
>
> Brick9: glusterfs3sds:/ws/disk3/ws_brick
>
> Brick10: glusterfs1sds:/ws/disk4/ws_brick
>
> Brick11: glusterfs2sds:/ws/disk4/ws_brick
>
> Brick12: glusterfs3sds:/ws/disk4/ws_brick
>
> Brick13: glusterfs1sds:/ws/disk5/ws_brick
>
> Brick14: glusterfs2sds:/ws/disk5/ws_brick
>
> Brick15: glusterfs3sds:/ws/disk5/ws_brick
>
> Brick16: glusterfs1sds:/ws/disk6/ws_brick
>
> Brick17: glusterfs2sds:/ws/disk6/ws_brick
>
> Brick18: glusterfs3sds:/ws/disk6/ws_brick
>
> Brick19: glusterfs1sds:/ws/disk7/ws_brick
>
> Brick20: glusterfs2sds:/ws/disk7/ws_brick
>
> Brick21: glusterfs3sds:/ws/disk7/ws_brick
>
> Brick22: glusterfs1sds:/ws/disk8/ws_brick
&

Re: [Gluster-devel] create restrictions xlator

[Pranith Kumar Karampuri]

On Thu, Jul 13, 2017 at 10:11 AM, Taehwa Lee <alghost@gmail.com> wrote:

> Thank you for response quickly
>
>
> I went through dht_get_du_info before I start developing this.
>
> at that time, I think that this functionality should be independent module.
>
>
> so, I will move this into DHT without new statfs.
>

Let's here from dht folks also what they think about this change before you
make modifications. I included some of the dht folks to the thread.


>
> and then, will suggest it on gerrit !
>
>
> Thank you so much.
>
>
> -
> Taehwa Lee
> Gluesys Co.,Ltd.
> alghost@gmail.com
> +82-10-3420-6114, +82-70-8785-6591
> -----
>
> 2017. 7. 13. 오후 1:06, Pranith Kumar Karampuri <pkara...@redhat.com> 작성:
>
> hey,
>   I went through the patch. I see that statfs is always wound for
> create fop. So number of network operations increase and performance will
> be less even in normal case. I think similar functionality is in DHT, may
> be you should take a look at that?
>
> Check dht_get_du_info() which is used by dht_mknod(). It keeps refreshing
> this info every X seconds. I will let DHT guys comment a bit more about
> this. One more thing to check is if we can have just one implementation
> that satisfied everyone's requirements. i.e. move out this functionality
> from DHT to this xlator or, move the functionality of this xlator into DHT.
>
>
> On Thu, Jul 13, 2017 at 8:18 AM, Taehwa Lee <alghost@gmail.com> wrote:
>
>> Hi all,
>>
>> I’ve been developing a xlator that create is rejected when used capacity
>> of a volume higher than threshold.
>>
>>
>> the reason why I’m doing is that I got problems when LV is used fully.
>>
>> this patch is in the middle of develop.
>>
>> just I want to know whether my approach is pretty correct to satisfy my
>> requirement.
>>
>> so, when you guys have a little spare time, please review my patch and
>> tell me WHATEVER you’re thinking.
>>
>>
>> and If you guys think that it is useful for glusterfs, I’m gonna do
>> process to merge into glusterfs.
>>
>>
>> thanks in advance
>>
>>
>>
>>
>>
>> -
>> Taehwa Lee
>> Gluesys Co.,Ltd.
>> alghost@gmail.com
>> +82-10-3420-6114, +82-70-8785-6591
>> -
>>
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> Pranith
>
>
>


-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] create restrictions xlator

[Pranith Kumar Karampuri]

hey,
  I went through the patch. I see that statfs is always wound for
create fop. So number of network operations increase and performance will
be less even in normal case. I think similar functionality is in DHT, may
be you should take a look at that?

Check dht_get_du_info() which is used by dht_mknod(). It keeps refreshing
this info every X seconds. I will let DHT guys comment a bit more about
this. One more thing to check is if we can have just one implementation
that satisfied everyone's requirements. i.e. move out this functionality
from DHT to this xlator or, move the functionality of this xlator into DHT.


On Thu, Jul 13, 2017 at 8:18 AM, Taehwa Lee  wrote:

> Hi all,
>
> I’ve been developing a xlator that create is rejected when used capacity
> of a volume higher than threshold.
>
>
> the reason why I’m doing is that I got problems when LV is used fully.
>
> this patch is in the middle of develop.
>
> just I want to know whether my approach is pretty correct to satisfy my
> requirement.
>
> so, when you guys have a little spare time, please review my patch and
> tell me WHATEVER you’re thinking.
>
>
> and If you guys think that it is useful for glusterfs, I’m gonna do
> process to merge into glusterfs.
>
>
> thanks in advance
>
>
>
>
>
> -
> Taehwa Lee
> Gluesys Co.,Ltd.
> alghost@gmail.com
> +82-10-3420-6114, +82-70-8785-6591
> -
>
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] gfid and volume-id extended attributes lost

[Pranith Kumar Karampuri]

Ram,
   As per the code, self-heal was the only candidate which *can* do it.
Could you check logs of self-heal daemon and the mount to check if there
are any metadata heals on root?


+Sanoj

Sanoj,
   Is there any systemtap script we can use to detect which process is
removing these xattrs?

On Sat, Jul 8, 2017 at 2:58 AM, Ankireddypalle Reddy <are...@commvault.com>
wrote:

> We lost the attributes on all the bricks on servers glusterfs2 and
> glusterfs3 again.
>
>
>
> [root@glusterfs2 Log_Files]# gluster volume info
>
>
>
> Volume Name: StoragePool
>
> Type: Distributed-Disperse
>
> Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
>
> Status: Started
>
> Number of Bricks: 20 x (2 + 1) = 60
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: glusterfs1sds:/ws/disk1/ws_brick
>
> Brick2: glusterfs2sds:/ws/disk1/ws_brick
>
> Brick3: glusterfs3sds:/ws/disk1/ws_brick
>
> Brick4: glusterfs1sds:/ws/disk2/ws_brick
>
> Brick5: glusterfs2sds:/ws/disk2/ws_brick
>
> Brick6: glusterfs3sds:/ws/disk2/ws_brick
>
> Brick7: glusterfs1sds:/ws/disk3/ws_brick
>
> Brick8: glusterfs2sds:/ws/disk3/ws_brick
>
> Brick9: glusterfs3sds:/ws/disk3/ws_brick
>
> Brick10: glusterfs1sds:/ws/disk4/ws_brick
>
> Brick11: glusterfs2sds:/ws/disk4/ws_brick
>
> Brick12: glusterfs3sds:/ws/disk4/ws_brick
>
> Brick13: glusterfs1sds:/ws/disk5/ws_brick
>
> Brick14: glusterfs2sds:/ws/disk5/ws_brick
>
> Brick15: glusterfs3sds:/ws/disk5/ws_brick
>
> Brick16: glusterfs1sds:/ws/disk6/ws_brick
>
> Brick17: glusterfs2sds:/ws/disk6/ws_brick
>
> Brick18: glusterfs3sds:/ws/disk6/ws_brick
>
> Brick19: glusterfs1sds:/ws/disk7/ws_brick
>
> Brick20: glusterfs2sds:/ws/disk7/ws_brick
>
> Brick21: glusterfs3sds:/ws/disk7/ws_brick
>
> Brick22: glusterfs1sds:/ws/disk8/ws_brick
>
> Brick23: glusterfs2sds:/ws/disk8/ws_brick
>
> Brick24: glusterfs3sds:/ws/disk8/ws_brick
>
> Brick25: glusterfs4sds.commvault.com:/ws/disk1/ws_brick
>
> Brick26: glusterfs5sds.commvault.com:/ws/disk1/ws_brick
>
> Brick27: glusterfs6sds.commvault.com:/ws/disk1/ws_brick
>
> Brick28: glusterfs4sds.commvault.com:/ws/disk10/ws_brick
>
> Brick29: glusterfs5sds.commvault.com:/ws/disk10/ws_brick
>
> Brick30: glusterfs6sds.commvault.com:/ws/disk10/ws_brick
>
> Brick31: glusterfs4sds.commvault.com:/ws/disk11/ws_brick
>
> Brick32: glusterfs5sds.commvault.com:/ws/disk11/ws_brick
>
> Brick33: glusterfs6sds.commvault.com:/ws/disk11/ws_brick
>
> Brick34: glusterfs4sds.commvault.com:/ws/disk12/ws_brick
>
> Brick35: glusterfs5sds.commvault.com:/ws/disk12/ws_brick
>
> Brick36: glusterfs6sds.commvault.com:/ws/disk12/ws_brick
>
> Brick37: glusterfs4sds.commvault.com:/ws/disk2/ws_brick
>
> Brick38: glusterfs5sds.commvault.com:/ws/disk2/ws_brick
>
> Brick39: glusterfs6sds.commvault.com:/ws/disk2/ws_brick
>
> Brick40: glusterfs4sds.commvault.com:/ws/disk3/ws_brick
>
> Brick41: glusterfs5sds.commvault.com:/ws/disk3/ws_brick
>
> Brick42: glusterfs6sds.commvault.com:/ws/disk3/ws_brick
>
> Brick43: glusterfs4sds.commvault.com:/ws/disk4/ws_brick
>
> Brick44: glusterfs5sds.commvault.com:/ws/disk4/ws_brick
>
> Brick45: glusterfs6sds.commvault.com:/ws/disk4/ws_brick
>
> Brick46: glusterfs4sds.commvault.com:/ws/disk5/ws_brick
>
> Brick47: glusterfs5sds.commvault.com:/ws/disk5/ws_brick
>
> Brick48: glusterfs6sds.commvault.com:/ws/disk5/ws_brick
>
> Brick49: glusterfs4sds.commvault.com:/ws/disk6/ws_brick
>
> Brick50: glusterfs5sds.commvault.com:/ws/disk6/ws_brick
>
> Brick51: glusterfs6sds.commvault.com:/ws/disk6/ws_brick
>
> Brick52: glusterfs4sds.commvault.com:/ws/disk7/ws_brick
>
> Brick53: glusterfs5sds.commvault.com:/ws/disk7/ws_brick
>
> Brick54: glusterfs6sds.commvault.com:/ws/disk7/ws_brick
>
> Brick55: glusterfs4sds.commvault.com:/ws/disk8/ws_brick
>
> Brick56: glusterfs5sds.commvault.com:/ws/disk8/ws_brick
>
> Brick57: glusterfs6sds.commvault.com:/ws/disk8/ws_brick
>
> Brick58: glusterfs4sds.commvault.com:/ws/disk9/ws_brick
>
> Brick59: glusterfs5sds.commvault.com:/ws/disk9/ws_brick
>
> Brick60: glusterfs6sds.commvault.com:/ws/disk9/ws_brick
>
> Options Reconfigured:
>
> performance.readdir-ahead: on
>
> diagnostics.client-log-level: INFO
>
> auth.allow: glusterfs1sds,glusterfs2sds,glusterfs3sds,glusterfs4sds.
> commvault.com,glusterfs5sds.commvault.com,glusterfs6sds.commvault.com
>
>
>
> Thanks and Regards,
>
> Ram
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Friday, July 07, 2017 12:15 PM
>
> *To:* Ankireddypalle Reddy
> *Cc:* Gluster Devel (gluster-devel@gluster.org); gluster-us...@gluster

Re: [Gluster-devel] gfid and volume-id extended attributes lost

[Pranith Kumar Karampuri]

On Fri, Jul 7, 2017 at 9:25 PM, Ankireddypalle Reddy <are...@commvault.com>
wrote:

> 3.7.19
>

These are the only callers for removexattr and only _posix_remove_xattr has
the potential to do removexattr as posix_removexattr already makes sure
that it is not gfid/volume-id. And surprise surprise _posix_remove_xattr
happens only from healing code of afr/ec. And this can only happen if the
source brick doesn't have gfid, which doesn't seem to match with the
situation you explained.

   #   line  filename / context / line
   1   1234  xlators/mgmt/glusterd/src/glusterd-quota.c
<>
 ret = sys_lremovexattr (abspath, QUOTA_LIMIT_KEY);
   2   1243  xlators/mgmt/glusterd/src/glusterd-quota.c
<>
 ret = sys_lremovexattr (abspath, QUOTA_LIMIT_OBJECTS_KEY);
   3   6102  xlators/mgmt/glusterd/src/glusterd-utils.c
<>
 sys_lremovexattr (path, "trusted.glusterfs.test");
   4 80  xlators/storage/posix/src/posix-handle.h <>
 op_ret = sys_lremovexattr (path, key); \
   5   5026  xlators/storage/posix/src/posix.c <<_posix_remove_xattr>>
 op_ret = sys_lremovexattr (filler->real_path, key);
   6   5101  xlators/storage/posix/src/posix.c <>
 op_ret = sys_lremovexattr (real_path, name);
   7   6811  xlators/storage/posix/src/posix.c <>
 sys_lremovexattr (dir_data->data, "trusted.glusterfs.test");

So there are only two possibilities:
1) Source directory in ec/afr doesn't have gfid
2) Something else removed these xattrs.

What is your volume info? May be that will give more clues.

 PS: sys_fremovexattr is called only from posix_fremovexattr(), so that
doesn't seem to be the culprit as it also have checks to guard against
gfid/volume-id removal.


>
> Thanks and Regards,
>
> Ram
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Friday, July 07, 2017 11:54 AM
>
> *To:* Ankireddypalle Reddy
> *Cc:* Gluster Devel (gluster-devel@gluster.org); gluster-us...@gluster.org
> *Subject:* Re: [Gluster-devel] gfid and volume-id extended attributes lost
>
>
>
>
>
>
>
> On Fri, Jul 7, 2017 at 9:20 PM, Ankireddypalle Reddy <are...@commvault.com>
> wrote:
>
> Pranith,
>
>  Thanks for looking in to the issue. The bricks were
> mounted after the reboot. One more thing that I noticed was when the
> attributes were manually set when glusterd was up then on starting the
> volume the attributes were again lost. Had to stop glusterd set attributes
> and then start glusterd. After that the volume start succeeded.
>
>
>
> Which version is this?
>
>
>
>
>
> Thanks and Regards,
>
> Ram
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Friday, July 07, 2017 11:46 AM
> *To:* Ankireddypalle Reddy
> *Cc:* Gluster Devel (gluster-devel@gluster.org); gluster-us...@gluster.org
> *Subject:* Re: [Gluster-devel] gfid and volume-id extended attributes lost
>
>
>
> Did anything special happen on these two bricks? It can't happen in the
> I/O path:
> posix_removexattr() has:
>   0 if (!strcmp (GFID_XATTR_KEY, name))
> {
>
>
>   1 gf_msg (this->name, GF_LOG_WARNING, 0,
> P_MSG_XATTR_NOT_REMOVED,
>   2 "Remove xattr called on gfid for file %s",
> real_path);
>   3 op_ret = -1;
>
>   4 goto out;
>
>   5 }
>
>   6 if (!strcmp (GF_XATTR_VOL_ID_KEY, name))
> {
>   7 gf_msg (this->name, GF_LOG_WARNING, 0,
> P_MSG_XATTR_NOT_REMOVED,
>   8 "Remove xattr called on volume-id for file
> %s",
>   9 real_path);
>
>  10 op_ret = -1;
>
>  11 goto out;
>
>  12 }
>
> I just found that op_errno is not set correctly, but it can't happen in
> the I/O path, so self-heal/rebalance are off the hook.
>
> I also grepped for any removexattr of trusted.gfid from glusterd and
> didn't find any.
>
> So one thing that used to happen was that sometimes when machines reboot,
> the brick mounts wouldn't happen and this would lead to absence of both
> trusted.gfid and volume-id. So at the moment this is my wild guess.
>
>
>
>
>
> On Fri, Jul 7, 2017 at 8:39 PM, Ankireddypalle Reddy <are...@commvault.com>
> wrote:
>
> Hi,
>
>We faced an issue in the production today. We had to stop the
> volume and reboot all the servers in the cluster.  Once the servers
> rebooted starting of the volume failed because the following extended
> attributes were not present on all the bricks on 2 servers.
>
> 1)  tr

Re: [Gluster-devel] gfid and volume-id extended attributes lost

[Pranith Kumar Karampuri]

On Fri, Jul 7, 2017 at 9:20 PM, Ankireddypalle Reddy <are...@commvault.com>
wrote:

> Pranith,
>
>  Thanks for looking in to the issue. The bricks were
> mounted after the reboot. One more thing that I noticed was when the
> attributes were manually set when glusterd was up then on starting the
> volume the attributes were again lost. Had to stop glusterd set attributes
> and then start glusterd. After that the volume start succeeded.
>

Which version is this?


>
>
> Thanks and Regards,
>
> Ram
>
>
>
> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> *Sent:* Friday, July 07, 2017 11:46 AM
> *To:* Ankireddypalle Reddy
> *Cc:* Gluster Devel (gluster-devel@gluster.org); gluster-us...@gluster.org
> *Subject:* Re: [Gluster-devel] gfid and volume-id extended attributes lost
>
>
>
> Did anything special happen on these two bricks? It can't happen in the
> I/O path:
> posix_removexattr() has:
>   0 if (!strcmp (GFID_XATTR_KEY, name))
> {
>
>
>   1 gf_msg (this->name, GF_LOG_WARNING, 0,
> P_MSG_XATTR_NOT_REMOVED,
>   2 "Remove xattr called on gfid for file %s",
> real_path);
>   3 op_ret = -1;
>
>   4 goto out;
>
>   5 }
>
>   6 if (!strcmp (GF_XATTR_VOL_ID_KEY, name))
> {
>   7 gf_msg (this->name, GF_LOG_WARNING, 0,
> P_MSG_XATTR_NOT_REMOVED,
>   8 "Remove xattr called on volume-id for file
> %s",
>   9 real_path);
>
>  10 op_ret = -1;
>
>  11 goto out;
>
>  12 }
>
> I just found that op_errno is not set correctly, but it can't happen in
> the I/O path, so self-heal/rebalance are off the hook.
>
> I also grepped for any removexattr of trusted.gfid from glusterd and
> didn't find any.
>
> So one thing that used to happen was that sometimes when machines reboot,
> the brick mounts wouldn't happen and this would lead to absence of both
> trusted.gfid and volume-id. So at the moment this is my wild guess.
>
>
>
>
>
> On Fri, Jul 7, 2017 at 8:39 PM, Ankireddypalle Reddy <are...@commvault.com>
> wrote:
>
> Hi,
>
>We faced an issue in the production today. We had to stop the
> volume and reboot all the servers in the cluster.  Once the servers
> rebooted starting of the volume failed because the following extended
> attributes were not present on all the bricks on 2 servers.
>
> 1)  trusted.gfid
>
> 2)  trusted.glusterfs.volume-id
>
>
>
> We had to manually set these extended attributes to start the volume.  Are
> there any such known issues.
>
>
>
> Thanks and Regards,
>
> Ram
>
> ***Legal Disclaimer***
>
> "This communication may contain confidential and privileged material for
> the
>
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
>
> by others is strictly prohibited. If you have received the message by
> mistake,
>
> please advise the sender by reply email and delete the message. Thank you."
>
> **
>
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
>
> --
>
> Pranith
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] gfid and volume-id extended attributes lost

[Pranith Kumar Karampuri]

On Fri, Jul 7, 2017 at 9:15 PM, Pranith Kumar Karampuri <pkara...@redhat.com
> wrote:

> Did anything special happen on these two bricks? It can't happen in the
> I/O path:
> posix_removexattr() has:
>   0 if (!strcmp (GFID_XATTR_KEY, name))
> {
>
>
>   1 gf_msg (this->name, GF_LOG_WARNING, 0,
> P_MSG_XATTR_NOT_REMOVED,
>   2 "Remove xattr called on gfid for file %s",
> real_path);
>   3 op_ret = -1;
>
>   4 goto out;
>
>   5 }
>
>   6 if (!strcmp (GF_XATTR_VOL_ID_KEY, name))
> {
>   7 gf_msg (this->name, GF_LOG_WARNING, 0,
> P_MSG_XATTR_NOT_REMOVED,
>   8 "Remove xattr called on volume-id for file
> %s",
>   9 real_path);
>
>  10 op_ret = -1;
>
>  11 goto out;
>
>  12 }
>
> I just found that op_errno is not set correctly, but it can't happen in
> the I/O path, so self-heal/rebalance are off the hook.
>
> I also grepped for any removexattr of trusted.gfid from glusterd and
> didn't find any.
>
> So one thing that used to happen was that sometimes when machines reboot,
> the brick mounts wouldn't happen and this would lead to absence of both
> trusted.gfid and volume-id. So at the moment this is my wild guess.
>

Fix for this was to mount the bricks. But considering that you guys did
setting of the xattrs instead, I am guessing the other data was intact and
only these particular xattrs were missing? I wonder what new problem this
is.


>
>
> On Fri, Jul 7, 2017 at 8:39 PM, Ankireddypalle Reddy <are...@commvault.com
> > wrote:
>
>> Hi,
>>
>>We faced an issue in the production today. We had to stop the
>> volume and reboot all the servers in the cluster.  Once the servers
>> rebooted starting of the volume failed because the following extended
>> attributes were not present on all the bricks on 2 servers.
>>
>> 1)  trusted.gfid
>>
>> 2)  trusted.glusterfs.volume-id
>>
>>
>>
>> We had to manually set these extended attributes to start the volume.
>> Are there any such known issues.
>>
>>
>>
>> Thanks and Regards,
>>
>> Ram
>> ***Legal Disclaimer***
>> "This communication may contain confidential and privileged material for
>> the
>> sole use of the intended recipient. Any unauthorized review, use or
>> distribution
>> by others is strictly prohibited. If you have received the message by
>> mistake,
>> please advise the sender by reply email and delete the message. Thank
>> you."
>> **
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> Pranith
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] gfid and volume-id extended attributes lost

[Pranith Kumar Karampuri]

Did anything special happen on these two bricks? It can't happen in the I/O
path:
posix_removexattr() has:
  0 if (!strcmp (GFID_XATTR_KEY, name))
{

  1 gf_msg (this->name, GF_LOG_WARNING, 0,
P_MSG_XATTR_NOT_REMOVED,
  2 "Remove xattr called on gfid for file %s",
real_path);
  3 op_ret =
-1;
  4 goto
out;
  5
}
  6 if (!strcmp (GF_XATTR_VOL_ID_KEY, name))
{
  7 gf_msg (this->name, GF_LOG_WARNING, 0,
P_MSG_XATTR_NOT_REMOVED,
  8 "Remove xattr called on volume-id for file
%s",
  9
real_path);
 10 op_ret =
-1;
 11 goto
out;
 12 }

I just found that op_errno is not set correctly, but it can't happen in the
I/O path, so self-heal/rebalance are off the hook.

I also grepped for any removexattr of trusted.gfid from glusterd and didn't
find any.

So one thing that used to happen was that sometimes when machines reboot,
the brick mounts wouldn't happen and this would lead to absence of both
trusted.gfid and volume-id. So at the moment this is my wild guess.


On Fri, Jul 7, 2017 at 8:39 PM, Ankireddypalle Reddy 
wrote:

> Hi,
>
>We faced an issue in the production today. We had to stop the
> volume and reboot all the servers in the cluster.  Once the servers
> rebooted starting of the volume failed because the following extended
> attributes were not present on all the bricks on 2 servers.
>
> 1)  trusted.gfid
>
> 2)  trusted.glusterfs.volume-id
>
>
>
> We had to manually set these extended attributes to start the volume.  Are
> there any such known issues.
>
>
>
> Thanks and Regards,
>
> Ram
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel