date:20161101

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

2016-11-01 Thread Lian, George (Nokia - CN/Hangzhou)

Hi,

I will test it with your patches and update to you when I have result.

Thanks a lots

Best Regards,
George

From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Monday, October 31, 2016 11:23 AM
To: Lian, George (Nokia - CN/Hangzhou) 
Cc: Raghavendra Gowdappa ; Zhang, Bingxuan (Nokia - 
CN/Hangzhou) ; Gluster-devel@gluster.org; Zizka, Jan 
(Nokia - CZ/Prague) 
Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the 
really size of the syslog file

Removing 
i_ext_mbb_wcdma_swd3_da1_mat...@internal.nsn.com,
 it is causing mail delivery problems for me.
George,
 Raghavendra and I made some progress on this issue. We were in 
parallel working on another issue which is similar where elastic search indices 
are getting corrupted because of wrong stat sizes in our opinion. So I have 
been running different translator stacks in identifying the problematic xlators 
which are leading to indices corruption.
  We found the list to be 1) Write-behind, 2) Quick-read, 3) 
Readdir-ahead. Raghavendra and I just had a chat and we are suspecting that 
lack of lookup/readdirp implementation in write-behind could be the reason for 
this problem. Similar problems may exist in other two xlators too. But we are 
working on write-behind with priority.
Our theory is this:
If we do a 4KB write for example and it is cached in write-behind and we do a 
lookup on the file/do a readdirp on the directory with this file we send out 
wrong stat value to the kernel. There are different caches between kernel and 
gluster which may lead to fstat never coming till write-behind. So we need to 
make sure that we don't get into this situation.
Action items:
 At the moment Raghavendra is working on a patch to implement lookup/readdirp 
in write-behind. I am going to test the same for elastic search. Will it be 
possible for you to test your application against the same patch and confirm 
that the patch fixes the problem?

On Fri, Oct 28, 2016 at 12:08 PM, Pranith Kumar Karampuri 
> wrote:
hi George,
   It would help if we can identify the bare minimum xlators which are 
contributing to the issue like Raghavendra was mentioning earlier. We were 
wondering if it is possible for you to help us in identifying the issue by 
running the workload on a modified setup? We can suggest testing out using 
custom volfiles so that we can slowly build the graph which could be causing 
this issue. We would like you guys to try out this problem with just 
posix-xlator and fuse and nothing else.

On Thu, Oct 27, 2016 at 1:40 PM, Lian, George (Nokia - CN/Hangzhou) 
> wrote:
Hi, Raghavendra,

Could you please give some suggestion for this issue? we try to find the clue 
for this issue for a long time, but it has no progress:(

Thanks & Best Regards,
George

-Original Message-
From: Lian, George (Nokia - CN/Hangzhou)
Sent: Wednesday, October 19, 2016 4:40 PM
To: 'Raghavendra Gowdappa' >
Cc: Gluster-devel@gluster.org; 
I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS 
>;
 Zhang, Bingxuan (Nokia - CN/Hangzhou) 
>; Zizka, Jan (Nokia 
- CZ/Prague) >
Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the 
really size of the syslog file

Hi, Raghavendra

Just now, we test it with glusterfs log with debug-level "TRACE", and let some 
application trigger "glusterfs" produce large log, in that case, when we set 
write-behind and stat-prefetch both OFF,
Tail the glusterfs log such like mnt-{VOLUME-NAME}.log, it still failed with 
"file truncated",

So that means if file's IO in huge amount, the issue will still be there even 
write-behind and stat-prefetch both OFF.

Best Regards,
George

-Original Message-
From: Raghavendra Gowdappa 
[mailto:rgowd...@redhat.com]
Sent: Wednesday, October 19, 2016 2:54 PM
To: Lian, George (Nokia - CN/Hangzhou) 
>
Cc: Gluster-devel@gluster.org; 
I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS 
>;
 Zhang, Bingxuan (Nokia - CN/Hangzhou) 
>; Zizka, Jan (Nokia 
- CZ/Prague) >
Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the 
really size of the syslog file

- Original Message -
> From: "George

[Gluster-devel] A question of GlusterFS dentries!

2016-11-01 Thread Keiviw

Hi,
In GlusterFS distributed volumes, listing a non-empty directory was slow. 
Then I read the dht codes and found the reasons. But I was confused that 
GlusterFS dht travesed all the bricks(in the volume) sequentially,why not use 
multi-thread to read dentries from multiple bricks simultaneously. That's a 
question that's always puzzled me, Couly you please tell me something about 
this??? ___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

2016-11-01 Thread Raghavendra G

Hi George,

You can try your tests in two scenarios:

1. Switch off following performance translators:
a. write-behind
b. quick-read
c. readdir-ahead

Once you see that the issue goes away in the above scenario, you can apply
write-behind fix [1]  and turn write-behind on.

We have a fix in quick-read at [2].

You can also try to test with the quick-read fix and turning quick-read on.
However, since your issue is stale stat I doubt quick-read is not the
culprit here (The issue with quick-read is stale data).

We see that readdir-ahead is not updating/flushing the stats stored along
with dentries when write or file modifications happen. That seem to be the
issue with readdir-ahead. For now, during both of the above two scenarios,
please keep readdir-ahead off.

[1] http://review.gluster.org/15757/2
[2] http://review.gluster.org/#/c/15759/2

regards,
Raghavendra

On Tue, Nov 1, 2016 at 7:55 AM, Pranith Kumar Karampuri  wrote:

>
>
> On Tue, Nov 1, 2016 at 7:32 AM, Lian, George (Nokia - CN/Hangzhou) <
> george.l...@nokia.com> wrote:
>
>> Hi,
>>
>>
>>
>> I will test it with your patches and update to you when I have result.
>>
>
> hi George,
>   Please use http://review.gluster.org/#/c/15757/2 i.e. second
> version of Raghavendra's patch. I tested it and it worked fine. We are
> still trying to figure out quick-read and readdir-ahead as I type this mail.
>
>
>>
>> Thanks a lots
>>
>>
>>
>> Best Regards,
>>
>> George
>>
>>
>>
>> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
>> *Sent:* Monday, October 31, 2016 11:23 AM
>> *To:* Lian, George (Nokia - CN/Hangzhou) 
>> *Cc:* Raghavendra Gowdappa ; Zhang, Bingxuan (Nokia
>> - CN/Hangzhou) ; Gluster-devel@gluster.org;
>> Zizka, Jan (Nokia - CZ/Prague) 
>>
>> *Subject:* Re: [Gluster-devel] Issue about the size of fstat is less
>> than the really size of the syslog file
>>
>>
>>
>> Removing i_ext_mbb_wcdma_swd3_da1_mat...@internal.nsn.com, it is causing
>> mail delivery problems for me.
>>
>> George,
>>
>>  Raghavendra and I made some progress on this issue. We were in
>> parallel working on another issue which is similar where elastic search
>> indices are getting corrupted because of wrong stat sizes in our opinion.
>> So I have been running different translator stacks in identifying the
>> problematic xlators which are leading to indices corruption.
>>
>>   We found the list to be 1) Write-behind, 2) Quick-read, 3)
>> Readdir-ahead. Raghavendra and I just had a chat and we are suspecting that
>> lack of lookup/readdirp implementation in write-behind could be the reason
>> for this problem. Similar problems may exist in other two xlators too. But
>> we are working on write-behind with priority.
>>
>> Our theory is this:
>>
>> If we do a 4KB write for example and it is cached in write-behind and we
>> do a lookup on the file/do a readdirp on the directory with this file we
>> send out wrong stat value to the kernel. There are different caches between
>> kernel and gluster which may lead to fstat never coming till write-behind.
>> So we need to make sure that we don't get into this situation.
>>
>> Action items:
>>
>>  At the moment Raghavendra is working on a patch to implement
>> lookup/readdirp in write-behind. I am going to test the same for elastic
>> search. Will it be possible for you to test your application against the
>> same patch and confirm that the patch fixes the problem?
>>
>>
>>
>> On Fri, Oct 28, 2016 at 12:08 PM, Pranith Kumar Karampuri <
>> pkara...@redhat.com> wrote:
>>
>> hi George,
>>
>>It would help if we can identify the bare minimum xlators which
>> are contributing to the issue like Raghavendra was mentioning earlier. We
>> were wondering if it is possible for you to help us in identifying the
>> issue by running the workload on a modified setup? We can suggest testing
>> out using custom volfiles so that we can slowly build the graph which could
>> be causing this issue. We would like you guys to try out this problem with
>> just posix-xlator and fuse and nothing else.
>>
>>
>>
>> On Thu, Oct 27, 2016 at 1:40 PM, Lian, George (Nokia - CN/Hangzhou) <
>> george.l...@nokia.com> wrote:
>>
>> Hi, Raghavendra,
>>
>> Could you please give some suggestion for this issue? we try to find the
>> clue for this issue for a long time, but it has no progress:(
>>
>> Thanks & Best Regards,
>> George
>>
>> -Original Message-
>> From: Lian, George (Nokia - CN/Hangzhou)
>> Sent: Wednesday, October 19, 2016 4:40 PM
>> To: 'Raghavendra Gowdappa' 
>> Cc: Gluster-devel@gluster.org; I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS <
>> i_ext_mbb_wcdma_swd3_da1_mat...@internal.nsn.com>; Zhang, Bingxuan
>> (Nokia - CN/Hangzhou) ; Zizka, Jan (Nokia -
>> CZ/Prague) 
>> Subject: RE: [Gluster-devel] Issue about the size of fstat is less than
>> the really

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

2016-11-01 Thread Raghavendra Gowdappa



- Original Message -
> From: "Raghavendra G" 
> To: "Pranith Kumar Karampuri" 
> Cc: Gluster-devel@gluster.org, "Jan Zizka (Nokia - CZ/Prague)" 
> , "Bingxuan Zhang (Nokia -
> CN/Hangzhou)" 
> Sent: Tuesday, November 1, 2016 6:55:31 PM
> Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the 
> really size of the syslog file
> 
> Hi George,
> 
> You can try your tests in two scenarios:
> 
> 1. Switch off following performance translators:
> a. write-behind
> b. quick-read
> c. readdir-ahead
> 
> Once you see that the issue goes away in the above scenario, you can apply
> write-behind fix [1] and turn write-behind on.
> 
> We have a fix in quick-read at [2].
> 
> You can also try to test with the quick-read fix and turning quick-read on.
> However, since your issue is stale stat I doubt quick-read is not the

s/quick-read is not the culprit/quick-read is the culprit/

> culprit here (The issue with quick-read is stale data).
> 
> We see that readdir-ahead is not updating/flushing the stats stored along
> with dentries when write or file modifications happen. That seem to be the
> issue with readdir-ahead. For now, during both of the above two scenarios,
> please keep readdir-ahead off.
> 
> [1] http://review.gluster.org/15757/2
> [2] http://review.gluster.org/#/c/15759/2
> 
> regards,
> Raghavendra
> 
> On Tue, Nov 1, 2016 at 7:55 AM, Pranith Kumar Karampuri < pkara...@redhat.com
> > wrote:
> 
> 
> 
> 
> 
> On Tue, Nov 1, 2016 at 7:32 AM, Lian, George (Nokia - CN/Hangzhou) <
> george.l...@nokia.com > wrote:
> 
> 
> 
> 
> 
> Hi,
> 
> 
> 
> I will test it with your patches and update to you when I have result.
> 
> hi George,
> Please use http://review.gluster.org/#/c/15757/2 i.e. second version of
> Raghavendra's patch. I tested it and it worked fine. We are still trying to
> figure out quick-read and readdir-ahead as I type this mail.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks a lots
> 
> 
> 
> Best Regards,
> 
> George
> 
> 
> 
> From: Pranith Kumar Karampuri [mailto: pkara...@redhat.com ]
> Sent: Monday, October 31, 2016 11:23 AM
> To: Lian, George (Nokia - CN/Hangzhou) < george.l...@nokia.com >
> Cc: Raghavendra Gowdappa < rgowd...@redhat.com >; Zhang, Bingxuan (Nokia -
> CN/Hangzhou) < bingxuan.zh...@nokia.com >; Gluster-devel@gluster.org ;
> Zizka, Jan (Nokia - CZ/Prague) < jan.zi...@nokia.com >
> 
> Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the
> really size of the syslog file
> 
> 
> 
> 
> 
> 
> 
> 
> Removing i_ext_mbb_wcdma_swd3_da1_mat...@internal.nsn.com , it is causing
> mail delivery problems for me.
> 
> 
> George,
> 
> 
> Raghavendra and I made some progress on this issue. We were in parallel
> working on another issue which is similar where elastic search indices are
> getting corrupted because of wrong stat sizes in our opinion. So I have been
> running different translator stacks in identifying the problematic xlators
> which are leading to indices corruption.
> 
> 
> We found the list to be 1) Write-behind, 2) Quick-read, 3) Readdir-ahead.
> Raghavendra and I just had a chat and we are suspecting that lack of
> lookup/readdirp implementation in write-behind could be the reason for this
> problem. Similar problems may exist in other two xlators too. But we are
> working on write-behind with priority.
> 
> 
> Our theory is this:
> 
> 
> If we do a 4KB write for example and it is cached in write-behind and we do a
> lookup on the file/do a readdirp on the directory with this file we send out
> wrong stat value to the kernel. There are different caches between kernel
> and gluster which may lead to fstat never coming till write-behind. So we
> need to make sure that we don't get into this situation.
> 
> 
> Action items:
> 
> 
> At the moment Raghavendra is working on a patch to implement lookup/readdirp
> in write-behind. I am going to test the same for elastic search. Will it be
> possible for you to test your application against the same patch and confirm
> that the patch fixes the problem?
> 
> 
> 
> 
> 
> On Fri, Oct 28, 2016 at 12:08 PM, Pranith Kumar Karampuri <
> pkara...@redhat.com > wrote:
> 
> 
> 
> 
> hi George,
> 
> 
> It would help if we can identify the bare minimum xlators which are
> contributing to the issue like Raghavendra was mentioning earlier. We were
> wondering if it is possible for you to help us in identifying the issue by
> running the workload on a modified setup? We can suggest testing out using
> custom volfiles so that we can slowly build the graph which could be causing
> this issue. We would like you guys to try out this problem with just
> posix-xlator and fuse and nothing else.
> 
> 
> 
> 
> 
> On Thu, Oct 27, 2016 at 1:40 PM, Lian, George (Nokia - CN/Hangzhou) <
> george.l...@nokia.com > wrote:
> 
> 
> 
> Hi, Raghavendra,
> 
> Could you please give some suggestion for this issue? we try to find the clue
>

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

2016-11-01 Thread Raghavendra Gowdappa



- Original Message -
> From: "George Lian (Nokia - CN/Hangzhou)" 
> To: "Pranith Kumar Karampuri" 
> Cc: "Raghavendra Gowdappa" , "Bingxuan Zhang (Nokia - 
> CN/Hangzhou)" ,
> Gluster-devel@gluster.org, "Jan Zizka (Nokia - CZ/Prague)" 
> 
> Sent: Wednesday, November 2, 2016 8:52:32 AM
> Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the 
> really size of the syslog file
> 
> Hi,
> 
> As I used the patch from http://review.gluster.org/#/c/15757/2
> The issue still exist,

Was readdir-ahead turned off when you did the test? Please refer to my last 
mail in this thread for more details.

> 
> # tail -f syslog >syslog.2.log
> tail: syslog: file truncated
> 
> One more information, we used GlusterFS version with 3.6.9.
> 
> FYI,
> 
> Best Regards,
> George
> From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
> Sent: Tuesday, November 01, 2016 10:25 AM
> To: Lian, George (Nokia - CN/Hangzhou) 
> Cc: Raghavendra Gowdappa ; Zhang, Bingxuan (Nokia -
> CN/Hangzhou) ; Gluster-devel@gluster.org; Zizka,
> Jan (Nokia - CZ/Prague) 
> Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the
> really size of the syslog file
> 
> 
> 
> On Tue, Nov 1, 2016 at 7:32 AM, Lian, George (Nokia - CN/Hangzhou)
> > wrote:
> Hi,
> 
> I will test it with your patches and update to you when I have result.
> 
> hi George,
>   Please use http://review.gluster.org/#/c/15757/2 i.e. second version of
>   Raghavendra's patch. I tested it and it worked fine. We are still
>   trying to figure out quick-read and readdir-ahead as I type this mail.
> 
> Thanks a lots
> 
> Best Regards,
> George
> 
> From: Pranith Kumar Karampuri
> [mailto:pkara...@redhat.com]
> Sent: Monday, October 31, 2016 11:23 AM
> To: Lian, George (Nokia - CN/Hangzhou)
> >
> Cc: Raghavendra Gowdappa >;
> Zhang, Bingxuan (Nokia - CN/Hangzhou)
> >;
> Gluster-devel@gluster.org; Zizka, Jan
> (Nokia - CZ/Prague) >
> 
> Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the
> really size of the syslog file
> 
> Removing
> i_ext_mbb_wcdma_swd3_da1_mat...@internal.nsn.com,
> it is causing mail delivery problems for me.
> George,
>  Raghavendra and I made some progress on this issue. We were in
>  parallel working on another issue which is similar where elastic
>  search indices are getting corrupted because of wrong stat sizes in
>  our opinion. So I have been running different translator stacks in
>  identifying the problematic xlators which are leading to indices
>  corruption.
>   We found the list to be 1) Write-behind, 2) Quick-read, 3)
>   Readdir-ahead. Raghavendra and I just had a chat and we are
>   suspecting that lack of lookup/readdirp implementation in
>   write-behind could be the reason for this problem. Similar
>   problems may exist in other two xlators too. But we are working on
>   write-behind with priority.
> Our theory is this:
> If we do a 4KB write for example and it is cached in write-behind and we do a
> lookup on the file/do a readdirp on the directory with this file we send out
> wrong stat value to the kernel. There are different caches between kernel
> and gluster which may lead to fstat never coming till write-behind. So we
> need to make sure that we don't get into this situation.
> Action items:
>  At the moment Raghavendra is working on a patch to implement lookup/readdirp
>  in write-behind. I am going to test the same for elastic search. Will it be
>  possible for you to test your application against the same patch and
>  confirm that the patch fixes the problem?
> 
> On Fri, Oct 28, 2016 at 12:08 PM, Pranith Kumar Karampuri
> > wrote:
> hi George,
>It would help if we can identify the bare minimum xlators which are
>contributing to the issue like Raghavendra was mentioning earlier. We
>were wondering if it is possible for you to help us in identifying
>the issue by running the workload on a modified setup? We can suggest
>testing out using custom volfiles so that we can slowly build the
>graph which could be causing this issue. We would like you guys to
>try out this problem with just posix-xlator and fuse and nothing
>else.
> 
> On Thu, Oct 27, 2016 at 1:40 PM, Lian, George

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

2016-11-01 Thread Lian, George (Nokia - CN/Hangzhou)

Hi,

As I used the patch from http://review.gluster.org/#/c/15757/2
The issue still exist,

# tail -f syslog >syslog.2.log
tail: syslog: file truncated

One more information, we used GlusterFS version with 3.6.9.

FYI,

Best Regards,
George
From: Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
Sent: Tuesday, November 01, 2016 10:25 AM
To: Lian, George (Nokia - CN/Hangzhou) 
Cc: Raghavendra Gowdappa ; Zhang, Bingxuan (Nokia - 
CN/Hangzhou) ; Gluster-devel@gluster.org; Zizka, Jan 
(Nokia - CZ/Prague) 
Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the 
really size of the syslog file

On Tue, Nov 1, 2016 at 7:32 AM, Lian, George (Nokia - CN/Hangzhou) 
> wrote:
Hi,

I will test it with your patches and update to you when I have result.

hi George,
  Please use http://review.gluster.org/#/c/15757/2 i.e. second version of 
Raghavendra's patch. I tested it and it worked fine. We are still trying to 
figure out quick-read and readdir-ahead as I type this mail.

Thanks a lots

Best Regards,
George

From: Pranith Kumar Karampuri 
[mailto:pkara...@redhat.com]
Sent: Monday, October 31, 2016 11:23 AM
To: Lian, George (Nokia - CN/Hangzhou) 
>
Cc: Raghavendra Gowdappa >; 
Zhang, Bingxuan (Nokia - CN/Hangzhou) 
>; 
Gluster-devel@gluster.org; Zizka, Jan (Nokia 
- CZ/Prague) >

Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the 
really size of the syslog file

Removing 
i_ext_mbb_wcdma_swd3_da1_mat...@internal.nsn.com,
 it is causing mail delivery problems for me.
George,
 Raghavendra and I made some progress on this issue. We were in 
parallel working on another issue which is similar where elastic search indices 
are getting corrupted because of wrong stat sizes in our opinion. So I have 
been running different translator stacks in identifying the problematic xlators 
which are leading to indices corruption.
  We found the list to be 1) Write-behind, 2) Quick-read, 3) 
Readdir-ahead. Raghavendra and I just had a chat and we are suspecting that 
lack of lookup/readdirp implementation in write-behind could be the reason for 
this problem. Similar problems may exist in other two xlators too. But we are 
working on write-behind with priority.
Our theory is this:
If we do a 4KB write for example and it is cached in write-behind and we do a 
lookup on the file/do a readdirp on the directory with this file we send out 
wrong stat value to the kernel. There are different caches between kernel and 
gluster which may lead to fstat never coming till write-behind. So we need to 
make sure that we don't get into this situation.
Action items:
 At the moment Raghavendra is working on a patch to implement lookup/readdirp 
in write-behind. I am going to test the same for elastic search. Will it be 
possible for you to test your application against the same patch and confirm 
that the patch fixes the problem?

On Fri, Oct 28, 2016 at 12:08 PM, Pranith Kumar Karampuri 
> wrote:
hi George,
   It would help if we can identify the bare minimum xlators which are 
contributing to the issue like Raghavendra was mentioning earlier. We were 
wondering if it is possible for you to help us in identifying the issue by 
running the workload on a modified setup? We can suggest testing out using 
custom volfiles so that we can slowly build the graph which could be causing 
this issue. We would like you guys to try out this problem with just 
posix-xlator and fuse and nothing else.

On Thu, Oct 27, 2016 at 1:40 PM, Lian, George (Nokia - CN/Hangzhou) 
> wrote:
Hi, Raghavendra,

Could you please give some suggestion for this issue? we try to find the clue 
for this issue for a long time, but it has no progress:(

Thanks & Best Regards,
George

-Original Message-
From: Lian, George (Nokia - CN/Hangzhou)
Sent: Wednesday, October 19, 2016 4:40 PM
To: 'Raghavendra Gowdappa' >
Cc: Gluster-devel@gluster.org; 
I_EXT_MBB_WCDMA_SWD3_DA1_MATRIX_GMS 
>;
 Zhang, Bingxuan (Nokia - CN/Hangzhou) 
>; Zizka, Jan (Nokia 
- CZ/Prague) >
Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the 
really size of the syslog file

Re: [Gluster-devel] A question of GlusterFS dentries!

2016-11-01 Thread Raghavendra Gowdappa



- Original Message -
> From: "Keiviw" 
> To: gluster-devel@gluster.org
> Sent: Tuesday, November 1, 2016 12:41:02 PM
> Subject: [Gluster-devel] A question of GlusterFS dentries!
> 
> Hi,
> In GlusterFS distributed volumes, listing a non-empty directory was slow.
> Then I read the dht codes and found the reasons. But I was confused that
> GlusterFS dht travesed all the bricks(in the volume) sequentially,why not
> use multi-thread to read dentries from multiple bricks simultaneously.
> That's a question that's always puzzled me, Couly you please tell me
> something about this???

readdir across subvols is sequential mostly because we have to support 
rewinddir(3). We need to maintain the mapping of offset and dentry across 
multiple invocations of readdir. In other words if someone did a rewinddir to 
an offset corresponding to earlier dentry, subsequent readdirs should return 
same set of dentries what the earlier invocation of readdir returned. For 
example, in an hypothetical scenario, readdir returned following dentries:

1. a, off=10
2. b, off=2
3. c, off=5
4. d, off=15
5. e, off=17
6. f, off=13

Now if we did rewinddir to off 5 and issue readdir again we should get 
following dentries:
(c, off=5), (d, off=15), (e, off=17), (f, off=13)

Within a subvol backend filesystem provides rewinddir guarantee for the 
dentries present on that subvol. However, across subvols it is the 
responsibility of DHT to provide the above guarantee. Which means we should've 
some well defined order in which we send readdir calls (Note that order is not 
well defined if we do a parallel readdir across all subvols). So, DHT has 
sequential readdir which is a well defined order of reading dentries.

To give an example if we have another subvol - subvol2 - (in addiction to the 
subvol above - say subvol1) with following listing:
1. g, off=16
2. h, off=20
3. i, off=3
4. j, off=19

With parallel readdir we can have many ordering like - (a, b, g, h, i, c, d, e, 
f, j), (g, h, a, b, c, i, j, d, e, f) etc. Now if we do (with readdir done 
parallely):

1. A complete listing of the directory (which can be any one of 10P1 = 10 ways 
- I hope math is correct here).
2. Do rewinddir (20)

We cannot predict what are the set of dentries that come _after_ offset 20. 
However, if we do a readdir sequentially across subvols there is only one 
directory listing i.e, (a, b, c, d, e, f, g, h, i, j). So, its easier to 
support rewinddir.

If there is no POSIX requirement for rewinddir support, I think a parallel 
readdir can easily be implemented (which improves performance too). But 
unfortunately rewinddir is still a POSIX requirement. This also opens up 
another possibility of a "no-rewinddir-support" option in DHT, which if enabled 
results in parallel readdirs across subvols. What I am not sure is how many 
users still use rewinddir? If there is a critical mass which wants performance 
with a tradeoff of no rewinddir support this can be a good feature.

+gluster-users to get an opinion on this.

regards,
Raghavendra

> 
> 
> 
> 
> 
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Following up on Community Bootstrap Challenge

2016-11-01 Thread Amye Scavarda

On Mon, Oct 24, 2016 at 10:20 AM, Amye Scavarda  wrote:

> Notes from our Gluster Developer Summit 2016 in Berlin!
>
> Videos
> Slides
> Flickr Group
> Public Etherpad
> Bootstrapping Challenge
>
> All of the videos from Gluster Developer Summit are now live on our
> YouTube channel, and slides are available in our Slideshare accounts. We've
> also created a Flickr group, please add your photos of the event!
>
> https://www.youtube.com/user/GlusterCommunity
> http://www.slideshare.net/GlusterCommunity
> https://www.flickr.com/groups/glusterdevelopersummit2016/
>
> We've also got a public etherpad for our comments from the event:
> https://public.pad.fsfe.org/p/gluster-developer-summit-2016
>
> Please feel free to add to this and help keep our momentum from this
> event! I'm looking for the community maintainers to take a strong hand in
> here to be able to tell us what they're focusing on this from this event
> over the next three months.
>
> One thing that we didn't get to that I wanted to was a Community Bootstrap
> Challenge, so let's do this as a hangout after the Community Meeting on
> November 2nd. I'll send out a separate email on this describing the event,
> and we'll all join in at 1pm UTC.
>
> As we're still working on a 3.9 release, and this would fit perfectly
within a 3.9 release plan, I'll post about this again more directly as we
get there.
Watch for more!

- amye


> Anything I missed?
>
> Happy to take suggestions and comments about what else we'd want to see in
> a Gluster Developer Summit!
>
> -- amye
>
> --
> Amye Scavarda | a...@redhat.com | Gluster Community Lead
>

Editing to add:
As we're still working on a 3.9 release, and that would be a fantastic
Community Bootstrap Challenge, I'm moving this around a bit.
Rest assured, we'll do a hangout around this.

For

-- 
Amye Scavarda | a...@redhat.com | Gluster Community Lead
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

2016-11-01 Thread Raghavendra Gowdappa

Can you try with following xlators turned off?

1. write-behind
2. readdir-ahead
3. quick-read

regards,
Raghavendra

- Original Message -
> From: "George Lian (Nokia - CN/Hangzhou)" 
> To: "Raghavendra Gowdappa" , "Raghavendra G" 
> 
> Cc: Gluster-devel@gluster.org, "Jan Zizka (Nokia - CZ/Prague)" 
> , "Bingxuan Zhang (Nokia -
> CN/Hangzhou)" 
> Sent: Wednesday, November 2, 2016 10:26:08 AM
> Subject: RE: [Gluster-devel] Issue about the size of fstat is less than the 
> really size of the syslog file
> 
> Hi,
> 
> Sorry, it seems the issue still there with readdir-ahead off.
> 
> # tail -f syslog >syslog.2.log
> tail: syslog: file truncated
> tail: syslog: file truncated
> tail: syslog: file truncated
> tail: syslog: file truncated
> tail: syslog: file truncated
> 
> 
> 
> 
> 
> 
> 
> tail: syslog: file truncated
> 
> The last line is new truncated error message after the following set with
> option readdir-ahead.
> 
> 
> # gluster volume set log performance.readdir-ahead off
> volume set: success
> [root@SN-0(RNC-124) /root]
> # gluster volume info log
> 
> Volume Name: log
> Type: Replicate
> Volume ID: 9d987a23-1d64-4857-980a-548f1dd0d1e6
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 169.254.0.12:/mnt/bricks/log/brick
> Brick2: 169.254.0.13:/mnt/bricks/log/brick
> Options Reconfigured:
> performance.readdir-ahead: off
> 
> BRs
> George
> 
> -Original Message-
> From: gluster-devel-boun...@gluster.org
> [mailto:gluster-devel-boun...@gluster.org] On Behalf Of Raghavendra Gowdappa
> Sent: Tuesday, November 01, 2016 9:29 PM
> To: Raghavendra G 
> Cc: Gluster-devel@gluster.org; Zizka, Jan (Nokia - CZ/Prague)
> ; Zhang, Bingxuan (Nokia - CN/Hangzhou)
> 
> Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the
> really size of the syslog file
> 
> 
> 
> - Original Message -
> > From: "Raghavendra G" 
> > To: "Pranith Kumar Karampuri" 
> > Cc: Gluster-devel@gluster.org, "Jan Zizka (Nokia - CZ/Prague)"
> > , "Bingxuan Zhang (Nokia -
> > CN/Hangzhou)" 
> > Sent: Tuesday, November 1, 2016 6:55:31 PM
> > Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the
> > really size of the syslog file
> > 
> > Hi George,
> > 
> > You can try your tests in two scenarios:
> > 
> > 1. Switch off following performance translators:
> > a. write-behind
> > b. quick-read
> > c. readdir-ahead
> > 
> > Once you see that the issue goes away in the above scenario, you can apply
> > write-behind fix [1] and turn write-behind on.
> > 
> > We have a fix in quick-read at [2].
> > 
> > You can also try to test with the quick-read fix and turning quick-read on.
> > However, since your issue is stale stat I doubt quick-read is not the
> 
> s/quick-read is not the culprit/quick-read is the culprit/
> 
> > culprit here (The issue with quick-read is stale data).
> > 
> > We see that readdir-ahead is not updating/flushing the stats stored along
> > with dentries when write or file modifications happen. That seem to be the
> > issue with readdir-ahead. For now, during both of the above two scenarios,
> > please keep readdir-ahead off.
> > 
> > [1] http://review.gluster.org/15757/2
> > [2] http://review.gluster.org/#/c/15759/2
> > 
> > regards,
> > Raghavendra
> > 
> > On Tue, Nov 1, 2016 at 7:55 AM, Pranith Kumar Karampuri <
> > pkara...@redhat.com
> > > wrote:
> > 
> > 
> > 
> > 
> > 
> > On Tue, Nov 1, 2016 at 7:32 AM, Lian, George (Nokia - CN/Hangzhou) <
> > george.l...@nokia.com > wrote:
> > 
> > 
> > 
> > 
> > 
> > Hi,
> > 
> > 
> > 
> > I will test it with your patches and update to you when I have result.
> > 
> > hi George,
> > Please use http://review.gluster.org/#/c/15757/2 i.e. second version of
> > Raghavendra's patch. I tested it and it worked fine. We are still trying to
> > figure out quick-read and readdir-ahead as I type this mail.
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > Thanks a lots
> > 
> > 
> > 
> > Best Regards,
> > 
> > George
> > 
> > 
> > 
> > From: Pranith Kumar Karampuri [mailto: pkara...@redhat.com ]
> > Sent: Monday, October 31, 2016 11:23 AM
> > To: Lian, George (Nokia - CN/Hangzhou) < george.l...@nokia.com >
> > Cc: Raghavendra Gowdappa < rgowd...@redhat.com >; Zhang, Bingxuan (Nokia -
> > CN/Hangzhou) < bingxuan.zh...@nokia.com >; Gluster-devel@gluster.org ;
> > Zizka, Jan (Nokia - CZ/Prague) < jan.zi...@nokia.com >
> > 
> > Subject: Re: [Gluster-devel] Issue about the size of fstat is less than the
> > really size of the syslog file
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > Removing i_ext_mbb_wcdma_swd3_da1_mat...@internal.nsn.com , it is causing
> > mail delivery problems for me.
> > 
> > 
> > George,
> > 
> >

[Gluster-devel] Reminder to add meeting updates

2016-11-01 Thread Kaushal M

Hi all,

This is a reminder to all to add their updates to weekly meeting pad
[1]. Please make sure you find some time to add updates about your
components, features or things you are working on.

Thanks,
Kaushal

[1] https://public.pad.fsfe.org/p/gluster-community-meetings
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

[Gluster-devel] A question of GlusterFS dentries!

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

Re: [Gluster-devel] A question of GlusterFS dentries!

[Gluster-devel] Following up on Community Bootstrap Challenge

Re: [Gluster-devel] Issue about the size of fstat is less than the really size of the syslog file

[Gluster-devel] Reminder to add meeting updates

10 matches

Site Navigation

Mail list logo

Footer information