Re: [Gluster-devel] [NFS-Ganesha-Devel] Re: Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Soumya Koduri]

On 10/12/18 12:04 PM, Soumya Koduri wrote:
1. Cache it separately as posix ACLs (a new option maybe 
"cache-glusterfs-acl" is added);

 And make sure _posix_xattr_get_set fills them when lookup requests.



I am not sure if posix layer can handle it. Virtual xattrs are 
in-memory and not stored on disk. They are converted to/from posix-acl 
in posix-acl xlator. So FWIU, posix-acl xlator should handle setting 
these attributes as part of LOOKUP response if needed. Same shall 
apply for any virtual xattr cached in md-cache. Request Poornima to 
comment.


Posix-acl can hand it correctly now.


Okay.





At a time, any gfapi consumer would use either posix-acl or virtual 
glusterfs ACLs. So having two options to selectively choose which one 
of them to cache sounds better to me instead of unnecessarily storing 
two different representations of the same ACL.


Make sense.
I will add another option for virtual glusterfs ACLs in md-cache.


Cool. thanks!

-Soumya



thanks,
Kinglong Mee

___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [NFS-Ganesha-Devel] Re: Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Soumya Koduri]

On 10/12/18 7:22 AM, Kinglong Mee wrote:

On 2018/10/11 19:09, Soumya Koduri wrote:

NFS-Ganesha's md-cache layer already does extensive caching of attributes and 
ACLs of each file looked upon. Do you see any additional benefit with turning 
on gluster md-cache as well?  More replies inline..


Yes, I think.

The logical is different between md-cache and ganesha's cache,
Ganesha caches xattr data depends on timeout, if timeout, ganesha get it from 
back-end glusterfs;
Md-cache caches depneds on timeout too, but md-cache can delay the timeout for 
some cases.


Could you please list out which xattrs fall into that category. AFAIK, 
like iatt, even all xattrs are invalidated post timeout in md-cache xlator.


By turning on both these caches, the process shall consume more memory 
and CPUs to store and invalidate same set of attributes at two different 
layers right. Do you see much better performance when compared to that 
cost?


Thanks,
Soumya






On 10/11/18 7:47 AM, Kinglong Mee wrote:

Cc nfs-ganesha,

Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs 
ACLs
("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.

But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup 
requests.
So, md-cache caches bad virtual glusterfs ACLs.

After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client 
gets many EIO errors.

https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305

There are two chooses for cache virtual glusterfs ACLs in md-cache,
1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" 
is added);
     And make sure _posix_xattr_get_set fills them when lookup requests.



I am not sure if posix layer can handle it. Virtual xattrs are in-memory and 
not stored on disk. They are converted to/from posix-acl in posix-acl xlator. 
So FWIU, posix-acl xlator should handle setting these attributes as part of 
LOOKUP response if needed. Same shall apply for any virtual xattr cached in 
md-cache. Request Poornima to comment.


Posix-acl can hand it correctly now.



At a time, any gfapi consumer would use either posix-acl or virtual glusterfs 
ACLs. So having two options to selectively choose which one of them to cache 
sounds better to me instead of unnecessarily storing two different 
representations of the same ACL.


Make sense.
I will add another option for virtual glusterfs ACLs in md-cache.

thanks,
Kinglong Mee



Thanks,
Soumya


2. Does not cache it, only cache posix ACLs;
     If gfapi request it, md-cache lookup according posix ACL at cache,
     if exist, make the virtual glusterfs ACL locally and return to gfapi;
     otherwise, send the request to glusterfsd.

Virtual glusterfs ACLs are another format of posix ACLs, there are larger than 
posix ACLs,
and always exist no matter the really posix ACL exist or not.




So, I'd prefer #2.
Any comments are welcome.

thanks,
Kinglong Mee

___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel




___
Devel mailing list -- de...@lists.nfs-ganesha.org
To unsubscribe send an email to devel-le...@lists.nfs-ganesha.org


___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Release 5: GA and what are we waiting on

[Raghavendra Gowdappa]

On Thu, Oct 11, 2018 at 9:16 PM Krutika Dhananjay 
wrote:

>
>
> On Thu, Oct 11, 2018 at 8:55 PM Shyam Ranganathan 
> wrote:
>
>> So we are through with a series of checks and tasks on release-5 (like
>> ensuring all backports to other branches are present in 5, upgrade
>> testing, basic performance testing, Package testing, etc.), but still
>> need the following resolved else we stand to delay the release GA
>> tagging, which I hope to get done over the weekend or by Monday 15th
>> morning (EDT).
>>
>> 1) Fix for libgfapi-python related blocker on Gluster:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1630804
>
>
I'll take a look and respond over the bz.


>
>>
>> @ppai, who needs to look into this?
>>
>> 2) Release notes for options added to the code (see:
>> https://lists.gluster.org/pipermail/gluster-devel/2018-October/055563.html
>> )
>>
>> @du, @krutika can we get some text for the options referred in the mail
>> above?
>>
>
I just responded in the other thread.


>
>>
> Replied here -
> https://lists.gluster.org/pipermail/gluster-devel/2018-October/055577.html
>
> -Krutika
>
> 3) Python3 testing
>> - Heard back from Kotresh on geo-rep passing and saw that we have
>> handled cliutils issues
>> - Anything more to cover? (@aravinda, @kotresh, @ppai?)
>> - We are attempting to get a regression run on a Python3 platform, but
>> that maybe a little ways away from the release (see:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1638030 )
>>
>> Request attention to the above, to ensure we are not breaking things
>> with the release.
>>
>> Thanks,
>> Shyam
>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 5: Missing option documentation (need inputs)

[Raghavendra Gowdappa]

On Wed, Oct 10, 2018 at 8:30 PM Shyam Ranganathan 
wrote:

> The following options were added post 4.1 and are part of 5.0 as the
> first release for the same. They were added in as part of bugs, and
> hence looking at github issues to track them as enhancements did not
> catch the same.
>
> We need to document it in the release notes (and also the gluster doc.
> site ideally), and hence I would like a some details on what to write
> for the same (or release notes commits) for them.
>
> Option: cluster.daemon-log-level
> Attention: @atin
> Review: https://review.gluster.org/c/glusterfs/+/20442
>
> Option: ctime-invalidation
> Attention: @Du
> Review: https://review.gluster.org/c/glusterfs/+/20286


Quick-read by default uses mtime to identify changes to file data. However
there are applications like rsync which explicitly set mtime making it
unreliable for the purpose of identifying change in file content. Since
ctime also changes when content of a file changes and it cannot be set
explicitly, it becomes suitable for identifying staleness of cached data.
This option makes quick-read to prefer ctime over mtime to validate its
cache. However, using ctime can result in false positives as ctime changes
with just attribute changes like permission without changes to file data.
So, use this option only when mtime is not reliable.


>
> Option: shard-lru-limit
> Attention: @krutika
> Review: https://review.gluster.org/c/glusterfs/+/20544
>
> Option: shard-deletion-rate
> Attention: @krutika
> Review: https://review.gluster.org/c/glusterfs/+/19970
>
> Please send in the required text ASAP, as we are almost towards the end
> of the release.
>
> Thanks,
> Shyam
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Kinglong Mee]

On 2018/10/11 19:09, Soumya Koduri wrote:
> NFS-Ganesha's md-cache layer already does extensive caching of attributes and 
> ACLs of each file looked upon. Do you see any additional benefit with turning 
> on gluster md-cache as well?  More replies inline..

Yes, I think.

The logical is different between md-cache and ganesha's cache,
Ganesha caches xattr data depends on timeout, if timeout, ganesha get it from 
back-end glusterfs;
Md-cache caches depneds on timeout too, but md-cache can delay the timeout for 
some cases.

> 
> On 10/11/18 7:47 AM, Kinglong Mee wrote:
>> Cc nfs-ganesha,
>>
>> Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
>> ("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs 
>> ACLs
>> ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
>>
>> But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup 
>> requests.
>> So, md-cache caches bad virtual glusterfs ACLs.
>>
>> After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs 
>> client gets many EIO errors.
>>
>> https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305
>>
>> There are two chooses for cache virtual glusterfs ACLs in md-cache,
>> 1. Cache it separately as posix ACLs (a new option maybe 
>> "cache-glusterfs-acl" is added);
>>     And make sure _posix_xattr_get_set fills them when lookup requests.
>>
> 
> I am not sure if posix layer can handle it. Virtual xattrs are in-memory and 
> not stored on disk. They are converted to/from posix-acl in posix-acl xlator. 
> So FWIU, posix-acl xlator should handle setting these attributes as part of 
> LOOKUP response if needed. Same shall apply for any virtual xattr cached in 
> md-cache. Request Poornima to comment.

Posix-acl can hand it correctly now.

> 
> At a time, any gfapi consumer would use either posix-acl or virtual glusterfs 
> ACLs. So having two options to selectively choose which one of them to cache 
> sounds better to me instead of unnecessarily storing two different 
> representations of the same ACL.

Make sense.
I will add another option for virtual glusterfs ACLs in md-cache.

thanks,
Kinglong Mee

> 
> Thanks,
> Soumya
> 
>> 2. Does not cache it, only cache posix ACLs;
>>     If gfapi request it, md-cache lookup according posix ACL at cache,
>>     if exist, make the virtual glusterfs ACL locally and return to gfapi;
>>     otherwise, send the request to glusterfsd.
>>
>> Virtual glusterfs ACLs are another format of posix ACLs, there are larger 
>> than posix ACLs,
>> and always exist no matter the really posix ACL exist or not.
> 
>>
>> So, I'd prefer #2.
>> Any comments are welcome.
>>
>> thanks,
>> Kinglong Mee
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Release 5: GA and what are we waiting on

[Shyam Ranganathan]

On 10/11/2018 11:25 AM, Shyam Ranganathan wrote:
> 1) Fix for libgfapi-python related blocker on Gluster:
> https://bugzilla.redhat.com/show_bug.cgi?id=1630804


@du @krutika, the root cause for the above issue is from the commit,

commit c9bde3021202f1d5c5a2d19ac05a510fc1f788ac
https://review.gluster.org/c/glusterfs/+/20639

performance/readdir-ahead: keep stats of cached dentries in sync with
modifications

I have updated the bug with the required findings, please take a look
and let us know if we can get a fix in time for release-5.

Thanks,
Shyam
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Release 5: GA and what are we waiting on

[Krutika Dhananjay]

On Thu, Oct 11, 2018 at 8:55 PM Shyam Ranganathan 
wrote:

> So we are through with a series of checks and tasks on release-5 (like
> ensuring all backports to other branches are present in 5, upgrade
> testing, basic performance testing, Package testing, etc.), but still
> need the following resolved else we stand to delay the release GA
> tagging, which I hope to get done over the weekend or by Monday 15th
> morning (EDT).
>
> 1) Fix for libgfapi-python related blocker on Gluster:
> https://bugzilla.redhat.com/show_bug.cgi?id=1630804
>
> @ppai, who needs to look into this?
>
> 2) Release notes for options added to the code (see:
> https://lists.gluster.org/pipermail/gluster-devel/2018-October/055563.html
> )
>
> @du, @krutika can we get some text for the options referred in the mail
> above?
>
>
Replied here -
https://lists.gluster.org/pipermail/gluster-devel/2018-October/055577.html

-Krutika

3) Python3 testing
> - Heard back from Kotresh on geo-rep passing and saw that we have
> handled cliutils issues
> - Anything more to cover? (@aravinda, @kotresh, @ppai?)
> - We are attempting to get a regression run on a Python3 platform, but
> that maybe a little ways away from the release (see:
> https://bugzilla.redhat.com/show_bug.cgi?id=1638030 )
>
> Request attention to the above, to ensure we are not breaking things
> with the release.
>
> Thanks,
> Shyam
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 5: Missing option documentation (need inputs)

[Krutika Dhananjay]

On Wed, Oct 10, 2018 at 8:30 PM Shyam Ranganathan 
wrote:

> The following options were added post 4.1 and are part of 5.0 as the
> first release for the same. They were added in as part of bugs, and
> hence looking at github issues to track them as enhancements did not
> catch the same.
>
> We need to document it in the release notes (and also the gluster doc.
> site ideally), and hence I would like a some details on what to write
> for the same (or release notes commits) for them.
>
> Option: cluster.daemon-log-level
> Attention: @atin
> Review: https://review.gluster.org/c/glusterfs/+/20442
>
> Option: ctime-invalidation
> Attention: @Du
> Review: https://review.gluster.org/c/glusterfs/+/20286
>
> Option: shard-lru-limit
> Attention: @krutika
> Review: https://review.gluster.org/c/glusterfs/+/20544


I added this option solely to make it easier to hit shard's in-memory lru
limit and enable testing of different cases that arise when the limit is
reached.
For this reason, this option is also marked "NO_DOC" in the code. So we
don't need to document it in the release notes.


>
> Option: shard-deletion-rate
> Attention: @krutika
> Review: https://review.gluster.org/c/glusterfs/+/19970
>
> Please send in the required text ASAP, as we are almost towards the end
>
of the release.
>

This option is used to configure the number of shards to delete in parallel
when the original file is deleted. The default value is 100. But it can
always be increased to delete more shards in parallel for faster freeing up
of space. The upper limit is yet to be fixed.  But use it with caution as a
very large number will cause serious lock contention issues on the bricks
(in locks translator). As an example, in our testing, an upper limit of
125000 was enough to cause timeouts and hangs in the gluster processes due
to lock contention.

-Krutika


> Thanks,
> Shyam
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Release 5: GA and what are we waiting on

[Shyam Ranganathan]

So we are through with a series of checks and tasks on release-5 (like
ensuring all backports to other branches are present in 5, upgrade
testing, basic performance testing, Package testing, etc.), but still
need the following resolved else we stand to delay the release GA
tagging, which I hope to get done over the weekend or by Monday 15th
morning (EDT).

1) Fix for libgfapi-python related blocker on Gluster:
https://bugzilla.redhat.com/show_bug.cgi?id=1630804

@ppai, who needs to look into this?

2) Release notes for options added to the code (see:
https://lists.gluster.org/pipermail/gluster-devel/2018-October/055563.html )

@du, @krutika can we get some text for the options referred in the mail
above?

3) Python3 testing
- Heard back from Kotresh on geo-rep passing and saw that we have
handled cliutils issues
- Anything more to cover? (@aravinda, @kotresh, @ppai?)
- We are attempting to get a regression run on a Python3 platform, but
that maybe a little ways away from the release (see:
https://bugzilla.redhat.com/show_bug.cgi?id=1638030 )

Request attention to the above, to ensure we are not breaking things
with the release.

Thanks,
Shyam
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 5: Missing option documentation (need inputs)

[Niels de Vos]

On Thu, Oct 11, 2018 at 09:00:50AM -0400, Shyam Ranganathan wrote:
> On 10/10/2018 11:20 PM, Atin Mukherjee wrote:
> > 
> > 
> > On Wed, 10 Oct 2018 at 20:30, Shyam Ranganathan  > > wrote:
> > 
> > The following options were added post 4.1 and are part of 5.0 as the
> > first release for the same. They were added in as part of bugs, and
> > hence looking at github issues to track them as enhancements did not
> > catch the same.
> > 
> > We need to document it in the release notes (and also the gluster doc.
> > site ideally), and hence I would like a some details on what to write
> > for the same (or release notes commits) for them.
> > 
> > Option: cluster.daemon-log-level
> > Attention: @atin
> > Review: https://review.gluster.org/c/glusterfs/+/20442
> > 
> > 
> > This option has to be used based on extreme need basis and this is why
> > it has been mentioned as GLOBAL_NO_DOC. So ideally this shouldn't be
> > documented.
> > 
> > Do we still want to capture it in the release notes?
> 
> This is an interesting catch-22, when we want users to use the option
> (say to provide better logs for troubleshooting), we have nothing to
> point to, and it would be instructions (repeated over the course of
> time) over mails.
> 
> I would look at adding this into an options section in the docs, but the
> best I can find in there is
> https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/
> 
> I would say we need to improve the way we deal with options and the
> required submissions around the same.
> 
> Thoughts?

Maybe this should be documented under
https://docs.gluster.org/en/latest/Troubleshooting/ and not the general
"Managing Volumes" part of the docs.

Having it documented *somewhere* is definitely needed. And because it
seems to be related to debugging particular components, the
Troubleshooting section seems appropriate.

Niels
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Release 5: Missing option documentation (need inputs)

[Nithya Balachandran]

On 11 October 2018 at 18:30, Shyam Ranganathan  wrote:

> On 10/10/2018 11:20 PM, Atin Mukherjee wrote:
> >
> >
> > On Wed, 10 Oct 2018 at 20:30, Shyam Ranganathan  > > wrote:
> >
> > The following options were added post 4.1 and are part of 5.0 as the
> > first release for the same. They were added in as part of bugs, and
> > hence looking at github issues to track them as enhancements did not
> > catch the same.
> >
> > We need to document it in the release notes (and also the gluster
> doc.
> > site ideally), and hence I would like a some details on what to write
> > for the same (or release notes commits) for them.
> >
> > Option: cluster.daemon-log-level
> > Attention: @atin
> > Review: https://review.gluster.org/c/glusterfs/+/20442
> >
> >
> > This option has to be used based on extreme need basis and this is why
> > it has been mentioned as GLOBAL_NO_DOC. So ideally this shouldn't be
> > documented.
> >
> > Do we still want to capture it in the release notes?
>
> This is an interesting catch-22, when we want users to use the option
> (say to provide better logs for troubleshooting), we have nothing to
> point to, and it would be instructions (repeated over the course of
> time) over mails.
>
> I would look at adding this into an options section in the docs, but the
> best I can find in there is
> https://docs.gluster.org/en/latest/Administrator%20Guide/
> Managing%20Volumes/
>
> I would say we need to improve the way we deal with options and the
> required submissions around the same.
>
> No argument there. I will take a look and get back on what we can improve
in the docs.



> Thoughts?
>
> >
> > 
> >
> > Option: ctime-invalidation
> > Attention: @Du
> > Review: https://review.gluster.org/c/glusterfs/+/20286
> >
> > Option: shard-lru-limit
> > Attention: @krutika
> > Review: https://review.gluster.org/c/glusterfs/+/20544
> >
> > Option: shard-deletion-rate
> > Attention: @krutika
> > Review: https://review.gluster.org/c/glusterfs/+/19970
> >
> > Please send in the required text ASAP, as we are almost towards the
> end
> > of the release.
> >
> > Thanks,
> > Shyam
> >
> ___
> maintainers mailing list
> maintain...@gluster.org
> https://lists.gluster.org/mailman/listinfo/maintainers
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] when there are dangling entry(without gfid) in only one brick dir, the glusterfs heal info will keep showing the entry, glustershd can not really remove this entry from brick .

[Zhou, Cynthia (NSB - CN/Hangzhou)]

Hi glusterfs expert,
I meet one problem in my test bed (3 brick on 3 sn nodes), the "/" is always in 
glusterfs v heal info output. In my ftest+reboot-sn-nodes-randomly test, the 
gluster v heal info output keeps showing entry "/" even for hours, and even you 
do some touch or ls of /mnt/mstate , it will not help to solve this issue.

[root@sn-0:/mnt/bricks/mstate/brick]
# gluster v heal mstate info
Brick sn-0.local:/mnt/bricks/mstate/brick
/
Status: Connected
Number of entries: 1

Brick sn-2.local:/mnt/bricks/mstate/brick
/
Status: Connected
Number of entries: 1

Brick sn-1.local:/mnt/bricks/mstate/brick
/
Status: Connected
Number of entries: 1


>From sn glustershd.log I find following prints
[2018-10-10 08:13:00.005250] I [MSGID: 108026] 
[afr-self-heald.c:432:afr_shd_index_heal] 0-mstate-replicate-0: got entry: 
----0001 from mstate-client-0
[2018-10-10 08:13:00.006077] I [MSGID: 108026] 
[afr-self-heald.c:341:afr_shd_selfheal] 0-mstate-replicate-0: entry: path /, 
gfid: ----0001
[2018-10-10 08:13:00.011599] I [MSGID: 108026] 
[afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-mstate-replicate-0: 
performing entry selfheal on ----0001
[2018-10-10 08:16:28.722059] W [MSGID: 108015] 
[afr-self-heal-entry.c:47:afr_selfheal_entry_delete] 0-mstate-replicate-0: 
expunging dir 
----0001/fstest_76f272545249be5d71359f06962e069b 
(----) on mstate-client-0
[2018-10-10 08:16:28.722975] W [MSGID: 114031] 
[client-rpc-fops.c:670:client3_3_rmdir_cbk] 0-mstate-client-0: remote operation 
failed [No such file or directory]


When I check the env I find that fstest_76f272545249be5d71359f06962e069b only 
exists on sn-0 node brick only and the getfattr of this is empty!

[root@sn-0:/mnt/bricks/mstate/brick]
# getfattr -m . -d -e hex fstest_76f272545249be5d71359f06962e069b//return 
is empty output
[root@sn-0:/mnt/bricks/mstate/brick]
# getfattr -m . -d -e hex .
# file: .
trusted.afr.dirty=0x
trusted.afr.mstate-client-1=0x
trusted.afr.mstate-client-2=0x02a7
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0xbf7aad0e4ce44196aa9b0a33928ea2ff

[root@sn-1:/root]
# stat /mnt/bricks/mstate/brick/fstest_76f272545249be5d71359f06962e069b
stat: cannot stat 
'/mnt/bricks/mstate/brick/fstest_76f272545249be5d71359f06962e069b': No such 
file or directory
[root@sn-1:/root]
[root@sn-1:/mnt/bricks/mstate/brick]
# getfattr -m . -d -e hex .
# file: .
trusted.afr.dirty=0x
trusted.afr.mstate-client-0=0x0006
trusted.afr.mstate-client-1=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0xbf7aad0e4ce44196aa9b0a33928ea2ff

[root@sn-2:/mnt/bricks/mstate/brick]
# stat /mnt/bricks/mstate/brick/fstest_76f272545249be5d71359f06962e069b
stat: cannot stat 
'/mnt/bricks/mstate/brick/fstest_76f272545249be5d71359f06962e069b': No such 
file or directory
[root@sn-2:/mnt/bricks/mstate/brick]

[root@sn-2:/mnt/bricks/mstate/brick]
# getfattr -m . -d -e hex .
# file: .
trusted.afr.dirty=0x
trusted.afr.mstate-client-0=0x0006
trusted.afr.mstate-client-2=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0xbf7aad0e4ce44196aa9b0a33928ea2ff


I think the entry fstest_76f272545249be5d71359f06962e069b should either be 
assigned gfid or be removed, from the glustershd.log it shows clearly that 
glustershd on sn-0 try to remove this dangling entry but meet some error. And 
when I do some gdb I find that in this case the entry is not assigned to gfid, 
because the __afr_selfheal_heal_dirent input param source is 1, so  
replies[source].op_ret == -1, and can not assign gfid to it. my question is in 
this case there is no fstest_76f272545249be5d71359f06962e069b on sn-1 and sn-2, 
so if want to remove this dangling entry on sn-0 can not use syncop_rmdir, I 
would like your opinion on this issue, thanks!

Thread 12 "glustershdheal" hit Breakpoint 1, __afr_selfheal_heal_dirent 
(frame=0x7f5aec009350, this=0x7f5b1001d8d0, fd=0x7f5b0800c8a0,
name=0x7f5b08059db0 "fstest_76f272545249be5d71359f06962e069b", 
inode=0x7f5aec001e80, source=1, sources=0x7f5af8fefb20 "",
healed_sinks=0x7f5af8fefae0 "\001", locked_on=0x7f5af8fefac0 
"\001\001\001\370Z\177", replies=0x7f5af8fef190) at afr-self-heal-entry.c:172
172 afr-self-heal-entry.c: No such file or directory.
(gdb) print name
$17 = 0x7f5b08059db0 "fstest_76f272545249be5d71359f06962e069b"
(gdb) print source
$18 = 1
(gdb) print replies[0].op_ret
$19 = 0
(gdb) print replies[1].op_ret
$20 = -1
(gdb) print replies[2].op_ret
$21 = -1
(gdb) 

Re: [Gluster-devel] [Gluster-Maintainers] Release 5: Missing option documentation (need inputs)

[Shyam Ranganathan]

On 10/10/2018 11:20 PM, Atin Mukherjee wrote:
> 
> 
> On Wed, 10 Oct 2018 at 20:30, Shyam Ranganathan  > wrote:
> 
> The following options were added post 4.1 and are part of 5.0 as the
> first release for the same. They were added in as part of bugs, and
> hence looking at github issues to track them as enhancements did not
> catch the same.
> 
> We need to document it in the release notes (and also the gluster doc.
> site ideally), and hence I would like a some details on what to write
> for the same (or release notes commits) for them.
> 
> Option: cluster.daemon-log-level
> Attention: @atin
> Review: https://review.gluster.org/c/glusterfs/+/20442
> 
> 
> This option has to be used based on extreme need basis and this is why
> it has been mentioned as GLOBAL_NO_DOC. So ideally this shouldn't be
> documented.
> 
> Do we still want to capture it in the release notes?

This is an interesting catch-22, when we want users to use the option
(say to provide better logs for troubleshooting), we have nothing to
point to, and it would be instructions (repeated over the course of
time) over mails.

I would look at adding this into an options section in the docs, but the
best I can find in there is
https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/

I would say we need to improve the way we deal with options and the
required submissions around the same.

Thoughts?

> 
> 
> 
> Option: ctime-invalidation
> Attention: @Du
> Review: https://review.gluster.org/c/glusterfs/+/20286
> 
> Option: shard-lru-limit
> Attention: @krutika
> Review: https://review.gluster.org/c/glusterfs/+/20544
> 
> Option: shard-deletion-rate
> Attention: @krutika
> Review: https://review.gluster.org/c/glusterfs/+/19970
> 
> Please send in the required text ASAP, as we are almost towards the end
> of the release.
> 
> Thanks,
> Shyam
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Soumya Koduri]

NFS-Ganesha's md-cache layer already does extensive caching of 
attributes and ACLs of each file looked upon. Do you see any additional 
benefit with turning on gluster md-cache as well?  More replies inline..


On 10/11/18 7:47 AM, Kinglong Mee wrote:

Cc nfs-ganesha,

Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
("system.posix_acl_access"/"system.posix_acl_default") and virtual glusterfs 
ACLs
("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.

But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when lookup 
requests.
So, md-cache caches bad virtual glusterfs ACLs.

After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs client 
gets many EIO errors.

https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305

There are two chooses for cache virtual glusterfs ACLs in md-cache,
1. Cache it separately as posix ACLs (a new option maybe "cache-glusterfs-acl" 
is added);
And make sure _posix_xattr_get_set fills them when lookup requests.



I am not sure if posix layer can handle it. Virtual xattrs are in-memory 
and not stored on disk. They are converted to/from posix-acl in 
posix-acl xlator. So FWIU, posix-acl xlator should handle setting these 
attributes as part of LOOKUP response if needed. Same shall apply for 
any virtual xattr cached in md-cache. Request Poornima to comment.


At a time, any gfapi consumer would use either posix-acl or virtual 
glusterfs ACLs. So having two options to selectively choose which one of 
them to cache sounds better to me instead of unnecessarily storing two 
different representations of the same ACL.


Thanks,
Soumya


2. Does not cache it, only cache posix ACLs;
If gfapi request it, md-cache lookup according posix ACL at cache,
if exist, make the virtual glusterfs ACL locally and return to gfapi;
otherwise, send the request to glusterfsd.

Virtual glusterfs ACLs are another format of posix ACLs, there are larger than 
posix ACLs,
and always exist no matter the really posix ACL exist or not.




So, I'd prefer #2.
Any comments are welcome.

thanks,
Kinglong Mee

___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel


___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Problems about cache virtual glusterfs ACLs for ganesha in md-cache

[Kinglong Mee]

On 2018/10/11 12:10, Jiffin Tony Thottan wrote:> On Thursday 11 October 2018 
08:10 AM, Raghavendra Gowdappa wrote:
>> On Thu, Oct 11, 2018 at 7:47 AM Kinglong Mee > > wrote:
>>
>> Cc nfs-ganesha,
>>
>> Md-cache has option "cache-posix-acl" that controls caching of posix ACLs
>> ("system.posix_acl_access"/"system.posix_acl_default") and virtual 
>> glusterfs ACLs
>> ("glusterfs.posix.acl"/"glusterfs.posix.default_acl") now.
>>
>>
>> Not sure how virtual xattrs are consumed or who generates them. +Raghavendra 
>> Talur  +Thottan, Jiffin 
>>  - acl maintainers.
> 
> The currently only consumers of this virtual xattr is nfs-ganesha. Nfsv4 acls 
> were sent from client and ganesha converts to posix acl
> 
> and sent as virtual xattr to glusterfs bricks using pub_glfs_h_acl_set/get 
> api's. AFAIR  in samba vfs module they convert windows acl to
> 
> posix acl and sent as actual getxattr/setxattr calls on "system.posixl-acl"
> 
> --
> 
> Jiffin
> 
>>
>>
>> But, _posix_xattr_get_set does not fill virtual glusterfs ACLs when 
>> lookup requests.
>> So, md-cache caches bad virtual glusterfs ACLs.
>>
>> After I turn on "cache-posix-acl" option to cache ACLs at md-cache, nfs 
>> client gets many EIO errors.
>>
>> https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/427305

Hi Jiffin,

Thanks for your explain.

I'm testing the cache of virtual glusterfs ACLs in md-cache for ganesha's nfsv4.
I wanna to know the plan of ACLs in the future.
Are both posix ACLs (used by samba) and virtual glusterfs ACLs (used by 
ganesha's nfsv4) reserved?
Or Is only virtual glusterfs ACLs reserved?

If both are reserved, there are two chooses for md-cache caches virtual 
glusterfs ACLs.
>> There are two chooses for cache virtual glusterfs ACLs in md-cache,
>> 1. Cache it separately as posix ACLs (a new option maybe 
>> "cache-glusterfs-acl" is added);
>>    And make sure _posix_xattr_get_set fills them when lookup requests.
>>
>> 2. Does not cache it, only cache posix ACLs;
>>    If gfapi request it, md-cache lookup according posix ACL at cache,
>>    if exist, make the virtual glusterfs ACL locally and return to gfapi;
>>    otherwise, send the request to glusterfsd.
>>
>> Virtual glusterfs ACLs are another format of posix ACLs, there are 
>> larger than posix ACLs,
>> and always exist no matter the really posix ACL exist or not.
>>
>> So, I'd prefer #2.

If only virtual glusterfs ACLs is reserved, I'd like chose the #1 implement for 
md-cache.
Md-cache reserves caching posix acl until samba moving to the virtual glusterfs 
ACLs.

thanks,
Kinglong Mee
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel