Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Prashanth Pai 
readdirp needs to be explicitly disabled at 3 places if I remember correctly.
At fuse mount option, at dht and at md-cache level.

 -Prashanth Pai

- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Raghavendra Gowdappa" 
> Cc: "Gluster Devel" 
> Sent: Friday, 30 September, 2016 9:53:44 AM
> Subject: Re: [Gluster-devel] Dht readdir filtering out names
> 
> 
> 
> On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa < rgowd...@redhat.com >
> wrote:
> 
> 
> 
> 
> - Original Message -
> > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > To: "Raghavendra Gowdappa" < rgowd...@redhat.com >
> > Cc: "Shyam Ranganathan" < srang...@redhat.com >, "Nithya Balachandran" <
> > nbala...@redhat.com >, "Gluster Devel"
> > < gluster-devel@gluster.org >
> > Sent: Friday, September 30, 2016 9:15:04 AM
> > Subject: Re: Dht readdir filtering out names
> > 
> > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa < rgowd...@redhat.com
> > >
> > wrote:
> > 
> > > dht_readdirp_cbk has different behaviour for directories and files.
> > > 
> > > 1. If file, pick the dentry (passed from subvols as part of readdirp
> > > response) if the it corresponds to data file.
> > > 2. If directory pick the dentry if readdirp response is from
> > > hashed-subvol.
> > > 
> > > In all other cases, the dentry is skipped and not passed to higher
> > > layers/application. To elaborate, the dentries which are ignored are:
> > > 1. dentries corresponding to linkto files.
> > > 2. dentries from non-hashed subvols corresponding to directories.
> > > 
> > > Since the behaviour is different for different filesystem objects, dht
> > > needs ia_type to choose its behaviour.
> > > 
> > > - Original Message -
> > > > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > > > To: "Shyam Ranganathan" < srang...@redhat.com >, "Raghavendra Gowdappa"
> > > > <
> > > rgowd...@redhat.com >, "Nithya Balachandran"
> > > > < nbala...@redhat.com >
> > > > Cc: "Gluster Devel" < gluster-devel@gluster.org >
> > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > > Subject: Dht readdir filtering out names
> > > > 
> > > > hi,
> > > > In dht_readdirp_cbk() there is a check about skipping files
> > > without
> > > > ia_type. Could you help me understand why this check is added? There
> > > > are
> > > > times when users have to delete gfid of the entries and trigger
> > > > something
> > > > like 'find . | xargs stat' to heal the gfids. This case would fail if
> > > > we
> > > > skip entries without gfid, if the lower xlators don't send stat
> > > information
> > > > for them.
> > > 
> > > Probably we can make readdirp_cbk not rely on ia_type and pass _all_
> > > dentries received by subvols to application without filtering. However we
> > > should make this behaviour optional and use this only for recovery
> > > setups.
> > > If we don't rely on ia_type (during non error scenarios), applications
> > > end
> > > up seeing duplicate dentries in readdir listing.
> > > 
> > 
> > That means dht_readdir() gives duplicate entries? As per the code it seems
> > like it...
> 
> No. It follows the filtering logic of "pick dentry only from hashed subvol".
> This logic doesn't need ia_type. Now, that you brought the topic of
> dht_readdir, I've another solution for your use case (Basically don't use
> readdirp :) ):
> 
> 1. mount glusterfs with "--use-readdirp=no" option.
> 2. disable md-cache/stat-prefetch as it converts all readdir calls into
> readdirp calls
> 
> Probably the ones in dht as well? i.e. use-readdirp option.
> 
> 
> 
> Use this only for recovery setups :).
> 
> > 
> > 
> > > 
> > > > 
> > > > --
> > > > Pranith
> > > > 
> > > 
> > > regards,
> > > Raghavendra
> > > 
> > 
> > 
> > 
> > --
> > Pranith
> > 
> 
> 
> 
> --
> Pranith
> 
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Pranith Kumar Karampuri 
What if the lower xlators want to set the entry->inode to NULL and clear
the entry->d_stat to force a lookup on the name? i.e.
gfid-split-brain/ia_type mismatches.

On Fri, Sep 30, 2016 at 10:00 AM, Raghavendra Gowdappa 
wrote:

>
>
> - Original Message -
> > From: "Raghavendra Gowdappa" 
> > To: "Pranith Kumar Karampuri" 
> > Cc: "Shyam Ranganathan" , "Nithya Balachandran" <
> nbala...@redhat.com>, "Gluster Devel"
> > 
> > Sent: Friday, September 30, 2016 9:58:34 AM
> > Subject: Re: Dht readdir filtering out names
> >
> >
> >
> > - Original Message -
> > > From: "Pranith Kumar Karampuri" 
> > > To: "Raghavendra Gowdappa" 
> > > Cc: "Shyam Ranganathan" , "Nithya Balachandran"
> > > , "Gluster Devel"
> > > 
> > > Sent: Friday, September 30, 2016 9:53:44 AM
> > > Subject: Re: Dht readdir filtering out names
> > >
> > > On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa <
> rgowd...@redhat.com>
> > > wrote:
> > >
> > > >
> > > >
> > > > - Original Message -
> > > > > From: "Pranith Kumar Karampuri" 
> > > > > To: "Raghavendra Gowdappa" 
> > > > > Cc: "Shyam Ranganathan" , "Nithya
> Balachandran" <
> > > > nbala...@redhat.com>, "Gluster Devel"
> > > > > 
> > > > > Sent: Friday, September 30, 2016 9:15:04 AM
> > > > > Subject: Re: Dht readdir filtering out names
> > > > >
> > > > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> > > > rgowd...@redhat.com>
> > > > > wrote:
> > > > >
> > > > > > dht_readdirp_cbk has different behaviour for directories and
> files.
> > > > > >
> > > > > > 1. If file, pick the dentry (passed from subvols as part of
> readdirp
> > > > > > response) if the it corresponds to data file.
> > > > > > 2. If directory pick the dentry if readdirp response is from
> > > > hashed-subvol.
> > > > > >
> > > > > > In all other cases, the dentry is skipped and not passed to
> higher
> > > > > > layers/application. To elaborate, the dentries which are ignored
> are:
> > > > > > 1. dentries corresponding to linkto files.
> > > > > > 2. dentries from non-hashed subvols corresponding to directories.
> > > > > >
> > > > > > Since the behaviour is different for different filesystem
> objects,
> > > > > > dht
> > > > > > needs ia_type to choose its behaviour.
> > > > > >
> > > > > > - Original Message -
> > > > > > > From: "Pranith Kumar Karampuri" 
> > > > > > > To: "Shyam Ranganathan" , "Raghavendra
> > > > Gowdappa" <
> > > > > > rgowd...@redhat.com>, "Nithya Balachandran"
> > > > > > > 
> > > > > > > Cc: "Gluster Devel" 
> > > > > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > > > > > Subject: Dht readdir filtering out names
> > > > > > >
> > > > > > > hi,
> > > > > > >In dht_readdirp_cbk() there is a check about skipping
> files
> > > > > > without
> > > > > > > ia_type. Could you help me understand why this check is added?
> > > > > > > There
> > > > are
> > > > > > > times when users have to delete gfid of the entries and trigger
> > > > something
> > > > > > > like 'find . | xargs stat' to heal the gfids. This case would
> fail
> > > > if we
> > > > > > > skip entries without gfid, if the lower xlators don't send stat
> > > > > > information
> > > > > > > for them.
> > > > > >
> > > > > > Probably we can make readdirp_cbk not rely on ia_type and pass
> _all_
> > > > > > dentries received by subvols to application without filtering.
> > > > > > However
> > > > we
> > > > > > should make this behaviour optional and use this only for
> recovery
> > > > setups.
> > > > > > If we don't rely on ia_type (during non error scenarios),
> > > > > > applications
> > > > end
> > > > > > up seeing duplicate dentries in readdir listing.
> > > > > >
> > > > >
> > > > > That means dht_readdir() gives duplicate entries? As per the code
> it
> > > > seems
> > > > > like it...
> > > >
> > > > No. It follows the filtering logic of "pick dentry only from hashed
> > > > subvol". This logic doesn't need ia_type. Now, that you brought the
> topic
> > > > of dht_readdir, I've another solution for your use case (Basically
> don't
> > > > use readdirp :) ):
> > > >
> > > > 1. mount glusterfs with "--use-readdirp=no" option.
> > > > 2. disable md-cache/stat-prefetch as it converts all readdir calls
> into
> > > > readdirp calls
> > > >
> > >
> > > Probably the ones in dht as well? i.e. use-readdirp option.
> >
> > No. dht doesn't convert a readdir into readdirp. The option you are
> referring
> > to might be "readdir-optimize" which is something different.
>
> Sorry. I was wrong. There is an option in dht too, to force using
> readdirp. As you said, we should disable that too.
>
> >
> > >
> > >
> > > >
> > > > Use this only for recovery setups :).
> > > >
> > > > >
> > > > >
> > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Pranith
> > > > > > >
> > > > > >
> > > > > > regards,
> > > > > > Raghavendra
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Pranith
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Pranith
> > >
> >
>



-- 
Pranith

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Raghavendra Gowdappa 


- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Raghavendra Gowdappa" 
> Cc: "Shyam Ranganathan" , "Nithya Balachandran" 
> , "Gluster Devel"
> 
> Sent: Friday, September 30, 2016 9:15:04 AM
> Subject: Re: Dht readdir filtering out names
> 
> On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa 
> wrote:
> 
> > dht_readdirp_cbk has different behaviour for directories and files.
> >
> > 1. If file, pick the dentry (passed from subvols as part of readdirp
> > response) if the it corresponds to data file.
> > 2. If directory pick the dentry if readdirp response is from hashed-subvol.
> >
> > In all other cases, the dentry is skipped and not passed to higher
> > layers/application. To elaborate, the dentries which are ignored are:
> > 1. dentries corresponding to linkto files.
> > 2. dentries from non-hashed subvols corresponding to directories.
> >
> > Since the behaviour is different for different filesystem objects, dht
> > needs ia_type to choose its behaviour.
> >
> > - Original Message -
> > > From: "Pranith Kumar Karampuri" 
> > > To: "Shyam Ranganathan" , "Raghavendra Gowdappa" <
> > rgowd...@redhat.com>, "Nithya Balachandran"
> > > 
> > > Cc: "Gluster Devel" 
> > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > Subject: Dht readdir filtering out names
> > >
> > > hi,
> > >In dht_readdirp_cbk() there is a check about skipping files
> > without
> > > ia_type. Could you help me understand why this check is added? There are
> > > times when users have to delete gfid of the entries and trigger something
> > > like 'find . | xargs stat' to heal the gfids. This case would fail if we
> > > skip entries without gfid, if the lower xlators don't send stat
> > information
> > > for them.
> >
> > Probably we can make readdirp_cbk not rely on ia_type and pass _all_
> > dentries received by subvols to application without filtering. However we
> > should make this behaviour optional and use this only for recovery setups.
> > If we don't rely on ia_type (during non error scenarios), applications end
> > up seeing duplicate dentries in readdir listing.
> >
> 
> That means dht_readdir() gives duplicate entries? As per the code it seems
> like it...

No. It follows the filtering logic of "pick dentry only from hashed subvol". 
This logic doesn't need ia_type. Now, that you brought the topic of 
dht_readdir, I've another solution for your use case (Basically don't use 
readdirp :) ):

1. mount glusterfs with "--use-readdirp=no" option.
2. disable md-cache/stat-prefetch as it converts all readdir calls into 
readdirp calls

Use this only for recovery setups :).

> 
> 
> >
> > >
> > > --
> > > Pranith
> > >
> >
> > regards,
> > Raghavendra
> >
> 
> 
> 
> --
> Pranith
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Poornima Gurusiddaiah 
In gfapi, we pass down readdirp, irrespective of whether the application called 
readdir/readdirp. 
Hence the behaviour will be same for samba and Ganesha i suppose. 

Regards, 
Poornima 

- Original Message -

> From: "Pranith Kumar Karampuri" 
> To: "Raghavendra Gowdappa" , "Poornima Gurusiddaiah"
> , "Raghavendra Talur" , "Soumya
> Koduri" 
> Cc: "Shyam Ranganathan" , "Nithya Balachandran"
> , "Gluster Devel" 
> Sent: Friday, September 30, 2016 12:38:06 AM
> Subject: Re: Dht readdir filtering out names

> Does samba/gfapi/nfs-ganesha have options to disable readdirp?

> On Fri, Sep 30, 2016 at 10:04 AM, Pranith Kumar Karampuri <
> pkara...@redhat.com > wrote:

> > What if the lower xlators want to set the entry->inode to NULL and clear
> > the
> > entry->d_stat to force a lookup on the name? i.e. gfid-split-brain/ia_type
> > mismatches.
> 

> > On Fri, Sep 30, 2016 at 10:00 AM, Raghavendra Gowdappa <
> > rgowd...@redhat.com
> > > wrote:
> 

> > > - Original Message -
> > 
> 
> > > > From: "Raghavendra Gowdappa" < rgowd...@redhat.com >
> > 
> 
> > > > To: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > 
> 
> > > > Cc: "Shyam Ranganathan" < srang...@redhat.com >, "Nithya Balachandran"
> > > > <
> > > > nbala...@redhat.com >, "Gluster Devel"
> > 
> 
> > > > < gluster-devel@gluster.org >
> > 
> 
> > > > Sent: Friday, September 30, 2016 9:58:34 AM
> > 
> 
> > > > Subject: Re: Dht readdir filtering out names
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > >
> > 
> 
> > > > - Original Message -
> > 
> 
> > > > > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > 
> 
> > > > > To: "Raghavendra Gowdappa" < rgowd...@redhat.com >
> > 
> 
> > > > > Cc: "Shyam Ranganathan" < srang...@redhat.com >, "Nithya
> > > > > Balachandran"
> > 
> 
> > > > > < nbala...@redhat.com >, "Gluster Devel"
> > 
> 
> > > > > < gluster-devel@gluster.org >
> > 
> 
> > > > > Sent: Friday, September 30, 2016 9:53:44 AM
> > 
> 
> > > > > Subject: Re: Dht readdir filtering out names
> > 
> 
> > > > >
> > 
> 
> > > > > On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa <
> > > > > rgowd...@redhat.com >
> > 
> 
> > > > > wrote:
> > 
> 
> > > > >
> > 
> 
> > > > > >
> > 
> 
> > > > > >
> > 
> 
> > > > > > - Original Message -
> > 
> 
> > > > > > > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > 
> 
> > > > > > > To: "Raghavendra Gowdappa" < rgowd...@redhat.com >
> > 
> 
> > > > > > > Cc: "Shyam Ranganathan" < srang...@redhat.com >, "Nithya
> > > > > > > Balachandran" <
> > 
> 
> > > > > > nbala...@redhat.com >, "Gluster Devel"
> > 
> 
> > > > > > > < gluster-devel@gluster.org >
> > 
> 
> > > > > > > Sent: Friday, September 30, 2016 9:15:04 AM
> > 
> 
> > > > > > > Subject: Re: Dht readdir filtering out names
> > 
> 
> > > > > > >
> > 
> 
> > > > > > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> > 
> 
> > > > > > rgowd...@redhat.com >
> > 
> 
> > > > > > > wrote:
> > 
> 
> > > > > > >
> > 
> 
> > > > > > > > dht_readdirp_cbk has different behaviour for directories and
> > > > > > > > files.
> > 
> 
> > > > > > > >
> > 
> 
> > > > > > > > 1. If file, pick the dentry (passed from subvols as part of
> > > > > > > > readdirp
> > 
> 
> > > > > > > > response) if the it corresponds to data file.
> > 
> 
> > > > > > > > 2. If directory pick the dentry if readdirp response is from
> > 
> 
> > > > > > hashed-subvol.
> > 
> 
> > > > > > > >
> > 
> 
> > > > > > > > In all other cases, the dentry is skipped and not passed to
> > > > > > > > higher
> > 
> 
> > > > > > > > layers/application. To elaborate, the dentries which are
> > > > > > > > ignored
> > > > > > > > are:
> > 
> 
> > > > > > > > 1. dentries corresponding to linkto files.
> > 
> 
> > > > > > > > 2. dentries from non-hashed subvols corresponding to
> > > > > > > > directories.
> > 
> 
> > > > > > > >
> > 
> 
> > > > > > > > Since the behaviour is different for different filesystem
> > > > > > > > objects,
> > 
> 
> > > > > > > > dht
> > 
> 
> > > > > > > > needs ia_type to choose its behaviour.
> > 
> 
> > > > > > > >
> > 
> 
> > > > > > > > - Original Message -
> > 
> 
> > > > > > > > > From: "Pranith Kumar Karampuri" < pkara...@redhat.com >
> > 
> 
> > > > > > > > > To: "Shyam Ranganathan" < srang...@redhat.com >, "Raghavendra
> > 
> 
> > > > > > Gowdappa" <
> > 
> 
> > > > > > > > rgowd...@redhat.com >, "Nithya Balachandran"
> > 
> 
> > > > > > > > > < nbala...@redhat.com >
> > 
> 
> > > > > > > > > Cc: "Gluster Devel" < gluster-devel@gluster.org >
> > 
> 
> > > > > > > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > 
> 
> > > > > > > > > Subject: Dht readdir filtering out names
> > 
> 
> > > > > > > > >
> > 
> 
> > > > > > > > > hi,
> > 
> 
> > > > > > > > > In dht_readdirp_cbk() there is a check about skipping files
> > 
> 
> > > > > > > > without
> > 
> 
> > > > > > > > > ia_type. Could you help me understand why this check is
> > > > > > > > > added?
> > 
> 
> > > > > > > > > There
> > 
> 
> > > > > > a

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Raghavendra Gowdappa 


- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Raghavendra Gowdappa" 
> Cc: "Shyam Ranganathan" , "Nithya Balachandran" 
> , "Gluster Devel"
> 
> Sent: Friday, September 30, 2016 9:53:44 AM
> Subject: Re: Dht readdir filtering out names
> 
> On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa 
> wrote:
> 
> >
> >
> > - Original Message -
> > > From: "Pranith Kumar Karampuri" 
> > > To: "Raghavendra Gowdappa" 
> > > Cc: "Shyam Ranganathan" , "Nithya Balachandran" <
> > nbala...@redhat.com>, "Gluster Devel"
> > > 
> > > Sent: Friday, September 30, 2016 9:15:04 AM
> > > Subject: Re: Dht readdir filtering out names
> > >
> > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> > rgowd...@redhat.com>
> > > wrote:
> > >
> > > > dht_readdirp_cbk has different behaviour for directories and files.
> > > >
> > > > 1. If file, pick the dentry (passed from subvols as part of readdirp
> > > > response) if the it corresponds to data file.
> > > > 2. If directory pick the dentry if readdirp response is from
> > hashed-subvol.
> > > >
> > > > In all other cases, the dentry is skipped and not passed to higher
> > > > layers/application. To elaborate, the dentries which are ignored are:
> > > > 1. dentries corresponding to linkto files.
> > > > 2. dentries from non-hashed subvols corresponding to directories.
> > > >
> > > > Since the behaviour is different for different filesystem objects, dht
> > > > needs ia_type to choose its behaviour.
> > > >
> > > > - Original Message -
> > > > > From: "Pranith Kumar Karampuri" 
> > > > > To: "Shyam Ranganathan" , "Raghavendra
> > Gowdappa" <
> > > > rgowd...@redhat.com>, "Nithya Balachandran"
> > > > > 
> > > > > Cc: "Gluster Devel" 
> > > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > > > Subject: Dht readdir filtering out names
> > > > >
> > > > > hi,
> > > > >In dht_readdirp_cbk() there is a check about skipping files
> > > > without
> > > > > ia_type. Could you help me understand why this check is added? There
> > are
> > > > > times when users have to delete gfid of the entries and trigger
> > something
> > > > > like 'find . | xargs stat' to heal the gfids. This case would fail
> > if we
> > > > > skip entries without gfid, if the lower xlators don't send stat
> > > > information
> > > > > for them.
> > > >
> > > > Probably we can make readdirp_cbk not rely on ia_type and pass _all_
> > > > dentries received by subvols to application without filtering. However
> > we
> > > > should make this behaviour optional and use this only for recovery
> > setups.
> > > > If we don't rely on ia_type (during non error scenarios), applications
> > end
> > > > up seeing duplicate dentries in readdir listing.
> > > >
> > >
> > > That means dht_readdir() gives duplicate entries? As per the code it
> > seems
> > > like it...
> >
> > No. It follows the filtering logic of "pick dentry only from hashed
> > subvol". This logic doesn't need ia_type. Now, that you brought the topic
> > of dht_readdir, I've another solution for your use case (Basically don't
> > use readdirp :) ):
> >
> > 1. mount glusterfs with "--use-readdirp=no" option.
> > 2. disable md-cache/stat-prefetch as it converts all readdir calls into
> > readdirp calls
> >
> 
> Probably the ones in dht as well? i.e. use-readdirp option.

No. dht doesn't convert a readdir into readdirp. The option you are referring 
to might be "readdir-optimize" which is something different.

> 
> 
> >
> > Use this only for recovery setups :).
> >
> > >
> > >
> > > >
> > > > >
> > > > > --
> > > > > Pranith
> > > > >
> > > >
> > > > regards,
> > > > Raghavendra
> > > >
> > >
> > >
> > >
> > > --
> > > Pranith
> > >
> >
> 
> 
> 
> --
> Pranith
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Pranith Kumar Karampuri 
hi,
   In dht_readdirp_cbk() there is a check about skipping files without
ia_type. Could you help me understand why this check is added? There are
times when users have to delete gfid of the entries and trigger something
like 'find . | xargs stat' to heal the gfids. This case would fail if we
skip entries without gfid, if the lower xlators don't send stat information
for them.

-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Pranith Kumar Karampuri 
On Fri, Sep 30, 2016 at 9:58 AM, Raghavendra Gowdappa 
wrote:

>
>
> - Original Message -
> > From: "Pranith Kumar Karampuri" 
> > To: "Raghavendra Gowdappa" 
> > Cc: "Shyam Ranganathan" , "Nithya Balachandran" <
> nbala...@redhat.com>, "Gluster Devel"
> > 
> > Sent: Friday, September 30, 2016 9:53:44 AM
> > Subject: Re: Dht readdir filtering out names
> >
> > On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa <
> rgowd...@redhat.com>
> > wrote:
> >
> > >
> > >
> > > - Original Message -
> > > > From: "Pranith Kumar Karampuri" 
> > > > To: "Raghavendra Gowdappa" 
> > > > Cc: "Shyam Ranganathan" , "Nithya
> Balachandran" <
> > > nbala...@redhat.com>, "Gluster Devel"
> > > > 
> > > > Sent: Friday, September 30, 2016 9:15:04 AM
> > > > Subject: Re: Dht readdir filtering out names
> > > >
> > > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> > > rgowd...@redhat.com>
> > > > wrote:
> > > >
> > > > > dht_readdirp_cbk has different behaviour for directories and files.
> > > > >
> > > > > 1. If file, pick the dentry (passed from subvols as part of
> readdirp
> > > > > response) if the it corresponds to data file.
> > > > > 2. If directory pick the dentry if readdirp response is from
> > > hashed-subvol.
> > > > >
> > > > > In all other cases, the dentry is skipped and not passed to higher
> > > > > layers/application. To elaborate, the dentries which are ignored
> are:
> > > > > 1. dentries corresponding to linkto files.
> > > > > 2. dentries from non-hashed subvols corresponding to directories.
> > > > >
> > > > > Since the behaviour is different for different filesystem objects,
> dht
> > > > > needs ia_type to choose its behaviour.
> > > > >
> > > > > - Original Message -
> > > > > > From: "Pranith Kumar Karampuri" 
> > > > > > To: "Shyam Ranganathan" , "Raghavendra
> > > Gowdappa" <
> > > > > rgowd...@redhat.com>, "Nithya Balachandran"
> > > > > > 
> > > > > > Cc: "Gluster Devel" 
> > > > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > > > > Subject: Dht readdir filtering out names
> > > > > >
> > > > > > hi,
> > > > > >In dht_readdirp_cbk() there is a check about skipping
> files
> > > > > without
> > > > > > ia_type. Could you help me understand why this check is added?
> There
> > > are
> > > > > > times when users have to delete gfid of the entries and trigger
> > > something
> > > > > > like 'find . | xargs stat' to heal the gfids. This case would
> fail
> > > if we
> > > > > > skip entries without gfid, if the lower xlators don't send stat
> > > > > information
> > > > > > for them.
> > > > >
> > > > > Probably we can make readdirp_cbk not rely on ia_type and pass
> _all_
> > > > > dentries received by subvols to application without filtering.
> However
> > > we
> > > > > should make this behaviour optional and use this only for recovery
> > > setups.
> > > > > If we don't rely on ia_type (during non error scenarios),
> applications
> > > end
> > > > > up seeing duplicate dentries in readdir listing.
> > > > >
> > > >
> > > > That means dht_readdir() gives duplicate entries? As per the code it
> > > seems
> > > > like it...
> > >
> > > No. It follows the filtering logic of "pick dentry only from hashed
> > > subvol". This logic doesn't need ia_type. Now, that you brought the
> topic
> > > of dht_readdir, I've another solution for your use case (Basically
> don't
> > > use readdirp :) ):
> > >
> > > 1. mount glusterfs with "--use-readdirp=no" option.
> > > 2. disable md-cache/stat-prefetch as it converts all readdir calls into
> > > readdirp calls
> > >
> >
> > Probably the ones in dht as well? i.e. use-readdirp option.
>
> No. dht doesn't convert a readdir into readdirp. The option you are
> referring to might be "readdir-optimize" which is something different.
>

It seems to do it.


>
> >
> >
> > >
> > > Use this only for recovery setups :).
> > >
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > --
> > > > > > Pranith
> > > > > >
> > > > >
> > > > > regards,
> > > > > Raghavendra
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Pranith
> > > >
> > >
> >
> >
> >
> > --
> > Pranith
> >
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Raghavendra Gowdappa 
dht_readdirp_cbk has different behaviour for directories and files.

1. If file, pick the dentry (passed from subvols as part of readdirp response) 
if the it corresponds to data file.
2. If directory pick the dentry if readdirp response is from hashed-subvol.

In all other cases, the dentry is skipped and not passed to higher 
layers/application. To elaborate, the dentries which are ignored are:
1. dentries corresponding to linkto files.
2. dentries from non-hashed subvols corresponding to directories.

Since the behaviour is different for different filesystem objects, dht needs 
ia_type to choose its behaviour.

- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Shyam Ranganathan" , "Raghavendra Gowdappa" 
> , "Nithya Balachandran"
> 
> Cc: "Gluster Devel" 
> Sent: Friday, September 30, 2016 8:39:28 AM
> Subject: Dht readdir filtering out names
> 
> hi,
>In dht_readdirp_cbk() there is a check about skipping files without
> ia_type. Could you help me understand why this check is added? There are
> times when users have to delete gfid of the entries and trigger something
> like 'find . | xargs stat' to heal the gfids. This case would fail if we
> skip entries without gfid, if the lower xlators don't send stat information
> for them.

Probably we can make readdirp_cbk not rely on ia_type and pass _all_ dentries 
received by subvols to application without filtering. However we should make 
this behaviour optional and use this only for recovery setups. If we don't rely 
on ia_type (during non error scenarios), applications end up seeing duplicate 
dentries in readdir listing.

> 
> --
> Pranith
> 

regards,
Raghavendra
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Soumya Koduri 



On 09/30/2016 10:08 AM, Pranith Kumar Karampuri wrote:

Does samba/gfapi/nfs-ganesha have options to disable readdirp?


AFAIK, currently there is no option to disable/enable readdirp in gfapi 
& nfs-ganesha (not sure about samba). But looks like nfs-ganesha seem to 
be always using readdir, which I plan to change it to readdirp in the 
near future to check if it improves performance of stat on small-files. 
Could you please summarize the issues with using readdirp?


Thanks,
Soumya



On Fri, Sep 30, 2016 at 10:04 AM, Pranith Kumar Karampuri
mailto:pkara...@redhat.com>> wrote:

What if the lower xlators want to set the entry->inode to NULL and
clear the entry->d_stat to force a lookup on the name? i.e.
gfid-split-brain/ia_type mismatches.

On Fri, Sep 30, 2016 at 10:00 AM, Raghavendra Gowdappa
mailto:rgowd...@redhat.com>> wrote:



- Original Message -
> From: "Raghavendra Gowdappa" mailto:rgowd...@redhat.com>>
> To: "Pranith Kumar Karampuri" mailto:pkara...@redhat.com>>
> Cc: "Shyam Ranganathan" mailto:srang...@redhat.com>>, "Nithya
Balachandran" mailto:nbala...@redhat.com>>, "Gluster Devel"
> mailto:gluster-devel@gluster.org>>
> Sent: Friday, September 30, 2016 9:58:34 AM
> Subject: Re: Dht readdir filtering out names
>
>
>
> - Original Message -
> > From: "Pranith Kumar Karampuri" mailto:pkara...@redhat.com>>
> > To: "Raghavendra Gowdappa" mailto:rgowd...@redhat.com>>
> > Cc: "Shyam Ranganathan" mailto:srang...@redhat.com>>, "Nithya Balachandran"
> > mailto:nbala...@redhat.com>>, "Gluster
Devel"
> > mailto:gluster-devel@gluster.org>>
> > Sent: Friday, September 30, 2016 9:53:44 AM
> > Subject: Re: Dht readdir filtering out names
> >
> > On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa
mailto:rgowd...@redhat.com>>
> > wrote:
> >
> > >
> > >
> > > - Original Message -
> > > > From: "Pranith Kumar Karampuri" mailto:pkara...@redhat.com>>
> > > > To: "Raghavendra Gowdappa" mailto:rgowd...@redhat.com>>
> > > > Cc: "Shyam Ranganathan" mailto:srang...@redhat.com>>, "Nithya Balachandran" <
> > > nbala...@redhat.com >,
"Gluster Devel"
> > > > mailto:gluster-devel@gluster.org>>
> > > > Sent: Friday, September 30, 2016 9:15:04 AM
> > > > Subject: Re: Dht readdir filtering out names
> > > >
> > > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> > > rgowd...@redhat.com >
> > > > wrote:
> > > >
> > > > > dht_readdirp_cbk has different behaviour for
directories and files.
> > > > >
> > > > > 1. If file, pick the dentry (passed from subvols as
part of readdirp
> > > > > response) if the it corresponds to data file.
> > > > > 2. If directory pick the dentry if readdirp response
is from
> > > hashed-subvol.
> > > > >
> > > > > In all other cases, the dentry is skipped and not
passed to higher
> > > > > layers/application. To elaborate, the dentries which
are ignored are:
> > > > > 1. dentries corresponding to linkto files.
> > > > > 2. dentries from non-hashed subvols corresponding to
directories.
> > > > >
> > > > > Since the behaviour is different for different
filesystem objects,
> > > > > dht
> > > > > needs ia_type to choose its behaviour.
> > > > >
> > > > > - Original Message -
> > > > > > From: "Pranith Kumar Karampuri" mailto:pkara...@redhat.com>>
> > > > > > To: "Shyam Ranganathan" mailto:srang...@redhat.com>>, "Raghavendra
> > > Gowdappa" <
> > > > > rgowd...@redhat.com >,
"Nithya Balachandran"
> > > > > > mailto:nbala...@redhat.com>>
> > > > > > Cc: "Gluster Devel" mailto:gluster-devel@gluster.org>>
> > > > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > > > > Subject: Dht readdir filtering out names
> > > > > >
> > > > > > hi,
> > > > > >In dht_readdirp_cbk() there is a check about
skipping files
> > > > > without
> > > > > > ia_type. Could you help me understand why this check
is added?
> > > > > > There
> > > are
> > > > > > times when users have to delete gfid of the entries
and trigger
> > > something
> > > > > > like 'find . | xargs stat' to heal the gfids. This
case would fail
> > > if we
> > > > > > skip entries without gfid, if the lower xlators
don't send stat
> > > > > information
> > > > > > for them.

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Raghavendra Gowdappa 


- Original Message -
> From: "Pranith Kumar Karampuri" 
> To: "Raghavendra Gowdappa" 
> Cc: "Shyam Ranganathan" , "Nithya Balachandran" 
> , "Gluster Devel"
> 
> Sent: Friday, September 30, 2016 10:04:29 AM
> Subject: Re: Dht readdir filtering out names
> 
> What if the lower xlators want to set the entry->inode to NULL and clear
> the entry->d_stat to force a lookup on the name? i.e.
> gfid-split-brain/ia_type mismatches.

Didn't get you. If you are using readdirp, as you pointed out dht_readdirp 
won't work properly if entry->d_stat is zeroed out. So, the entry won't be 
listed and if it is not listed, we cannot force a lookup on that.

> 
> On Fri, Sep 30, 2016 at 10:00 AM, Raghavendra Gowdappa 
> wrote:
> 
> >
> >
> > - Original Message -
> > > From: "Raghavendra Gowdappa" 
> > > To: "Pranith Kumar Karampuri" 
> > > Cc: "Shyam Ranganathan" , "Nithya Balachandran" <
> > nbala...@redhat.com>, "Gluster Devel"
> > > 
> > > Sent: Friday, September 30, 2016 9:58:34 AM
> > > Subject: Re: Dht readdir filtering out names
> > >
> > >
> > >
> > > - Original Message -
> > > > From: "Pranith Kumar Karampuri" 
> > > > To: "Raghavendra Gowdappa" 
> > > > Cc: "Shyam Ranganathan" , "Nithya Balachandran"
> > > > , "Gluster Devel"
> > > > 
> > > > Sent: Friday, September 30, 2016 9:53:44 AM
> > > > Subject: Re: Dht readdir filtering out names
> > > >
> > > > On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa <
> > rgowd...@redhat.com>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > - Original Message -
> > > > > > From: "Pranith Kumar Karampuri" 
> > > > > > To: "Raghavendra Gowdappa" 
> > > > > > Cc: "Shyam Ranganathan" , "Nithya
> > Balachandran" <
> > > > > nbala...@redhat.com>, "Gluster Devel"
> > > > > > 
> > > > > > Sent: Friday, September 30, 2016 9:15:04 AM
> > > > > > Subject: Re: Dht readdir filtering out names
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> > > > > rgowd...@redhat.com>
> > > > > > wrote:
> > > > > >
> > > > > > > dht_readdirp_cbk has different behaviour for directories and
> > files.
> > > > > > >
> > > > > > > 1. If file, pick the dentry (passed from subvols as part of
> > readdirp
> > > > > > > response) if the it corresponds to data file.
> > > > > > > 2. If directory pick the dentry if readdirp response is from
> > > > > hashed-subvol.
> > > > > > >
> > > > > > > In all other cases, the dentry is skipped and not passed to
> > higher
> > > > > > > layers/application. To elaborate, the dentries which are ignored
> > are:
> > > > > > > 1. dentries corresponding to linkto files.
> > > > > > > 2. dentries from non-hashed subvols corresponding to directories.
> > > > > > >
> > > > > > > Since the behaviour is different for different filesystem
> > objects,
> > > > > > > dht
> > > > > > > needs ia_type to choose its behaviour.
> > > > > > >
> > > > > > > - Original Message -
> > > > > > > > From: "Pranith Kumar Karampuri" 
> > > > > > > > To: "Shyam Ranganathan" , "Raghavendra
> > > > > Gowdappa" <
> > > > > > > rgowd...@redhat.com>, "Nithya Balachandran"
> > > > > > > > 
> > > > > > > > Cc: "Gluster Devel" 
> > > > > > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > > > > > > Subject: Dht readdir filtering out names
> > > > > > > >
> > > > > > > > hi,
> > > > > > > >In dht_readdirp_cbk() there is a check about skipping
> > files
> > > > > > > without
> > > > > > > > ia_type. Could you help me understand why this check is added?
> > > > > > > > There
> > > > > are
> > > > > > > > times when users have to delete gfid of the entries and trigger
> > > > > something
> > > > > > > > like 'find . | xargs stat' to heal the gfids. This case would
> > fail
> > > > > if we
> > > > > > > > skip entries without gfid, if the lower xlators don't send stat
> > > > > > > information
> > > > > > > > for them.
> > > > > > >
> > > > > > > Probably we can make readdirp_cbk not rely on ia_type and pass
> > _all_
> > > > > > > dentries received by subvols to application without filtering.
> > > > > > > However
> > > > > we
> > > > > > > should make this behaviour optional and use this only for
> > recovery
> > > > > setups.
> > > > > > > If we don't rely on ia_type (during non error scenarios),
> > > > > > > applications
> > > > > end
> > > > > > > up seeing duplicate dentries in readdir listing.
> > > > > > >
> > > > > >
> > > > > > That means dht_readdir() gives duplicate entries? As per the code
> > it
> > > > > seems
> > > > > > like it...
> > > > >
> > > > > No. It follows the filtering logic of "pick dentry only from hashed
> > > > > subvol". This logic doesn't need ia_type. Now, that you brought the
> > topic
> > > > > of dht_readdir, I've another solution for your use case (Basically
> > don't
> > > > > use readdirp :) ):
> > > > >
> > > > > 1. mount glusterfs with "--use-readdirp=no" option.
> > > > > 2. disable md-cache/stat-prefetch as it converts all readdir calls
> > into
> > > > > readdirp calls
> > > > >
>

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Pranith Kumar Karampuri 
On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa 
wrote:

> dht_readdirp_cbk has different behaviour for directories and files.
>
> 1. If file, pick the dentry (passed from subvols as part of readdirp
> response) if the it corresponds to data file.
> 2. If directory pick the dentry if readdirp response is from hashed-subvol.
>
> In all other cases, the dentry is skipped and not passed to higher
> layers/application. To elaborate, the dentries which are ignored are:
> 1. dentries corresponding to linkto files.
> 2. dentries from non-hashed subvols corresponding to directories.
>
> Since the behaviour is different for different filesystem objects, dht
> needs ia_type to choose its behaviour.
>
> - Original Message -
> > From: "Pranith Kumar Karampuri" 
> > To: "Shyam Ranganathan" , "Raghavendra Gowdappa" <
> rgowd...@redhat.com>, "Nithya Balachandran"
> > 
> > Cc: "Gluster Devel" 
> > Sent: Friday, September 30, 2016 8:39:28 AM
> > Subject: Dht readdir filtering out names
> >
> > hi,
> >In dht_readdirp_cbk() there is a check about skipping files
> without
> > ia_type. Could you help me understand why this check is added? There are
> > times when users have to delete gfid of the entries and trigger something
> > like 'find . | xargs stat' to heal the gfids. This case would fail if we
> > skip entries without gfid, if the lower xlators don't send stat
> information
> > for them.
>
> Probably we can make readdirp_cbk not rely on ia_type and pass _all_
> dentries received by subvols to application without filtering. However we
> should make this behaviour optional and use this only for recovery setups.
> If we don't rely on ia_type (during non error scenarios), applications end
> up seeing duplicate dentries in readdir listing.
>

That means dht_readdir() gives duplicate entries? As per the code it seems
like it...


>
> >
> > --
> > Pranith
> >
>
> regards,
> Raghavendra
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Raghavendra Gowdappa 


- Original Message -
> From: "Raghavendra Gowdappa" 
> To: "Pranith Kumar Karampuri" 
> Cc: "Shyam Ranganathan" , "Nithya Balachandran" 
> , "Gluster Devel"
> 
> Sent: Friday, September 30, 2016 9:58:34 AM
> Subject: Re: Dht readdir filtering out names
> 
> 
> 
> - Original Message -
> > From: "Pranith Kumar Karampuri" 
> > To: "Raghavendra Gowdappa" 
> > Cc: "Shyam Ranganathan" , "Nithya Balachandran"
> > , "Gluster Devel"
> > 
> > Sent: Friday, September 30, 2016 9:53:44 AM
> > Subject: Re: Dht readdir filtering out names
> > 
> > On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa 
> > wrote:
> > 
> > >
> > >
> > > - Original Message -
> > > > From: "Pranith Kumar Karampuri" 
> > > > To: "Raghavendra Gowdappa" 
> > > > Cc: "Shyam Ranganathan" , "Nithya Balachandran" <
> > > nbala...@redhat.com>, "Gluster Devel"
> > > > 
> > > > Sent: Friday, September 30, 2016 9:15:04 AM
> > > > Subject: Re: Dht readdir filtering out names
> > > >
> > > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> > > rgowd...@redhat.com>
> > > > wrote:
> > > >
> > > > > dht_readdirp_cbk has different behaviour for directories and files.
> > > > >
> > > > > 1. If file, pick the dentry (passed from subvols as part of readdirp
> > > > > response) if the it corresponds to data file.
> > > > > 2. If directory pick the dentry if readdirp response is from
> > > hashed-subvol.
> > > > >
> > > > > In all other cases, the dentry is skipped and not passed to higher
> > > > > layers/application. To elaborate, the dentries which are ignored are:
> > > > > 1. dentries corresponding to linkto files.
> > > > > 2. dentries from non-hashed subvols corresponding to directories.
> > > > >
> > > > > Since the behaviour is different for different filesystem objects,
> > > > > dht
> > > > > needs ia_type to choose its behaviour.
> > > > >
> > > > > - Original Message -
> > > > > > From: "Pranith Kumar Karampuri" 
> > > > > > To: "Shyam Ranganathan" , "Raghavendra
> > > Gowdappa" <
> > > > > rgowd...@redhat.com>, "Nithya Balachandran"
> > > > > > 
> > > > > > Cc: "Gluster Devel" 
> > > > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > > > > Subject: Dht readdir filtering out names
> > > > > >
> > > > > > hi,
> > > > > >In dht_readdirp_cbk() there is a check about skipping files
> > > > > without
> > > > > > ia_type. Could you help me understand why this check is added?
> > > > > > There
> > > are
> > > > > > times when users have to delete gfid of the entries and trigger
> > > something
> > > > > > like 'find . | xargs stat' to heal the gfids. This case would fail
> > > if we
> > > > > > skip entries without gfid, if the lower xlators don't send stat
> > > > > information
> > > > > > for them.
> > > > >
> > > > > Probably we can make readdirp_cbk not rely on ia_type and pass _all_
> > > > > dentries received by subvols to application without filtering.
> > > > > However
> > > we
> > > > > should make this behaviour optional and use this only for recovery
> > > setups.
> > > > > If we don't rely on ia_type (during non error scenarios),
> > > > > applications
> > > end
> > > > > up seeing duplicate dentries in readdir listing.
> > > > >
> > > >
> > > > That means dht_readdir() gives duplicate entries? As per the code it
> > > seems
> > > > like it...
> > >
> > > No. It follows the filtering logic of "pick dentry only from hashed
> > > subvol". This logic doesn't need ia_type. Now, that you brought the topic
> > > of dht_readdir, I've another solution for your use case (Basically don't
> > > use readdirp :) ):
> > >
> > > 1. mount glusterfs with "--use-readdirp=no" option.
> > > 2. disable md-cache/stat-prefetch as it converts all readdir calls into
> > > readdirp calls
> > >
> > 
> > Probably the ones in dht as well? i.e. use-readdirp option.
> 
> No. dht doesn't convert a readdir into readdirp. The option you are referring
> to might be "readdir-optimize" which is something different.

Sorry. I was wrong. There is an option in dht too, to force using readdirp. As 
you said, we should disable that too.

> 
> > 
> > 
> > >
> > > Use this only for recovery setups :).
> > >
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > --
> > > > > > Pranith
> > > > > >
> > > > >
> > > > > regards,
> > > > > Raghavendra
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Pranith
> > > >
> > >
> > 
> > 
> > 
> > --
> > Pranith
> > 
> 
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Pranith Kumar Karampuri 
On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa 
wrote:

>
>
> - Original Message -
> > From: "Pranith Kumar Karampuri" 
> > To: "Raghavendra Gowdappa" 
> > Cc: "Shyam Ranganathan" , "Nithya Balachandran" <
> nbala...@redhat.com>, "Gluster Devel"
> > 
> > Sent: Friday, September 30, 2016 9:15:04 AM
> > Subject: Re: Dht readdir filtering out names
> >
> > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> rgowd...@redhat.com>
> > wrote:
> >
> > > dht_readdirp_cbk has different behaviour for directories and files.
> > >
> > > 1. If file, pick the dentry (passed from subvols as part of readdirp
> > > response) if the it corresponds to data file.
> > > 2. If directory pick the dentry if readdirp response is from
> hashed-subvol.
> > >
> > > In all other cases, the dentry is skipped and not passed to higher
> > > layers/application. To elaborate, the dentries which are ignored are:
> > > 1. dentries corresponding to linkto files.
> > > 2. dentries from non-hashed subvols corresponding to directories.
> > >
> > > Since the behaviour is different for different filesystem objects, dht
> > > needs ia_type to choose its behaviour.
> > >
> > > - Original Message -
> > > > From: "Pranith Kumar Karampuri" 
> > > > To: "Shyam Ranganathan" , "Raghavendra
> Gowdappa" <
> > > rgowd...@redhat.com>, "Nithya Balachandran"
> > > > 
> > > > Cc: "Gluster Devel" 
> > > > Sent: Friday, September 30, 2016 8:39:28 AM
> > > > Subject: Dht readdir filtering out names
> > > >
> > > > hi,
> > > >In dht_readdirp_cbk() there is a check about skipping files
> > > without
> > > > ia_type. Could you help me understand why this check is added? There
> are
> > > > times when users have to delete gfid of the entries and trigger
> something
> > > > like 'find . | xargs stat' to heal the gfids. This case would fail
> if we
> > > > skip entries without gfid, if the lower xlators don't send stat
> > > information
> > > > for them.
> > >
> > > Probably we can make readdirp_cbk not rely on ia_type and pass _all_
> > > dentries received by subvols to application without filtering. However
> we
> > > should make this behaviour optional and use this only for recovery
> setups.
> > > If we don't rely on ia_type (during non error scenarios), applications
> end
> > > up seeing duplicate dentries in readdir listing.
> > >
> >
> > That means dht_readdir() gives duplicate entries? As per the code it
> seems
> > like it...
>
> No. It follows the filtering logic of "pick dentry only from hashed
> subvol". This logic doesn't need ia_type. Now, that you brought the topic
> of dht_readdir, I've another solution for your use case (Basically don't
> use readdirp :) ):
>
> 1. mount glusterfs with "--use-readdirp=no" option.
> 2. disable md-cache/stat-prefetch as it converts all readdir calls into
> readdirp calls
>

Probably the ones in dht as well? i.e. use-readdirp option.


>
> Use this only for recovery setups :).
>
> >
> >
> > >
> > > >
> > > > --
> > > > Pranith
> > > >
> > >
> > > regards,
> > > Raghavendra
> > >
> >
> >
> >
> > --
> > Pranith
> >
>



-- 
Pranith
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Soumya Koduri 



On 09/30/2016 03:02 PM, Poornima Gurusiddaiah wrote:

In gfapi, we pass down readdirp, irrespective of whether the application
called readdir/readdirp.
Hence the behaviour will be same for samba and Ganesha i suppose.


But in gfapi, I see clear distinction for readdir and readdirp calls



if (plus)
ret = syncop_readdirp (subvol, fd, 131072, glfd->offset,
   &entries, NULL, NULL);
else
ret = syncop_readdir (subvol, fd, 131072, glfd->offset,
  &entries, NULL, NULL);
DECODE_SYNCOP_ERR (ret);


And nfs-ganesha doesn't set 'plus' boolean atm. So I assume think it 
doesn't get converted to readdirp. Or is it that syncop_readdir is 
converted to readdirp in any of the underlying xlators?


Thanks,
Soumya



Regards,
Poornima



*From: *"Pranith Kumar Karampuri" 
*To: *"Raghavendra Gowdappa" , "Poornima
Gurusiddaiah" , "Raghavendra Talur"
, "Soumya Koduri" 
*Cc: *"Shyam Ranganathan" , "Nithya
Balachandran" , "Gluster Devel"

*Sent: *Friday, September 30, 2016 12:38:06 AM
*Subject: *Re: Dht readdir filtering out names

Does samba/gfapi/nfs-ganesha have options to disable readdirp?

On Fri, Sep 30, 2016 at 10:04 AM, Pranith Kumar Karampuri
mailto:pkara...@redhat.com>> wrote:

What if the lower xlators want to set the entry->inode to NULL
and clear the entry->d_stat to force a lookup on the name? i.e.
gfid-split-brain/ia_type mismatches.

On Fri, Sep 30, 2016 at 10:00 AM, Raghavendra Gowdappa
mailto:rgowd...@redhat.com>> wrote:



- Original Message -
> From: "Raghavendra Gowdappa" mailto:rgowd...@redhat.com>>
> To: "Pranith Kumar Karampuri" mailto:pkara...@redhat.com>>
> Cc: "Shyam Ranganathan" mailto:srang...@redhat.com>>, "Nithya
Balachandran" mailto:nbala...@redhat.com>>, "Gluster Devel"
> mailto:gluster-devel@gluster.org>>
> Sent: Friday, September 30, 2016 9:58:34 AM
> Subject: Re: Dht readdir filtering out names
>
>
>
> - Original Message -
> > From: "Pranith Kumar Karampuri" mailto:pkara...@redhat.com>>
> > To: "Raghavendra Gowdappa" mailto:rgowd...@redhat.com>>
> > Cc: "Shyam Ranganathan" mailto:srang...@redhat.com>>, "Nithya Balachandran"
> > mailto:nbala...@redhat.com>>,
"Gluster Devel"
> > mailto:gluster-devel@gluster.org>>
> > Sent: Friday, September 30, 2016 9:53:44 AM
> > Subject: Re: Dht readdir filtering out names
> >
> > On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa
mailto:rgowd...@redhat.com>>
> > wrote:
> >
> > >
> > >
> > > - Original Message -
> > > > From: "Pranith Kumar Karampuri" mailto:pkara...@redhat.com>>
> > > > To: "Raghavendra Gowdappa" mailto:rgowd...@redhat.com>>
> > > > Cc: "Shyam Ranganathan" mailto:srang...@redhat.com>>, "Nithya Balachandran" <
> > > nbala...@redhat.com >,
"Gluster Devel"
> > > > mailto:gluster-devel@gluster.org>>
> > > > Sent: Friday, September 30, 2016 9:15:04 AM
> > > > Subject: Re: Dht readdir filtering out names
> > > >
> > > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
> > > rgowd...@redhat.com >
> > > > wrote:
> > > >
> > > > > dht_readdirp_cbk has different behaviour for
directories and files.
> > > > >
> > > > > 1. If file, pick the dentry (passed from subvols
as part of readdirp
> > > > > response) if the it corresponds to data file.
> > > > > 2. If directory pick the dentry if readdirp
response is from
> > > hashed-subvol.
> > > > >
> > > > > In all other cases, the dentry is skipped and not
passed to higher
> > > > > layers/application. To elaborate, the dentries
which are ignored are:
> > > > > 1. dentries corresponding to linkto files.
> > > > > 2. dentries from non-hashed subvols corresponding
to directories.
> > > > >
> > > > > Since the behaviour is different for different
filesystem objects,
> > > > > dht
> > > > > needs ia_type to choose its behaviour.
> > > > >
> > > > > - Original Message -
> > > > > > From: "Pranith Kumar Karampuri"
mailto:pkara...@redh

Re: [Gluster-devel] Dht readdir filtering out names

2016-09-30 Thread Pranith Kumar Karampuri 
Does samba/gfapi/nfs-ganesha have options to disable readdirp?

On Fri, Sep 30, 2016 at 10:04 AM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

> What if the lower xlators want to set the entry->inode to NULL and clear
> the entry->d_stat to force a lookup on the name? i.e.
> gfid-split-brain/ia_type mismatches.
>
> On Fri, Sep 30, 2016 at 10:00 AM, Raghavendra Gowdappa <
> rgowd...@redhat.com> wrote:
>
>>
>>
>> - Original Message -
>> > From: "Raghavendra Gowdappa" 
>> > To: "Pranith Kumar Karampuri" 
>> > Cc: "Shyam Ranganathan" , "Nithya Balachandran" <
>> nbala...@redhat.com>, "Gluster Devel"
>> > 
>> > Sent: Friday, September 30, 2016 9:58:34 AM
>> > Subject: Re: Dht readdir filtering out names
>> >
>> >
>> >
>> > - Original Message -
>> > > From: "Pranith Kumar Karampuri" 
>> > > To: "Raghavendra Gowdappa" 
>> > > Cc: "Shyam Ranganathan" , "Nithya Balachandran"
>> > > , "Gluster Devel"
>> > > 
>> > > Sent: Friday, September 30, 2016 9:53:44 AM
>> > > Subject: Re: Dht readdir filtering out names
>> > >
>> > > On Fri, Sep 30, 2016 at 9:50 AM, Raghavendra Gowdappa <
>> rgowd...@redhat.com>
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > - Original Message -
>> > > > > From: "Pranith Kumar Karampuri" 
>> > > > > To: "Raghavendra Gowdappa" 
>> > > > > Cc: "Shyam Ranganathan" , "Nithya
>> Balachandran" <
>> > > > nbala...@redhat.com>, "Gluster Devel"
>> > > > > 
>> > > > > Sent: Friday, September 30, 2016 9:15:04 AM
>> > > > > Subject: Re: Dht readdir filtering out names
>> > > > >
>> > > > > On Fri, Sep 30, 2016 at 9:13 AM, Raghavendra Gowdappa <
>> > > > rgowd...@redhat.com>
>> > > > > wrote:
>> > > > >
>> > > > > > dht_readdirp_cbk has different behaviour for directories and
>> files.
>> > > > > >
>> > > > > > 1. If file, pick the dentry (passed from subvols as part of
>> readdirp
>> > > > > > response) if the it corresponds to data file.
>> > > > > > 2. If directory pick the dentry if readdirp response is from
>> > > > hashed-subvol.
>> > > > > >
>> > > > > > In all other cases, the dentry is skipped and not passed to
>> higher
>> > > > > > layers/application. To elaborate, the dentries which are
>> ignored are:
>> > > > > > 1. dentries corresponding to linkto files.
>> > > > > > 2. dentries from non-hashed subvols corresponding to
>> directories.
>> > > > > >
>> > > > > > Since the behaviour is different for different filesystem
>> objects,
>> > > > > > dht
>> > > > > > needs ia_type to choose its behaviour.
>> > > > > >
>> > > > > > - Original Message -
>> > > > > > > From: "Pranith Kumar Karampuri" 
>> > > > > > > To: "Shyam Ranganathan" , "Raghavendra
>> > > > Gowdappa" <
>> > > > > > rgowd...@redhat.com>, "Nithya Balachandran"
>> > > > > > > 
>> > > > > > > Cc: "Gluster Devel" 
>> > > > > > > Sent: Friday, September 30, 2016 8:39:28 AM
>> > > > > > > Subject: Dht readdir filtering out names
>> > > > > > >
>> > > > > > > hi,
>> > > > > > >In dht_readdirp_cbk() there is a check about skipping
>> files
>> > > > > > without
>> > > > > > > ia_type. Could you help me understand why this check is added?
>> > > > > > > There
>> > > > are
>> > > > > > > times when users have to delete gfid of the entries and
>> trigger
>> > > > something
>> > > > > > > like 'find . | xargs stat' to heal the gfids. This case would
>> fail
>> > > > if we
>> > > > > > > skip entries without gfid, if the lower xlators don't send
>> stat
>> > > > > > information
>> > > > > > > for them.
>> > > > > >
>> > > > > > Probably we can make readdirp_cbk not rely on ia_type and pass
>> _all_
>> > > > > > dentries received by subvols to application without filtering.
>> > > > > > However
>> > > > we
>> > > > > > should make this behaviour optional and use this only for
>> recovery
>> > > > setups.
>> > > > > > If we don't rely on ia_type (during non error scenarios),
>> > > > > > applications
>> > > > end
>> > > > > > up seeing duplicate dentries in readdir listing.
>> > > > > >
>> > > > >
>> > > > > That means dht_readdir() gives duplicate entries? As per the code
>> it
>> > > > seems
>> > > > > like it...
>> > > >
>> > > > No. It follows the filtering logic of "pick dentry only from hashed
>> > > > subvol". This logic doesn't need ia_type. Now, that you brought the
>> topic
>> > > > of dht_readdir, I've another solution for your use case (Basically
>> don't
>> > > > use readdirp :) ):
>> > > >
>> > > > 1. mount glusterfs with "--use-readdirp=no" option.
>> > > > 2. disable md-cache/stat-prefetch as it converts all readdir calls
>> into
>> > > > readdirp calls
>> > > >
>> > >
>> > > Probably the ones in dht as well? i.e. use-readdirp option.
>> >
>> > No. dht doesn't convert a readdir into readdirp. The option you are
>> referring
>> > to might be "readdir-optimize" which is something different.
>>
>> Sorry. I was wrong. There is an option in dht too, to force using
>> readdirp. As you said, we should disable that too.
>>
>> >
>> > >
>> > >
>> > > >
>> > > > Use this only for recovery set