Re: [Gluster-devel] [Gluster-Maintainers] Proposed Protocol changes for 4.0: Need feedback.
On Fri, Aug 11, 2017 at 06:04:36PM +0530, Amar Tumballi wrote: > Hi All, > > Below are the proposed protocol changes (ie, XDR changes on the wire) we > are thinking for Gluster 4.0. One of the things that is missing in this list, is to have dictionary entries have a (key, type, value) format. This makes it possible to have all the values XDR encoded/decoded correctly. When new values get transported over the network, we can force the need for XDR encoding functions for these types. I thought there was a GitHub issue for this already (or maybe a bug?), but I fail to find it. Niels ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Coverity covscan for 2017-08-11-811c7ba4 (master branch)
GlusterFS Coverity covscan results are available from http://download.gluster.org/pub/gluster/glusterfs/static-analysis/master/glusterfs-coverity/2017-08-11-811c7ba4 ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Proposed Protocol changes for 4.0: Need feedback.
Hi All, Below are the proposed protocol changes (ie, XDR changes on the wire) we are thinking for Gluster 4.0. - rchecksum/fsetattr: Add 'gfid' field on wire Basic work already done at https://review.gluster.org/#/c/3956/ . Considering its 5yrs old patch, I refreshed it at https://review.gluster.org/17656 for experimental branch, and it is all working fine. This patch also helps in creating new RPC program etc, so for all other XDR changes, we only need to handle the specific change in the patch. STATUS: GREEN - statx() support https://github.com/gluster/glusterfs/issues/273 talks more on it. We can consider to make XDR changes for sure even if we don't implement the fops in all other xlator IMO, so there won't be clients' compatibility issue. STATUS: RED (As no work has been started yet) - Changes to 'gf_flock' structure on wire More on it @ https://review.gluster.org/#/c/15698. Can be handled without this change by using xdata, but adding this field in the XDR will make it faster, and less error prone. STATUS: YELLOW - Changes in few of the 'fops' to get struct iatt in _cbk This is needed for mainly handling the cases of fail over during rebalance, self-heal etc. Today, because we don't have a protocol support, our cross architecture compatibility is broken, mainly because we send iatt as binary in xdata dict, which is not desirable. STATUS: RED (Red as we need to hear from team on what are the changes needed, and it would be significant change as we may have to change the fops signature itself). - fadvise() As per the email thread http://lists.gluster.org/pipermail/gluster-devel/2017-August/053457.html, if we implement the fop, we would need a new XDR for it. STATUS: RED (It is red as it is still in discovery phase) - Misc We don't have any other proposal for protocol change for now, other than a suggestion from Jeff Darcy about taking out the common flags we use across the board, inside xdata, and make them as 'flags' itself in XDR, for better perf, and manageability. STATUS: RED (Mainly because the work is about discovery and changing the fops signature itself). This is the good time to highlight if you need any further changes in protocol itself, and start towards getting it implemented and tested. I volunteer to review all such patches, and happy to co-ordinate it on experimental branch before sending them as a single patch (or multiple dependent, granular patches) on master when we feel it is ready. Regards, Amar -- Amar Tumballi (amarts) ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] How commonly applications make use of fadvise?
On Fri, Aug 11, 2017 at 05:50:47PM +0530, Ravishankar N wrote: > > > On 08/11/2017 04:51 PM, Niels de Vos wrote: > > On Fri, Aug 11, 2017 at 12:47:47AM -0400, Raghavendra Gowdappa wrote: > > > Hi all, > > > > > > In a conversation between me, Milind and Csaba, Milind pointed out > > > fadvise(2) [1] and its potential benefits to Glusterfs' caching > > > translators like read-ahead etc. After discussing about it, we agreed > > > that our performance translators can leverage the hints to provide > > > better performance. Now the question is how commonly applications > > > actually provide hints? Is it something that is used quite frequently? > > > If yes, we can think of implementing this in glusterfs (probably > > > kernel-fuse too?). If no, there is not much of an advantage in > > > spending our energies here. Your inputs will help us to prioritize > > > this feature. > > If functionality like this is available, we would add support in > > libgfapi.so as well. NFS-Ganesha is prepared for consuming this > > (fsal_obj_ops->io_advise), so applications running on top of NFS will > > benefit. I failed to see if the standard Samba/vfs can use it. > > > > A quick check in QEMU does not suggest it is used by the block drivers. > > > > I don't think Linux/FUSE supports fadvise though. So this is an > > oppertunity for a Gluster developer to get their name in the Linux > > kernel :-) Feature additions like this have been done before by us, and > > we should continue where we can. It is a relatively easy entry for > > contributing to the Linux kernel. > > To me it looks like fadvise (mm/fadvise.c) affects only the linux page cache > behavior and is decoupled from the filesystem itself. What this means for > fuse is that the 'advise' is only to the content that the fuse kernel > module has stored in that machine's page cache. Exposing it as a FOP would > likely involve adding a new fop to struct file_operations that is common > across the entire VFS and likely won't fly with the kernel folks. I could > be wrong in understanding all of this. :-) Thanks for checking! If that is the case, we need a good use-case to add a fadvise function pointer to the file_operations. It is not impossible to convince the Linux VFS developers, but it would not be as trivial as adding it to FUSE only (but that requires the VFS infrastructure to be there). Niels > Regards, > Ravi > > > [1] https://linux.die.net/man/2/fadvise > > As well as local man-pages for fadvise64/posix_fadvise. > > > > Showing that we have support for this, suggests that the filesystem > > becomes more mature and gains advanced features. This should impress > > users and might open up more interest for certain (HPC?) use-cases. > > > > Thanks, > > Niels > > > > > > > regards, > > > Raghavendra > > > ___ > > > Gluster-devel mailing list > > > Gluster-devel@gluster.org > > > http://lists.gluster.org/mailman/listinfo/gluster-devel > > ___ > > Gluster-devel mailing list > > Gluster-devel@gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-devel > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] How commonly applications make use of fadvise?
On 08/11/2017 04:51 PM, Niels de Vos wrote: On Fri, Aug 11, 2017 at 12:47:47AM -0400, Raghavendra Gowdappa wrote: Hi all, In a conversation between me, Milind and Csaba, Milind pointed out fadvise(2) [1] and its potential benefits to Glusterfs' caching translators like read-ahead etc. After discussing about it, we agreed that our performance translators can leverage the hints to provide better performance. Now the question is how commonly applications actually provide hints? Is it something that is used quite frequently? If yes, we can think of implementing this in glusterfs (probably kernel-fuse too?). If no, there is not much of an advantage in spending our energies here. Your inputs will help us to prioritize this feature. If functionality like this is available, we would add support in libgfapi.so as well. NFS-Ganesha is prepared for consuming this (fsal_obj_ops->io_advise), so applications running on top of NFS will benefit. I failed to see if the standard Samba/vfs can use it. A quick check in QEMU does not suggest it is used by the block drivers. I don't think Linux/FUSE supports fadvise though. So this is an oppertunity for a Gluster developer to get their name in the Linux kernel :-) Feature additions like this have been done before by us, and we should continue where we can. It is a relatively easy entry for contributing to the Linux kernel. To me it looks like fadvise (mm/fadvise.c) affects only the linux page cache behavior and is decoupled from the filesystem itself. What this means for fuse is that the 'advise' is only to the content that the fuse kernel module has stored in that machine's page cache. Exposing it as a FOP would likely involve adding a new fop to struct file_operations that is common across the entire VFS and likely won't fly with the kernel folks. I could be wrong in understanding all of this. :-) Regards, Ravi [1] https://linux.die.net/man/2/fadvise As well as local man-pages for fadvise64/posix_fadvise. Showing that we have support for this, suggests that the filesystem becomes more mature and gains advanced features. This should impress users and might open up more interest for certain (HPC?) use-cases. Thanks, Niels regards, Raghavendra ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] How commonly applications make use of fadvise?
Hi, Both swift[1] and gluster-swift[2] calls posix_fadvise64() with POSIX_FADV_DONTNEED right after calling fdatasync() or fsync(). This is only performed during object PUT operation (writing an object to disk) suggesting the kernel that the data written is no longer needed with the hopes that cached pages that are more useful are not discarded instead. [1]: https://github.com/openstack/swift/blob/2105ad2a7f6e56cabea0e0b71c5cae94088fd597/swift/common/utils.py#L833 [2]: https://github.com/gluster/gluster-swift/blob/master/gluster/swift/obj/diskfile.py#L300 On Fri, Aug 11, 2017 at 10:17 AM, Raghavendra Gowdappawrote: > Hi all, > > In a conversation between me, Milind and Csaba, Milind pointed out > fadvise(2) [1] and its potential benefits to Glusterfs' caching translators > like read-ahead etc. After discussing about it, we agreed that our > performance translators can leverage the hints to provide better > performance. Now the question is how commonly applications actually provide > hints? Is it something that is used quite frequently? If yes, we can think > of implementing this in glusterfs (probably kernel-fuse too?). If no, there > is not much of an advantage in spending our energies here. Your inputs will > help us to prioritize this feature. > > [1] https://linux.die.net/man/2/fadvise > > regards, > Raghavendra > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-devel > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel