On Fri, Feb 9, 2018 at 4:30 PM Raghavendra Gowdappa <rgowd...@redhat.com> wrote:
> > > ----- Original Message ----- > > From: "Pranith Kumar Karampuri" <pkara...@redhat.com> > > To: "Raghavendra G" <raghaven...@gluster.com> > > Cc: "Gluster Devel" <gluster-devel@gluster.org> > > Sent: Friday, February 9, 2018 2:30:59 PM > > Subject: Re: [Gluster-devel] Glusterfs and Structured data > > > > > > > > On Thu, Feb 8, 2018 at 12:05 PM, Raghavendra G < raghaven...@gluster.com > > > > wrote: > > > > > > > > > > > > On Tue, Feb 6, 2018 at 8:15 PM, Vijay Bellur < vbel...@redhat.com > > wrote: > > > > > > > > > > > > On Sun, Feb 4, 2018 at 3:39 AM, Raghavendra Gowdappa < > rgowd...@redhat.com > > > wrote: > > > > > > All, > > > > One of our users pointed out to the documentation that glusterfs is not > good > > for storing "Structured data" [1], while discussing an issue [2]. > > > > > > As far as I remember, the content around structured data in the Install > Guide > > is from a FAQ that was being circulated in Gluster, Inc. indicating the > > startup's market positioning. Most of that was based on not wanting to > get > > into performance based comparisons of storage systems that are frequently > > seen in the structured data space. > > > > > > Does any of you have more context on the feasibility of storing > "structured > > data" on Glusterfs? Is one of the reasons for such a suggestion > "staleness > > of metadata" as encountered in bugs like [3]? > > > > > > There are challenges that distributed storage systems face when exposed > to > > applications that were written for a local filesystem interface. We have > > encountered problems with applications like tar [4] that are not in the > > realm of "Structured data". If we look at the common theme across all > these > > problems, it is related to metadata & read after write consistency issues > > with the default translator stack that gets exposed on the client side. > > While the default stack is optimal for other scenarios, it does seem > that a > > category of applications needing strict metadata consistency is not well > > served by that. We have observed that disabling a few performance > > translators and tuning cache timeouts for VFS/FUSE have helped to > overcome > > some of them. The WIP effort on timestamp consistency across the > translator > > stack, patches that have been merged as a result of the bugs that you > > mention & other fixes for outstanding issues should certainly help in > > catering to these workloads better with the file interface. > > > > There are deployments that I have come across where glusterfs is used for > > storing structured data. gluster-block & qemu-libgfapi overcome the > metadata > > consistency problem by exposing a file as a block device & by disabling > most > > of the performance translators in the default stack. Workloads that have > > been deemed problematic with the file interface for the reasons alluded > > above, function well with the block interface. > > > > I agree that gluster-block due to its usage of a subset of glusterfs fops > > (mostly reads/writes I guess), runs into less number of consistency > issues. > > However, as you've mentioned we seem to disable perf xlator stack in our > > tests/use-cases till now. Note that perf xlator stack is one of worst > > offenders as far as the metadata consistency is concerned (relatively > less > > scenarios of data inconsistency). So, I wonder, > > * what would be the scenario if we enable perf xlator stack for > > gluster-block? > > * Is performance on gluster-block satisfactory so that we don't need > these > > xlators? > > - Or is it that these xlators are not useful for the workload usually > run on > > gluster-block (For random read/write workload, read/write caching xlators > > offer less or no advantage)? > > > > Yes. They are not useful. Block/VM files are opened with O_DIRECT, so we > > don't enable caching at any layer in glusterfs. md-cache could be useful > for > > serving fstat from glusterfs. But apart from that I don't see any other > > xlator contributing much. > > > > > > > > - Or theoretically the workload is ought to benefit from perf xlators, > but we > > don't see them in our results (there are open bugs to this effect)? > > > > I am asking these questions to ascertain priority on fixing perf xlators > for > > (meta)data inconsistencies. If we offer a different solution for these > > workloads, the need for fixing these issues will be less. > > > > My personal opinion is that both block and fs should work correctly. i.e. > > caching xlators shouldn't lead to inconsistency issues. > > +1. That's my personal opinion too. We'll try to fix these issues. > However, we need to qualify the fixes. It would be helpful if community can > help here. We'll let community know when the fixes are in. > There has been some progress on this. Details can be found at: https://www.mail-archive.com/gluster-devel@gluster.org/msg14877.html > > It would be better > > if we are in a position where we choose a workload on block vs fs based > on > > their performance for that workload and nothing else. Block/VM usecases > > change the workload of the application for glusterfs, so for small file > > operations the kind of performance you see on block can never be > achieved by > > glusterfs with the current architecture/design. > > > > > > > > > > > > > > > > I feel that we have come a long way from the time the install guide was > > written and an update for removing the "staleness of content" might be in > > order there :-). > > > > Regards, > > Vijay > > > > [4] https://bugzilla.redhat.com/show_bug.cgi?id=1058526 > > > > > > > > [1] http://docs.gluster.org/en/latest/Install-Guide/Overview/ > > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1512691 > > [3] https://bugzilla.redhat.com/show_bug.cgi?id=1390050 > > > > regards, > > Raghavendra > > _______________________________________________ > > Gluster-devel mailing list > > Gluster-devel@gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-devel > > > > > > _______________________________________________ > > Gluster-devel mailing list > > Gluster-devel@gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-devel > > > > > > > > -- > > Raghavendra G > > > > _______________________________________________ > > Gluster-devel mailing list > > Gluster-devel@gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-devel > > > > > > > > -- > > Pranith > > > > _______________________________________________ > > Gluster-devel mailing list > > Gluster-devel@gluster.org > > http://lists.gluster.org/mailman/listinfo/gluster-devel > _______________________________________________ > Gluster-devel mailing list > Gluster-devel@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-devel >
_______________________________________________ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel