Re: [Gluster-devel] [Gluster-users] [Gluster-Maintainers] Proposal to change gNFSstatus

[Xie Changlong]

~~Get in a word~~

Hi,  Kaleb Keithley. Could you give some comments on below url ? We 
encounter it in practice.


https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/465149

Thanks

在 2019/11/22 5:14, Kaleb Keithley 写道:
I personally wouldn't call three years ago — when we started to 
deprecate it, in glusterfs-3.9 — a recent change.


As a community the decision was made to move to NFS-Ganesha as the 
preferred NFS solution, but it was agreed to keep the old code in the 
tree for those who wanted it. There have been plans to drop it from 
the community packages for most of those three years, but we didn't 
follow through across the board until fairly recently. Perhaps the 
most telling piece of data is that it's been gone from the packages in 
the CentOS Storage SIG in glusterfs-4.0, -4.1, -5, -6, and -7 with no 
complaints ever, that I can recall.


Ganesha is a preferable solution because it supports NFSv4, NFSv4.1, 
NFSv4.2, and pNFS, in addition to legacy NFSv3. More importantly, it 
is actively developed, maintained, and supported, both in the 
community and commercially. There are several vendors selling it, or 
support for it; and there are community packages for it for all the 
same distributions that Gluster packages are available for.


Out in the world, the default these days is NFSv4. Specifically v4.2 
or v4.1 depending on how recent your linux kernel is. In the linux 
kernel, client mounts start negotiating for v4.2 and work down to 
v4.1, v4.0, and only as a last resort v3. NFSv3 client support in the 
linux kernel largely exists at this point only because of the large 
number of legacy servers still running that can't do anything higher 
than v3. The linux NFS developers would drop the v3 support in a 
heartbeat if they could.


IMO, providing it, and calling it maintained, only encourages people 
to keep using a dead end solution. Anyone in favor of bringing back 
NFSv2, SSHv1, or X10R4? No? I didn't think so.


The recent issue[1] where someone built gnfs in glusterfs-7.0 on 
CentOS7 strongly suggests to me that gnfs is not actually working 
well. Three years of no maintenance seems to have taken its toll.


Other people are more than welcome to build their own packages from 
the src.rpms and/or tarballs that are available from gluster — and 
support them. It's still in the source and there are no plans to 
remove it. (Unlike most of the other deprecated features which were 
recently removed in glusterfs-7.)




[1] https://github.com/gluster/glusterfs/issues/764

On Thu, Nov 21, 2019 at 5:31 AM Amar Tumballi > wrote:


Hi All,

As per the discussion on https://review.gluster.org/23645,
recently we changed the status of gNFS (gluster's native NFSv3
support) feature to 'Depricated / Orphan' state. (ref:
https://github.com/gluster/glusterfs/blob/master/MAINTAINERS#L185..L189).
With this email, I am proposing to change the status again to 'Odd
Fixes' (ref:
https://github.com/gluster/glusterfs/blob/master/MAINTAINERS#L22)

TL;DR;

I understand the current maintainers are not able to focus on
maintaining it as the focus of the project, as earlier described,
is keeping NFS-Ganesha based integration with glusterfs. But, I am
volunteering along with Xie Changlong (currently working at
Chinamobile), to keep the feature running as it used to in
previous versions. Hence the status of 'Odd Fixes'.

Before sending the patch to make these changes, I am proposing it
here now, as gNFS is not even shipped with latest glusterfs-7.0
releases. I have heard from some users that it was working great
for them with earlier releases, as all they wanted was NFS v3
support, and not much of features from gNFS. Also note that, even
though the packages are not built, none of the regression tests
using gNFS are stopped with latest master, so it is working same
from at least last 2 years.

I request the package maintainers to please add '--with gnfs' (or
--enable-gnfs) back to their release script through this email, so
those users wanting to use gNFS happily can continue to use it.
Also points to users/admins is that, the status is 'Odd Fixes', so
don't expect any 'enhancements' on the features provided by gNFS.

Happy to hear feedback, if any.

Regards,
Amar

___
maintainers mailing list
maintain...@gluster.org 
https://lists.gluster.org/mailman/listinfo/maintainers




Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
gluster-us...@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
___

Community Meeting 

Re: [Gluster-devel] [Gluster-Maintainers] Proposal to change gNFS status

[Niels de Vos]

On Thu, Nov 21, 2019 at 04:01:23PM +0530, Amar Tumballi wrote:
> Hi All,
> 
> As per the discussion on https://review.gluster.org/23645, recently we
> changed the status of gNFS (gluster's native NFSv3 support) feature to
> 'Depricated / Orphan' state. (ref:
> https://github.com/gluster/glusterfs/blob/master/MAINTAINERS#L185..L189).
> With this email, I am proposing to change the status again to 'Odd Fixes'
> (ref: https://github.com/gluster/glusterfs/blob/master/MAINTAINERS#L22)

I'd recommend against re-surrecting gNFS. The server is not very
extensible and adding new features is pretty tricky without breaking
other (mostly undocumented) use-cases. Eventhough NFSv3 is stateless,
the actual usage of NFSv3, mounting and locking is definitely not. The
server keeps track of which clients have an export mounted, and which
clients received grants for locks. These things are currently not very
reliable in combination with high-availability. And there is also the by
default disabled duplicate-reply-cache (DRC) that has always been very
buggy (and neither cluster-aware).

If we enable gNFS by default again, we're sending out an incorrect
message to our users. gNFS works fine for certain workloads and
environments, but it should not be advertised as 'clustered NFS'.

Instead of going the gNFS route, I suggest to make it easier to deploy
NFS-Ganesha as that is a more featured, well maintained and can be
configured for much more reliable high-availability than gNFS.

If someone really wants to maintain gNFS, I won't object much, but they
should know that previous maintainers have had many difficulties just
keeping it working well while other components evolved. Addressing some
of the bugs/limitations will be extremely difficult and may require
large rewrites of parts of gNFS.

Until now, I have not read convincing arguments in this thread that gNFS
is stable enough to be consumed by anyone in the community. Users should
be aware of its limitations and be careful what workloads to run on it.

HTH,
Niels

___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968


NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-Maintainers] Modifying gluster's logging mechanism

[Ravishankar N]

On 22/11/19 3:13 pm, Barak Sason Rofman wrote:
This is actually one of the main reasons I wanted to bring this up for 
discussion - will it be fine with the community to run a dedicated 
tool to reorder the logs offline?


I think it is a bad idea to log without ordering and later relying on an 
external tool to sort it.  This is definitely not something I would want 
to do while doing test and development or debugging field issues.  
Structured logging  is definitely useful for gathering statistics and 
post-processing to make reports and charts and what not,  but from a 
debugging point of view, maintaining causality of messages and working 
with command line text editors and tools on log files is important IMO.


I had a similar concerns when  brick multiplexing feature was developed 
where a single log file was used for logging all multiplexed bricks' 
logs.  So much extra work to weed out messages of 99 processes to read 
the log of the 1 process you are interested in.


Regards,
Ravi

___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968


NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Modifying gluster's logging mechanism

[Yaniv Kaul]

On Fri, Nov 22, 2019 at 11:45 AM Barak Sason Rofman 
wrote:

> Thank you for your input Atin and Xie Changlong.
>
> Regarding log ordering - my initial thought was to do it offline using a
> dedicated too. Should be straight forward, as the logs have time stamp
> composed of seconds and microseconds, so ordering them using this value is
> definitely possible.
> This is actually one of the main reasons I wanted to bring this up for
> discussion - will it be fine with the community to run a dedicated tool to
> reorder the logs offline?
> Reordering the logs offline will allow us to gain the most performance
> improvement, as keeping the logs order online will have some cost (probably
> through stronger synchronization).
> Moreover, we can take log viewing one step further and maybe create some
> GUI system (JAVA based?) to view and handle logs (e.g. one window to show
> the combined order logs, other windows to show logs per thread etc').
>

If you need an external tool (please not Java - let's not add another
language to the project), you might as well move to binary logging.
I believe we need to be able to do this sort using Linux command line tools
only.

>
> Regarding the test method - my initial testing was done by removing all
> logging from regression. Modifying the method "skip_logging" to return
> 'true' in all cases seems to remove most of the logs (though not all, "to
> be on the safe side", really removing all logging related methods is
> probably even better).
>

This is not a fair comparison:
1. The regression tests are running with debug log
2. Not logging at all != replacing the logging framework - the new one will
have its own overhead as well.
3. I believe there's a lot of code that is not real life scenarios there -
such as dumping a lot of data there (which causes a lot of calls to
inode_find() and friends - for example).
Y.

As regression tests use mostly single-node tests, some additional testing
> was needed. I've written a couple of very basic scripts to create large
> number of files / big files, read / write to / from them, move them around
> and perform some other basic functionality.
> I'd actually be glad to test this in some 'real world' use cases - if you
> have specific use cases that you use frequently, we can model them and
> benchmark against - this will likely offer an even more accurate benchmark.
>
> On Fri, Nov 22, 2019 at 7:27 AM Xie Changlong  wrote:
>
> >
> > 在 2019/11/21 21:04, Barak Sason Rofman 写道:
> >
> > I see two design / implementation problems with that mechanism:
> >
> >1.
> >
> >The mutex that guards the log file is likely under constant contention
___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968


NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Modifying gluster's logging mechanism

[Xie Changlong]

在 2019/11/22 17:43, Barak Sason Rofman 写道:

Thank you for your input Atin and Xie Changlong.

Regarding log ordering - my initial thought was to do it offline using 
a dedicated too. Should be straight forward, as the logs have time 
stamp composed of seconds and microseconds, so ordering them using 
this value is definitely possible.
This is actually one of the main reasons I wanted to bring this up for 
discussion - will it be fine with the community to run a dedicated 
tool to reorder the logs offline?
Reordering the logs offline will allow us to gain the most performance 
improvement, as keeping the logs order online will have some cost 
(probably through stronger synchronization).
Moreover, we can take log viewing one step further and maybe create 
some GUI system (JAVA based?) to view and handle logs (e.g. one window 
to show the combined order logs, other windows to show logs per thread 
etc').


Regarding the test method - my initial testing was done by removing 
all logging from regression. Modifying the method "skip_logging" to 
return 'true' in all cases seems to remove most of the logs (though 
not all, "to be on the safe side", really removing all logging related 
methods is probably even better).



Thanks for sharing, i'll go back to your method and do some perf test : )


As regression tests use mostly single-node tests, some additional 
testing was needed. I've written a couple of very basic scripts to 
create large number of files / big files, read / write to / from them, 
move them around and perform some other basic functionality.
I'd actually be glad to test this in some 'real world' use cases - if 
you have specific use cases that you use frequently, we can model them 
and benchmark against - this will likely offer an even more accurate 
benchmark.


On Fri, Nov 22, 2019 at 7:27 AM Xie Changlong > wrote:



在 2019/11/21 21:04, Barak Sason Rofman 写道:


I see two design / implementation problems with that mechanism:

1.

The mutex that guards the log file is likely under constant
contention.

2.

The fact that each worker thread perform the IO by himself,
thus slowing his "real" work.


Initial tests, done by *removing logging from the regression
testing, shows an improvement of about 20% in run time*. This
indicates we’re taking a pretty heavy performance hit just
because of the logging activity.



Hi Barak Sason Rofman.  Amazing perf improvement! Could show me
the detail test method ?

Thanks

    -Xie


In addition to these problems, the logging module is due for an
upgrade:

1.

There are dozens of APIs in the logger, much of them are
deprecated - this makes it very hard for new developers to
keep evolving the project.

2.

One of the key points for Gluster-X, presented in October at
Bangalore, is the switch to a structured logging all across
gluster.




--
*Barak Sason Rofman*

Gluster Storage Development

Red Hat Israel 

34 Jerusalem rd. Ra'anana, 43501

bsaso...@redhat.com   T: _+972-9-7692304_
M: _+972-52-4326355_



___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968


NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] Modifying gluster's logging mechanism

[Barak Sason Rofman]

Thank you for your input Atin and Xie Changlong.

Regarding log ordering - my initial thought was to do it offline using a
dedicated too. Should be straight forward, as the logs have time stamp
composed of seconds and microseconds, so ordering them using this value is
definitely possible.
This is actually one of the main reasons I wanted to bring this up for
discussion - will it be fine with the community to run a dedicated tool to
reorder the logs offline?
Reordering the logs offline will allow us to gain the most performance
improvement, as keeping the logs order online will have some cost (probably
through stronger synchronization).
Moreover, we can take log viewing one step further and maybe create some
GUI system (JAVA based?) to view and handle logs (e.g. one window to show
the combined order logs, other windows to show logs per thread etc').

Regarding the test method - my initial testing was done by removing all
logging from regression. Modifying the method "skip_logging" to return
'true' in all cases seems to remove most of the logs (though not all, "to
be on the safe side", really removing all logging related methods is
probably even better).
As regression tests use mostly single-node tests, some additional testing
was needed. I've written a couple of very basic scripts to create large
number of files / big files, read / write to / from them, move them around
and perform some other basic functionality.
I'd actually be glad to test this in some 'real world' use cases - if you
have specific use cases that you use frequently, we can model them and
benchmark against - this will likely offer an even more accurate benchmark.

On Fri, Nov 22, 2019 at 7:27 AM Xie Changlong  wrote:

>
> 在 2019/11/21 21:04, Barak Sason Rofman 写道:
>
> I see two design / implementation problems with that mechanism:
>
>1.
>
>The mutex that guards the log file is likely under constant contention.
>2.
>
>The fact that each worker thread perform the IO by himself, thus
>slowing his "real" work.
>
>
> Initial tests, done by *removing logging from the regression testing,
> shows an improvement of about 20% in run time*. This indicates we’re
> taking a pretty heavy performance hit just because of the logging activity.
>
> Hi Barak Sason Rofman.  Amazing perf improvement! Could show me the detail
> test method ?
>
> Thanks
>
> -Xie
>
> In addition to these problems, the logging module is due for an upgrade:
>
>1.
>
>There are dozens of APIs in the logger, much of them are deprecated -
>this makes it very hard for new developers to keep evolving the project.
>2.
>
>One of the key points for Gluster-X, presented in October at
>Bangalore, is the switch to a structured logging all across gluster.
>
>

-- 
*Barak Sason Rofman*

Gluster Storage Development

Red Hat Israel 

34 Jerusalem rd. Ra'anana, 43501

bsaso...@redhat.com T: *+972-9-7692304*
M: *+972-52-4326355*

___

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968


NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel