Re: [Gluster-devel] 回复： Glusterfs crash when enable quota on Arm aarch 64platform.
It's my pleasure, CC devel to confirm this issue.
?? 2019/12/3 14:13, PSC :
Thank you for your help!
It seems that it is my fault, I didn't disable quota and enable it
again, so it seems work fine on commits before 2fb445ba, when you
write file to a volume which had already reached its usage limit, it
tells Disk quota exceeded. But actually, on former
commits(before?0?22fb445ba), if I disable quota and enable it again, the
volume is crash. You are right! 2fb445ba addressed the quota issue!
Thank you very much!!
--?0?2?0?2--
*??:*?0?2"zgrep";
*:*?0?22019??11??29??(??) 6:35
*??:*?0?2"Yaniv Kaul";
*:*?0?2"PSC"<1173701...@qq.com>;"Gluster
Devel";
*:*?0?2Re: [Gluster-devel] Glusterfs crash when enable quota on Arm
aarch 64platform.
NO, this issue disappered since commit 2fb445ba.
?? 2019/11/29 18:12, Yaniv Kaul :
Does it happen on master?
On Fri, 29 Nov 2019, 12:06 Xie Changlong <mailto:zg...@139.com>> wrote:
Hi, PSC
We encounter the same issue a few month ago, and git bisect says
the first bad commit is 2fb445ba. This patch is not quota
related, but it addressed the quota issue!
Maybe it's gcc issue??
commit 2fb445babdd621b71676e40804fe98e95fc9084d
Author: Xavi Hernandez
<mailto:xhernan...@redhat.com>
Date:?0?2?0?2 Thu Jan 31 08:13:58 2019 +0100
?0?2?0?2?0?2 syncop: remove unnecessary call to gf_backtrace_save()
?0?2?0?2?0?2 A call to gf_backtrace_save() was done on each context switch
of a
?0?2?0?2?0?2 synctask. The backtrace is generated writing to the
filesystem, so it
?0?2?0?2?0?2 can have an important impact on latency.
?0?2?0?2?0?2 The generated backtrace was not used anywhere, so it's been
removed.
?0?2?0?2?0?2 Change-Id: I399a93b932c5b6e981c696c72c3e1ef44710ba52
?0?2?0?2?0?2 Updates: bz#1193929
?0?2?0?2?0?2 Signed-off-by: Xavi Hernandez
<mailto:xhernan...@redhat.com>
diff --git a/libglusterfs/src/glusterfs/syncop.h
b/libglusterfs/src/glusterfs/syncop.h
index 7a6167b..e0f1017 100644
--- a/libglusterfs/src/glusterfs/syncop.h
+++ b/libglusterfs/src/glusterfs/syncop.h
@@ -73,7 +73,6 @@ struct synctask {
?0?2?0?2?0?2?0?2 int done;
?0?2?0?2?0?2?0?2 struct list_head waitq; /* can wait only "once" at a time
*/
-?0?2?0?2?0?2 char btbuf[GF_BACKTRACE_LEN];
?0?2};
?0?2struct syncproc {
diff --git a/libglusterfs/src/syncop.c b/libglusterfs/src/syncop.c
index bf70daf..6206d4c 100644
--- a/libglusterfs/src/syncop.c
+++ b/libglusterfs/src/syncop.c
@@ -327,7 +327,6 @@ synctask_yield(struct synctask *task)
?0?2?0?2?0?2?0?2 if (task->state != SYNCTASK_DONE) {
?0?2?0?2?0?2?0?2?0?2?0?2?0?2?0?2 task->state = SYNCTASK_SUSPEND;
-?0?2?0?2?0?2?0?2?0?2?0?2?0?2 (void)gf_backtrace_save(task->btbuf);
?0?2?0?2?0?2?0?2 }
?0?2?0?2?0?2?0?2 if (swapcontext(>ctx, >proc->sched) < 0) {
?0?2?0?2?0?2?0?2?0?2?0?2?0?2?0?2 gf_msg("syncop", GF_LOG_ERROR, errno,
LG_MSG_SWAPCONTEXT_FAILED,
?? 2019/11/29 16:12, PSC :
Hi, I am studying on port glusterfs to ARM platform. I compiled
and ran it. Most of its functions works fine, however, when I
enable quota on any kind of volume, and mount it, and try to
read or write anything to the volume. It will run into crash,
and tell me the "Transport endpoint is not connected".
The version of glusterfs I am using is GlusterFS 3.12.2.
And I test it from version 3.12.2 to 6.0. This problem was
solved on 6.0, but I didn't found any addressed Bugs relate to
quota.
I also do some test on x86 servers, quota works fine.
On both of x86 and ARM servers, I am using CentOS 7.6.
For stability reason, I had been told that I should use
GlusterFS 3.12.2, rather than new versions. So I need to find
out how to address this bug.
Please offer me some help. Thank you very much!!
___
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge:https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge:https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org>
Re: [Gluster-devel] Glusterfs crash when enable quota on Arm aarch 64platform.
NO, this issue disappered since commit 2fb445ba.
在 2019/11/29 18:12, Yaniv Kaul 写道:
Does it happen on master?
On Fri, 29 Nov 2019, 12:06 Xie Changlong <mailto:zg...@139.com>> wrote:
Hi, PSC
We encounter the same issue a few month ago, and git bisect says
the first bad commit is 2fb445ba. This patch is not quota related,
but it addressed the quota issue!
Maybe it's gcc issue??
commit 2fb445babdd621b71676e40804fe98e95fc9084d
Author: Xavi Hernandez
<mailto:xhernan...@redhat.com>
Date: Thu Jan 31 08:13:58 2019 +0100
syncop: remove unnecessary call to gf_backtrace_save()
A call to gf_backtrace_save() was done on each context switch of a
synctask. The backtrace is generated writing to the
filesystem, so it
can have an important impact on latency.
The generated backtrace was not used anywhere, so it's been
removed.
Change-Id: I399a93b932c5b6e981c696c72c3e1ef44710ba52
Updates: bz#1193929
Signed-off-by: Xavi Hernandez
<mailto:xhernan...@redhat.com>
diff --git a/libglusterfs/src/glusterfs/syncop.h
b/libglusterfs/src/glusterfs/syncop.h
index 7a6167b..e0f1017 100644
--- a/libglusterfs/src/glusterfs/syncop.h
+++ b/libglusterfs/src/glusterfs/syncop.h
@@ -73,7 +73,6 @@ struct synctask {
int done;
struct list_head waitq; /* can wait only "once" at a time */
- char btbuf[GF_BACKTRACE_LEN];
};
struct syncproc {
diff --git a/libglusterfs/src/syncop.c b/libglusterfs/src/syncop.c
index bf70daf..6206d4c 100644
--- a/libglusterfs/src/syncop.c
+++ b/libglusterfs/src/syncop.c
@@ -327,7 +327,6 @@ synctask_yield(struct synctask *task)
if (task->state != SYNCTASK_DONE) {
task->state = SYNCTASK_SUSPEND;
- (void)gf_backtrace_save(task->btbuf);
}
if (swapcontext(>ctx, >proc->sched) < 0) {
gf_msg("syncop", GF_LOG_ERROR, errno,
LG_MSG_SWAPCONTEXT_FAILED,
在 2019/11/29 16:12, PSC 写道:
Hi, I am studying on port glusterfs to ARM platform. I compiled
and ran it. Most of its functions works fine, however, when I
enable quota on any kind of volume, and mount it, and try to read
or write anything to the volume. It will run into crash, and tell
me the "Transport endpoint is not connected".
The version of glusterfs I am using is GlusterFS 3.12.2.
And I test it from version 3.12.2 to 6.0. This problem was solved
on 6.0, but I didn't found any addressed Bugs relate to quota.
I also do some test on x86 servers, quota works fine.
On both of x86 and ARM servers, I am using CentOS 7.6.
For stability reason, I had been told that I should use GlusterFS
3.12.2, rather than new versions. So I need to find out how to
address this bug.
Please offer me some help. Thank you very much!!
___
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge:https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge:https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@gluster.org <mailto:Gluster-devel@gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Glusterfs crash when enable quota on Arm aarch 64platform.
Hi, PSC
We encounter the same issue a few month ago, and git bisect says the
first bad commit is 2fb445ba. This patch is not quota related, but it
addressed the quota issue!
Maybe it's gcc issue??
commit 2fb445babdd621b71676e40804fe98e95fc9084d
Author: Xavi Hernandez
Date: Thu Jan 31 08:13:58 2019 +0100
syncop: remove unnecessary call to gf_backtrace_save()
A call to gf_backtrace_save() was done on each context switch of a
synctask. The backtrace is generated writing to the filesystem, so it
can have an important impact on latency.
The generated backtrace was not used anywhere, so it's been removed.
Change-Id: I399a93b932c5b6e981c696c72c3e1ef44710ba52
Updates: bz#1193929
Signed-off-by: Xavi Hernandez
diff --git a/libglusterfs/src/glusterfs/syncop.h
b/libglusterfs/src/glusterfs/syncop.h
index 7a6167b..e0f1017 100644
--- a/libglusterfs/src/glusterfs/syncop.h
+++ b/libglusterfs/src/glusterfs/syncop.h
@@ -73,7 +73,6 @@ struct synctask {
int done;
struct list_head waitq; /* can wait only "once" at a time */
- char btbuf[GF_BACKTRACE_LEN];
};
struct syncproc {
diff --git a/libglusterfs/src/syncop.c b/libglusterfs/src/syncop.c
index bf70daf..6206d4c 100644
--- a/libglusterfs/src/syncop.c
+++ b/libglusterfs/src/syncop.c
@@ -327,7 +327,6 @@ synctask_yield(struct synctask *task)
if (task->state != SYNCTASK_DONE) {
task->state = SYNCTASK_SUSPEND;
- (void)gf_backtrace_save(task->btbuf);
}
if (swapcontext(>ctx, >proc->sched) < 0) {
gf_msg("syncop", GF_LOG_ERROR, errno, LG_MSG_SWAPCONTEXT_FAILED,
在 2019/11/29 16:12, PSC 写道:
Hi, I am studying on port glusterfs to ARM platform. I compiled and
ran it. Most of its functions works fine, however, when I enable quota
on any kind of volume, and mount it, and try to read or write anything
to the volume. It will run into crash, and tell me the "Transport
endpoint is not connected".
The version of glusterfs I am using is GlusterFS 3.12.2.
And I test it from version 3.12.2 to 6.0. This problem was solved on
6.0, but I didn't found any addressed Bugs relate to quota.
I also do some test on x86 servers, quota works fine.
On both of x86 and ARM servers, I am using CentOS 7.6.
For stability reason, I had been told that I should use GlusterFS
3.12.2, rather than new versions. So I need to find out how to address
this bug.
Please offer me some help. Thank you very much!!
___
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] [Gluster-Maintainers] Proposal to change gNFSstatus
~~Get in a word~~ Hi, Kaleb Keithley. Could you give some comments on below url ? We encounter it in practice. https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/465149 Thanks 在 2019/11/22 5:14, Kaleb Keithley 写道: I personally wouldn't call three years ago — when we started to deprecate it, in glusterfs-3.9 — a recent change. As a community the decision was made to move to NFS-Ganesha as the preferred NFS solution, but it was agreed to keep the old code in the tree for those who wanted it. There have been plans to drop it from the community packages for most of those three years, but we didn't follow through across the board until fairly recently. Perhaps the most telling piece of data is that it's been gone from the packages in the CentOS Storage SIG in glusterfs-4.0, -4.1, -5, -6, and -7 with no complaints ever, that I can recall. Ganesha is a preferable solution because it supports NFSv4, NFSv4.1, NFSv4.2, and pNFS, in addition to legacy NFSv3. More importantly, it is actively developed, maintained, and supported, both in the community and commercially. There are several vendors selling it, or support for it; and there are community packages for it for all the same distributions that Gluster packages are available for. Out in the world, the default these days is NFSv4. Specifically v4.2 or v4.1 depending on how recent your linux kernel is. In the linux kernel, client mounts start negotiating for v4.2 and work down to v4.1, v4.0, and only as a last resort v3. NFSv3 client support in the linux kernel largely exists at this point only because of the large number of legacy servers still running that can't do anything higher than v3. The linux NFS developers would drop the v3 support in a heartbeat if they could. IMO, providing it, and calling it maintained, only encourages people to keep using a dead end solution. Anyone in favor of bringing back NFSv2, SSHv1, or X10R4? No? I didn't think so. The recent issue[1] where someone built gnfs in glusterfs-7.0 on CentOS7 strongly suggests to me that gnfs is not actually working well. Three years of no maintenance seems to have taken its toll. Other people are more than welcome to build their own packages from the src.rpms and/or tarballs that are available from gluster — and support them. It's still in the source and there are no plans to remove it. (Unlike most of the other deprecated features which were recently removed in glusterfs-7.) [1] https://github.com/gluster/glusterfs/issues/764 On Thu, Nov 21, 2019 at 5:31 AM Amar Tumballi <mailto:ama...@gmail.com>> wrote: Hi All, As per the discussion on https://review.gluster.org/23645, recently we changed the status of gNFS (gluster's native NFSv3 support) feature to 'Depricated / Orphan' state. (ref: https://github.com/gluster/glusterfs/blob/master/MAINTAINERS#L185..L189). With this email, I am proposing to change the status again to 'Odd Fixes' (ref: https://github.com/gluster/glusterfs/blob/master/MAINTAINERS#L22) TL;DR; I understand the current maintainers are not able to focus on maintaining it as the focus of the project, as earlier described, is keeping NFS-Ganesha based integration with glusterfs. But, I am volunteering along with Xie Changlong (currently working at Chinamobile), to keep the feature running as it used to in previous versions. Hence the status of 'Odd Fixes'. Before sending the patch to make these changes, I am proposing it here now, as gNFS is not even shipped with latest glusterfs-7.0 releases. I have heard from some users that it was working great for them with earlier releases, as all they wanted was NFS v3 support, and not much of features from gNFS. Also note that, even though the packages are not built, none of the regression tests using gNFS are stopped with latest master, so it is working same from at least last 2 years. I request the package maintainers to please add '--with gnfs' (or --enable-gnfs) back to their release script through this email, so those users wanting to use gNFS happily can continue to use it. Also points to users/admins is that, the status is 'Odd Fixes', so don't expect any 'enhancements' on the features provided by gNFS. Happy to hear feedback, if any. Regards, Amar ___ maintainers mailing list maintain...@gluster.org <mailto:maintain...@gluster.org> https://lists.gluster.org/mailman/listinfo/maintainers Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-users mailing list gluster-us...@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users ___ Co
Re: [Gluster-devel] Modifying gluster's logging mechanism
在 2019/11/22 17:43, Barak Sason Rofman 写道: Thank you for your input Atin and Xie Changlong. Regarding log ordering - my initial thought was to do it offline using a dedicated too. Should be straight forward, as the logs have time stamp composed of seconds and microseconds, so ordering them using this value is definitely possible. This is actually one of the main reasons I wanted to bring this up for discussion - will it be fine with the community to run a dedicated tool to reorder the logs offline? Reordering the logs offline will allow us to gain the most performance improvement, as keeping the logs order online will have some cost (probably through stronger synchronization). Moreover, we can take log viewing one step further and maybe create some GUI system (JAVA based?) to view and handle logs (e.g. one window to show the combined order logs, other windows to show logs per thread etc'). Regarding the test method - my initial testing was done by removing all logging from regression. Modifying the method "skip_logging" to return 'true' in all cases seems to remove most of the logs (though not all, "to be on the safe side", really removing all logging related methods is probably even better). Thanks for sharing, i'll go back to your method and do some perf test : ) As regression tests use mostly single-node tests, some additional testing was needed. I've written a couple of very basic scripts to create large number of files / big files, read / write to / from them, move them around and perform some other basic functionality. I'd actually be glad to test this in some 'real world' use cases - if you have specific use cases that you use frequently, we can model them and benchmark against - this will likely offer an even more accurate benchmark. On Fri, Nov 22, 2019 at 7:27 AM Xie Changlong <mailto:zg...@139.com>> wrote: 在 2019/11/21 21:04, Barak Sason Rofman 写道: I see two design / implementation problems with that mechanism: 1. The mutex that guards the log file is likely under constant contention. 2. The fact that each worker thread perform the IO by himself, thus slowing his "real" work. Initial tests, done by *removing logging from the regression testing, shows an improvement of about 20% in run time*. This indicates we’re taking a pretty heavy performance hit just because of the logging activity. Hi Barak Sason Rofman. Amazing perf improvement! Could show me the detail test method ? Thanks -Xie In addition to these problems, the logging module is due for an upgrade: 1. There are dozens of APIs in the logger, much of them are deprecated - this makes it very hard for new developers to keep evolving the project. 2. One of the key points for Gluster-X, presented in October at Bangalore, is the switch to a structured logging all across gluster. -- *Barak Sason Rofman* Gluster Storage Development Red Hat Israel <https://www.redhat.com/> 34 Jerusalem rd. Ra'anana, 43501 bsaso...@redhat.com <mailto:a...@redhat.com> T: _+972-9-7692304_ M: _+972-52-4326355_ <https://red.ht/sig> ___ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Modifying gluster's logging mechanism
在 2019/11/21 21:04, Barak Sason Rofman 写道: I see two design / implementation problems with that mechanism: 1. The mutex that guards the log file is likely under constant contention. 2. The fact that each worker thread perform the IO by himself, thus slowing his "real" work. Initial tests, done by *removing logging from the regression testing, shows an improvement of about 20% in run time*. This indicates we’re taking a pretty heavy performance hit just because of the logging activity. Hi Barak Sason Rofman. Amazing perf improvement! Could show me the detail test method ? Thanks -Xie In addition to these problems, the logging module is due for an upgrade: 1. There are dozens of APIs in the logger, much of them are deprecated - this makes it very hard for new developers to keep evolving the project. 2. One of the key points for Gluster-X, presented in October at Bangalore, is the switch to a structured logging all across gluster. ___ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-Maintainers] Proposal to change gNFSstatus
在 2019/11/22 5:14, Kaleb Keithley 写道: I personally wouldn't call three years ago — when we started to deprecate it, in glusterfs-3.9 — a recent change. As a community the decision was made to move to NFS-Ganesha as the preferred NFS solution, but it was agreed to keep the old code in the tree for those who wanted it. There have been plans to drop it from the community packages for most of those three years, but we didn't follow through across the board until fairly recently. Perhaps the most telling piece of data is that it's been gone from the packages in the CentOS Storage SIG in glusterfs-4.0, -4.1, -5, -6, and -7 with no complaints ever, that I can recall. Ganesha is a preferable solution because it supports NFSv4, NFSv4.1, NFSv4.2, and pNFS, in addition to legacy NFSv3. More importantly, it is actively developed, maintained, and supported, both in the community and commercially. There are several vendors selling it, or support for it; and there are community packages for it for all the same distributions that Gluster packages are available for. Out in the world, the default these days is NFSv4. Specifically v4.2 or v4.1 depending on how recent your linux kernel is. In the linux kernel, client mounts start negotiating for v4.2 and work down to v4.1, v4.0, and only as a last resort v3. NFSv3 client support in the linux kernel largely exists at this point only because of the large number of legacy servers still running that can't do anything higher than v3. The linux NFS developers would drop the v3 support in a heartbeat if they could. IMO, providing it, and calling it maintained, only encourages people to keep using a dead end solution. Anyone in favor of bringing back NFSv2, SSHv1, or X10R4? No? I didn't think so. The recent issue[1] where someone built gnfs in glusterfs-7.0 on CentOS7 strongly suggests to me that gnfs is not actually working well. Three years of no maintenance seems to have taken its toll. Other people are more than welcome to build their own packages from the src.rpms and/or tarballs that are available from gluster — and support them. It's still in the source and there are no plans to remove it. (Unlike most of the other deprecated features which were recently removed in glusterfs-7.) [1] https://github.com/gluster/glusterfs/issues/764 It seems https://bugzilla.redhat.com/show_bug.cgi?id=1727248 has resolved this issue. Here i'll talk about something from commerical company view. For security reasons most government procurement projects only allow universal storage protocol(nfs, cifs etc) what means fuse will be excluded. Consindering performance requirements, the only option is nfs. Nfsv4 is stateful protocol, but i see on performance improvement. Trust me, nfs-ganesha(v3, v4) shows ~30% performance degradation versus gnfs for either small or big files r/w in practice. Further, many customers prefer nfs client than cifs in windows, because the poor cifs performance, AFAIK nfs-ganesha is not going well with windows nfs client. Gnfs is stable enough, we have about ~1000 servers, 4~24 servers for a gluster cluster, about ~2000 nfs clients, all works fine till the last two years expect some memleak issue. Thanks -Xie On Thu, Nov 21, 2019 at 5:31 AM Amar Tumballi <mailto:ama...@gmail.com>> wrote: Hi All, As per the discussion on https://review.gluster.org/23645, recently we changed the status of gNFS (gluster's native NFSv3 support) feature to 'Depricated / Orphan' state. (ref: https://github.com/gluster/glusterfs/blob/master/MAINTAINERS#L185..L189). With this email, I am proposing to change the status again to 'Odd Fixes' (ref: https://github.com/gluster/glusterfs/blob/master/MAINTAINERS#L22) TL;DR; I understand the current maintainers are not able to focus on maintaining it as the focus of the project, as earlier described, is keeping NFS-Ganesha based integration with glusterfs. But, I am volunteering along with Xie Changlong (currently working at Chinamobile), to keep the feature running as it used to in previous versions. Hence the status of 'Odd Fixes'. Before sending the patch to make these changes, I am proposing it here now, as gNFS is not even shipped with latest glusterfs-7.0 releases. I have heard from some users that it was working great for them with earlier releases, as all they wanted was NFS v3 support, and not much of features from gNFS. Also note that, even though the packages are not built, none of the regression tests using gNFS are stopped with latest master, so it is working same from at least last 2 years. I request the package maintainers to please add '--with gnfs' (or --enable-gnfs) back to their release script through this email, so those users wanting to use gNFS happily can continue to use it. Also points to users/admins is that, the status is 'Odd F
Re: [Gluster-devel] Glusterfs performance regression with quota enabled
Hi changwei, i'm sure if you enable quota, the performance will sharply drop. There is a issue to implement glusterfs project quota just like xfs project quota, but no progress for a long time. Pls ref: https://github.com/gluster/glusterfs/issues/184 发件人: Changwei Ge 时间: 2019/08/16(星期五)17:09 收件人: gluster-devel; 主题: [Gluster-devel] Glusterfs performance regression with quota enabledHi, I am using glusterfs-5.6 with quota enabled. I observed a obvious performance regression about 30% against a certain vdbench workload[1]. But I didn't set up a hard-limit or soft-limit to any particular volume or directory yet, just enable quota. After disabling features/quota, glusterfs performance went normal. Is it normal that qutoa will make the performance sharply drop? Thanks, Changwei [1]: messagescan=no fsd=fsd1,anchor=/mnt/q8,depth=1,width=1,files=5,size=(32k,20,64k,30,128k,30,256k,20) fwd=fwd1,fsd=fsd1,operation=read,rdpct=80,xfersize=8k,fileio=random,fileselect=random,threads=8 rd=rd1,fwd=fwd1,fwdrate=max,format=yes,elapsed=100,interval=1 ___ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/836554017 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/486278655 Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel ___ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/836554017 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/486278655 Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] GETXATTR op pending on index xlator for more than 10 hours
Hi all, Today, i found gnfs GETXATTR bailing out on gluster release 3.12.0. I have a simple 4*2 Distributed-Rep volume. [2019-06-03 19:58:33.085880] E [rpc-clnt.c:185:Call_bail] 0-cl25vol01-client-4: bailing out frame type(GlusterFS 3.3) op(GETXATTR(18)) xid=0x21de4275 sent = 2019-06-03 19:28:30.552356. timeout = 1800 for 10.3.133.57:49153 xid= 0x21de4275 = 568214133 Then i try to dump brick 10.3.133.57:49153, and find the GETXATTR op pending on index xlator for more than 10 hours! MicrosoftInternetExplorer402DocumentNotSpecified7.8 磅Normal0 [root@node0001 gluster]# grep -rn 568214133 gluster-brick-1-cl25vol01.6078.dump.15596* gluster-brick-1-cl25vol01.6078.dump.1559617125:5093:unique=568214133 gluster-brick-1-cl25vol01.6078.dump.1559618121:5230:unique=568214133 gluster-brick-1-cl25vol01.6078.dump.1559618912:5434:unique=568214133 gluster-brick-1-cl25vol01.6078.dump.1559628467:6921:unique=568214133 [root@node0001 gluster]# date -d @1559617125 Tue Jun 4 10:58:45 CST 2019 [root@node0001 gluster]# date -d @1559628467 Tue Jun 4 14:07:47 CST 2019 MicrosoftInternetExplorer402DocumentNotSpecified7.8 磅Normal0 [root@node0001 gluster]# [global.callpool.stack.115] stack=0x7f8b342623c0 uid=500 gid=500 pid=-6 unique=568214133 lk-owner=faff op=stack type=0 cnt=4 [global.callpool.stack.115.frame.1] frame=0x7f8b1d6fb540 ref_count=0 translator=cl25vol01-index complete=0 parent=cl25vol01-quota wind_from=quota_getxattr wind_to=(this->children->xlator)->fops->getxattr unwind_to=default_getxattr_cbk [global.callpool.stack.115.frame.2] frame=0x7f8b30a14da0 ref_count=1 translator=cl25vol01-quota complete=0 parent=cl25vol01-io-stats wind_from=io_stats_getxattr wind_to=(this->children->xlator)->fops->getxattr unwind_to=io_stats_getxattr_cbk [global.callpool.stack.115.frame.3] frame=0x7f8b6debada0 ref_count=1 translator=cl25vol01-io-stats complete=0 parent=cl25vol01-server wind_from=server_getxattr_resume wind_to=FIRST_CHILD(this)->fops->getxattr unwind_to=server_getxattr_cbk [global.callpool.stack.115.frame.4] frame=0x7f8b21962a60 ref_count=1 translator=cl25vol01-server complete=0 I've checked the code logic and got nothing, any advice? I still have the scene on my side, so we can dig more. Thanks ___ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/836554017 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/486278655 Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Questions] why shard feature doesn't support quota
Hi Krutika Thanks for your quick response! 在 6/29/2017 3:24 PM, Krutika Dhananjay 写道: This is based on my high-level understanding of quota in gluster - supporting quota on sharded volumes will require us to make quota xlator account for shards residing under the hidden ".shard" directory per file and adding this to the quota-xattr representing aggregated consumed size on the parent directory of the original file. This isn't hard to do. But at the moment we support sharding only for VM image store use case. Why sharding only support VM large images? It's interesting. More, are there other limits for sharding usage? We want to reduce data reconstruction time with sharding, as you know it takes too long time to recover data with raid, and sharding is a good choice to us. There are plans to make shard useful for more generic use cases in the near term. When we get there, we will support quota too on sharded files. That's a *good* news. BTW we also have a plan to implement this part. Any suggestions? -Krutika -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] [Questions] why shard feature doesn't support quota
Hi all I did a small test on gluster 3.8.4, and found shard feature does not support quota. Here is the step: $ gluster volume set s1 features.shard on $ gluster volume quota s1 enable $ gluster volume quota s1 limit-usage /ttt 1GB $ gluster volume quota s1 list $ cd /mnt/s1/ttt; fallocate -l 2GB testfile Then, i found testfile created sucessully. So, It seems shard doesn't support quota. It's a technical issue or there is a ongoing plan? -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Qusetions] How profile affect glusterfs performance?
在 6/7/2017 4:47 PM, Pranith Kumar Karampuri 写道: Yes, they do. The patches I gave earlier provide the facility to this in io-stats. Do let us know if you have any doubts. Sorry for slowness, but i think i catch up with U now. -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Qusetions] How profile affect glusterfs performance?
在 6/7/2017 3:08 PM, Pranith Kumar Karampuri 写道: On Tue, Jun 6, 2017 at 8:14 AM, Xie Changlong <xiechanglon...@gmail.com> wrote: 在 6/5/2017 6:30 PM, Pranith Kumar Karampuri 写道: I meant what are you using gluster for? VM workload or image/video file creation or lots of small files etc etc. 1) We use glusterfs for general purpose, no limit to image/video file creation or small files etc. Okay, this is good. What is the cluster size? Is it replica 3 or replica 2 or arbiter or is it EC volume? I use replica 2 on my test. But in the real world, "we have deployed more than 100 glusterfs nodes in producion(20 nodes for the biggest single cluster)" per Liu Yuan. Replica 2/3, or EC are optional. Please don't mind me asking so many details. I am delighted to see you guys You are always welcome! active in the community because I have seen Xiubo Li's work on tcmu-runner who is also from chinamobile and his work is pretty good :-). Yeah, i would be very pleasant to convey your praise. :) 2) We just want a low performance affect way to calculate each brick's iops/bandwidth for the upper layer management app. BTW, is there any other way to get iops/bandwith for each brick besides profile? At the moment we have these stats using profile commands. As per Facebook patches they enable json capture of io-stats on their volumes and measure these. They have it enabled always. If that is not enough for you, then we I think you mean FB guys gather data based on io-stats with profile always on, am i right? BTW, are these patches open to everyone? so i can dig into it. should probably look at enhancing things. Do send patches if you think something would make things better :-). -- Thanks -Xie -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Qusetions] How profile affect glusterfs performance?
在 6/5/2017 6:30 PM, Pranith Kumar Karampuri 写道: I meant what are you using gluster for? VM workload or image/video file creation or lots of small files etc etc. 1) We use glusterfs for general purpose, no limit to image/video file creation or small files etc. 2) We just want a low performance affect way to calculate each brick's iops/bandwidth for the upper layer management app. BTW, is there any other way to get iops/bandwith for each brick besides profile? -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Qusetions] How profile affect glusterfs performance?
在 6/5/2017 3:10 PM, Xie Changlong 写道: 在 6/5/2017 2:34 PM, Pranith Kumar Karampuri 写道: What is your usecase if I may ask? Only fio/iozone now, maybe you can give me better test sugguestion :) seq read # dd if=/dev/zero of=test bs=4k count=524288 #fio --filename=test -iodepth=64 -ioengine=libaio --direct=1 --rw=read --bs=1m --size=2g --numjobs=4 --runtime=10 --group_reporting --name=test-read seq write # fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=1m -size=2g -numjobs=4 -runtime=20 -group_reporting -name=test-write random read #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=randread -bs=4k -size=2G -numjobs=64 -runtime=20 -group_reporting -name=test-rand-read random write #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=randwrite -bs=4k -size=2G -numjobs=64 -runtime=20 -group_reporting -name=test-rand-write iozone 1 thread # ./iozone -s 10g -r 1m -i 0 -i 1 -f /mnt/test-volume/test -Rb /tmp/test_iozone_1.xls iozone 2 multi-threads # ./iozone -s 10g -r 1m -i 0 -i 1 -f /mnt/test-volume/test -t 8 -Rb /tmp/test_iozone_2.xls Re-declare, For fio, i compare "bw" and "iops" fileds. For iozone, i compare "write" "rewrite" "read" "re-read" fileds. On Mon, Jun 5, 2017 at 11:08 AM, Xie Changlong<xiechanglon...@gmail.com> wrote: 在 6/5/2017 1:22 PM, Pranith Kumar Karampuri 写道: On Mon, Jun 5, 2017 at 8:29 AM, Xie Changlong<xiechanglon...@gmail.com> wrote: 在 6/5/2017 10:50 AM, Pranith Kumar Karampuri 写道: I think there have been improvements here to use special instructions to -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Qusetions] How profile affect glusterfs performance?
在 6/5/2017 2:34 PM, Pranith Kumar Karampuri 写道: What is your usecase if I may ask? Only fio/iozone now, maybe you can give me better test sugguestion :) seq read # dd if=/dev/zero of=test bs=4k count=524288 #fio --filename=test -iodepth=64 -ioengine=libaio --direct=1 --rw=read --bs=1m --size=2g --numjobs=4 --runtime=10 --group_reporting --name=test-read seq write # fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=1m -size=2g -numjobs=4 -runtime=20 -group_reporting -name=test-write random read #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=randread -bs=4k -size=2G -numjobs=64 -runtime=20 -group_reporting -name=test-rand-read random write #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=randwrite -bs=4k -size=2G -numjobs=64 -runtime=20 -group_reporting -name=test-rand-write iozone 1 thread # ./iozone -s 10g -r 1m -i 0 -i 1 -f /mnt/test-volume/test -Rb /tmp/test_iozone_1.xls iozone 2 multi-threads # ./iozone -s 10g -r 1m -i 0 -i 1 -f /mnt/test-volume/test -t 8 -Rb /tmp/test_iozone_2.xls On Mon, Jun 5, 2017 at 11:08 AM, Xie Changlong<xiechanglon...@gmail.com> wrote: 在 6/5/2017 1:22 PM, Pranith Kumar Karampuri 写道: On Mon, Jun 5, 2017 at 8:29 AM, Xie Changlong<xiechanglon...@gmail.com> wrote: 在 6/5/2017 10:50 AM, Pranith Kumar Karampuri 写道: I think there have been improvements here to use special instructions to -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Qusetions] How profile affect glusterfs performance?
在 6/5/2017 1:22 PM, Pranith Kumar Karampuri 写道: On Mon, Jun 5, 2017 at 8:29 AM, Xie Changlong <xiechanglon...@gmail.com> wrote: 在 6/5/2017 10:50 AM, Pranith Kumar Karampuri 写道: I think there have been improvements here to use special instructions to if "special instructions" means profile command or more? Also can you point which commit? I meant Intel CPU extentions You should look at http://review.gluster.org/12209 http://review.gluster.org/12210 https://review.gluster.org/16963 https://review.gluster.org/17009 Useful enough for me, thanks for your patience. do the increments instead of taking spin-locks and doing increments. So may be it doesn't affect performance as much anymore. I think if you don't see a difference, then the enhancements are doing a good job :-). Which version of gluster are you using? Our version is gluster 3.8.4 -- Thanks -Xie -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Qusetions] How profile affect glusterfs performance?
在 6/5/2017 10:50 AM, Pranith Kumar Karampuri 写道: I think there have been improvements here to use special instructions to if "special instructions" means profile command or more? Also can you point which commit? do the increments instead of taking spin-locks and doing increments. So may be it doesn't affect performance as much anymore. I think if you don't see a difference, then the enhancements are doing a good job :-). Which version of gluster are you using? Our version is gluster 3.8.4 -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] [Qusetions] How profile affect glusterfs performance?
Hi all It's said[1] that profile based on io-stats, if you enable this feature, it can affect system performance while the profile information is being collected. I do some tests on my two linux+vmware virtual machine with replica(lack of resources ). And the results shows no diffrence to me, following is the test case #dd if=/dev/zero of=test bs=4k count=524288 #fio --filename=test -iodepth=64 -ioengine=libaio --direct=1 --rw=read --bs=1m --size=2g --numjobs=4 --runtime=10 --group_reporting --name=test-read #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=1m -size=2g -numjobs=4 -runtime=20 -group_reporting -name=test-write #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=randread -bs=4k -size=2G -numjobs=64 -runtime=20 -group_reporting -name=test-rand-read #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=randwrite -bs=4k -size=2G -numjobs=64 -runtime=20 -group_reporting -name=test-rand-write It's said that fio is only for lagre files, also i suspect that the test infrastructure is too small. The question is that, if you guys have detailed data for how profile affect performance? More, we wanna gain the detail r/w iops/bandwidth data for each brick. It seems that only profile can provide relatived data to be calculated?if i'm wrong pls corrent me. If profile really affect peformance so much, would you mind a new command such as "gluster volume io [nfs]" to acquire brick r/w fops/data? Or just help us review it? [1] https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/chap-monitoring_red_hat_storage_workload#sect-Running_the_Volume_Profile_Command -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] [Qusetions] How profile influnce glusterfs performance?
Hi all It's said[1] that profile based on io-stats, if you enable this feature, it can affect system performance while the profile information is being collected. I do some tests on my two linux+vmware virtual machine with replica(lack of resources :( ). And the results shows no diffrence to me, following is the test case #dd if=/dev/zero of=test bs=4k count=524288 #fio --filename=test -iodepth=64 -ioengine=libaio --direct=1 --rw=read --bs=1m --size=2g --numjobs=4 --runtime=10 --group_reporting --name=test-read #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=write -bs=1m -size=2g -numjobs=4 -runtime=20 -group_reporting -name=test-write #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=randread -bs=4k -size=2G -numjobs=64 -runtime=20 -group_reporting -name=test-rand-read #fio -filename=test -iodepth=64 -ioengine=libaio -direct=1 -rw=randwrite -bs=4k -size=2G -numjobs=64 -runtime=20 -group_reporting -name=test-rand-write It's said that fio is only for lagre files, also i suspect that the test infrastructure is too small. The question is that, if you guys have detailed data for how profile affect performance? More, we wanna gain the detail r/w iops/bandwidth data for each brick. It seems that only profile can provide relatived data to be calculated? if i'm wrong pls corrent me. If profile really affect peformance so much, would you mind a new command such as "gluster volume io [nfs]" to acquire brick r/w fops/data? Or just help us review it? [1] https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.2/html/administration_guide/chap-monitoring_red_hat_storage_workload#sect-Running_the_Volume_Profile_Command -- Thanks -Xie ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
