Re: [Gluster-devel] Release 3.10 testing update: Older client unable to mount post upgrade
I think this is because of the commit 96fb3562. In this 3 options, which were already in readdir-ahead but not exposed as volume set, were exposed as volume set options(allowed to be set with op-version GD_OP_VERSION_3_9_1): rda-request-size, rda-low-wmark, rda-cache-limit, rda-high-wmark Along with this, the type of rda-request-size was changed from INT to SIZE_t. I think the new volfiles contain 'rda-request-size 128KB' but the old client can only understand INT. This cannot be documented and left, it has to be fixed. One way would be to retain the type of rda-request-size as INT, but may be there is a better way need to check. Regards, Poornima - Original Message - > From: "Shyam" > To: "Gluster Devel" , "Raghavendra Gowdappa" > , "Poornima > Gurusiddaiah" , "Atin Mukherjee" > Sent: Friday, February 17, 2017 5:36:07 AM > Subject: Release 3.10 testing update: Older client unable to mount post > upgrade > > Hi, > > Post upgrading brick nodes from 3.8.8 to 3.10rc0 (sort of), I tried a to > mount from a 3.8.8 client (as I had not upgraded the client till then). > The mount failed with the following in the logs (at the end of the mail). > > The issue was that I did an rpm -U to get the latest version, so all vol > files were upgraded (from the spec file, glusterd --xlator-option > *.upgrade=on -N), this resulted in client vol files being upgraded as > well, and now the client does not understand the option as below, > > 0-testvolssd-readdir-ahead: invalid number format "128KB" in option > "rda-request-size" > 0-testvolssd-readdir-ahead: validate of rda-request-size returned -1 > 0-testvolssd-readdir-ahead: validation failed: invalid number format > "128KB" in option "rda-request-size" > > So is this a bug? Or something to document? or, am I missing something? > Poornima/Du/Atin? > > Thanks, > Shyam > > ::Full logs:: > # mount -t glusterfs 10.21.76.10:/testvolssd /mnt/testvol1 > Mount failed. Please check the log file for more details. > > # cat /var/log/glusterfs/mnt-testvol1.log > [2017-02-16 17:37:12.703983] I [MSGID: 100030] [glusterfsd.c:2454:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.8 > (args: /usr/sbin/glusterfs --volfile-server=10.21.76.10 > --volfile-id=/testvolssd /mnt/testvol1) > [2017-02-16 17:37:12.726213] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [2017-02-16 17:37:12.744187] E [MSGID: 101002] > [options.c:69:xlator_option_validate_int] 0-testvolssd-readdir-ahead: > invalid number format "128KB" in option "rda-request-size" > [2017-02-16 17:37:12.744221] W [MSGID: 101029] > [options.c:945:xl_opt_validate] 0-testvolssd-readdir-ahead: validate of > rda-request-size returned -1 > [2017-02-16 17:37:12.744235] E [MSGID: 101090] > [graph.c:301:glusterfs_graph_validate_options] > 0-testvolssd-readdir-ahead: validation failed: invalid number format > "128KB" in option "rda-request-size" > [2017-02-16 17:37:12.744248] E [MSGID: 101090] > [graph.c:665:glusterfs_graph_activate] 0-graph: validate options failed > [2017-02-16 17:37:12.753368] W [glusterfsd.c:1327:cleanup_and_exit] > (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3c1) [0x7fbf4eb38eb1] > -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x172) [0x7fbf4eb335d2] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbf4eb32b4b] ) 0-: > received signum (1), shutting down > [2017-02-16 17:37:12.753420] I [fuse-bridge.c:5794:fini] 0-fuse: > Unmounting '/mnt/testvol1'. > [2017-02-16 17:37:12.753957] W [glusterfsd.c:1327:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fbf4d49edc5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fbf4eb32cd5] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbf4eb32b4b] ) 0-: > received signum (15), shutting down > ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Release 3.10 testing update: Older client unable to mount post upgrade
This is definitely a bug and we have broken the older client support. [1] is the one which has caused it. We have changed the type of rda-request-size from int to size_t because of which the older clients are unable to understand the KB format. I remember when this patch was in its implementation this change had to be done to avoid a crash [2] . We'd have to revert back the data type to int to ensure backward compatibility, now how do we solve the crash which we were seeing earlier is something to be looked at. [1] https://review.gluster.org/#/c/16137/ [2] https://build.gluster.org/job/centos6-regression/2388/consoleFull On Fri, Feb 17, 2017 at 5:36 AM, Shyam wrote: > Hi, > > Post upgrading brick nodes from 3.8.8 to 3.10rc0 (sort of), I tried a to > mount from a 3.8.8 client (as I had not upgraded the client till then). The > mount failed with the following in the logs (at the end of the mail). > > The issue was that I did an rpm -U to get the latest version, so all vol > files were upgraded (from the spec file, glusterd --xlator-option > *.upgrade=on -N), this resulted in client vol files being upgraded as well, > and now the client does not understand the option as below, > > 0-testvolssd-readdir-ahead: invalid number format "128KB" in option > "rda-request-size" > 0-testvolssd-readdir-ahead: validate of rda-request-size returned -1 > 0-testvolssd-readdir-ahead: validation failed: invalid number format > "128KB" in option "rda-request-size" > > So is this a bug? Or something to document? or, am I missing something? > Poornima/Du/Atin? > > Thanks, > Shyam > > ::Full logs:: > # mount -t glusterfs 10.21.76.10:/testvolssd /mnt/testvol1 > Mount failed. Please check the log file for more details. > > # cat /var/log/glusterfs/mnt-testvol1.log > [2017-02-16 17:37:12.703983] I [MSGID: 100030] [glusterfsd.c:2454:main] > 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.8 > (args: /usr/sbin/glusterfs --volfile-server=10.21.76.10 > --volfile-id=/testvolssd /mnt/testvol1) > [2017-02-16 17:37:12.726213] I [MSGID: 101190] > [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread > with index 1 > [2017-02-16 17:37:12.744187] E [MSGID: 101002] > [options.c:69:xlator_option_validate_int] 0-testvolssd-readdir-ahead: > invalid number format "128KB" in option "rda-request-size" > [2017-02-16 17:37:12.744221] W [MSGID: 101029] > [options.c:945:xl_opt_validate] 0-testvolssd-readdir-ahead: validate of > rda-request-size returned -1 > [2017-02-16 17:37:12.744235] E [MSGID: 101090] > [graph.c:301:glusterfs_graph_validate_options] > 0-testvolssd-readdir-ahead: validation failed: invalid number format > "128KB" in option "rda-request-size" > [2017-02-16 17:37:12.744248] E [MSGID: 101090] > [graph.c:665:glusterfs_graph_activate] 0-graph: validate options failed > [2017-02-16 17:37:12.753368] W [glusterfsd.c:1327:cleanup_and_exit] > (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3c1) [0x7fbf4eb38eb1] > -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x172) [0x7fbf4eb335d2] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbf4eb32b4b] ) 0-: > received signum (1), shutting down > [2017-02-16 17:37:12.753420] I [fuse-bridge.c:5794:fini] 0-fuse: > Unmounting '/mnt/testvol1'. > [2017-02-16 17:37:12.753957] W [glusterfsd.c:1327:cleanup_and_exit] > (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fbf4d49edc5] > -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fbf4eb32cd5] > -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbf4eb32b4b] ) 0-: > received signum (15), shutting down > -- ~ Atin (atinm) ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] 3.10 testing update: Deadlock in gf_timer invocation from a signal handler and possible inode ctx corruption issue in AFR
Pranith, Ravi, While testing 3.10 I faced a hung FUSE mount, and on further debugging, it looks like there are 2 problems here, 1) A deadlock in gf_timer locks, when invoked from a fault signal handler 2) The fault itself being caused due to AFR inode ctx pointer being invalid, and hence a call to gf_timer_call_cancel causes the segfault. More details are captured here, https://bugzilla.redhat.com/show_bug.cgi?id=1423065 Request your attention on this for 3.10. Thanks, Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] 3.10 testing update: readdir-ahead is noisy in the logs
Poornima/Du, I encountered a lot of logs from readdir-ahead, when running a ls -l performance benchmark test as follows, [2017-02-16 18:57:10.358205] E [MSGID: 129006] [readdir-ahead.c:576:rda_opendir] 0-testvolssd-readdir-ahead: Dict get of key:readdir-filter-directories failed with :-2 The message "E [MSGID: 129006] [readdir-ahead.c:576:rda_opendir] 0-testvolssd-readdir-ahead: Dict get of key:readdir-filter-directories failed with :-2" repeated 8 times between [2017-02-16 18:57:10.358205] and [2017-02-16 18:59:02.529601] This was introduced with, commit c1cafe6e314f01d3f07229c0972af5f1017c62cf Author: Poornima G Date: Thu Dec 8 16:08:40 2016 +0530 Readdir-ahead : Honour readdir-optimise option of dht Do we need this at ERROR level, or can this be made DEBUG/TRACE? If this needs a change, request a bug and the whole nine yards for the same. Thanks, Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Release 3.10 testing update: glusterfs-fuse RPM now depends on gfapi?
On 02/16/2017 07:23 PM, Jeff Darcy wrote: It's *primarily* a test tool. It could also be useful to sysadmins IMO, hence its inclusion, but I guess I could live with it being left out. Oh! we can add it no problem, I thought it was a left over by accident. It might make more sense for gf_attach to be packaged as part of glusterfs-server, since it communicates with server processes. Would that help at all? It probably still requires some specfile changes, but shouldn't introduce a new dependency since lots of server stuff already depends on GFAPI. Yup, glusterfs-server already has this dependency, so we can add it to that. I filed this [1] bug Jeff, in case you get to it first, fine, else one of us will pick up the fix. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1423063 ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Release 3.10 testing update: glusterfs-fuse RPM now depends on gfapi?
> On an upgrade test, when upgrading clients, glusterfs-fuse RPM now > needed libgfapi, this is due to gf_attach being packaged as a part of > glusterfs-fuse. > > Jeff, we do not need to package gf_attach, right? This is a test tool, > if I am correct. It's *primarily* a test tool. It could also be useful to sysadmins IMO, hence its inclusion, but I guess I could live with it being left out. > Niels, you did add this to .gitignore Samikshan did. > but looks like it is getting > packaged into the glusterfs-fuse RPM, guess the Makefile needs some > modifications for this to be ignored? > > It is built from here: ./glusterfsd/src/Makefile.am > > Let me know if this is the case, I can file a bug and we can take this > forward. It might make more sense for gf_attach to be packaged as part of glusterfs-server, since it communicates with server processes. Would that help at all? It probably still requires some specfile changes, but shouldn't introduce a new dependency since lots of server stuff already depends on GFAPI. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Release 3.10 testing update: glusterfs-fuse RPM now depends on gfapi?
On an upgrade test, when upgrading clients, glusterfs-fuse RPM now needed libgfapi, this is due to gf_attach being packaged as a part of glusterfs-fuse. Jeff, we do not need to package gf_attach, right? This is a test tool, if I am correct. Niels, you did add this to .gitignore, but looks like it is getting packaged into the glusterfs-fuse RPM, guess the Makefile needs some modifications for this to be ignored? It is built from here: ./glusterfsd/src/Makefile.am Let me know if this is the case, I can file a bug and we can take this forward. Thanks, Shyam ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Release 3.10 testing update: Older client unable to mount post upgrade
Hi, Post upgrading brick nodes from 3.8.8 to 3.10rc0 (sort of), I tried a to mount from a 3.8.8 client (as I had not upgraded the client till then). The mount failed with the following in the logs (at the end of the mail). The issue was that I did an rpm -U to get the latest version, so all vol files were upgraded (from the spec file, glusterd --xlator-option *.upgrade=on -N), this resulted in client vol files being upgraded as well, and now the client does not understand the option as below, 0-testvolssd-readdir-ahead: invalid number format "128KB" in option "rda-request-size" 0-testvolssd-readdir-ahead: validate of rda-request-size returned -1 0-testvolssd-readdir-ahead: validation failed: invalid number format "128KB" in option "rda-request-size" So is this a bug? Or something to document? or, am I missing something? Poornima/Du/Atin? Thanks, Shyam ::Full logs:: # mount -t glusterfs 10.21.76.10:/testvolssd /mnt/testvol1 Mount failed. Please check the log file for more details. # cat /var/log/glusterfs/mnt-testvol1.log [2017-02-16 17:37:12.703983] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.8 (args: /usr/sbin/glusterfs --volfile-server=10.21.76.10 --volfile-id=/testvolssd /mnt/testvol1) [2017-02-16 17:37:12.726213] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-02-16 17:37:12.744187] E [MSGID: 101002] [options.c:69:xlator_option_validate_int] 0-testvolssd-readdir-ahead: invalid number format "128KB" in option "rda-request-size" [2017-02-16 17:37:12.744221] W [MSGID: 101029] [options.c:945:xl_opt_validate] 0-testvolssd-readdir-ahead: validate of rda-request-size returned -1 [2017-02-16 17:37:12.744235] E [MSGID: 101090] [graph.c:301:glusterfs_graph_validate_options] 0-testvolssd-readdir-ahead: validation failed: invalid number format "128KB" in option "rda-request-size" [2017-02-16 17:37:12.744248] E [MSGID: 101090] [graph.c:665:glusterfs_graph_activate] 0-graph: validate options failed [2017-02-16 17:37:12.753368] W [glusterfsd.c:1327:cleanup_and_exit] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3c1) [0x7fbf4eb38eb1] -->/usr/sbin/glusterfs(glusterfs_process_volfp+0x172) [0x7fbf4eb335d2] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbf4eb32b4b] ) 0-: received signum (1), shutting down [2017-02-16 17:37:12.753420] I [fuse-bridge.c:5794:fini] 0-fuse: Unmounting '/mnt/testvol1'. [2017-02-16 17:37:12.753957] W [glusterfsd.c:1327:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fbf4d49edc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fbf4eb32cd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fbf4eb32b4b] ) 0-: received signum (15), shutting down ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Logging in a multi-brick daemon
On Feb 15, 2017 5:39 PM, "Jeff Darcy" wrote: One of the issues that has come up with multiplexing is that all of the bricks in a process end up sharing a single log file. The reaction from both of the people who have mentioned this is that we should find a way to give each brick its own log even when they're in the same process, and make sure gf_log etc. are able to direct messages to the correct one. I can think of ways to do this, but it doesn't seem optimal to me. It will certainly use up a lot of file descriptors. I think it will use more memory. And then there's the issue of whether this would really be better for debugging. Often it's necessary to look at multiple brick logs while trying to diagnose this problem, so it's actually kind of handy to have them all in one file. Which would you rather do? (a) Weave together entries in multiple logs, either via a script or in your head? (b) Split or filter entries in a single log, according to which brick they're from? +1 for a single log file with tagging, combined with necessary grep-fu. Plus I like the idea of an included script or other facility to aid said grepping. To me, (b) seems like a much more tractable problem. I'd say that what we need is not multiple logs, but *marking of entries* so that everything pertaining to one brick can easily be found. One way to do this would be to modify volgen so that a brick ID (not name because that's a path and hence too long) is appended/prepended to the name of every translator in the brick. Grep for that brick ID, and voila! You now have all log messages for that brick and no other. A variant of this would be to leave the names alone and modify gf_log so that it adds the brick ID automagically (based on a thread-local variable similar to THIS). Same effect, other than making translator names longer, so I'd kind of prefer this approach. Before I start writing the code, does anybody else have any opinions, preferences, or alternatives I haven't mentioned yet? ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Reviews needed
On Thu, Feb 16, 2017 at 3:43 AM, Xavier Hernandez wrote: > Hi everyone, > > I would need some reviews if you have some time: > > A memory leak fix in fuse: > * Patch already merged in master and 3.10 > * Backport to 3.9: https://review.gluster.org/16402 > * Backport to 3.8: https://review.gluster.org/16403 > > A safe fallback for dynamic code generation in EC: > * Master: https://review.gluster.org/16614 > > A fix for incompatibilities with FreeBSD: > * Master: https://review.gluster.org/16417 > > A fix for FreeBSD's statvfs(): > * Patch already merged in master > * Backport to 3.10: https://review.gluster.org/16631 > * Backport to 3.9: https://review.gluster.org/16632 > * Backport to 3.8: https://review.gluster.org/16634 > > I also have two reviews for 3.7 but I think it won't have any new > releases, right ? > > That is right. We can also avoid additional backports to 3.9 as it is going to be EOL once 3.10 is out. Thanks, Vijay ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Mixing style and other changes in a patch
I may not be a code contributor, but I do tend to read the code a lot to figure out how things work out to track down a bug. I find that format changes that are not related to code changes within the same commit just make the whole commit more complex and harder to read. My preference would be that formatting fixes to existing code should be a separate commit. On February 16, 2017 6:45:28 AM PST, Niels de Vos wrote: >On Thu, Feb 16, 2017 at 08:41:23AM -0500, Jeff Darcy wrote: >> In the last few days, I've seen both of these kinds of review >comments >> (not necessarily on my own patches or from the same reviewers). >> >> (a) "Please fix the style in the entire function where you changed >one line." >> >> (b) "This style change should be in a separate patch." >> >> It's clearly not helpful to have patches delayed for both reasons. >> Which should prevail? I think our general practice has been more >> toward (b) and that's also my personal preference. In that case >> instances of (a) should not occur. Or maybe people feel it should be >> the other way around. Can we get a consensus here? > >I do not like to have patches change the coding style alone. Only for >the lines at the beginning of the function I prefer to see (a) being >followed, but blocking a patch should not be the case. Still, if some >of >the regular contributors do not follow the coding-style, I tend to get >annoyed with it and my give them a -1 in the hope they do repeat that >in >the future. If it is really the only thing that bothers me, I tend to >send an updated patch for them (after a chat on IRC). > >Doing coding-style fixes on lines that have no other modifications tend >to make 'git blame' more difficult to follow. I use that a lot when >trying to figure out when a problem has been introduced. So in general, >I really do not like (b). > >Niels -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Release 3.10: Request fix status for RC1 tagging
On 02/16/2017 03:35 AM, Xavier Hernandez wrote: - 1421649: Ashis/Niels when can we expect a fix to land for this? I think this will require more thinking and participation from experts on security and selinux to come up with a good and clean solution. Not sure if this can be done before 3.10 release. There is a workaround (set disperse.cpu-extensions = none) and a mitigating solution (patch https://review.gluster.org/16614) though. Acknowledged, tracker updated accordingly. - 1421956: Xavi, I guess you would backport the fix on mainline once that is merged into 3.10, right? Yes. It's on review on master (https://review.gluster.org/16614). As soon as it's merged I'll backport it to 3.10. Pranith/Anoop/Ashish, can one of you finish the review of the above patch on master please, so that we can get the backport going! ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Release 3.10: Request fix status for RC1 tagging
On 02/16/2017 05:07 AM, Niels de Vos wrote: On Wed, Feb 15, 2017 at 08:47:20PM -0500, Shyam wrote: Current bug list [2]: - 1415226: Kaleb/Niels do we need to do more for the python dependency or is the last fix in? There was an email where these python2 changes were causing problems with upgrading. As far as I know this has not been fixed yet. I need to dig through emails to see who reported it (and where?) and who was working on getting it resolved. Updated the same bug with email contents and assigned it to you Niels. I would assume it is either you or Kaleb who addresses this, let me know otherwise. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Mixing style and other changes in a patch
On Thu, Feb 16, 2017 at 08:41:23AM -0500, Jeff Darcy wrote: > In the last few days, I've seen both of these kinds of review comments > (not necessarily on my own patches or from the same reviewers). > > (a) "Please fix the style in the entire function where you changed one line." > > (b) "This style change should be in a separate patch." > > It's clearly not helpful to have patches delayed for both reasons. > Which should prevail? I think our general practice has been more > toward (b) and that's also my personal preference. In that case > instances of (a) should not occur. Or maybe people feel it should be > the other way around. Can we get a consensus here? I do not like to have patches change the coding style alone. Only for the lines at the beginning of the function I prefer to see (a) being followed, but blocking a patch should not be the case. Still, if some of the regular contributors do not follow the coding-style, I tend to get annoyed with it and my give them a -1 in the hope they do repeat that in the future. If it is really the only thing that bothers me, I tend to send an updated patch for them (after a chat on IRC). Doing coding-style fixes on lines that have no other modifications tend to make 'git blame' more difficult to follow. I use that a lot when trying to figure out when a problem has been introduced. So in general, I really do not like (b). Niels signature.asc Description: PGP signature ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Logging in a multi-brick daemon
On 02/16/2017 05:27 AM, Rajesh Joseph wrote: On Thu, Feb 16, 2017 at 9:46 AM, Ravishankar N wrote: On 02/16/2017 04:09 AM, Jeff Darcy wrote: One of the issues that has come up with multiplexing is that all of the bricks in a process end up sharing a single log file. The reaction from both of the people who have mentioned this is that we should find a way to give each brick its own log even when they're in the same process, and make sure gf_log etc. are able to direct messages to the correct one. I can think of ways to do this, but it doesn't seem optimal to me. It will certainly use up a lot of file descriptors. I think it will use more memory. And then there's the issue of whether this would really be better for debugging. Often it's necessary to look at multiple brick logs while trying to diagnose this problem, so it's actually kind of handy to have them all in one file. Which would you rather do? (a) Weave together entries in multiple logs, either via a script or in your head? (b) Split or filter entries in a single log, according to which brick they're from? To me, (b) seems like a much more tractable problem. I'd say that what we need is not multiple logs, but *marking of entries* so that everything pertaining to one brick can easily be found. One way to do this would be to modify volgen so that a brick ID (not name because that's a path and hence too long) is appended/prepended to the name of every translator in the brick. Grep for that brick ID, and voila! You now have all log messages for that brick and no other. A variant of this would be to leave the names alone and modify gf_log so that it adds the brick ID automagically (based on a thread-local variable similar to THIS). Same effect, other than making translator names longer, so I'd kind of prefer this approach. Before I start writing the code, does anybody else have any opinions, preferences, or alternatives I haven't mentioned yet? ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel My vote is for having separate log files per brick. Even in separate log files that we have today, I find it difficult to mentally ignore irrelevant messages in a single log file as I am sifting through it to look for errors that are related to the problem at hand. Having entries from multiple bricks and then grepping it would only make things harder. I cannot think of a case where having entries from all bricks in one file would particularly be beneficial for debugging since what happens in one brick is independent of the other bricks (at least until we move client xlators to server side and run them in the brick process). As for file descriptor count/memory usage, I think we should be okay as it is not any worse than that in the non-multiplexed approach we have today. On a side-note, I think the problem is not having too many log files but having them in multiple nodes. Having a log-aggregation solution where all messages are logged to a single machine (but still in separate files) would make it easier to monitor/debug issues. -Ravi I believe the logs are not just from one volume but from all. In that case merging them into a single log file may not be great for debugging. Especially in container use cases there can be multiple volumes. Yes, with some tagging and scripting we can separate the logs and still live with it. In container world, *I believe* centralized logging (using something like an ELK/EFK stack) would be the way to go, than collecting logs from each gluster (or application mount) container/node. In these situations we are going to get logs from different volumes anyway, or at best a filtered list from whichever stack is used for the centralized logging. So, I would think (as being described) we would need to have enough identifiers in the log message, such that we can filter appropriately and that should take care of the debugging concern. Of course building these scripts out from the beginning and possibly even shipping them with our RPMs may help a great deal, than having to roll one out when we get into troubleshooting or debugging a setup. What about the log levels? Each volume can configure different log levels. Will you carve out a separate process in case log levels are changed for a volume? How is this handled here? -Rajesh ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Mixing style and other changes in a patch
In the last few days, I've seen both of these kinds of review comments (not necessarily on my own patches or from the same reviewers). (a) "Please fix the style in the entire function where you changed one line." (b) "This style change should be in a separate patch." It's clearly not helpful to have patches delayed for both reasons. Which should prevail? I think our general practice has been more toward (b) and that's also my personal preference. In that case instances of (a) should not occur. Or maybe people feel it should be the other way around. Can we get a consensus here? ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Logging in a multi-brick daemon
> Debugging will involve getting far more/bigger files from customers
> unless we have a script (?) to grep out only those messages pertaining
> to the volume in question. IIUC, this would just be grepping for the
> volname and then determining which brick each message pertains to
> based on the brick id, correct?
Correct. There would also be some possibly-interesting messages that
aren't specifically tied to any one brick, e.g. in protocol/server or
various parts of libglusterfs, so we'd probably always want those no
matter what brick(s) we're interested in.
> Would brick ids remain constant across add/remove brick operations? An
> easy way would probably be just to use the client xlator number as the
> brick id which would make it easy to map the brick to client
> connection.
Brick IDs should be constant across add/remove operations, which I
suppose means they'll need to be more globally unique than they are now
(client translator indices can clash across volumes).
> With several brick processes all writing to the same log file, can
> there be problems with interleaving messages?
AFAICT the "atom" of logging is a line, so there shouldn't be problems
of interleaving within a line ("foo" + "bar" won't become "fboaro").
However, when code tries to log multiple lines together - e.g. DHT
layouts or AFR self-heal info - that could end up being interleaved with
another brick doing the same. They'd still be distinct according to
brick ID, but when looking at an unfiltered log it could look a bit
confusing.
> Logrotate might kick in faster as well causing us to lose debugging
> data if only a limited number of files are saved, as all those files
> would now hold less log data per volume. The logrotate config options
> would need to be changed to keep more files.
True.
> Having all messages for the bricks of the same volume in a single file
> would definitely be helpful. Still thinking through logging all
> messages for all bricks in a single file. :)
Something to keep in mind is that multiplexing everything into only one
process is only temporary. Very soon, we'll be multiplexing into
multiple processes, with the number of processes proportional to the
number of cores in the system. So, for a node with 200 bricks and 24
cores, we might have 24 processes each containing ~8 bricks. In that
case, it would make sense to keep bricks for the same volume separate as
much as possible. A process is a failure domain, and having multiple
related bricks in the same failure domain is undesirable (though
unavoidable in some cases). The consequence for logging is that you'd
still have to look at multiple files.
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Logging in a multi-brick daemon
> What about the log levels? Each volume can configure different log > levels. Will you carve > out a separate process in case log levels are changed for a volume? I don't think we need to go that far, but you do raise a good point. Log levels would need to be fetched from a brick-specific location (probably similar to what I suggested for the brick ID) instead of from the shared context. Thanks for pointing that out. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Logging in a multi-brick daemon
> As for file descriptor count/memory usage, I think we should be okay > as it is not any worse than that in the non-multiplexed approach we > have today. I don't think that's true. Each logging context allocates a certain amount of memory. Let's call that X. With N bricks in separate processes, our total memory is N*X. With N bricks in one process, each using its own logging context, we're still at N*X. However, with N bricks in one process sharing a context, we'd be down to X. Even more importantly, with separate contexts we'd be touching N pages all the time, increasing the amount of *physical* memory that is devoted to logging. With respect to file descriptors, having N file descriptors *in one process* is more problematic than having one per process in N processes. We already have approximately N times as many file descriptors per process for client connections; adding another N for logging descriptors isn't going to be helpful when N might be in the hundreds. These might not be big enough effects to rule out the separate-log approach, but they're not zero either. Since reducing memory usage is a primary goal of multiplexing, I don't think we should lightly forego opportunities to do so. ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Logging in a multi-brick daemon
On Thu, Feb 16, 2017 at 9:46 AM, Ravishankar N wrote: > On 02/16/2017 04:09 AM, Jeff Darcy wrote: >> >> One of the issues that has come up with multiplexing is that all of the >> bricks in a process end up sharing a single log file. The reaction from >> both of the people who have mentioned this is that we should find a way to >> give each brick its own log even when they're in the same process, and make >> sure gf_log etc. are able to direct messages to the correct one. I can >> think of ways to do this, but it doesn't seem optimal to me. It will >> certainly use up a lot of file descriptors. I think it will use more >> memory. And then there's the issue of whether this would really be better >> for debugging. Often it's necessary to look at multiple brick logs while >> trying to diagnose this problem, so it's actually kind of handy to have them >> all in one file. Which would you rather do? >> >> (a) Weave together entries in multiple logs, either via a script or in >> your head? >> >> (b) Split or filter entries in a single log, according to which brick >> they're from? >> >> To me, (b) seems like a much more tractable problem. I'd say that what we >> need is not multiple logs, but *marking of entries* so that everything >> pertaining to one brick can easily be found. One way to do this would be to >> modify volgen so that a brick ID (not name because that's a path and hence >> too long) is appended/prepended to the name of every translator in the >> brick. Grep for that brick ID, and voila! You now have all log messages >> for that brick and no other. A variant of this would be to leave the names >> alone and modify gf_log so that it adds the brick ID automagically (based on >> a thread-local variable similar to THIS). Same effect, other than making >> translator names longer, so I'd kind of prefer this approach. Before I >> start writing the code, does anybody else have any opinions, preferences, or >> alternatives I haven't mentioned yet? >> >> ___ >> Gluster-devel mailing list >> Gluster-devel@gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-devel > > My vote is for having separate log files per brick. Even in separate log > files > that we have today, I find it difficult to mentally ignore irrelevant > messages > in a single log file as I am sifting through it to look for errors that are > related to the problem at hand. Having entries from multiple bricks and then > grepping it would only make things harder. I cannot think of a case where > having > entries from all bricks in one file would particularly be beneficial for > debugging since what happens in one brick is independent of the other bricks > (at least until we move client xlators to server side and run them in the > brick process). > As for file descriptor count/memory usage, I think we should be okay > as it is not any worse than that in the non-multiplexed approach we have > today. > > On a side-note, I think the problem is not having too many log files but > having > them in multiple nodes. Having a log-aggregation solution where all messages > are > logged to a single machine (but still in separate files) would make it > easier to > monitor/debug issues. > -Ravi > I believe the logs are not just from one volume but from all. In that case merging them into a single log file may not be great for debugging. Especially in container use cases there can be multiple volumes. Yes, with some tagging and scripting we can separate the logs and still live with it. What about the log levels? Each volume can configure different log levels. Will you carve out a separate process in case log levels are changed for a volume? How is this handled here? -Rajesh ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Release 3.10: Request fix status for RC1 tagging
On Wed, Feb 15, 2017 at 08:47:20PM -0500, Shyam wrote: > Hi, > > The 3.10 release tracker [1], shows 6 bugs needing a fix in 3.10. We need to > get RC1 out so that we can start tracking the same for a potential release. > > Request folks on these bugs to provide a date by when we can expect a fix > for these issues. > > Request others to add any other bug to the tracker as appropriate. > > Current bug list [2]: .. > - 1417915: Vitaly Lipatov/Niels, I assume one of you would do the backport > for this one into 3.10 Just posted https://review.gluster.org/16640 Niel signature.asc Description: PGP signature ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Release 3.10: Request fix status for RC1 tagging
On Wed, Feb 15, 2017 at 08:47:20PM -0500, Shyam wrote: > Hi, > > The 3.10 release tracker [1], shows 6 bugs needing a fix in 3.10. We need to > get RC1 out so that we can start tracking the same for a potential release. > > Request folks on these bugs to provide a date by when we can expect a fix > for these issues. > > Request others to add any other bug to the tracker as appropriate. > > Current bug list [2]: > - 1415226: Kaleb/Niels do we need to do more for the python dependency or > is the last fix in? There was an email where these python2 changes were causing problems with upgrading. As far as I know this has not been fixed yet. I need to dig through emails to see who reported it (and where?) and who was working on getting it resolved. > - 1417915: Vitaly Lipatov/Niels, I assume one of you would do the backport > for this one into 3.10 ... > - 1421649: Ashis/Niels when can we expect a fix to land for this? I do not think this should be a blocker. Xavi explained it in https://bugzilla.redhat.com/show_bug.cgi?id=1421649#c5 as well. Bug 1421956 is the correct blocker. (and that is the next one on your list) > - 1421956: Xavi, I guess you would backport the fix on mainline once that > is merged into 3.10, right? .. > [1] Tracker bug: https://bugzilla.redhat.com/show_bug.cgi?id=1416031 I like names much more than numbers ;-) https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.10.0 > [2] Open bugs against the tracker: > https://bugzilla.redhat.com/buglist.cgi?quicksearch=1415226%201417915%201421590%201421649%201421956%201422363&list_id=7089913 This is the dynamic list, and will automatically be updated when new blockers are added: https://bugzilla.redhat.com/showdependencytree.cgi?id=glusterfs-3.10.0&maxdepth=1&hide_resolved=1 Niels signature.asc Description: PGP signature ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Gluster Community Meeting 2017-02-15
We had a well attended and active (and productive) meeting this time. Thank you everyone for your attendance. We discussed 3.10, 3.8.9 and an upcoming infra downtime. shyam once again reminded everyone about the very close release date for 3.10. 3.10.0 is still expected on the 21st. A RC1 release is planned, which is expected to be tested by maintainers. shyam has been reaching out to maintainers to get feedback. ndevos brought to notice the problem of packaging 3.8.9. Our one man packaging superman, kkeithley, is on vacation. As a result the download.gluster.org, Ubuntu PPA and SuSE packages haven't built. We decided to postpone building these packages for 3.8.9 till kkeithley returns, but still go ahead with the 3.8.9 announcement. We ended up in a long discussion about how we can fix this problem for the future. More details are in the logs and minutes. nigelb brought to notice of a planned infra downtime for an expected 72hours, due to data center migration. The exact date is still not set, but the earliest should be March 14. Our next meeting is scheduled for 1500UTC on March 1, 2017. I've attached a calendar invite for this. The meeting pad is available for your updates and topics of discussion at https://bit.ly/gluster-community-meetings See you all later. ~kaushal ## Logs - Minutes: https://meetbot.fedoraproject.org/gluster-meeting/2017-02-15/community_meeting_2017-02-15.2017-02-15-15.01.html - Minutes (text): https://meetbot.fedoraproject.org/gluster-meeting/2017-02-15/community_meeting_2017-02-15.2017-02-15-15.01.txt - Log: https://meetbot.fedoraproject.org/gluster-meeting/2017-02-15/community_meeting_2017-02-15.2017-02-15-15.01.log.html ## Topics of Discussion The meeting is an open floor, open for discussion of any topic entered below. - Reminder: Last week before 3.10 release - Post your test results - Post any blocker bugs ASAP, so that we can build RC1 - Check the release notes if you have any features delivered in this release - Package maintainers gear up for RC1 (if any gear up is needed) - glusterfs-3.8.9 has been tagged, release (hopefully) later today - Kaleb is on holidays, other packagers for community .rpm and .deb? - ndevos can take care of the Fedora packages - debs and rpms on download.gluster.org need help - no documentation exists, and not clear that required permissions can be obtained without kkeithley - nigelb will begin documenting the process when kkeithley is back - helps with automating everything - 3.8.9 packages for dependent distros delayed till kkeithely returns - 3.8.9 will be announced later today as planned - We need to fix this problem before the march release cycle - Get a rotating set of packagers who know how to build the packages - Probably have the release-maintainers package - Too much work to package for all distributions - What all pacakges do we build? - https://gluster.readthedocs.io/en/latest/Install-Guide/Community_Packages/ - This page is not informative enough - Docs need fixing - Docs hackday in Malta? - Infra Outage in March - Details still being finalized - Possibly around March 14th for 72h. - RH Community Cage is being moved to a different DC. - review.g.o, build.g.o, fstat.g.o, centos-ci will be offline - list of servers/VM in the cage: https://github.com/gluster/gluster.org_ansible_configuration/blob/master/hosts#L27 ### Next edition's meeting host - kshlm ## Updates > NOTE : Updates will not be discussed during meetings. Any important or > noteworthy update will be announced at the end of the meeting ### Action Items from the last meeting - jdarcy will work with nigelb to make the infra for reverts easier. - _No updates_ - shyam will send out a note on 3.10 deadlines and get someone in BLR to nag in person. - Done. ### Releases GlusterFS 4.0 - Tracker bug : https://bugzilla.redhat.com/showdependencytree.cgi?id=glusterfs-4.0 - Roadmap : https://www.gluster.org/community/roadmap/4.0/ - Updates: - GD2: - FOSDEM 2017 presentation - https://fosdem.org/2017/schedule/event/glusterd2/ - Prashanth is refactoring some of the sunrpc code - Aravinda started a discussion and a POC around plugins - https://github.com/gluster/glusterd2/pull/241 GlusterFS 3.10 - Maintainers : shyam, kkeithley, rtalur - Next release : 3.10.0 - Target date: February 21, 2017 - Release tracker : https://github.com/gluster/glusterfs/milestone/1 - Updates: - Awaiting test updates to github issues - Link to testing issues: http://bit.ly/2kDCR8M - Awaiting final release blocker fixes - Tracker: https://bugzilla.redhat.com/show_bug.cgi?id=1416031 - Merge pending dashboard: - http://bit.ly/2ky6VPI GlusterFS 3.9 - Maintainers : pranithk, aravindavk, dblack - Current release : 3.9.1 - Next release : 3.9.2
[Gluster-devel] Reviews needed
Hi everyone, I would need some reviews if you have some time: A memory leak fix in fuse: * Patch already merged in master and 3.10 * Backport to 3.9: https://review.gluster.org/16402 * Backport to 3.8: https://review.gluster.org/16403 A safe fallback for dynamic code generation in EC: * Master: https://review.gluster.org/16614 A fix for incompatibilities with FreeBSD: * Master: https://review.gluster.org/16417 A fix for FreeBSD's statvfs(): * Patch already merged in master * Backport to 3.10: https://review.gluster.org/16631 * Backport to 3.9: https://review.gluster.org/16632 * Backport to 3.8: https://review.gluster.org/16634 I also have two reviews for 3.7 but I think it won't have any new releases, right ? Thank you very much :) Xavi ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Release 3.10: Request fix status for RC1 tagging
Hi Shyam, On 16/02/17 02:47, Shyam wrote: Hi, The 3.10 release tracker [1], shows 6 bugs needing a fix in 3.10. We need to get RC1 out so that we can start tracking the same for a potential release. Request folks on these bugs to provide a date by when we can expect a fix for these issues. Request others to add any other bug to the tracker as appropriate. Current bug list [2]: - 1415226: Kaleb/Niels do we need to do more for the python dependency or is the last fix in? - 1417915: Vitaly Lipatov/Niels, I assume one of you would do the backport for this one into 3.10 - 1421590: Jeff, this needs a fix? Also, Samikshan can you provide Jeff with a .t that can reproduce this (if possible)? - 1421649: Ashis/Niels when can we expect a fix to land for this? I think this will require more thinking and participation from experts on security and selinux to come up with a good and clean solution. Not sure if this can be done before 3.10 release. There is a workaround (set disperse.cpu-extensions = none) and a mitigating solution (patch https://review.gluster.org/16614) though. - 1421956: Xavi, I guess you would backport the fix on mainline once that is merged into 3.10, right? Yes. It's on review on master (https://review.gluster.org/16614). As soon as it's merged I'll backport it to 3.10. Xavi - 1422363: Poornima, I am awaiting a merge of the same into mainline and also an update of the commit message for the backport to 3.10, before merging this into 3.10, request you to take care of the same. Pranith, is a bug filed and added to the tracker for the mail below? - http://lists.gluster.org/pipermail/maintainers/2017-February/002221.html Thanks, Shyam [1] Tracker bug: https://bugzilla.redhat.com/show_bug.cgi?id=1416031 [2] Open bugs against the tracker: https://bugzilla.redhat.com/buglist.cgi?quicksearch=1415226%201417915%201421590%201421649%201421956%201422363&list_id=7089913 ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
