Re: Is Apache getting too patchy?
On 10/29/2015 11:05 PM, Christophe JAILLET wrote: To give you some ideas of what can be looked at easily by new comers: Hi Christophe, Thanks for the list! Is there any way to get these into the STATUS, or into a document that the STATUS points to? (The current "Contributors looking for a mission" section tries to give a list of all the things that _can_ be done, but as a newbie I'm looking for "here's what the devs _want_ you to focus on for this release".) EASY ... - Speedup/cleanup code --> See http://svn.apache.org/r1701717 It might be "easy" for me to micro-optimize sections of code, but I don't really have the experience yet to know which optimizations are actually useful and which are just noise. Do you have a test setup with ApacheBench/something else that I can attempt to emulate? Or known hotspots that you're wanting additional eyes on, maybe in a certain module or MPM? Hoping these few lines can help or can be a starting point for new comers. These are very helpful, at least to me. Knowing what the dev team would like more hands on "right now" is much nicer than grepping through TODOs. --Jacob
Re: Is Apache getting too patchy?
Le 27/10/2015 17:27, Jacob Champion a écrit : > >> The most exciting work on httpd for me has been Christophe Jaillet’s >> memory optimisation patches. With each patch the code gets cleaner >> and the server gets faster. We need more of this kind of stuff. > > How would you improve the process and encourage those sorts of patches, then, from someone who _isn't_ in the httpd committers list? It seems to me that optimization needs to be done by experts in a codebase, not by newbies. And code cleanup is difficult when you don't already have a clear direction to push the code in. > > --Jacob > Hi, I understand your point. Personally, when I first looked at httpd, it was more or less in the same spirit as what is done on the kernel Janitor project. This project is aimed at cleaning and fixing potential issues in Linux kernel. So I started looking at what could, according to me, be improved in httpd. Not an easy task when, as you say, you don't have any direction to push. To give you some ideas of what can be looked at easily by new comers: EASY - Fix coding style issue: [1] (even if I don't really think that the 'indent' command line given there is that good. It generates way too much changes) --> See http://svn.apache.org/r1700332 - Identify where standard C functions or apr_ or ap_ functions can be used in order to remove code duplication. If familiar with it, 'coccinelle' can be used for that. --> See PR 39518 or 42006 as some examples - Speedup/cleanup code --> See http://svn.apache.org/r1701717 - Some parts of the server generate some HTML pages (in modules/generators, mod_proxy_balancer...). The generated output does not always comply with the HTML standard and could be clean-up. --> See http://marc.info/?l=apache-httpd-dev&m=142593723110939&w=2 - Improve existing documentation [2] - Improve (or sometimes fix) code documentation generated with doxygen - Look at comments added by users in the online documentation and propose patches to take these comments into account MEDIUM == - Identify and improve memory usage. For that, you must first know and understand the way APR memory pools work. --> see PR 52648 or http://svn.apache.org/r1681685 - Identify places where 'varbuf' could be used to improve memory usage - Fix compilation warning (if any) - Use and fix items reported by static analyzer such as cpp_check or sparse - Check bug reports in bugzilla, try to reproduce the issues, try the (sometimes) proposed patch. Mark bug reports with the 'PatchAvailable' keyword when it is missing - Look at differences between trunk and 2.4.x branch to identify and propose what can be backported. The closer they are, the easier it will be to backport other proposals HARD - Write tests to add in the test framework [3] It could also be a good idea to tag some bugzilla reports with something like [good first bug] (such as what is done in the bugzilla project itself or LibreOffice). This would give the opportunity to new comers to have a starting point. This mean that some maintainers should tag these bugs report and LEAVE THEM OPEN, EVEN IF EASY TO FIX. The best is to open a report on bugzilla for each individual patch you submit (see [4]). You can use the Keyword 'PatchAvailable' to identify them even more easily. It help us tracking them and they don't get lost in a mailing list. I'll be glad (and I know some others will also) to have a look at it and push it to the repository. If you are interested by one of the few ideas above and need more inputs on where/how exactly to start, do not hesitate to ask. Other (old) ideas can be found at http://httpd.apache.org/dev/todo.html Hoping these few lines can help or can be a starting point for new comers. [1] http://httpd.apache.org/dev/styleguide.html [2] https://httpd.apache.org/docs/2.4/ [3] http://httpd.apache.org/test/ [4] http://httpd.apache.org/dev/patches.html
Re: Is Apache getting too patchy?
On 10/27/2015 03:26 AM, Graham Leggett wrote: On 27 Oct 2015, at 12:40 AM, Tim Bannister wrote: That may not be easy. You need to find someone who'll be interested in an idea or patch, and has the time to review it. Plus, the community as a whole to agree it's a good idea, or at least not actively oppose it. I wonder if workflow would be improved if we had named maintainers for particular parts of the codebase - for example individual modules? Not necessarily to do all the work, but to take primary responsibility to see that your ideas don't just fall through the gaps and get ignored? How does the word “sponsor” sound? It rhymes with “bureaucracy”. :) Remember the end goal is to improve quality without causing stagnation, I agree that adding process always carries the risk of red tape (more so when people like me, outside an organization, propose processes for people inside it). We all want to avoid a stagnant codebase. I would also like to de-stagnate the communication, that's all. and giving people the impression that they need to ask a “sponsor” first who may no longer be involved with the project will cause that stagnation. My first thought is, "then don't give people that impression." To me, Nick and Tim's suggestions sounded more like "community liaison" than "code owner". The most exciting work on httpd for me has been Christophe Jaillet’s memory optimisation patches. With each patch the code gets cleaner and the server gets faster. We need more of this kind of stuff. How would you improve the process and encourage those sorts of patches, then, from someone who _isn't_ in the httpd committers list? It seems to me that optimization needs to be done by experts in a codebase, not by newbies. And code cleanup is difficult when you don't already have a clear direction to push the code in. --Jacob
Re: Is Apache getting too patchy?
On 27 Oct 2015, at 12:40 AM, Tim Bannister wrote: >> That may not be easy. You need to find someone who'll be interested in an >> idea or patch, and has the time to review it. >> Plus, the community as a whole to agree it's a good idea, or at least not >> actively oppose it. >> >> I wonder if workflow would be improved if we had named maintainers for >> particular parts of the codebase - for example individual modules? Not >> necessarily to do all the work, but to take primary responsibility to see >> that your ideas don't just fall through the gaps and get ignored? > > How does the word “sponsor” sound? It rhymes with “bureaucracy”. :) Remember the end goal is to improve quality without causing stagnation, and giving people the impression that they need to ask a “sponsor” first who may no longer be involved with the project will cause that stagnation. The most exciting work on httpd for me has been Christophe Jaillet’s memory optimisation patches. With each patch the code gets cleaner and the server gets faster. We need more of this kind of stuff. Regards, Graham —
Re: Is Apache getting too patchy?
On 10/26/2015 03:23 PM, Nick Kew wrote: On Mon, 26 Oct 2015 09:51:43 -0700 Jacob Champion wrote: I'd rather not feel like I'm just annoying dev@ until you submit my stuff -- I want to *talk* about it, and improve the server. That may not be easy. You need to find someone who'll be interested in an idea or patch, and has the time to review it. Plus, the community as a whole to agree it's a good idea, or at least not actively oppose it. That's fine by me. I don't think it necessarily needs to be _easy_ -- making it too easy to submit code loosens the codebase, which I think is what Jim's original mail is asking about -- but at the same time it shouldn't be like talking to a wall. In other words, I find it very difficult to keep a conversation going here, and in my experience, it's the conversations that keep codebases healthy. I wonder if workflow would be improved if we had named maintainers for particular parts of the codebase - for example individual modules? Not necessarily to do all the work, but to take primary responsibility to see that your ideas don't just fall through the gaps and get ignored? See my reply to Tim. --Jacob
Re: Is Apache getting too patchy?
On 10/26/2015 03:40 PM, Tim Bannister wrote: On 26 Oct 2015, at 22:23, Nick Kew wrote: I wonder if workflow would be improved if we had named maintainers for particular parts of the codebase - for example individual modules? Not necessarily to do all the work, but to take primary responsibility to see that your ideas don't just fall through the gaps and get ignored? How does the word “sponsor” sound? Someone who encourages and champions the development activity around a particular feature (and is also very welcome to contribute). The existing and more formal mechanisms for approving commits seem to work fine as a way of controlling the quality of code. I like "sponsor". In contrast, "primary maintainer" implies (at least to me) a lot of ownership over a section of code, which doesn't seem to fit into the Apache style. Improving the workflow means, to me, coaching and leadership, and different kinds of code review. Someone who isn't very good at C (like me) might well want to make a code contribution but not be sure how. I saw recently how much perseverance Yingqi Lu put in towards getting SO_REUSEPORT support into trunk and then into 2.4.17 – and that's great. It's unfortunate that the same perseverance also offers a lesson about the kind of barriers that a would-be contributor might encounter. So, sponsorship can be about encouraging participation and progress. I'm imagining someone who rarely has to settle a decision – those should stay consensual and democratic - but often leads discussions and moves things on. I like this. So your sponsor's qualities (coaching/servant leadership/facilitation) are primarily to break down the barriers to contribution. I think there are now two topics of conversation, though. To wrap this back around to Jim's question, which was about the "patchiness" of the server (sorry, Jim, for somewhat derailing your thread)... In addition to a sponsor, having someone with "the grand vision" for a section of the server would be good too. Someone who tries to be involved with related patches, to ensure that work in one area does not conflict with other (un)related work... basically, to make sure that a piece of the server is moving in one direction at once. (I would call such a person an "architect", though I think that term is probably loaded with a lot of baggage for those of you in the corporate world.) I don't know if a sponsor and an architect could/should/would be the same person in httpd's case; I don't even know if either of those roles makes sense in Apache culture. And I understand that this is a volunteer project and not a corporation. But from the outside looking in, it would be nice if both roles were filled for various parts of the server. Maybe they already are filled, and I just can't see them? --Jacob
Re: Is Apache getting too patchy?
Hello Jim, As Bill pointed out, it would be helpful if you could identify some specific details within the general trend you are observing. But the kind of thing that you are mentioning is something that I would expect to happen far more often than can be observed in the Apache HTTP Server project. If individual contributors are each trying to "scratch their own itch", then there would be a tendency for code to become less coherent over time. What keeps things from growing into chaos is the review by experienced contributors who are looking at the overall purpose and direction of the project as a whole. Another thing that helps the Apache HTTP Server project is the modular design of the code base. New and diverse features can be isolated into separate modules, be given time to mature and then eventually be better integrated with the core code. Over the last few months there simply has been a lot of activity. But there has also been a lot of quality debate around this activity. Some very impressive new functionality has been attempted. I personally have not had enough time (and possibly not enough skill) to know if the quality of the code is good or not. But from my limited perspective the discussions seem to be going in a good direction. Thanks, Mike Rumph On 10/24/2015 8:49 AM, Jim Jagielski wrote: Just some food for thought; let me know if I'm off the rails. Over the last several months, it's appeared to me that we have been adding patches that feel, well, very-patchy to me. They feel like cumbersome add-ons that create some level of fragility to our code, with special one-off considerations instead of a deeper more complete fix. In other words, it seems that httpd is becoming more crufty rather than planned and cohesive and consistent. Thoughts? Comments?
Re: Is Apache getting too patchy?
On 26 Oct 2015, at 22:23, Nick Kew wrote: > On Mon, 26 Oct 2015 09:51:43 -0700 > Jacob Champion wrote: > >> I'd rather not feel like I'm just annoying dev@ until you submit my >> stuff -- I want to *talk* about it, and improve the server. > > That may not be easy. You need to find someone who'll be interested in an > idea or patch, and has the time to review it. > Plus, the community as a whole to agree it's a good idea, or at least not > actively oppose it. > > I wonder if workflow would be improved if we had named maintainers for > particular parts of the codebase - for example individual modules? Not > necessarily to do all the work, but to take primary responsibility to see > that your ideas don't just fall through the gaps and get ignored? How does the word “sponsor” sound? Someone who encourages and champions the development activity around a particular feature (and is also very welcome to contribute). The existing and more formal mechanisms for approving commits seem to work fine as a way of controlling the quality of code. Improving the workflow means, to me, coaching and leadership, and different kinds of code review. Someone who isn't very good at C (like me) might well want to make a code contribution but not be sure how. I saw recently how much perseverance Yingqi Lu put in towards getting SO_REUSEPORT support into trunk and then into 2.4.17 – and that's great. It's unfortunate that the same perseverance also offers a lesson about the kind of barriers that a would-be contributor might encounter. So, sponsorship can be about encouraging participation and progress. I'm imagining someone who rarely has to settle a decision – those should stay consensual and democratic - but often leads discussions and moves things on. Comments very welcome. -- Tim Bannister – is...@c8h10n4o2.org.uk
Re: Is Apache getting too patchy?
On Mon, 26 Oct 2015 09:51:43 -0700 Jacob Champion wrote: > I'm somewhat on the outside looking in, as an httpd newbie. But my > standard experience (it's happened two or three times now) is this: > > 1) Non-trivial patch is proposed to the list with calls for > discussion/debate. > 2) Nothing happens. > 3) List is pinged for comments. > 4) Silence. > 5) Patch author either gives up or pings relentlessly until... > 6) ...patch is applied directly to trunk without discussion. Thanks for the perspective. Yes, it's sometimes uncomfortable. > I'd rather not feel like I'm just annoying dev@ until you submit my > stuff -- I want to *talk* about it, and improve the server. That may not be easy. You need to find someone who'll be interested in an idea or patch, and has the time to review it. Plus, the community as a whole to agree it's a good idea, or at least not actively oppose it. I wonder if workflow would be improved if we had named maintainers for particular parts of the codebase - for example individual modules? Not necessarily to do all the work, but to take primary responsibility to see that your ideas don't just fall through the gaps and get ignored? -- Nick Kew
Re: Is Apache getting too patchy?
On Sat, Oct 24, 2015 at 10:49 AM, Jim Jagielski wrote: > Just some food for thought; let me know if I'm off the rails. > Well, that depends on what you are commenting on... Over the last several months, it's appeared to me that we have > been adding patches that feel, well, very-patchy to me. They > feel like cumbersome add-ons that create some level of fragility > to our code, with special one-off considerations instead of a > deeper more complete fix. In other words, it seems that httpd is > becoming more crufty rather than planned and cohesive and > consistent. > So over an arbitrary window of time (several months = 12 weeks?) there are patches that you are uncomfortable with on trunk? 2.4? Perhaps even 2.2? They seem to be fingers in the dam, rather than reinforcing the wall of a solid code base? Rather than 'vaugebooking' (to borrow a phrase from another context), it might be helpful to point out the specific commits that have you concerned, and register specific complaints or feedback? If a -1 is called for, of course put that forward, but perhaps some coaching/coaxing might get the specific fixes into the sort of shape you feel is appropriate? If there is a pattern from specific committers, pointing out any issues on one patch is probably sufficient to coax them into reviewing their other patches for similar issues and avoid any similar issues in the future? Some parts of trunk are certainly due for refactoring, but that can be a multi-week task rather than the obvious quick fix. If there is a better or more comprehensive fix, calling it out on the list might solicit rework from yet another committer who is interested in the same issue at hand? As a committer and PMC member, it is yours, and mine, and all the rest of the project participants to point out problematic commits individually. That's the definition of commit-then-review, something you recently championed an expansion of for more aspects of the 2.4 branch. Please pipe up on the -r phase so we all learn which patches concerns you, and why?
Re: Is Apache getting too patchy?
On 10/26/2015 10:52 AM, Kurtis Rader wrote: Asshole was probably not the best term. I meant it in the sense that you shouldn't be afraid to tell someone in plain language what the problems are with their proposed change. Yes, the reviewer should be polite. No, they should not sugar-coat their feedback. Cool, we're in agreement then. --Jacob
Re: Is Apache getting too patchy?
On Mon, Oct 26, 2015 at 9:56 AM, Jacob Champion wrote: > On 10/24/2015 09:32 PM, Kurtis Rader wrote: > >> People reviewing changes to the existing code base have to be hard-nosed >> assholes. >> > > Kurtis, > > I agree with everything you said except this. IMHO, you've traded one bad > extreme (where few people care about the quality of the codebase) for > another (where the people who care about the codebase are rude and > uncooperative). You can be hard-nosed about code standards without being a > jerk. > > Maybe you didn't mean it like that, but I feel like there are way too many > people in open source communities who take an "I have to be an asshole" > mantra to heart. Asshole was probably not the best term. I meant it in the sense that you shouldn't be afraid to tell someone in plain language what the problems are with their proposed change. Yes, the reviewer should be polite. No, they should not sugar-coat their feedback. I've seen way too many instances (not limited to this project) of ill-advised or badly implemented changes being accepted because no one was willing to be the "bad guy" and say "No, that change will not be accepted." This has been a major problem with the Zsh project, for example. To the point where I'm seriously considering switching shells. I'd hate to see that happen to this project. -- Kurtis Rader Caretaker of the exceptional canines Junior and Hank
Re: Is Apache getting too patchy?
On 10/24/2015 09:32 PM, Kurtis Rader wrote: People reviewing changes to the existing code base have to be hard-nosed assholes. Kurtis, I agree with everything you said except this. IMHO, you've traded one bad extreme (where few people care about the quality of the codebase) for another (where the people who care about the codebase are rude and uncooperative). You can be hard-nosed about code standards without being a jerk. Maybe you didn't mean it like that, but I feel like there are way too many people in open source communities who take an "I have to be an asshole" mantra to heart. --Jacob
Re: Is Apache getting too patchy?
On 10/24/2015 08:49 AM, Jim Jagielski wrote: Thoughts? Comments? Jim, I'm somewhat on the outside looking in, as an httpd newbie. But my standard experience (it's happened two or three times now) is this: 1) Non-trivial patch is proposed to the list with calls for discussion/debate. 2) Nothing happens. 3) List is pinged for comments. 4) Silence. 5) Patch author either gives up or pings relentlessly until... 6) ...patch is applied directly to trunk without discussion. This being a volunteer project, I don't actually mind steps 1-5 so much. You're busy people, and my patches shouldn't be priority one for anyone. I'm more concerned about step 6, where conversations wither on the vine in favor of just applying the patch and seeing what happens. I think that could contribute to the "cruft instead of cohesion" that you're seeing. I'd rather not feel like I'm just annoying dev@ until you submit my stuff -- I want to *talk* about it, and improve the server. HTH, --Jacob
Re: Is Apache getting too patchy?
On Sat, Oct 24, 2015 at 8:49 AM, Jim Jagielski wrote: > Just some food for thought; let me know if I'm off the rails. > > Over the last several months, it's appeared to me that we have > been adding patches that feel, well, very-patchy to me. They > feel like cumbersome add-ons that create some level of fragility > to our code, with special one-off considerations instead of a > deeper more complete fix. In other words, it seems that httpd is > becoming more crufty rather than planned and cohesive and > consistent. > > Thoughts? Comments? > I agree. I recently contributed to the improvement of the Apache HTTP server source code when I noticed that mod_dumpio.c was badly broken (it does not correctly handle null bytes). Which caused me to subscribe to the dev mailing list and read some earlier messages. I have been underwhelmed with the quality of the arguments for a given change or the code itself. Frankly, I would not allow almost all of the proposed changes I've seen (including my first patch to mod_dumpio.c) to be accepted. People reviewing changes to the existing code base have to be hard-nosed assholes. You have to reject changes that are problematic unless there is a) a "TODO" comment explaining the unresolved issues and b) a reason to believe the contributor or someone might resolve those issues in the near future. Similarly style issues are not just nitpicking. They are show stoppers for accepting a change. Every time I look at code that has a style different from the surrounding code it makes it more likely I'll overlook a bug or misinterpret what the code does. -- Kurtis Rader Caretaker of the exceptional canines Junior and Hank
