Re: Another proposal for modernization of our infrastructure
On Tuesday, 3 February 2015 12:37:30 CEST, Martin Sandsmark wrote: So everyone with a KDE account will be able to push to any KDE project, bypassing Gerrit? Yes. -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Tue, Feb 03, 2015 at 12:02:40PM +0100, Martin Sandsmark wrote: Yes, but as I said this doesn't really solve it at all. As I said, for long discussions it still adds a lot of space and noise into the code, which makes following the flow of the code extremely hard to do (at least for me personally, and I suspect others). Disregard my other points, as they were based on faulty assumptions. :-) But this point is still pretty major, IMHO, and I suspect that it is not something that upstream would change easily. But we have had bugs slipping through because of this already (at work), so it's not really something minor IMHO. -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
On Sat, Jan 31, 2015 at 02:01:22PM +0100, Jan Kundrát wrote: Due to the nature of build jobs which constitute a pretty bursty load, renting VMs sounds like a cost-effective approach for our scale. I do not expect that it would make financial sense for us to procure enough physical HW to cover the peak demand -- hence the VMs. But this means that we would be even more dependent on financial donations to be able to keep a functioning infrastructure, if I understand correctly? Renting VMs also enables corporate sponsors to offer unused capacity for less money, or even for free. Has anyone been in contact with hosters/providers to see what kind of sponsoring we could get? And it still increases our reliance on sponsors. -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
On Tue, Jan 27, 2015 at 11:08:49AM +0100, Jan Kundrát wrote: Gerrit will act as a primary repository host. This will be completely transparent to the users. Developers who do not want to change their workflow will witness no user-visible changes. All existing clones will work, and developers will be able to direct push and bypass code review if they so choose. So everyone with a KDE account will be able to push to any KDE project, bypassing Gerrit? -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
14 messages in 90 minutes on a topic we are discussing for weeks now. Please realise that that is why people do not engage on this mailing list. It is not the choice of tools. Thanks, Mirko. On 03 Feb 2015, at 12:23, Martin Sandsmark martin.sandsm...@kde.org wrote: On Sat, Jan 31, 2015 at 02:01:22PM +0100, Jan Kundrát wrote: Due to the nature of build jobs which constitute a pretty bursty load, renting VMs sounds like a cost-effective approach for our scale. I do not expect that it would make financial sense for us to procure enough physical HW to cover the peak demand -- hence the VMs. But this means that we would be even more dependent on financial donations to be able to keep a functioning infrastructure, if I understand correctly? -- Mirko Boehm | mi...@kde.org | KDE e.V. FSFE Fellow, FSFE Team Germany Qt Certified Specialist Request a meeting: https://doodle.com/mirkoboehm
Re: Another proposal for modernization of our infrastructure
On Tue, Feb 03, 2015 at 12:40:01PM +0100, Mirko Boehm wrote: 14 messages in 90 minutes on a topic we are discussing for weeks now. Please realise that that is why people do not engage on this mailing list. It is not the choice of tools. Sorry for being late to the discussion, but I haven't had time to catch up on this until now, and I brought up issues I believe are important. I would like to spend more time contributing to KDE, and for that I feel that we should have a process that lowers the barrier for doing that (for everyone). I'm also sorry for working in burst-mode, I can slow down if you believe that is better? -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
Hello, First of all, thank you Boud for the wise words. On Tuesday 03 February 2015 11:17:59 Boudewijn Rempt wrote: On Mon, 2 Feb 2015, Milian Wolff wrote: Sigh, I find it highly sad to read this over and over again. Well, this whole discussion makes me extremely sad. What people have to learn is that _arguments_ only go so far. People can feel they're double-plus extra-super right, and still at one point they have to accept that they're not going to change the other people's point of view. And not respecting or ignoring other people's point of view will lead to bullying or pushing people away. For the record those numerous mega-threads already pushed some people away (I'm aware of several people whom motivation drastically declined just reading them). It'd be nice if everyone would take a deep breath and realize: it is just a tool dammit! So, here is my point of view, put very simply: * replacing reviewboard with gerrit will mean fewer contributors to many of the projects KDE hosts. * following from that, projects that want to stay alive and relevant will move away from the kde infrastructure. * which makes gerrit a Bad Thing. This is what I am sure _will_ happen, no matter how much anyone argues that gerrit is cool, can be cool, will be cool, won't be as uncool as it's for Qt, how lovely gerrit's git integration is, how nice it is to train people to contribute to Qt, that nobody has tested phabricator yet and so on ad infinitum. I've read it all, and I'm not convinced. There are people who like gerrit and would love to use it. I accept that. Agreed. I also accept your opinion that gerrit would be a Bad Thing even though I'm not convinced by that either. Simply put: I'm not convinced gerrit would be super-good or super-bad for us. It's likely something in between. But there is not going to be a broadly supported consensus that gerrit is cool and should be used by KDE in the development workflow. There is not going to be a consensus that gerrit should replace reviewboard, sorry, no matter over how many mails the same arguments get rehashed. That's something people who like gerrit have to accept. Hear! Hear! Also it is just a tool dammit! Don't forget: If there are people who don't like gerrit you're not a fool because you like it. Now, phabricator might be just as crappy, though if Blender uses it... (But that's the same argument as Qt uses it is for gerrit and just as invalid), but let's first _test_ it, as Ben proposed. Yes please... No need for more noise on the matter until we have the experiment done. Regards. -- Kévin Ottens, http://ervin.ipsquad.net KDAB - proud supporter of KDE, http://www.kdab.com signature.asc Description: This is a digitally signed message part.
Re: Another proposal for modernization of our infrastructure
On Tuesday, 3 February 2015 11:36:37 CEST, Martin Sandsmark wrote: On Fri, Jan 30, 2015 at 11:44:22PM -0200, Thiago Macieira wrote: Many of your complaints about usability (threading, replies, etc.) are solved or at least partially addressed in the new Gerrit UI, which versions like 2.7 have. It might not be the default on the installation, so check the settings and try to turn it on. Do you know of any Gerrit installations that have this enabled? Yes, the one we're testing in KDE is reasonably recent. It lives at https://gerrit.vesnicky.cesnet.cz/ , and it uses the new change screen by default. On other Gerrit servers = 2.8, you can go to (your name at upper right) - Settings - Preferences - Change view (second option from the bottom) and select New screen. We use Gerrit (2.7) at work, and the UI is still pretty horrible, The new change screen only got added in 2.8, and 2.7 doesn't have it. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Thu, Jan 29, 2015 at 06:49:28PM +0100, Eike Hein wrote: I disagree - having the comment in a floating popup instead of breaking up source code makes it easier to read the code for me. I just want to back up this point. As mentioned already, we've been using Gerrit at work for quite a while now, and having the code broken up by comments (sometimes many lines in case of a discussion) makes it extremely hard to actually follow the flow of the code. Do you know if upstream would accept to change this, or how hard it would be to change? -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
On Sat, Jan 31, 2015 at 01:16:05PM +0100, Jan Kundrát wrote: Your mail suggested that they apparently do not care about improving their UI, because if they did, they would have solved everything already. I disagree with that, and provide evidence which supports the idea that Gerrit upstream in fact also cares about users, including those who are not already experienced with its UI. I think the point was more that what Gerrit has fixed were simple UI glitches, not radical improvements that change the existing design to make it easier for less experienced or casual users (or even experienced users, but that's another discussion). :-) -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
On Tuesday, 3 February 2015 11:48:30 CEST, Martin Sandsmark wrote: As mentioned already, we've been using Gerrit at work for quite a while now, and having the code broken up by comments (sometimes many lines in case of a discussion) makes it extremely hard to actually follow the flow of the code. Do you know if upstream would accept to change this, or how hard it would be to change? I believe that this is fixed in the new change UI: - The diff viewer shows comments minimized/collapsed and in a way which consumes less space. - The review page shows file/line/range comments with a pointer to what file and what part of a file this is about. Now, one thing which is arguably missing and can be improved is adding a small chunk of actual file content to the comments shown on review page. I think that upstream will be happy to accept such a patch. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Sun, Feb 01, 2015 at 10:49:58AM -0200, Thiago Macieira wrote: They would have if they still had major problems with the usability of the tool. It probably just so happens that all the backers are used to the interface, however bad it might be, and don't feel the need to sponsor such a work. I think Ben's mail was more about the uncertainty whether this was actually the case (feel free to correct me on this, though). Do you know for a fact that noone has tried to push upstream radical UI changes to improve it, or is that just an assumption? Isn't it also possible that people that find the UI extremely bad haven't simply avoided Gerrit completely, or moved to something that's better? -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
On Tuesday, 3 February 2015 11:53:30 CEST, Martin Sandsmark wrote: I think the point was more that what Gerrit has fixed were simple UI glitches, not radical improvements that change the existing design to make it easier for less experienced or casual users (or even experienced users, but that's another discussion). :-) Thanks for explaining this to me. As they completely revamped the change screen UI in 2.8, I do not think that this point is true, either. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Tue, Feb 03, 2015 at 11:55:58AM +0100, Jan Kundrát wrote: I believe that this is fixed in the new change UI: We use 2.7, which I assume has the new UI. - The diff viewer shows comments minimized/collapsed and in a way which consumes less space. Yes, but as I said this doesn't really solve it at all. As I said, for long discussions it still adds a lot of space and noise into the code, which makes following the flow of the code extremely hard to do (at least for me personally, and I suspect others). - The review page shows file/line/range comments with a pointer to what file and what part of a file this is about. Still annoying to have to manually go to the source files, and scroll down to the right line. :-) I guess it could be better if it was turned into a link? Now, one thing which is arguably missing and can be improved is adding a small chunk of actual file content to the comments shown on review page. I think that upstream will be happy to accept such a patch. This would indeed be a big improvement, especially if it has the ability to expand up and down, like in the file diff view (which could also be improved IMHO, clicking the number is not very intuitive at all...). :-) -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
On Tue, Feb 03, 2015 at 11:46:10AM +0100, Jan Kundrát wrote: Yes, the one we're testing in KDE is reasonably recent. It lives at https://gerrit.vesnicky.cesnet.cz/ , and it uses the new change screen by default. Thanks! And now I see that the source/line from the comments is already a link. :-) IMHO it suffers a bit from information overload and seems generally confusing, but I guess that is solvable with some styling. I see however that most of the small issues with the UI are solved, though, which is very nice, I'll ask for our installation at work to be upgraded. :-) -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
On Tue, Feb 03, 2015 at 11:58:50AM +0100, Jan Kundrát wrote: As they completely revamped the change screen UI in 2.8, I do not think that this point is true, either. It doesn't really seem revamped, mostly just fixing all the glitches, not really changes in the basic assumptions about how it should work or the behaviour? Also adding much more to the already overloaded UI (IMHO). It would have been nice to try to decrease the information overload (collapse/hide stuff by default, for example). I'm not a UI designer, so I'm not exactly sure what the best way to do this is, but there seems to be a lot of information that isn't really necessary for most reviewers (commit hash, parent commit, change-id, commit dates, age, topic strategy, etc.). -- Martin Sandsmark
Re: Another proposal for modernization of our infrastructure
On Mon, 2 Feb 2015, Milian Wolff wrote: Sigh, I find it highly sad to read this over and over again. Well, this whole discussion makes me extremely sad. What people have to learn is that _arguments_ only go so far. People can feel they're double-plus extra-super right, and still at one point they have to accept that they're not going to change the other people's point of view. So, here is my point of view, put very simply: * replacing reviewboard with gerrit will mean fewer contributors to many of the projects KDE hosts. * following from that, projects that want to stay alive and relevant will move away from the kde infrastructure. * which makes gerrit a Bad Thing. This is what I am sure _will_ happen, no matter how much anyone argues that gerrit is cool, can be cool, will be cool, won't be as uncool as it's for Qt, how lovely gerrit's git integration is, how nice it is to train people to contribute to Qt, that nobody has tested phabricator yet and so on ad infinitum. I've read it all, and I'm not convinced. There are people who like gerrit and would love to use it. I accept that. But there is not going to be a broadly supported consensus that gerrit is cool and should be used by KDE in the development workflow. There is not going to be a consensus that gerrit should replace reviewboard, sorry, no matter over how many mails the same arguments get rehashed. That's something people who like gerrit have to accept. Now, phabricator might be just as crappy, though if Blender uses it... (But that's the same argument as Qt uses it is for gerrit and just as invalid), but let's first _test_ it, as Ben proposed. Boudewijn
Re: Another proposal for modernization of our infrastructure
On Fri, Jan 30, 2015 at 11:44:22PM -0200, Thiago Macieira wrote: Many of your complaints about usability (threading, replies, etc.) are solved or at least partially addressed in the new Gerrit UI, which versions like 2.7 have. It might not be the default on the installation, so check the settings and try to turn it on. Do you know of any Gerrit installations that have this enabled? We use Gerrit (2.7) at work, and the UI is still pretty horrible, unfortunately I can't play with the settings (I'm assuming they are some global settings, because I can't find anything for it in my personal settings). I also want to avoid setting up a personal Gerrit installation, it seems pretty painful to install. :-) -- Martin Sandsmark
Re: Re: Another proposal for modernization of our infrastructure
On Monday 02 February 2015 13:17:21 Milian Wolff wrote: On Saturday 31 January 2015 20:56:41 Martin Graesslin wrote: On Saturday 31 January 2015 20:37:31 Christoph Feck wrote: On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. I have to agree. Whenever I need to do a change for Qt I need to google for how to do it. Including putting serious thought in how the push command must look like, how I need to adjust the examples provided and for which ref/for/foo it has to be this time. This is something which seriously concerns me with the outlook of gerrit. git has a steep learning curve, gerrit adds quite some further steepness to it :-( Just add an alias for the refs, e.g. for 5.4 and dev, then use that in the future. `git qpushtab` and pick one, that's how I do it. E.g.: [alias] qpushdev = push gerrit HEAD:refs/for/dev And yes, this is something that kdesrc-build could automate for us. How would kdesrc-build know that we just created a new branch? To me that looks like I would have to manually edit every time we branch in every repo I use. meh, meh, meh Also we do not force new devs to use kdesrc-build. When I see new developers submitting there first patch, they hardly use kdesrc-build. So having that as a oh it will be so simple is something I do not buy :-( The rest of the above is independent of the actual tool. Even with reviewboard or phabricator, you'll need to know where to push your change. Bugfix? Latest stable release you want to support. New feature? dev. That's true and that's why I want to test out Phabricator to see how good the support is there. For review board at least I can just ignore the branch when pushing and fill in a free text form in the web ui. And yes, while it means interacting with two tools, I consider it as fairly simpler than gerrit's way. Cheers Martin signature.asc Description: This is a digitally signed message part.
Re: Another proposal for modernization of our infrastructure
On Saturday 31 January 2015 20:56:41 Martin Graesslin wrote: On Saturday 31 January 2015 20:37:31 Christoph Feck wrote: On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. I have to agree. Whenever I need to do a change for Qt I need to google for how to do it. Including putting serious thought in how the push command must look like, how I need to adjust the examples provided and for which ref/for/foo it has to be this time. This is something which seriously concerns me with the outlook of gerrit. git has a steep learning curve, gerrit adds quite some further steepness to it :-( Just add an alias for the refs, e.g. for 5.4 and dev, then use that in the future. `git qpushtab` and pick one, that's how I do it. E.g.: [alias] qpushdev = push gerrit HEAD:refs/for/dev And yes, this is something that kdesrc-build could automate for us. The rest of the above is independent of the actual tool. Even with reviewboard or phabricator, you'll need to know where to push your change. Bugfix? Latest stable release you want to support. New feature? dev. Bye -- Milian Wolff m...@milianw.de http://milianw.de
Re: Another proposal for modernization of our infrastructure
Jan Kundrát wrote: However, we also have people with little to no experience using Gerrit just fine. Shall we therefore focus on explaining that contributing through Gerrit is actually not painful? My two cents here: as an occasional contributor (and one drop in the ocean; take what I say with more than one grain of salt), I'm good enough with git, but the whole Gerrit approach doesn't feel too natural to me, and for sure, I'd prefer simpler ways. Not to say that RB was better, because I had my fights with rbt recently. BUt in that case at least there's fire and forget, or KDevelop integration (at least, used to work). The issue is: for people that commit every once in a while, the workflow is still a little too complex with Gerrit, at least the proposed one. Of course, if there are ways around the problem, then it becomes a moot point. -- Luca Beltrame - KDE Forums team KDE Science supporter GPG key ID: 6E1A4E79
Re: Another proposal for modernization of our infrastructure
On 02/02/2015 01:20 PM, Milian Wolff wrote: Sigh, I find it highly sad to read this over and over again. People keep confusing the flaky CI and the high quality barrier in Qt with gerrit itself... Seriously, gerrit the tool is OK, what makes it hard and what is the actual barrier to entry in Qt are the flaky CI which kills productivity and then sometimes the odd reviews with a pretty harsh tone that demand an extraordinary quality without holding your hand much. But that's not related to gerrit... These are different things, so please people - lets not confuse these things and say using gerrit means using it exactly like Qt is using it! 1) I wrote contribute to Qt intentionally there, not use gerrit, because I'm aware of the diff. Although no, my unhappiness is not limited to the CI; many of my problems are actually linked to the way JIRA and gerrit have been made to inter- play for auth. 2) I've posted other criticizims of gerrit in the thread for a fuller picture of how gerrit adds to the unhappiness. Jumping on this bit, hand- waving it away and claiming we're now left with gerrit is OK is incorrect. Bye Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On Saturday 31 January 2015 21:34:40 Eike Hein wrote: On 01/31/2015 09:25 PM, Boudewijn Rempt wrote: In short, Qt uses gerrit is a bogus argument in favor of gerrit. The argument isn't so much that gerrit is working well for Qt, but more that there's a certain simplicity in using the same tooling across the KDE/Qt stack, and that KDE benefits from having more KDE people active in Qt. But I actually find contributing to Qt pretty frus- trating (the infra is flaky, the process tends to break down here and there, and I sometimes have to step out- side gerrit and side-channel via email to get things moving again), yeah. Sigh, I find it highly sad to read this over and over again. People keep confusing the flaky CI and the high quality barrier in Qt with gerrit itself... Seriously, gerrit the tool is OK, what makes it hard and what is the actual barrier to entry in Qt are the flaky CI which kills productivity and then sometimes the odd reviews with a pretty harsh tone that demand an extraordinary quality without holding your hand much. But that's not related to gerrit... These are different things, so please people - lets not confuse these things and say using gerrit means using it exactly like Qt is using it! Bye -- Milian Wolff m...@milianw.de http://milianw.de
Re: Another proposal for modernization of our infrastructure
On Sat, January 31, 2015 8:25 pm, Boudewijn Rempt wrote: On Sat, 31 Jan 2015, Christoph Feck wrote: On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. Christoph Feck (kdepepo) Maybe quite a few KDE developers would want to contribute to Qt, and maybe Qt would like more contributors, but KDE is so much bigger -- I'd like to see some numbers, but I seriously doubt that the majority of KDE developers are potential Qt developers. Even if we have to work around Qt bugs quite often. In any case, if Qt wants more contributors out of the KDE developer pool, they'd better ease up their submission process and drop using gerrit. I know that I, and I've been a KDE developer for over a decade, won't do anything for Qt in my spare time as long as they have this gerrit-based workflow. If people are paying me for it, well, that's different, but no way am I going to submit to that process for fun and for for free. In short, Qt uses gerrit is a bogus argument in favor of gerrit. And I am pretty sure that if gerrit becomes a requirement for working on KDE projects, KDE will not just lose a lot of developers, it will lose a lot of projects. +1 I occasionally contributed patches in the past to Qt, but since the current gerrit setup was introduced I've never even contemplated doing so because it looks too much effort to get to grips with. It's far too off-putting for occasional contributors. -- David Jarvie. KDE developer. KAlarm author - http://www.astrojar.org.uk/kalarm
Re: Another proposal for modernization of our infrastructure
On Monday, 2 February 2015 11:22:57 CEST, David Jarvie wrote: I occasionally contributed patches in the past to Qt, but since the current gerrit setup was introduced I've never even contemplated doing so because it looks too much effort to get to grips with. It's far too off-putting for occasional contributors. Seems that the Qt project has a big problem in the *perceived* complexity of getting involved, then. I'm saying perceived because you said that you did not try; the mere fact that they use a tool which has a rumour of being hard to use, along with the state of their documentation, suggests to you and to others that it is apparently a big pain to get involved. I understand that people do not want KDE to end up that way. However, we also have people with little to no experience using Gerrit just fine. Shall we therefore focus on explaining that contributing through Gerrit is actually not painful? The fact that the Qt project uses an obsolete version of Gerrit along with a non-stadard workflow and keep changing the branch names certainly doesn't help. I can see why it's confusing to suddenly see no master branch, for example. Anyway, the proposal which I have for solving this problem is about writing nice documentation. Maybe even screencasts showing how easy it can be for a total beginner to send their first patch. With kind regards, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Saturday 31 January 2015 20:56:41 Martin Graesslin wrote: I have to agree. Whenever I need to do a change for Qt I need to google for how to do it. Including putting serious thought in how the push command must look like, how I need to adjust the examples provided and for which ref/for/foo it has to be this time. This is something which seriously concerns me with the outlook of gerrit. git has a steep learning curve, gerrit adds quite some further steepness to it Ossi has been working on a new script called git-gpush that will automate most of the pushing work for you. All you'll need to do is tell it which commits you want to push, like: git gpush HEAD +thiago +ossi +tronical It would have been ready if I had had the time to review it. You can find it in his dashboard and mine (in qt/qtrepotools). -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 signature.asc Description: This is a digitally signed message part.
Re: Another proposal for modernization of our infrastructure
On Friday 30 January 2015 10:57:33 Ben Cooksley wrote: Given that upstream has had multiple attempts now at an improved interface, I would question whether they would be willing to accept a user interface which is suitable for our needs. It appears that they are quite comfortable with an interface many find unintuitive or difficult to use. If they weren't, considering the number of backers it has - one would think someone would have sponsored such an interface. They would have if they still had major problems with the usability of the tool. It probably just so happens that all the backers are used to the interface, however bad it might be, and don't feel the need to sponsor such a work. I find myself in that category. Provided there is a one-page-diff support, Gerrit works very well for me and I don't find it clumsy at all. That's not to say I don't want it improved -- I do -- but I don't feel the pressing need to for someone to work on it. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358
Re: Another proposal for modernization of our infrastructure
On Saturday 31 January 2015 20:37:31 Christoph Feck wrote: On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. Please note the instructions stop at the Configuring Git heading. All of the rest is already done for anyone who is already using Git. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358
Re: Another proposal for modernization of our infrastructure
On Friday, 30 January 2015 03:30:55 CEST, Kevin Kofler wrote: Unfortunately, file level strikes me as a less than helpful default. Can this be changed to line-level merges in our instance? (I think the ideal would be to use git's native merging algorithm(s), but I expect some limitations due to the convenient web resolving UI.) Um, it seems that I managed to confuse myself here -- this feature already exists and is active on our instance. A content merge is already enabled. I'm afraid I never needeed it yet, so I cannot comment on what sorts of conflicts it can solve, and what conflicts require a human action to fix. Maybe someone already needed this and can provide more details? As a result, people who opt to disable JavaScript in their browser for whatever reason (e.g., security) will have: I agree with the sentiment in general, but at the same time, one could reasonably point out that Gerrit's choice of port 29184 for git-over-SSH might trip some corporate firewalls because it is not HTTP or HTTPS. Sure, disabling outbound traffic to insecure ports can increase security of our corporation. It is up to everyone to evaluate whether that particular benefit is something worth the trouble for them. In this context, I wonder what security benefits it brings when someone disables JavaScript for a trusted service where the entire set of JS code is free software. * the Gerrit web interface not working at all (or at least not until such an alternative web UI is implemented in a way not requiring client-side JavaScript and deployed on KDE infrastructure), * the integration between various utilities also not working, e.g., Bugzilla will not list pending review requests at all. To me, this contradicts the web maxim of graceful degradation. Note that even if people disable JS, they are still be able to do any of the following as soon as they get a change number from e.g. the project mailing list or an IRC channel: - pull changes from Gerrit for local testing, - upload patches and create new changes or push updates to existing ones, - record a result of their code review, including full voting and an eventual merge. Why can the work not be done on the server side? Especially for the integration between services, I would expect a simple API call for data lookup to be doable on the server side at least as easily as from client- side JavaScript. Yes, the technical options are assentially unlimited and someone /could/ write code doing just that. Maybe nobody sees a value in disabling JS to be compelling enough to commit their time. Or maybe people actually like JS and appreciate the feature set it brings. One benefit of having the UI implemented in JS is that the APIs are *guaranteed* to offer enough functionality to be able to implement an alternative client as, say, an IDE plugin. If Gerrit was generating static web pages, it would be very easy to accidentally introduce features which just could not be implemented in other clients because the required APIs were not made public by accident. These other clients exist today, btw. If a lack of support for JS-less browsers bothers you, may I suggest installing Gertty? It even has support for making patch review offline when on a plane, and bidirectionally synces stuff when you reconnect later. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. Christoph Feck (kdepepo)
Re: Another proposal for modernization of our infrastructure
On 01/31/2015 08:37 PM, Christoph Feck wrote: Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. This information could be pared down considerably and presented much more nicely - and would have to be if we decide to use gerrit. In addition, kdesrc-build could automate some of the local setup (Qt's ./init-repository script does as well) to get people started faster. I agree the barrier-to-entry with gerrit is high, but I don't consider that Qt page to be up to our standards if we were to put serious thought behind communicating the use of gerrit. Keep in mind though I've raged quite a bit about gerrit's UI in this thread, and yes, this is part of it. I think gerrit's useful, but I don't enjoy using it, personally. Christoph Feck (kdepepo) Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On 01/31/2015 08:57 PM, Alexander Neundorf wrote: hmm, looking back at our switch to git, I don't consider our standards for documentation of the developer workflow as very high unfortunately. :-/ Considering I wrote the majority of https://community.kde.org/Sysadmin/GitKdeOrgManual I guess I'll have to take that to heart, then ... My vision for how to do a use gerrit page well would be to lead with the minimal recipe for how to clone a project and upload a change for review. It should put visual focus on those couple of commands, and make explanation of what they do secondary. It should also encourage the use of kdesrc- build as an even quicker alternative. Most of the content on the Qt page should only be much further down and less in- your-face, or even on a secondary page, and framed as con- venience tweaks useful when contributing regularly. Basically similar to how programming languages or web frame- works will structure their website these days, leading with get you started info and putting effort into making that a fast path. It'd be nice to even make this recipe box a bit dynamic - click something to get a project selector, pick the KDE project, and get the commands shown adapted for copying with the right repo URL inserted. I think it's doable in principle, but I think the best- case scenario is still to make gerrit be not a hassle on coders; I think both its UI and the way you interact with it have fundamental problems when it comes to non-code contributors. I don't think this is shifting of goal posts either be- cause (a) ReviewBoard already has some screenshot functio- nality and we actually have policy in place to require showing screenshots for review requests that change UI (i.e. this is something gerrit will actually regress us on afaics) (b) the very reason we entertain shaking up our infra is because we're unhappy with limitations in our tools. Alex Cheers, Eike
Re: Another proposal for modernization of our infrastructure
2015-01-31 19:37 GMT+00:00 Christoph Feck cf...@kde.org: On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. I fully agree here! I only now managed to contribute my first patch to Qt because setting up gerrit really scared me. I would have moved on to another project if I hadn't been using+developing KDE for a few years. After having used it now I think gerrit is a great tool but the barrier of entry is just too high. Unless that somehow gets fixed I am strongly in favour of the Phabricator proposal. Alex
Re: Another proposal for modernization of our infrastructure
On 01/31/2015 08:57 PM, Alexander Neundorf wrote: hmm, looking back at our switch to git, I don't consider our standards for documentation of the developer workflow as very high unfortunately. :-/ Considering I wrote the majority of https://community.kde.org/Sysadmin/GitKdeOrgManual I guess I'll have to take that to heart, then ... For me as newcomer the mentioned wiki page and https://techbase.kde.org/Policies/Commit_Policy#Special_keywords_in_GIT_and_SVN_log_messages were valuable resources to get familiar with the KDE dev workflow. So it cannot be that bad. ;-)
Re: Another proposal for modernization of our infrastructure
On Saturday, January 31, 2015 21:11:19 Eike Hein wrote: On 01/31/2015 08:57 PM, Alexander Neundorf wrote: hmm, looking back at our switch to git, I don't consider our standards for documentation of the developer workflow as very high unfortunately. :-/ Considering I wrote the majority of https://community.kde.org/Sysadmin/GitKdeOrgManual I guess I'll have to take that to heart, then ... this page is the best we have for git, it contains a lot, but also things which I as a developer who just wants to contribute to some project, am not interested in, e.g. Commands related to repository importing, Commands for system adminisitrators, Commands to manage personal repositories and more. I was more referring to this one: https://techbase.kde.org/Development/Git , and that KDE still couldn't agree even on a set of git workflows to use, the wiki page still just lists a few proposed drafts. :-/ Alex
Re: Another proposal for modernization of our infrastructure
On 01/31/2015 10:37 AM, Jan Kundrát wrote: About the we could vs. we will in general, I have to admit I'm slightly confused by that. The proposal is careful to describe what is available today, and to make a clear difference in saying what needs to be done in future. Maybe some part needs clarification -- what parts do you think are more of the yes-this-would-be-nice-but-I'm-worried nature? I just wanted to make the point (but I agree I made it a bit poorly) that it's worth differentiating between features the proposed solutions are guaranteed to deliver right away, and features they may enable in the future, but only if certain other conditions are met, e.g. raw infrastructure expansion. That's not a knock against your proposal, though - it's much easier to e.g. ask for hardware donations when you can point to a clearly identifyable need. --- snip --- I'd like to summarize my current feelings on both proposals. Here's what I think gerrit's strong points are: * There's undeniably synergy and cultural alignment with middleware communities we depend on to be had. Qt is using gerrit and we intend to remain a major stakeholder in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway. KDE using gerrit lowers the barrier for KDE developers to work upstream, and for folks in the wider Qt community to involve themselves in KDE. Particularly for Frame- works development this could be a boon. * gerrit and the community around gerrit (which includes Jan) appears to have considerable chops when it comes to solving hard infrastructure problems like advanced CI and replication. This is stuff that can enhance the quality of our products by improving our processes, but it also stands to make KDE a more attractive envi- ronment for incubating new projects. Infrastructure should not be the sole platform we run recruitment on, but staying competitive is important, and one way to do that is to try and lift things only a community of KDE's size can lift -- and that can e.g. include gathering donations for fancy CI cluster setups. Here's what I think Phabricator's strong point is: * From what I've seen and played with so far, I really like the UI. This is a short bullet, but an important one. * In terms of cultural alignment, I see communities like Blender and Wikimedia pick Phabricator. Lowering the barrier for cross-pollination with communities like this seems very attractive to me, because there's types of talent engaged in those communities that we need more of in KDE, e.g. designers and artists. * I expect the above drivers to make it more likely that Phabricator will develop functionality that will make it more useful for these types of contributors. For example, good diff viewers for visual assets, good handling of screenshot attachments to change proposals, etc. I feel like it is less likely that gerrit will become similarly useful for non-code assets. * The conflation of change review and issue tracking in Phabricator makes sense in the above context as well, and could greatly improve our non-programming develop- ment processes. In short, I don't see gerrit benefit our non-code contributors at all, but I see a shot at Phabricator doing so. Given all of this, I find it really hard to have a firm opinion at this point. Both proposals promise strong benefits to different aspects of what the community is doing, which makes this an exercise in weighing those aspects and predicting future outcomes based on different ratios between them. That said, at this point, I have more experience work- ing with gerrit than with Phabricator, and I think that's something that needs to be addressed as part of this process. I think we should proceed with setting up a test instance and giving it a spin, this should help with getting things into clearer focus. Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On Saturday 31 January 2015 20:37:31 Christoph Feck wrote: On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. I have to agree. Whenever I need to do a change for Qt I need to google for how to do it. Including putting serious thought in how the push command must look like, how I need to adjust the examples provided and for which ref/for/foo it has to be this time. This is something which seriously concerns me with the outlook of gerrit. git has a steep learning curve, gerrit adds quite some further steepness to it :-( Cheers Martin signature.asc Description: This is a digitally signed message part.
Re: Another proposal for modernization of our infrastructure
On Saturday, January 31, 2015 20:52:44 Eike Hein wrote: On 01/31/2015 08:37 PM, Christoph Feck wrote: Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. This information could be pared down considerably and presented much more nicely - and would have to be if we decide to use gerrit. In addition, kdesrc-build could automate some of the local setup (Qt's ./init-repository script does as well) to get people started faster. I agree the barrier-to-entry with gerrit is high, but I don't consider that Qt page to be up to our standards if we were to put serious thought behind communicating the use of gerrit. hmm, looking back at our switch to git, I don't consider our standards for documentation of the developer workflow as very high unfortunately. :-/ Alex
Re: Another proposal for modernization of our infrastructure
On Sat, 31 Jan 2015, Christoph Feck wrote: On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. Christoph Feck (kdepepo) Maybe quite a few KDE developers would want to contribute to Qt, and maybe Qt would like more contributors, but KDE is so much bigger -- I'd like to see some numbers, but I seriously doubt that the majority of KDE developers are potential Qt developers. Even if we have to work around Qt bugs quite often. In any case, if Qt wants more contributors out of the KDE developer pool, they'd better ease up their submission process and drop using gerrit. I know that I, and I've been a KDE developer for over a decade, won't do anything for Qt in my spare time as long as they have this gerrit-based workflow. If people are paying me for it, well, that's different, but no way am I going to submit to that process for fun and for for free. In short, Qt uses gerrit is a bogus argument in favor of gerrit. And I am pretty sure that if gerrit becomes a requirement for working on KDE projects, KDE will not just lose a lot of developers, it will lose a lot of projects. Boudewijn
Re: Another proposal for modernization of our infrastructure
On 01/31/2015 09:25 PM, Boudewijn Rempt wrote: In short, Qt uses gerrit is a bogus argument in favor of gerrit. The argument isn't so much that gerrit is working well for Qt, but more that there's a certain simplicity in using the same tooling across the KDE/Qt stack, and that KDE benefits from having more KDE people active in Qt. But I actually find contributing to Qt pretty frus- trating (the infra is flaky, the process tends to break down here and there, and I sometimes have to step out- side gerrit and side-channel via email to get things moving again), yeah. It's one bullet from a long mail that addresses many other points, though ... I agree that the KDE community has concerns and ambitions unrelated to its relation- ship with Qt and a broader activity scope than the Qt project, and our tooling needs to be evaluated in that broader context. Boudewijn Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On Saturday, January 31, 2015 20:07:42 Eike Hein wrote: On 01/31/2015 10:37 AM, Jan Kundrát wrote: I'd like to summarize my current feelings on both proposals. Here's what I think gerrit's strong points are: * There's undeniably synergy and cultural alignment with middleware communities we depend on to be had. Qt is using gerrit and we intend to remain a major stakeholder in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway. KDE using gerrit lowers the barrier for KDE developers to work upstream, and for folks in the wider Qt community to involve themselves in KDE. Particularly for Frame- works development this could be a boon. Aside from the point that Christopher made that a too strictly restricted workflow will drive developers away, I also think this point is overstressed. The problem is not the developers that are good enough to actually contribute to Qt. The problem is the new people that will never stay around if they have to too much to learn until they can make their first contribution. It is one thing if there is one tool that is totally too weak to work for experienced people and one tool that is awesome but very difficult to learn. But that's not the situation we have here. I think we have one tool that is very good and then one that many have pointed out as cumbersome and difficult to learn for not very experienced people - but - with an edge when it comes to advanced git users. I know how long it took for me to get used ot git, and I think I'm pretty experienced. Adding to that burden is not the way to get new people. I'd like to liken this with myself as an emacs user. I have so far not seen one feature in any visual development environment that is not also available as a plugin feature for emacs. Emacs simply has a superset of all the features of all the other integrated devtools out there - but - it does it without graphics and with keyboard shortcuts. It's not without reason that emacs has sometimes been interpreted as escape-meta-alt-control-shift. :) Despite all this awesomeness, I have a hard time convincing people to use emacs. And why? Because it has a steep learning curve. So if I were to force people to use emacs they would run for the hills. I think I could outedit almost anybody when it comes to pure editing text files whatever the task. This is due to the sheer size of the emacs toolbox coupled with extreme versatility of the tools themselves. But this is not what makes people productive with development. Editing is but one of the many steps we have to do when developing our KDE applications. And it's the same way with patch review. Even if a complicated tool would make the top people's job easier, the cost to the community would be too great if we were to force this tool on everybody. So with the current choices I from now on will support Phabricator as the choice of the community. Caveat: The above choice is only valid if Phabricator is actually up to the task. I guess the tests will tell us this. * gerrit and the community around gerrit (which includes Jan) appears to have considerable chops when it comes to solving hard infrastructure problems like advanced CI and replication. This is stuff that can enhance the quality of our products by improving our processes, but it also stands to make KDE a more attractive envi- ronment for incubating new projects. Infrastructure should not be the sole platform we run recruitment on, but staying competitive is important, and one way to do that is to try and lift things only a community of KDE's size can lift -- and that can e.g. include gathering donations for fancy CI cluster setups. ...provided we will have any new people that can execute those processes. :) I'm still unconvinced that gerrit is the tool to grow our community. Here's what I think Phabricator's strong point is: * From what I've seen and played with so far, I really like the UI. This is a short bullet, but an important one. * In terms of cultural alignment, I see communities like Blender and Wikimedia pick Phabricator. Lowering the barrier for cross-pollination with communities like this seems very attractive to me, because there's types of talent engaged in those communities that we need more of in KDE, e.g. designers and artists. This is a good point that I actually didn't consider before. How does e.g. the Visual Design Group Work with review? Do they? And if not, would they if they had the proper tools? And how do Gerrit and Phabricator stack up here? * I expect the above drivers to make it more likely that Phabricator will develop functionality that will make it more useful for these types of contributors. For example, good diff viewers for visual assets, good handling of screenshot attachments to change proposals, etc.
Re: Another proposal for modernization of our infrastructure
... in fact, even if you consider Qt and KDE in symbiosis, you could say that KDE is the place you can do things that don't fit the narrower scope of Qt Project, and that calls for tooling that supports things gerrit doesn't support well enough. If gerrit is a constraint, then KDE picking tooling other than gerrit would arguably contribute to the combined ecosystem by making a resource available that is otherwise lacking. Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On Samstag, 31. Januar 2015 20:37:31 CEST, Christoph Feck wrote: On Saturday 31 January 2015 20:07:42 Eike Hein wrote: [...] Qt is using gerrit and we intend to remain a major stakeholde in Qt development, which means a sizable number of KDE developers need to be familiar with gerrit anyway [...] Excuse me, but if KDE developers will have to follow equivalent steps as described at http://qt-project.org/wiki/Setting-up-Gerrit to contribute, then I predict another big loss of developers. Aside that this is an exhaustive HowTo on git and gerrit*, there're apparently upload your plain diff webfrontends. (Though I think the question was brought up and not answered how follow-up patches are handled - eg. whether you've to pass some existing gerrit url to fetch the change-id from) Gerrit is however very git-centric, so if you managed the git challenge, there's relly no additional complexity (and that includes things to learn. Messy repo/branch setups are messy). If you however did W!!! GIT!!! (and I was close to that twice or thrice when KDE moved svn - git) - you're indeed lost (resp. have to follow a very cumbersome patch upload path that kills all user-side gerrit benefits at once) Cheers, Thomas * Setting up git, with creating an account, uploading a key and configuring git certainly /is/ a one-time job you've to fulfill, but it's git, resp. actually scm related, not gerrit related. The only** gerrit overhead here was really to dowload the change-id injecting hook. **Though the demo setup also required to add a second repo, but that's hardly the case if gerrit would do the entire scm instead of being replicated to the git.kde repo that I originally fetched the code from (and because that's also been a clone, I actually have three repos there ;-)
Re: Another proposal for modernization of our infrastructure
On Samstag, 31. Januar 2015 21:11:19 CEST, Eike Hein wrote: ReviewBoard already has some screenshot functio- nality and we actually have policy in place to require showing screenshots for review requests that change UI (i.e. this is something gerrit will actually regress us on afaics) I don't have a commit link, but asked Jan that in the context of some trojita patch and he promised that the current git version of gerrit already has the ability to upload illustrating screenshots. Cheers, Thomas
Re: Another proposal for modernization of our infrastructure
On Saturday, 31 January 2015 22:09:36 CEST, Thomas Lübking wrote: Aside that this is an exhaustive HowTo on git and gerrit*, there're apparently upload your plain diff webfrontends. (Though I think the question was brought up and not answered how follow-up patches are handled - eg. whether you've to pass some existing gerrit url to fetch the change-id from) The gerrit-patch-uploader says that you are supposed to copy-paste the Change-Id line from e.g. the change page, yes. Keep in mind that Gerrit 2.11 allows editing your change right from the Gerrit's web UI, so there is no need to upload another followup patch for a significant chunk of these changes. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Saturday, 31 January 2015 21:38:23 CEST, Inge Wallin wrote: It is one thing if there is one tool that is totally too weak to work for experienced people and one tool that is awesome but very difficult to learn. But that's not the situation we have here. I think we have one tool that is very good and then one that many have pointed out as cumbersome and difficult to learn for not very experienced people - but - with an edge when it comes to advanced git users. I am a bit surprised to read that Phabricator is apparently considered very good even though no KDE project has tried it yet. I was under the impression that the usual order of steps is: 1) install a tool, 2) play with it, 3) identify its strenghts and weeknesses, 4) make an informed opinion. Did I miss something? Which KDE project has been testing Phabricator? I know how long it took for me to get used ot git, and I think I'm pretty experienced. Adding to that burden is not the way to get new people. I'll repeat my request from earlier in this thread -- please quantify the expected increase of the barrier to entry. Different does not imply harder. In Trojita, we have GCI students submitting patches wihin 15 minutes, following the developer-oriented newbie-unfriendly documentation. 15 minutes for a high-schooler to start contributing. Translators have sent patches via Gerrit as well. Interestingly enough, the only complaint was from an experienced KDE developer, not from a newcomer. Maybe the newcomers just do not care whether they're learning about Phabricator, Reviewboard or Gerrit. Anyway, I don't really see a problem with the following steps: 0) Ignore any documentation, especially the clone command which already includes getting the hook. I don't read no documentation. 1) git clone ... 2) make changes make run 3) git commit -a 4) git push 5) Damn, that stupid Gerrit tells me on a command line that I should push to refs/for/master instead of just master. Oh well, stupid crap. 6) git push origin refs/for/master 7) Damn, now that stupid Gerrit tells me that I am supposed to copy-paste this line into my terminal for some crazy hook or what not, and to amend my commits. Annoying crap! 8) (copy-paste a line from Gerrit's output) 9) git commit --amend 10) git push origin refs/for/master For those who actually read the documentation, it's of course more straighforward, but we all know that nobody reads documentation, hence the extra detours. Besides, it has been already established that there will be support for direct patch upload. It would be very interesting to hear the VDG people's and the translators' and documentors' view on this. Given that the translators still use SVN, I sincerely hope that the proposed outcome does not involve us switching back to Subversion, or reducing our use of Git to an SVN equivalent. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
Hi, just a short note (don't want this to become a complete subthread distracting from the actual proposal-discussion) On Sat, Jan 31, 2015 at 9:38 PM, Alexander Neundorf neund...@kde.org wrote: that KDE still couldn't agree even on a set of git workflows to use, the wiki page still just lists a few proposed drafts. :-/ I think thats actually a good thing and would just not work for 'KDE' (i.e. the whole community with all projects). KDE is just to big and diverse to impose one particular git workflow onto all its projects, in particular since some of them also impose certain release management. Different projects move at different paces, they have different needs for their release management and the maintainers actually doing the work may have differing opinions or even just time. I always found it nice that 'KDE' did not impose too many rules onto its projects and their developers. Andreas
Re: Another proposal for modernization of our infrastructure
On Samstag, 31. Januar 2015 22:41:36 CEST, Jan Kundrát wrote: Maybe the newcomers just do not care whether they're learning about Phabricator, Reviewboard or Gerrit. Since it's always better to waste CPU time than to waste my time, we could btw. also provide a bash script that does all the required stuff (incl. the key generation and git aliases config) - and regardless of the actual tool in use (given Phabricator supports automation through some API for this), since the git process requires all that and we probably want to prevent contributors from downloading tarball after tarball. Cheers, Thomas
Re: Another proposal for modernization of our infrastructure
On Thursday, 29 January 2015 19:31:20 CEST, Eike Hein wrote: Just for the record: I consider you a KDE sysadmin (you're administrating infra used by KDE, after all), so I meant the kde.org more general. Thanks. I forgot about this mail, and I realize that I am not sure whether my reply was clear or if it left some space for a possible misunderstanding. Sorry for noise if we already understood each other. The following KDE people have root there: - Ben Cooksley - Frederik Gladhorn - Victor Blazques If there are others who need access, there's no problem adding them. The KDE server holding the PostgreSQL backups is tesla.kde.org. -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Saturday, January 31, 2015 22:52:22 Andreas Pakulat wrote: Hi, just a short note (don't want this to become a complete subthread distracting from the actual proposal-discussion) On Sat, Jan 31, 2015 at 9:38 PM, Alexander Neundorf neund...@kde.org wrote: that KDE still couldn't agree even on a set of git workflows to use, the wiki page still just lists a few proposed drafts. :-/ I think thats actually a good thing and would just not work for 'KDE' (i.e. the whole community with all projects). KDE is just to big and diverse to impose one particular git workflow onto all its projects, in particular since some of them also impose certain release management. Different projects move at different paces, they have different needs for their release management and the maintainers actually doing the work may have differing opinions or even just time. Even if it should not be possible to agree on ONE workflow, maybe it should be possible to agree on two or three (e.g. everything in master, or everything in feature branches, and maybe a third one) standard workflows, and document them properly. Then every project could simply state we are using workflow (A|B| C) and things would be clear. It wouldn't be too much IMO to make it mandatory for KDE projects to use one of two/three official KDE git workflows. Alex
Re: Another proposal for modernization of our infrastructure
On Saturday, January 31, 2015 22:41:36 Jan Kundrát wrote: On Saturday, 31 January 2015 21:38:23 CEST, Inge Wallin wrote: It is one thing if there is one tool that is totally too weak to work for experienced people and one tool that is awesome but very difficult to learn. But that's not the situation we have here. I think we have one tool that is very good and then one that many have pointed out as cumbersome and difficult to learn for not very experienced people - but - with an edge when it comes to advanced git users. I am a bit surprised to read that Phabricator is apparently considered very good even though no KDE project has tried it yet. I was under the impression that the usual order of steps is: This is true. It's not proven yet. What I described above was my impression taken from the comments by the people who did try it in other situations. The real status has to be determined by tests. So let me add a supposedly to the sentence above. No decision should be made without real tests, of course. 1) install a tool, 2) play with it, 3) identify its strenghts and weeknesses, 4) make an informed opinion. Did I miss something? Which KDE project has been testing Phabricator? I know how long it took for me to get used ot git, and I think I'm pretty experienced. Adding to that burden is not the way to get new people. I'll repeat my request from earlier in this thread -- please quantify the expected increase of the barrier to entry. Different does not imply harder. It does, actually. Not forever, of course, but the very fact that something is different than what you know before makes it more difficult to learn. But you knew that already, so I wonder why you were saying the above. And also, different from what? I bet that both Gerrit and Phabricator are quite different than Reviewboard. But that aside, I am a bit surprised by your request to quantify difficulty. What unit do you want it in? Brain-hours? Headaches? I would be interested in your own opinion about a quantified measure of difficulty to learn and use Gerrit compared to Phabricator. Or if you are unfamiliar with Phabricator, then the quantified measure of difficulty for Gerrit is enough. Anyway, I'll let it rest from here on. I suppose that the real tests will show us which tool is the best, and which weaknesses and strength they have.
Re: Another proposal for modernization of our infrastructure
On Saturday, January 31, 2015 23:14:01 Ben Cooksley wrote: About the only point left standing is that it doesn't check individual subcommits, but we've yet to see whether the KDE project as a whole sees this as necessary - especially considering that the vast majority of projects would use CI in an advisory role only for code reviews, and would regular developers continuing to push directly (necessitating post-push CI anyway). I don't know how the rest of KDE does it, but in Calligra we have a pretty strict new-features-in-branches policy. You develop a new feature in a branch, and then put it up for review when it's finished or at some important milestone. When I work in a feature branch, I often make 20-50 commits, smaller and bigger, before I'm done and 100 is far from unheard of. Naturally I make sure that my patches build before I commit anything but also naturally I don't test all configurations or all platforms. I think it would be VERY bad utilization of the KDE resources to check all my commits with the full power of CI. It's a good habit to make small and simple changes and frankly I think it would be impossible to do. Or have I misunderstood what people mean when they say subcommit? Cheers, Jan Regards, Ben -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Saturday, 31 January 2015 12:20:15 CEST, Inge Wallin wrote: Given how few of our community who have participated so far, I think it borders on pure falsehood to claim clear consensus on *anything*. I would put more like some people want it, and I can certainly see the appeal. Fair enough, you have a point -- I suspect there is no consensus that CI is useful, or that there is any value in having a clean git history without fix missing semicolon commits. I agree that having a per-commit CI coverage can be well considered an undesirable thing by some developers. Which is why I have no intention of pushing these to all KDE projects. What I am proposing is an opt-in for those who care. But from that to simply state the costs in HW are worth it (and conveniently forgetting cost in maintenance) is a very long step. I believe that the cost of maintenance is sufficiently covered by section 5 of the proposal, so I have to admit that I don't know what I am conventiently forgetting about. Could you please explain what maintenance cost you are worried about? Is it perhaps related to the number of build jobs being run, or the number of throwaway VMs we use for builds? Is it about the services which replace Jenkins? The scripting which is currently used for build VMs with Gerrit/Zuul lives at [1]. The bootstrapping part which turns a downloaded, minimal VM image into a build node is [2]. Cheers, Jan [1] http://quickgit.kde.org/?p=sysadmin%2Fgerrit-project-config.gita=tree [2] http://quickgit.kde.org/?p=sysadmin%2Fgerrit-project-config.gita=blobf=turbo-hipster%2Fcloud-init.sh -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Thursday, 29 January 2015 22:57:33 CEST, Ben Cooksley wrote: Given that upstream has had multiple attempts now at an improved interface, I would question whether they would be willing to accept a user interface which is suitable for our needs. It appears that they are quite comfortable with an interface many find unintuitive or difficult to use. If they weren't, considering the number of backers it has - one would think someone would have sponsored such an interface. I don't think this is an accurate and fair description of upstream. They fixed a UI usability glitch that Martin complained about in less than 12 hours. That sounds like they are pretty welcoming to 3rd-party feedback, IMHO. As for the CI backend, please mention what is wrong with Jenkins - if it would be integrated to check code review submissions. The reasons for considering another CI platform are described in my report in section 3.3. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Saturday, January 31, 2015 10:37:26 Jan Kundrát wrote: On Thursday, 29 January 2015 21:03:32 CEST, Eike Hein wrote: I think it's a real concern, and I'm wary of we can patch it away because carrying a huge custom patch delta for UI mods is what kept us from upgrading Bugzilla for multiple years. I think is it realistic that we can maintain this and keep up with upstream even if Ben or Jan get hit by a bus is an important question with regard to both proposals. That's a very good question, and a reason for why I am not patching Gerrit with stuff not accepted upstream. I agree that carrying big custom patches won't scale. So far, we don't have any patches at all. I'll be backporting stuff such as the show-headers-prior-to-cpp from 2.11 because it is super-easy to do so, and because 2.11 isn't released yet. We also have some JavaScript proof-of-concept for Bugzilla integration. You can check its complexity at [1]. I managed to write that over a Sunday, and I am definitely not a web guy. I had zero jQuery experience prior to this. I have similar concerns with some of the promised benefits in the proposal because they strike me more of we could, which is cool, but it's not we will. E.g. if test build- ing precombined patches takes an OpenStack cluster - do we have one? Where are we going to get that horsepower? Can we keep it? Designing contingency plans is indeed important (see section 5 of that proposal; it talks about managing infrastructure-as-code). You are also right that the current infrastructure is best-effort and that KDE won't get an SLA without paying for one. If we (KDE) need an SLA, we (the company the cluster is hosted at) will be happy to be asked for a quote :). Or we (KDE) can just host this stuff anywhere else and pay someone else. But it seems to me that we already have pretty clear consensus that we absolutely do want a pre-approval CI coverage, and that the costs in HW are worth it. The rest of the discussion aside, this is something I want to strongly object to. Given how few of our community who have participated so far, I think it borders on pure falsehood to claim clear consensus on *anything*. I would put more like some people want it, and I can certainly see the appeal. But from that to simply state the costs in HW are worth it (and conveniently forgetting cost in maintenance) is a very long step. Does someone from KDE e.V. know whether we could get some free HW resources from a commercial partner (hi RedHat/SuSE/Digia)? Do we have some backup cash to e.g. rent VM time from Amazon/Rackspace/whatever in an unlikely event that the current hosting platform is withdrawn with no prior notice? About the we could vs. we will in general, I have to admit I'm slightly confused by that. The proposal is careful to describe what is available today, and to make a clear difference in saying what needs to be done in future. Maybe some part needs clarification -- what parts do you think are more of the yes-this-would-be-nice-but-I'm-worried nature? With kind regards, Jan [1] https://gerrit.vesnicky.cesnet.cz/r/static/bugzilla.js
Re: Another proposal for modernization of our infrastructure
On Thursday, 29 January 2015 21:03:32 CEST, Eike Hein wrote: I think it's a real concern, and I'm wary of we can patch it away because carrying a huge custom patch delta for UI mods is what kept us from upgrading Bugzilla for multiple years. I think is it realistic that we can maintain this and keep up with upstream even if Ben or Jan get hit by a bus is an important question with regard to both proposals. That's a very good question, and a reason for why I am not patching Gerrit with stuff not accepted upstream. I agree that carrying big custom patches won't scale. So far, we don't have any patches at all. I'll be backporting stuff such as the show-headers-prior-to-cpp from 2.11 because it is super-easy to do so, and because 2.11 isn't released yet. We also have some JavaScript proof-of-concept for Bugzilla integration. You can check its complexity at [1]. I managed to write that over a Sunday, and I am definitely not a web guy. I had zero jQuery experience prior to this. I have similar concerns with some of the promised benefits in the proposal because they strike me more of we could, which is cool, but it's not we will. E.g. if test build- ing precombined patches takes an OpenStack cluster - do we have one? Where are we going to get that horsepower? Can we keep it? Designing contingency plans is indeed important (see section 5 of that proposal; it talks about managing infrastructure-as-code). You are also right that the current infrastructure is best-effort and that KDE won't get an SLA without paying for one. If we (KDE) need an SLA, we (the company the cluster is hosted at) will be happy to be asked for a quote :). Or we (KDE) can just host this stuff anywhere else and pay someone else. But it seems to me that we already have pretty clear consensus that we absolutely do want a pre-approval CI coverage, and that the costs in HW are worth it. Does someone from KDE e.V. know whether we could get some free HW resources from a commercial partner (hi RedHat/SuSE/Digia)? Do we have some backup cash to e.g. rent VM time from Amazon/Rackspace/whatever in an unlikely event that the current hosting platform is withdrawn with no prior notice? About the we could vs. we will in general, I have to admit I'm slightly confused by that. The proposal is careful to describe what is available today, and to make a clear difference in saying what needs to be done in future. Maybe some part needs clarification -- what parts do you think are more of the yes-this-would-be-nice-but-I'm-worried nature? With kind regards, Jan [1] https://gerrit.vesnicky.cesnet.cz/r/static/bugzilla.js -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Sat, Jan 31, 2015 at 10:53 PM, Jan Kundrát j...@kde.org wrote: On Thursday, 29 January 2015 22:57:33 CEST, Ben Cooksley wrote: Given that upstream has had multiple attempts now at an improved interface, I would question whether they would be willing to accept a user interface which is suitable for our needs. It appears that they are quite comfortable with an interface many find unintuitive or difficult to use. If they weren't, considering the number of backers it has - one would think someone would have sponsored such an interface. I don't think this is an accurate and fair description of upstream. They fixed a UI usability glitch that Martin complained about in less than 12 hours. That sounds like they are pretty welcoming to 3rd-party feedback, IMHO. I respectfully disagree. Fixing a usability glitch and accepting a radical redesign of your interface are completely different. We're not the first ones to complain about it's interface but they've not done anything to remedy this despite a complete redesign (which reduced the level of functionality interestingly). As for the CI backend, please mention what is wrong with Jenkins - if it would be integrated to check code review submissions. The reasons for considering another CI platform are described in my report in section 3.3. I see the following in that section: 1) A note that Jenkins is a glorified job launcher as we don't use any of it's advanced functionality (which I refuted - it is much more than that). 2) Some notes that a proposed patch may be based against a week old revision. This is making assumptions about how a Jenkins setup would be made - as we're in control of what it does there is nothing to stop us trying to apply the patch on top of the latest code in Git. In terms of checking dependency buildability, once again - this is possible but we don't do anything like this at the moment to preserve resources. As for it not having a declarative configuration, we're in the process of refining the Groovy based Job DSL script Scarlett wrote. This will shift the configuration of Jenkins jobs entirely into Git, and depending on how things work out - jobs could be automatically setup when new repositories come into existence or when branch metadata is revised. This is 100% declarative - and also integrated into our existing tooling reducing repetition and storage of information, without the need to convert or replicate the information even temporarily. We may even be able to eliminate the prepare-environment.py script, utilising Jenkins functionality for this instead - and allowing us to eliminate some of the commit hook complexity we have at the moment in the process. About the only point left standing is that it doesn't check individual subcommits, but we've yet to see whether the KDE project as a whole sees this as necessary - especially considering that the vast majority of projects would use CI in an advisory role only for code reviews, and would regular developers continuing to push directly (necessitating post-push CI anyway). Cheers, Jan Regards, Ben -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Saturday, 31 January 2015 11:14:01 CEST, Ben Cooksley wrote: Fixing a usability glitch and accepting a radical redesign of your interface are completely different. Your mail suggested that they apparently do not care about improving their UI, because if they did, they would have solved everything already. I disagree with that, and provide evidence which supports the idea that Gerrit upstream in fact also cares about users, including those who are not already experienced with its UI. We're not the first ones to complain about it's interface but they've not done anything to remedy this despite a complete redesign (which reduced the level of functionality interestingly). How does the ChangeScreen2 reduce the level of functionality? I see the following in that section: 1) A note that Jenkins is a glorified job launcher as we don't use any of it's advanced functionality (which I refuted - it is much more than that). You reiterated that it is cool to have graphs tracking number of failed tests. I proposed to fix the tests instead, and offered a solution which eliminates this problem for tests that are inherently unstable (see 3.3.2.). I also explained how running cppcheck and code coverage fits into this. The way I understand this reasoning is: we have failing tests - we got to have graphs so that people know whether any more tests start failing. That sounds rather suboptimal to me. I would much prefer to fix the actual cause of pain rather than to provide tools which plot the pain level and frequency of patient's seizures. Defect tracking is a tool, not a goal. If there is another tool which ensures that no additional defects can enter a repository, why not simply use that? (Please see the report for dealing with non-deterministic tests; this *is* covered.) 2) Some notes that a proposed patch may be based against a week old revision. This is making assumptions about how a Jenkins setup would be made - as we're in control of what it does there is nothing to stop us trying to apply the patch on top of the latest code in Git. You have to somehow account for the delay between a human reviews a patch and the time it gets merged. Any patch could have landed in the meanwhile. What builds a patch queue so that you have this covered? In terms of checking dependency buildability, once again - this is possible but we don't do anything like this at the moment to preserve resources. Given enough CPUs, how would you do this with our current Jenkins setup? This is absolutely not just a resource problem; you need something to build these projects against appropriate refs, and do this in an atomic manner. Zuul does it, and internally it's a ton of work. Jenkins does not do it. KDE's CI scripts do not do it, either. As for it not having a declarative configuration, we're in the process of refining the Groovy based Job DSL script Scarlett wrote. This will shift the configuration of Jenkins jobs entirely into Git, and depending on how things work out - jobs could be automatically setup when new repositories come into existence or when branch metadata is revised. The report said that there are automated tools which provide workarounds for this aspect of Jenkins. It's good to see KDE adopting one now. However, you are investing resources in making a tool with horrible configuration more usable. More power to you, it's your time, but this is exactly what the report says -- you are working around the tool's limitations here. About the only point left standing is that it doesn't check individual subcommits, but we've yet to see whether the KDE project as a whole sees this as necessary The fact that most of KDE's projects have no use of pre-merge CI does not imply that projects who want to opt-in should be punished. This is absolutely *not* about pushing an advanced workflow to everybody. It is about being *able* to accomodate such an advanced workflow at all. This is possible today with Gerrit+Zuul, and it was easy to configure that way. Our Zuul builds the dependencies (=projects outside of Gerrit) on a per-push basis, exactly how KDE's Jenkins does it. There is no time wasted on doing per-commit builds for these, because nobody could react on failures anymore -- a change is already in master by that point. What this is all about is enforcing that each commit which goes through code review is regression free. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Thursday, 29 January 2015 23:11:29 CEST, Eike Hein wrote: Maybe, but this is actually something I like from the Phabricator proposal: It provides an impression of our relationship with Phabricator upstream, which it says is a good and constructive one. I believe that our relation with the Gerrit upstream community is a good and constructive one, too. We also have patches to demonstrate this. Then we have the infra people of OpenStack which are an excellent group of highly-tallented people who know very well what they are doing, and yet are patient enough to help others and to explain why they are doing stuff the way they do to curious newcomers. That's one of the reasons why this proposal is modeled after the infrastructure which the OpenStack project uses. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Saturday, 31 January 2015 13:08:07 CEST, Inge Wallin wrote: Well, all of the above and more. Hosting, electricity, networking, I'm including all of the above as HW costs in my proposal. We (KDE) do not have our own datacenter after all. manual work as the number of physical machines increas Due to the nature of build jobs which constitute a pretty bursty load, renting VMs sounds like a cost-effective approach for our scale. I do not expect that it would make financial sense for us to procure enough physical HW to cover the peak demand -- hence the VMs. Renting VMs also enables corporate sponsors to offer unused capacity for less money, or even for free. Another benefit is that bootstrapping a new VM is about executing a single command (and this is not hype; it's how all of the current build VMs in use by Gerrit were launched). manual work as the complexity increases, and so on. I would prefer if we stopped using phrases such as increased complexity, and switched to expressing a specific and measurable difference in costs. The reason for this request is that increased complexity is almost always true, yet people have no way of actually judging what it is going to mean. For example, we have in-house expertise in Jenkins where Ben, Nicolas, Scarlett and Marko understand the stack we use. Switching to anything else will cost us some time to get the respective sysadmins up to speed to the new tools. I do not question this. However, chances are that at the same time adopting a new system could provide significant savings in future. For example, if a new system allowed easier updates to the build structure, it is reasonable to have a wiki page with instructions and to let developers propose changes for their projects. That way, sysadmin time could be freed for other tasks. So even though there is some initial investment, the long-term operation can easily get less demanding, and cheaper. In fact, a better tooling could attract more people. Hey, this is a nice system which provides almost exactly what I need; can I improve it? It also presents an opportunity to clean some of the cruft accumulated over years, thereby reducing the barrier of entry for new contributors and improving the bus factor. Finally, there is also that aspect of providing a better service to our developers. Running infrastructure costs something, yet we do it. IMHO, we should strive for offering the best service that we can afford with the resources we have today, and have a contingency plan for future. Right now, we have access to some nice CPUs for free. If it proves useful and we get used to it and if the resources stop being free at some future point and if we cannot obtain a free alternative from some other sponsor, then we should take a look on how much money it would cost us on a comercial basis at that point. If the price is too big, then the service is *not* that useful for us after all, and we will live without it just fine. If, however, we decide that the cost is worth the price and no free/cheaper altenrative exists, then we should go and pay. There is no vendor lock-in, and it is trivial to migrate to another provider. I just don't see why we should be actively prevented from using resources which we have today just because we might have to ask somewhere else in future. Turning off pre-merge CI would get us back to the current level of resource utilization immediately. That's what this proposal is all about, and please keep in mind that it's something which is already deployed and tested. It isn't an out-of-thin-blue proposal with uncertain cost of deployment. Everything can be automated in theory but in practice there is never a fully automatic system. Yes, which is why it's important to looks at the architecture and to spell out what our expectations of maintenance effort are. It might be reasonable to e.g. compare them with the current effort which goes into keeping Jenkins alive and modernizing it. Keeping the current state has its very real cost, too. With kind regards, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Thursday 29 January 2015 17:22:45 Thomas Lübking wrote: Maybe it's possible to borrow or upstream the Qt mod? See the repository qtqa/gerrit in the Qt infrastructure (Gitorious, Gerrit, etc.). See commits 6f3d74b7bda9d86a16d33ed16a0806b74482d57c, f6ec276bbd6980e4619e85abd3b3d62f7156fbfc, 2f6d90dc0ba7853d32a902dbaf92e69ba133c2b1, d124a6d287ef836b3290d92baa2f1e14b18a36bd and 66522d326988614cbe544103f99be14e415e44bd. Ismo, what's the status on upstreaming the one-page-review feature? What did upstream say last time this was attempted? -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358
Re: Another proposal for modernization of our infrastructure
On Friday 30 January 2015 10:04:01 Martin Graesslin wrote: Also I don't think it can be improved, this looks really fundamental in gerrit. I am not the only one who notices the problem and AFAIU Qt even patches around the issues. Given that I'm not confident that we can improve the software, that upstream accepts the changes and that upstream doesn't break the UI again in future releases. Many of your complaints about usability (threading, replies, etc.) are solved or at least partially addressed in the new Gerrit UI, which versions like 2.7 have. It might not be the default on the installation, so check the settings and try to turn it on. Note that there's no patch to make an single-page all-diff for the new UI. It only exists for the old UI, which is why the new UI is disabled for the Qt Gerrit. -- Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center PGP/GPG: 0x6EF45358; fingerprint: E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358 signature.asc Description: This is a digitally signed message part.
Re: Another proposal for modernization of our infrastructure
On Thursday 29 January 2015 12:25:57 Jan Kundrát wrote: On Wednesday, 28 January 2015 13:14:14 CEST, Martin Gräßlin wrote: Navigation through the code is difficult, you cannot see the complete change in one, but have to go through each file. This is something I consider as unfortunate as normally I prefer reading the changes to the header before the implementation, but due to alphabetic ordering we do not have this. Unfortunately when navigating through the change to get to the header file the implementation is marked as you have reviewed it. Hi Martin, thanks for an excellent idea, sorting headers before actual code changes makes a lot of sense. I have a quick'n'dirty patch at [1]. I didn't want to imply that my workflow should be duckttaped - I don't claim that I have the perfect workflow and I only used the example of how the underlying problem is affecting my reviewing workflow. Git has a config diff.orderfile option which might solve this reasonably well. Do you think that the following sorting order is reasonable for a KDE's default? CMake* cmake* src/*.h src/*.cpp *test*/*.h *test*/*.cpp What I also find bad is that you actually need to know the magic keyboard shortcuts to use it in a sane way. And the shortcuts are to be honest strange: ] is a key which is great to use on an English layout but on many other layouts it's just a very cumbersome to use key. Do you have some suggestions on how this could be improved? Upstream is quite friendly, and I'll be happy to offer patches which improve the user experience. I'm sorry but if I were able to improve UX, I would have enough to do on KDE software ;-) Also I don't think it can be improved, this looks really fundamental in gerrit. I am not the only one who notices the problem and AFAIU Qt even patches around the issues. Given that I'm not confident that we can improve the software, that upstream accepts the changes and that upstream doesn't break the UI again in future releases. Like Eike I consider this as very important as I spent quite some time each day creating review requests or reviewing other peoples code. Cheers Martin signature.asc Description: This is a digitally signed message part.
Re: Another proposal for modernization of our infrastructure
On Thursday, 29 January 2015 12:25:57 CEST, Jan Kundrát wrote: Hi Martin, thanks for an excellent idea, sorting headers before actual code changes makes a lot of sense. I have a quick'n'dirty patch at [1]. The patch has been merged upstream and will be released in next version (2.11). I'll also backport it when I upgrade our instance to 2.10 because it's reasonably self-contained, and because I also think that it's an excellent idea to have this better ordering. Thanks for suggesting it. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On Tuesday 27 January 2015 11:08:49 Jan Kundrát wrote: Hi, as promised, here is a proposal on how our infrastructure can be improved, with emphasis on service integration. There are screenshots inside. Feedback is very welcome. Hello Jan, thank you very much for this exhaustive overview of what could be done based on Gerrit and other tools. This, like the other report on Phabricator, educated me in many aspects. FWIW, this document reads like a fairy tale to me. The fact that so much is already tested and deployed, paired with the improved workflow (imo) and the potential improvements in the future (fix whitespace issues instead of nitpicking them? yeeehaaa!), really tips the scale in favor of your proposal - in my opinion. There are only two issues, which I think should definitely be addressed in the document in sec. 8 (Summary of Required Future Work): One is an improvement to the Gerrit Web UI for reviewing. This was reported by many people, and I agree with all of them. But, contrary to some of the others, I don't think this is a blocker that outranks all the other huge benefits. Rather, I think we can fix this ourselves, if required. Hell, it's just about improving the UI theme and implementing a better patch view (which is btw already done in the Qt Gerrit!). You need help? Count me in! Second, is a proposal for replacing ChiliProject for code browsing, i.e. something that can visualize the tree structured used by KDE and allows us to see short descriptions of projects and the maintainers of that codebase. You mention something in that regard in 3.2.12, but no actual tool to replace this - or am I missing that? I'm also aware that Phabricator would not have anything like that either, as it cannot cope with our tree structure. Furthermore, while the Zuul stuff sounds nice to have in the future, I'd also say we don't necessarily need to change our current Jenkins setup. I propose, to first fix the real issues we have (i.e. ChiliProject etc. pp.). Then, later, we can enable Zuul stuff for more and more projects, if they wish so. Bye -- Milian Wolff m...@milianw.de http://milianw.de
Re: Another proposal for modernization of our infrastructure
On 01/29/2015 06:16 PM, Milian Wolff wrote: FWIW, this document reads like a fairy tale to me. The fact that so much is already tested and deployed One thing I'm unclear on: Does the gerrit test instance run on machines administrated by kde.org these days? Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On Thursday 29 January 2015 16:27:52 Luca Beltrame wrote: Jan Kundrát wrote: as promised, here is a proposal on how our infrastructure can be improved, with emphasis on service integration. There are screenshots inside. I'm not sure if this is the right thread for it, but as someone who commits patches very occasionally either through CLI or the web interface, I find Gerrit's UI to be pretty much clumsy and hard to understand, not to mention when reading reviews (points already raised by Martin G). People can work around it, but IMO it's far too intimidating for people that only occasionally contribute patches. I agree. But is that such a serious blocker that outweights all other benefits? As I just wrote in the other mail, I think its a problem we, as a community, can fix if required. I volunteer (after I come back from vacation in ~2 months ;-))! Also note how parts of the issues raised by Martin, are fixed in e.g. the Qt Gerrit -- at least you can see all changes on one page. Furthermore, as someone who actually worked with Gerrit quite a bit in the last months, I don't think that the UI is particularly intimidating or hard to use. Yes, it takes some time getting used to it. But how is that different from any other tool? Change always implicates we must learn something new! To give an example: When you review code in Qt Gerrit, it's nice to see both the diff and the review comments inline. I find this much more useful, than the reviewboard UI which splits comments and actual code. Take e.g.: https://git.reviewboard.kde.org/r/122169/diff/# Here, I need to manually click on the commented-on lines to see what people wrote to a given changed line. Take this on the other hand: https://codereview.qt-project.org/#/c/89086/2//ALL Much nicer, I think! tl;dr; please do not rule-out Gerrit just because you don't know it (yet)! Bye -- Milian Wolff m...@milianw.de http://milianw.de
Re: Another proposal for modernization of our infrastructure
On Wednesday, 28 January 2015 13:14:14 CEST, Martin Gräßlin wrote: Navigation through the code is difficult, you cannot see the complete change in one, but have to go through each file. This is something I consider as unfortunate as normally I prefer reading the changes to the header before the implementation, but due to alphabetic ordering we do not have this. Unfortunately when navigating through the change to get to the header file the implementation is marked as you have reviewed it. Hi Martin, thanks for an excellent idea, sorting headers before actual code changes makes a lot of sense. I have a quick'n'dirty patch at [1]. Git has a config diff.orderfile option which might solve this reasonably well. Do you think that the following sorting order is reasonable for a KDE's default? CMake* cmake* src/*.h src/*.cpp *test*/*.h *test*/*.cpp What I also find bad is that you actually need to know the magic keyboard shortcuts to use it in a sane way. And the shortcuts are to be honest strange: ] is a key which is great to use on an English layout but on many other layouts it's just a very cumbersome to use key. Do you have some suggestions on how this could be improved? Upstream is quite friendly, and I'll be happy to offer patches which improve the user experience. Cheers, Jan [1] https://gerrit-review.googlesource.com/63812 -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
Jan Kundrát wrote: as promised, here is a proposal on how our infrastructure can be improved, with emphasis on service integration. There are screenshots inside. I'm not sure if this is the right thread for it, but as someone who commits patches very occasionally either through CLI or the web interface, I find Gerrit's UI to be pretty much clumsy and hard to understand, not to mention when reading reviews (points already raised by Martin G). People can work around it, but IMO it's far too intimidating for people that only occasionally contribute patches. -- Luca Beltrame - KDE Forums team KDE Science supporter GPG key ID: 6E1A4E79
Re: Another proposal for modernization of our infrastructure
On Thursday, 29 January 2015 18:22:35 CEST, Eike Hein wrote: One thing I'm unclear on: Does the gerrit test instance run on machines administrated by kde.org these days? The VM runs at my workplace. The KDE sysadmins have root access, PostgreSQL backups are automatically pushed to a KDE server twice a day, and Git is replicated to git.kde.org within seconds after each push. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Another proposal for modernization of our infrastructure
On 01/29/2015 06:51 PM, Jan Kundrát wrote: The VM runs at my workplace. The KDE sysadmins have root access, PostgreSQL backups are automatically pushed to a KDE server twice a day, and Git is replicated to git.kde.org within seconds after each push. Just for the record: I consider you a KDE sysadmin (you're administrating infra used by KDE, after all), so I meant the kde.org more general. Thanks. Cheers, Jan Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On 01/29/2015 06:24 PM, Milian Wolff wrote: Much nicer, I think! I disagree - having the comment in a floating popup instead of breaking up source code makes it easier to read the code for me. Personally, I agree that the gerrit UI is terrible to use. It's not just the diff viewer, either. The general review page layout is like someone spilled a jar of UI on the page. Basic functionality like Reply is placed oddly, the history view is useless-by-default by intially collapsing comments and muddying up the excerpts with redundant preambles, ... it's just really, really bad, and Phabricator blows it out of the water. For me, the strong suits of the gerrit proposal are: * There's the psychological/branding advantage of using tooling that much of our middleware stack uses, which is a crowd we historically aren't culturally close to, at a cost to us. This might offset the cost of the daily frus- tration of having to work with shit UI. In this sense gerrit is the IBM Thinkpad of collabora- tion tools: It looks nerd. It's actively alienating to people who aren't willing to look past the surface, and make no mistake, that's a comforting veneer to many engineers. Habituating developers to numb themselves to living with shit UI seems kinda dangerous in context of the software we are trying to make, though. * gerrit appears to solve some problems well that are no less hard than good UI design, like replication. Tapping into that work is definitely attractive. * I like that Jan is looking ahead at using tooling to improve our process, rather than just re-ticking boxes for things we already have. Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On Mittwoch, 28. Januar 2015 13:14:14 CET, Martin Gräßlin wrote: Ah. Web UI concerns. Yes. Share most of them. Navigation through the code is difficult, you cannot see the complete change in one, but have to go through each file. +1 Ideally one could have show the patch at once (for small ones across some files) or file-by-file (RB sometimes gets really slow on showing HGE patches, like sed'ing a symbol name etc.) Maybe it's possible to borrow or upstream the Qt mod? What I also find bad is that you actually need to know the magic keyboard shortcuts to use it in a sane way. And the shortcuts are to be honest strange Kids. =) Those actually look a lot like vi (not vim) shortcuts. But yes: to everybody not familiar w/ vi those are ... suboptimal (and i recall my thoughts on vi when being confronted with it for the first time) @Jan If the shortcuts can be freely chosen, one might want to make use of the modern inventions to keyboards like the page up/down or arrow keys (along a modifier, eg. ctrl) ?: Open shortcut help Alt + PgUp / PgDown:Previous comment / Next comment (kate bookmarks) Ctrl + f: Search (as present) F3 / Shift + F3:Next diff chunk or search result / Previous diff chunk j / k : Next line / Previous line (ok, because arrow up/down work as well) Right/Left Arrow: Expand or collapse comment (treeview style) Shift + Right/Left Arrow: Expand or collapse all comments on current line Alt+ Left/Right:Previous/Next file (typical browser history shortcut) I've no good or strong idea on: Shift + m : Mark patch as reviewed and go to next unreviewed patch seems ok. ,: Show diff preferences hardly ever used i: Toggle intraline difference hardly ever used a: Reply and score I've never sued that. - I do not like the comment threading of review board, but I consider gerrit even worse. What I like better on gerrit is that it doesn't spam the entire thread via mail (TOP QUOTING! ;-) All comments are collapsed and you have to go to the diff to read the comments on the code. +1 If any possible there should *really* be a brief context on the comment (ie. the code line reproduced) There is no threading at all involved. On review board we quite often have threaded discussions, but I cannot see how this could happen in gerrit. When you click reply, you create a quote, but that's hardly optimal. Also see https://groups.google.com/forum/m/#!topic/repo-discuss/VZuOTLzlDZ8 on the specific matter and/or here http://benpfaff.org/writings/gerrit.html for a more extensive list on improvable things. into git doesn't change much. Whether I do: git push something magic:something other magic or rbt post -o --parent=somethingmagic I'll disagree here and point my 50 commits review disaster (i canceled it immediately) git push something magic:something other magic is actually just a variant of the canonical form (example) git push origin master:master Shortcutting git push can cause you trouble unless you operate on one repo + branch only, so one should know the full form and then there's no magic left. Cheers, Thomas
Re: Another proposal for modernization of our infrastructure
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, On 29.01.2015 12:25, Jan Kundrát wrote: Git has a config diff.orderfile option which might solve this reasonably well. Do you think that the following sorting order is reasonable for a KDE's default? CMake* cmake* src/*.h src/*.cpp *test*/*.h *test*/*.cpp Nice, I didn't knew that config option yet, thanks! But as far as I can see, this would sort all .h in front of all .cpp files, while at least I would prefer to only have the matching .h in front of the .cpp file (so that I have: src/file1.h src/file1.cpp src/file2.h src/file2.cpp ) Is that somehow possible with that option? Cheers, fxrh -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJUyoRaAAoJEEOm4pk0rmQGtM0IALU5kID5AeTSgcJSJOQf55C7 9KXSusrEFAZOyYy7Tqya5NpivAykqWh+vb2Gu6Gt7eAStq6BldorYeM5r+DrwSyI bnsOaUxZsduZauqgH0AMSM/Y+7L1qevg++4y3QsfWbHHOrz3j8++vZEJAcARTcyh ex832D7boaUze8j68FxSG4cFUH0x9mrGqKoSPnkBn88lhnrW+kQyo9foSUzPAsH5 sIh2wkJx26t/MCNFiIsVv/zwGszTicNEr3nrLRzJpB5jjZFA6yEIm6EnQ35AzSyE xFhLGf3lIval9XtjVPKCb1D5MuBIeR5lO690D+z1Dm8sIu8i2+Ihn5D3DalSVa8= =WN0Q -END PGP SIGNATURE-
Re: Another proposal for modernization of our infrastructure
Milian Wolff wrote: I agree. But is that such a serious blocker that outweights all other benefits? As I just wrote in the other mail, I think its a problem we, as Perhaps not for frequent contributors, but for occasional ones (speaking for for myself, I send a patch every 3-5 months, at best) it's a daunting task. Personally I have no issue with git or rbt (and yes, I don't like rbt), but this puts an extra burden that might be unwanted. This is only the perspective of an occasional contributor, so perhaps it doesn't weigh as much. -- Luca Beltrame - KDE Forums team KDE Science supporter GPG key ID: 6E1A4E79
Re: Another proposal for modernization of our infrastructure
On 01/29/2015 08:54 PM, Luca Beltrame wrote: This is only the perspective of an occasional contributor, so perhaps it doesn't weigh as much. I think it's a real concern, and I'm wary of we can patch it away because carrying a huge custom patch delta for UI mods is what kept us from upgrading Bugzilla for multiple years. I think is it realistic that we can maintain this and keep up with upstream even if Ben or Jan get hit by a bus is an important question with regard to both proposals. I have similar concerns with some of the promised benefits in the proposal because they strike me more of we could, which is cool, but it's not we will. E.g. if test build- ing precombined patches takes an OpenStack cluster - do we have one? Where are we going to get that horsepower? Can we keep it? OTOH I expect that I *will* spend 1-2 hours a day five-six days a week in the review web UI, and Phabricator's is pretty nice off the shelf. It's tricky. Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On 01/29/2015 10:34 PM, Thomas Lübking wrote: Given the multiple concerns on the gerrit webfrontend (not only in this kcd thread) I however also assume that it should be not too hard to get a serious improvement upstream. That includes If we endup w/ a -hypothetical- decision between 'powerful, but the webfrontend sucks' and 'pretty ui, but the backend seriously falls short', i'd be happily willing to help on an improvement here. (At least from my POV, it should be simpler to fix some GUI than to get a well scaling replication and CI backend - by the order of some magnitudes ;-) Maybe, but this is actually something I like from the Phabricator proposal: It provides an impression of our relationship with Phabricator upstream, which it says is a good and constructive one. In my experience, this is worth tons. Consider the switch we did from autotools to CMake, where CMake partly won out over other solutions because upstream really wanted to work with us and did. We had similar luck with gitolite, where upstream did a ton of real feature work to address our needs. Redmine/Chili helped us a little bit, too, actually. So keep in mind that we need to fix this if we adopt it takes manpower, and historically, we've relied to some degree to getting that manpower *from upstream*. This is is in my book a world of difference from upstream might accept our patches ... maybe. Cheers, Eike
Re: Another proposal for modernization of our infrastructure
On Donnerstag, 29. Januar 2015 21:03:32 CET, Eike Hein wrote: I think it's a real concern, and I'm wary of we can patch it away because carrying a huge custom patch delta for UI mods is what kept us from upgrading Bugzilla for multiple years. Afaiu Jan's proposal, the default gerrit webui is just yet another frontend on a REST API (on a sidenote: looking at that CLI thing, I wonder whether and how hard it would be to have this integrated into kdevelop) - so I'm not *that* concerned here (assuming the API will remain rather stable across version iterations) Given the multiple concerns on the gerrit webfrontend (not only in this kcd thread) I however also assume that it should be not too hard to get a serious improvement upstream. That includes If we endup w/ a -hypothetical- decision between 'powerful, but the webfrontend sucks' and 'pretty ui, but the backend seriously falls short', i'd be happily willing to help on an improvement here. (At least from my POV, it should be simpler to fix some GUI than to get a well scaling replication and CI backend - by the order of some magnitudes ;-) cluster - do we have one? Where are we going to get that horsepower? Can we keep it? That indeed is a severe issue. Having CI on precombined patches is a *really* great feature and, walking around in a candy shop, I'd immediately drop a whatsoever shiny glossy button UI to get one (and regardless on the actual tool - I would seriously want that with phabricator, gerrit, reviewboard, ...), but if the feature is not affordable, it's not a feature at all :-( @Ben, Since this is mostly a peak performance limitation: are there any statistics on the arrival of review requests? (It doesn't help if we've the power during the week, but on Sunday afternoon, when everyone wants to submit patches, you'll have to wait until Tuesday for the results) - we might require some EC2 here to get this? Cheers, Thomas
Re: Another proposal for modernization of our infrastructure
On Fri, Jan 30, 2015 at 10:34 AM, Thomas Lübking thomas.luebk...@gmail.com wrote: On Donnerstag, 29. Januar 2015 21:03:32 CET, Eike Hein wrote: I think it's a real concern, and I'm wary of we can patch it away because carrying a huge custom patch delta for UI mods is what kept us from upgrading Bugzilla for multiple years. Afaiu Jan's proposal, the default gerrit webui is just yet another frontend on a REST API (on a sidenote: looking at that CLI thing, I wonder whether and how hard it would be to have this integrated into kdevelop) - so I'm not *that* concerned here (assuming the API will remain rather stable across version iterations) Please examine Phabricator's Conduit UI - it has quite a bit of functionality as well. Given the multiple concerns on the gerrit webfrontend (not only in this kcd thread) I however also assume that it should be not too hard to get a serious improvement upstream. Given that upstream has had multiple attempts now at an improved interface, I would question whether they would be willing to accept a user interface which is suitable for our needs. It appears that they are quite comfortable with an interface many find unintuitive or difficult to use. If they weren't, considering the number of backers it has - one would think someone would have sponsored such an interface. That includes If we endup w/ a -hypothetical- decision between 'powerful, but the webfrontend sucks' and 'pretty ui, but the backend seriously falls short', i'd be happily willing to help on an improvement here. Define seriously falls short (listing of features, etc). Please bear in mind that what one person deems a major issue may be minor or irrelevant to others because other functionality is more important to them. (At least from my POV, it should be simpler to fix some GUI than to get a well scaling replication and CI backend - by the order of some magnitudes ;-) Well designed user interfaces are easier said than done. Note that we already have plans for how a well scaling anongit replication system would be built. As for the CI backend, please mention what is wrong with Jenkins - if it would be integrated to check code review submissions. cluster - do we have one? Where are we going to get that horsepower? Can we keep it? That indeed is a severe issue. Having CI on precombined patches is a *really* great feature and, walking around in a candy shop, I'd immediately drop a whatsoever shiny glossy button UI to get one (and regardless on the actual tool - I would seriously want that with phabricator, gerrit, reviewboard, ...), but if the feature is not affordable, it's not a feature at all :-( Correct. With Reviewboard it is too hard to implement. Phabricator is more than capable of doing it with little issue. @Ben, Since this is mostly a peak performance limitation: are there any statistics on the arrival of review requests? (It doesn't help if we've the power during the week, but on Sunday afternoon, when everyone wants to submit patches, you'll have to wait until Tuesday for the results) - we might require some EC2 here to get this? I'm afraid that information isn't really available. Having a count on the arrival of review requests wouldnt be helpful anyway, as different projects have different resource costs. One review for Calligra/Krita would take more time than 10 for kcoreaddons for instance. Sort by time taken in Jenkins to get an idea of resource cost. Most projects are relatively lightweight thanks to the splitting. Also be aware that the power difference for the build nodes varies significantly - one of them has many times more power (CPU, memory and disk speed wise) compared to the others. This means the above numbers can fluctuate as well depending on the node used to perform the build. In my experience, the system is busiest during EU daytime on weekdays. Cheers, Thomas Regards, Ben
Re: Another proposal for modernization of our infrastructure
Hi, Jan Kundrát wrote: Feedback is very welcome. First of all, I would like to apologize for my overly negative tone in your prior feedback threads. I would also like to point out that I have absolutely no experience with Phabricator (the solution proposed by the competing proposal), and as such, I cannot really compare the 2 proposals nor give a personal preference. There are 2 points in your (Gerrit-based) proposal that I would like to comment on: 1. File-level conflict resolution 3.2.2 Conflicting Changes When a project is big enough, sooner or later there will be patches laying around which are mutually incompatible. By default, Gerrit uses merge algorithm that solves conflicts on a file level. A list of changes which modify the same files is shown within the UI. Changes which cannot be submitted due to a merge failure are clearly marked as needing a rebase in all UIs. That way, a developer can make sure that a conflict is solved in a meaningful way and without introducing bugs. Unfortunately, file level strikes me as a less than helpful default. Can this be changed to line-level merges in our instance? (I think the ideal would be to use git's native merging algorithm(s), but I expect some limitations due to the convenient web resolving UI.) In community-developed Free Software projects (also known as the Open Source Development Model or the bazaar model), very often, reviews modify files that have also been touched by other people before the review is processed, or an uploaded patch was generated against a release tarball and the file has since changed in master. I fear that having to manually merge as soon as there is a conflict at the level of the entire file is going to be really painful. This is different from some of the existing large-scale deployments (e.g., OpenStack), which, even where the end product is Free, are largely company- driven. I guess concurrent modification of a single file by different people is not so common in such corporate (cathedral) development models, which explains the default. 2. Reliance on client-side JavaScript Another thing I'm a bit concerned about is the widespread reliance on client-side JavaScript in both your proposal and upstream Gerrit. As you write in section 3.2.1: The bundled web UIs are implemented as a client-side JavaScript application that talk to Gerrit via the REST APIs. There is no artificial feature gap between what can be done with official tools and what is available to other UIs. Alternative web UIs using various modern web frameworks are under development. In addition, your existing and proposed code for integrating independent services between each other (section 4.1, and section 3.2.11 for the special case of Bugzilla) is (or will be) also written in client-side JavaScript. As a result, people who opt to disable JavaScript in their browser for whatever reason (e.g., security) will have: * the Gerrit web interface not working at all (or at least not until such an alternative web UI is implemented in a way not requiring client-side JavaScript and deployed on KDE infrastructure), * the integration between various utilities also not working, e.g., Bugzilla will not list pending review requests at all. To me, this contradicts the web maxim of graceful degradation. Why can the work not be done on the server side? Especially for the integration between services, I would expect a simple API call for data lookup to be doable on the server side at least as easily as from client- side JavaScript. Kevin Kofler
Re: Another proposal for modernization of our infrastructure
On Wednesday 28 January 2015 13:14:14 Martin Gräßlin wrote: I agree on what Martin says about some issues with the web interface of Gerrit, esp. in regard to shortcuts. Note though that the Qt gerrit has the ability (via custom code) to show the full patch. snip Given that code review is the core competence of gerrit I'm rather disappointed by the offerings. The fact that it's totally awesomely integrated into git doesn't change much. Whether I do: git push something magic:something other magic or rbt post -o --parent=somethingmagic doesn't matter to me. It's a command I have to learn by heart. So the fact that it's git all to the heart: meh, don't care. Just a small note here: With git, you'd do git review-push or similar. The git config for that is then: [alias] review-push = push gerrit HEAD:refs/for/master I use this in Qt, where I have qpush-dev and qpush-5.4 for bug fixes in stable branches. It's much less magic (imo). Also note that you do not: - need to ammend the commit to include a REVIEW: line - need to specify -g yes when updating reviews - need to specify the -r $REVIEWID line when updating reviews - need to update individual commits, each by running rbt post and specifing the git hash manually all in all, this makes it /much/ simpler to use than rbt. Bye -- Milian Wolff m...@milianw.de http://milianw.de
Re: Another proposal for modernization of our infrastructure
Hey, On Tue, Jan 27, 2015 at 11:08 AM, Jan Kundrát j...@kde.org wrote: Hi, as promised, here is a proposal on how our infrastructure can be improved, with emphasis on service integration. There are screenshots inside. Feedback is very welcome. Thanks for putting that together. One thing I still haven't seen anywhere in these threads is anything from the people currently testing the Gerrit on kio and plasma-framework and how they find that instance for usage in KDE. Their input should be quite valuable as they have real world experience with KDE + Gerrit which is the subject of this proposal. I would be quite interested to hear those experiences and opinions - guys, can you please write something up? Cheers -- Martin Klapetek | KDE Developer
Re: Another proposal for modernization of our infrastructure
On Tue, Jan 27, 2015 at 11:08 PM, Jan Kundrát j...@kde.org wrote: Hi, Hi Jan, as promised, here is a proposal on how our infrastructure can be improved, with emphasis on service integration. There are screenshots inside. Feedback is very welcome. A few comments. 1) Most applications integrate extremely poorly with LDAP. They basically take the details once on first login and don't sync the details again after that (this is what both Chiliproject and Reviewboard do). How does Gerrit perform here? 2) For this trivial scratch repository script, will it offer it's own user interface or will developers be required to pass arguments to it through some other means? The code you've presented thus far makes it appear some other means will be required. 3) We've used cGit in the past, and found it suffered from performance problems with our level of scale. Note that just because a solution scales with the size of repositories does not necessarily mean it scales with the number of repositories, which is what bites cGit. In light of this, what do you propose? 4) Has Gerrit's replication been demonstrated to handle 2000 Git repositories which consume 30gb of disk space? How is metadata such as repository descriptions (for repository browsers) replicated? 5) If Gerrit or it's hosting system were to develop problems how would the replication system handle this? From what I see it seems highly automated and bugs or glitches within Gerrit could rapidly be inflicted upon the anongit nodes with no apparent safe guards (which our present custom system has). Bear in mind that failure scenarios can always occur in the most unexpected ways, and the integrity of our repositories is of paramount importance. 6) Notifications: Does it support running various checks that our hooks do at the moment for license validity and the like? When these rules are tripped the author is emailed back on their own commits. 7) Storing information such as tree metadata location within individual Git repositories is a recipe for delivering a system that will eventually fail to scale, and will abuse resources. Due to the time it takes to fork out to Git, plus the disk access necessary for it to retrieve the information in question, I suspect your generation script will take several load intensive minutes to complete even if it only covers mainline repositories. This is comparable to the performance of Chiliproject in terms of generation at the moment. The original generation of our Git hooks invoked Git several times per commit, which meant the amount of time taken to process 1000 commits easily reached 10 minutes. I rebuilt them to invoke git only a handful of times per push - which is what we have now. 8) Shifting information such as branch assignments in the same manner will necessitate that someone have access to a copy of the Git repository to determine the branch to use. This is something the CI system cannot ensure, as it needs to determine this information for dependencies, and a given node may not have a workspace for the repository in question. It also makes it difficult to update rules which are common among a set of repositories such as those for Frameworks and Plasma (Workspace). I've no idea if it would cause problems for kdesrc-build, but that is also a possibility. 9) You've essentially said you are going to eliminate our existing hooks. Does Gerrit support: a) line ending checks, with exceptions for certain file types and repositories? b) Convenient deactivation of these checks if necessary. c) Checks on the author name to ensure it bears a resemblence to something actual. d) Prohibiting anyone from pushing certain types of email address such as *@localhost.localdomain? 10) You were aware of the DSL work Scarlett is doing and the fact this is Jenkins specific (as it generates Jenkins configuration). How can this work remain relevant? Additionally, Scarlett's work will introduce declarative configuration for jobs to KDE. 11) We actually do use some of Jenkins advanced features, and it offers quite a lot more than just a visual view of the last failure. As a quick overview: a) Tracked history for tests (you can determine if a single test is flaky and view a graph of it's pass/fail history). b) Log parsing to track the history of compiler warnings and other matters of significance (this is fully configurable based on regexes) c) Integrated cppcheck and code coverage reports, actively used by some projects within KDE. d) Intelligent dashboards which allow you to get an overview of a number of jobs easily. Bear in mind that these intelligent dashboards can be setup by anyone and are able to filter on a number of conditions. They can also provide RSS feeds and update automatically when a build completes. How would Zuul offer any of this? And how custom would this all have to be? Custom == maintenance cost. Addendum: the variations, etc. offered by the Zuul instance which already exists
Re: Another proposal for modernization of our infrastructure
On Wednesday, 28 January 2015 10:08:54 CEST, Ben Cooksley wrote: 1) Most applications integrate extremely poorly with LDAP. They basically take the details once on first login and don't sync the details again after that (this is what both Chiliproject and Reviewboard do). How does Gerrit perform here? Data are fetched from LDAP as needed. There's a local cache for speedup (with configurable TTL and support for explicit flushes). 2) For this trivial scratch repository script, will it offer it's own user interface or will developers be required to pass arguments to it through some other means? The code you've presented thus far makes it appear some other means will be required. I might not fully understand this question -- I thought we already dicussed this. The simplest method of invocation can be as easy as `ssh user@somehost create-personal-project foobar`, with SSH keys verified by OpenSSH. This is the same UI as our current setup. There are other options, some of them with fancy, web-based UIs. 3) We've used cGit in the past, and found it suffered from performance problems with our level of scale. Note that just because a solution scales with the size of repositories does not necessarily mean it scales with the number of repositories, which is what bites cGit. In light of this, what do you propose? An option which is suggested in the document is to use our current quickgit setup, i.e. GitPHP. If it works, there's no need to change it, IMHO, and sticking with it looks like a safe thing to me. But there are many additional choices (including gitiles). 4) Has Gerrit's replication been demonstrated to handle 2000 Git repositories which consume 30gb of disk space? How is metadata such as repository descriptions (for repository browsers) replicated? Yes, Gerrit scales far beyond that. See e.g. the thread at https://groups.google.com/forum/#!topic/repo-discuss/5JHwzednYkc for real users' feedback about large-scale deployments. 5) If Gerrit or it's hosting system were to develop problems how would the replication system handle this? From what I see it seems highly automated and bugs or glitches within Gerrit could rapidly be inflicted upon the anongit nodes with no apparent safe guards (which our present custom system has). Bear in mind that failure scenarios can always occur in the most unexpected ways, and the integrity of our repositories is of paramount importance. I agree that one needs proper, offline and off-site backups for critical data, and that any online Git replication is not a proper substitute for this. The plan for disaster recovery therefore is restore from backup. In terms of Gerrit, this means backing up all of the Git repositories and the dumping the PostgreSQL database, and storing these in a location which cannot be wiped out or modified by an attacker who has root on the main Git server, or by a software bug in our Git hosting. One cannot get that with just Git replication, of course. What are the safeguard mechanisms that you mentioned? What threats do they mitigate? I'm asking because e.g. the need for frequent branch deletion is minimized by Gerrit's code review process which uses branches inside. What risks do you expect to see here? 6) Notifications: Does it support running various checks that our hooks do at the moment for license validity and the like? When these rules are tripped the author is emailed back on their own commits. Yes, the proposed setup supports these. The best place for implementing them is via CI invocation through the ref-updated hook. My personal preference would be a ref-updated event handler in Zuul to ensure proper scalability, but there are other options. 7) Storing information such as tree metadata location within individual Git repositories is a recipe for delivering a system that will eventually fail to scale, and will abuse resources. Due to the time it takes to fork out to Git, Gerrit uses JGit, a Java implementation of Git. There are no forks. plus the disk access necessary for it to retrieve the information in question, I suspect your generation script will take several load intensive minutes to complete even if it only covers mainline repositories. This is comparable to the performance of Chiliproject in terms of generation at the moment. The yesterday-released Gerrit 2.10 adds a REST API for fetching arbitrary data from files stored in Git with aggressive caching. I would like to use that for generating that kde_projects.xml file. The original generation of our Git hooks invoked Git several times per commit, which meant the amount of time taken to process 1000 commits easily reached 10 minutes. I rebuilt them to invoke git only a handful of times per push - which is what we have now. Gerrit has a different architecture with no forks and aggressive caching. I'm all for benchmarking, though. Do you want a test repository to run your benchmarks against? 8) Shifting information
Re: Re: Another proposal for modernization of our infrastructure
On Wednesday 28 January 2015 11:52:17 Martin Klapetek wrote: Hey, On Tue, Jan 27, 2015 at 11:08 AM, Jan Kundrát j...@kde.org wrote: Hi, as promised, here is a proposal on how our infrastructure can be improved, with emphasis on service integration. There are screenshots inside. Feedback is very welcome. Thanks for putting that together. One thing I still haven't seen anywhere in these threads is anything from the people currently testing the Gerrit on kio and plasma-framework and how they find that instance for usage in KDE. Their input should be quite valuable as they have real world experience with KDE + Gerrit which is the subject of this proposal. I would be quite interested to hear those experiences and opinions - guys, can you please write something up? As someone who has used our gerrit setup as a reviewer I can provide some feedback, but in order to evaluate properly I also want to test Phabricator once we have the first projects using it. At the moment I must say that I find gerrit's web interface extremely cumbersome to use. This is something I experienced with both Qt's as well as KDE's setup. Navigation through the code is difficult, you cannot see the complete change in one, but have to go through each file. This is something I consider as unfortunate as normally I prefer reading the changes to the header before the implementation, but due to alphabetic ordering we do not have this. Unfortunately when navigating through the change to get to the header file the implementation is marked as you have reviewed it. This is to me quite a step back compared to review board's code navigation. What I also find bad is that you actually need to know the magic keyboard shortcuts to use it in a sane way. And the shortcuts are to be honest strange: ] is a key which is great to use on an English layout but on many other layouts it's just a very cumbersome to use key. I do not like the comment threading of review board, but I consider gerrit even worse. All comments are collapsed and you have to go to the diff to read the comments on the code. There is no threading at all involved. On review board we quite often have threaded discussions, but I cannot see how this could happen in gerrit. Similar issues marked as done: you get a nice list of done, done, done done, but actually have to go back to the code change to see what it's about. I think this is much better done in review board. Given that code review is the core competence of gerrit I'm rather disappointed by the offerings. The fact that it's totally awesomely integrated into git doesn't change much. Whether I do: git push something magic:something other magic or rbt post -o --parent=somethingmagic doesn't matter to me. It's a command I have to learn by heart. So the fact that it's git all to the heart: meh, don't care. I'm looking forward to try out the competitor to make myself a clear picture. Cheers Martin G. signature.asc Description: This is a digitally signed message part.
Re: Another proposal for modernization of our infrastructure
On Wednesday 28 January 2015 12:27:06 Jan Kundrát wrote: On Wednesday, 28 January 2015 10:08:54 CEST, Ben Cooksley wrote: 11) We actually do use some of Jenkins advanced features, and it offers quite a lot more than just a visual view of the last failure. As a quick overview: a) Tracked history for tests (you can determine if a single test is flaky and view a graph of it's pass/fail history). Please see section 3.3.2 which discusses possible ways on how to deal with flaky tests. IMHO, the key feature and our ultimate goal is let's handle flaky tests efficiently, not let's have a graph of failing tests (how would that work with a non-linear history of pre-merge CI?). b) Log parsing to track the history of compiler warnings and other matters of significance (this is fully configurable based on regexes) That's in section 3.3.3. One solution for using this is making the build warning-free on one well-known platform, and enforcing -Werror in there. I realise that the CI stuff is in some ways a side-issue, but I should point out that what these features really offer is *metrics* and, to a lesser extent, *motivation*. The motivation aspect is simple: it's nice to see bad numbers going down and good numbers going up, and for some people (myself included), that's a good motivator. And motivation is important in open source projects. The metrics aspect is a bit more nebulous, but it can be useful to see this information when making decisions - if a lot of tests have just recently started failing, that's a different situation than if they've been failing for a long time. Alex
Re: Another proposal for modernization of our infrastructure
On Wednesday 28 January 2015 13:14:14 Martin Gräßlin wrote: At the moment I must say that I find gerrit's web interface extremely cumbersome to use. This is something I experienced with both Qt's as well as KDE's setup. Navigation through the code is difficult, you cannot see the complete change in one, but have to go through each file. This is something I consider as unfortunate as normally I prefer reading the changes to the header before the implementation, but due to alphabetic ordering we do not have this. Unfortunately when navigating through the change to get to the header file the implementation is marked as you have reviewed it. This is to me quite a step back compared to review board's code navigation. What I also find bad is that you actually need to know the magic keyboard shortcuts to use it in a sane way. And the shortcuts are to be honest strange: ] is a key which is great to use on an English layout but on many other layouts it's just a very cumbersome to use key. I do not like the comment threading of review board, but I consider gerrit even worse. All comments are collapsed and you have to go to the diff to read the comments on the code. There is no threading at all involved. On review board we quite often have threaded discussions, but I cannot see how this could happen in gerrit. Similar issues marked as done: you get a nice list of done, done, done done, but actually have to go back to the code change to see what it's about. I think this is much better done in review board. On Wednesday 28 January 2015 14:07:34 Milian Wolff wrote: Also note that you do not: - need to ammend the commit to include a REVIEW: line - need to specify -g yes when updating reviews - need to specify the -r $REVIEWID line when updating reviews - need to update individual commits, each by running rbt post and specifing the git hash manually all in all, this makes it /much/ simpler to use than rbt. I kind of agree with both of these - gerrit makes following discussion an absolute pain, and reviewing code hardly pleasant, but it's automatic change- id assignation, rather than the error-prone REVIEW keyword, is a major plus point. rbt now provides a stamp method, and I believe phabricator's tool does as well, but I haven't tried them to see how easy they are (and I'd want them to be automagic). Alex
