Re: Sysadmin report on the modernization of our infrastructure
Hello, On Wednesday 21 January 2015 17:12:07 Ben Cooksley wrote: As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. It contains a detailed summary of what is broken with our existing systems, why change is necessary and an evaluation of the options we considered. We have also made a proposal based on our evaluations and the wishlist of functionality drawn up the community. Please find it attached - and let us know what you think. Feedback is welcome. At last some movement in that space that's really nice. I think I've been annoying the sysadmin team with our current tools for a while now, so I can only propose Zanshin as one of the test projects. I especially appreciate the global approach you took, indeed we have to go the forge approach not focus on code hosting + review only. It looks like Phabricator is a nice move in that direction to have something really consolidated and not plenty of disjoint tools. BTW, since we didn't open a kanboard for Zanshin yet, we're totally up to test the task tracking of Phabricator. In fact the more tools we get enabled for our test the better so that we can cover wide on the features. Regards. -- Kévin Ottens, http://ervin.ipsquad.net KDAB - proud supporter of KDE, http://www.kdab.com signature.asc Description: This is a digitally signed message part.
Re: State of kdesvn?
Christian Ehrlicher wrote: I recently tried to fix some bugs in kdesvn but got stuck because there is nobody who can review my patches. Rajko Albrecht as the original maintainer stopped the development of kdesvn sometime in 2013 and the other two project members (David Faure and Christophe Giboudeaux) have no time to dig deep into the subversion api for a proper review. Therefore I'm asking for help and suggestions how to go further. Maybe there are some people on this list who are still working with subversion and kdesvn and are interested in the further development of kdesvn. I'd say just commit/push the patches. If there's no maintainer, there's also nobody to complain. :-) (Wo kein Kläger, da auch kein Richter.) Kevin Kofler
Re: Sysadmin report on the modernization of our infrastructure
Some open questions: - Do we have access to Qt's (ie. KDE's major upstream) decision process (reasoning) towards gerrit? - The major concern about gerrit seems about scratch repos and I think I found a quite old bug/request on this which might cross trivial approaches w/ scripts? [1] Otoh, it seems Phabricator doesn't support scratch repos right now either. - The lack of non-repo driven (web interface) patch supplies in gerrit does not seem to have been addressed in the document, but there've been many calls for such. So what's the state on this? - Phabricator is suggested to have integration w/ a bugtracker - Can gerrit be easily integrated w/ bugzilla (or other trackers), notably by cross-referencing the review (once! ;-) for the BUG keyword? (link and tag that this bug is being worked on - tag must be removed in case the patch is abandoned) - and of course closing as well. Cheers, Thomas [1] https://code.google.com/p/gerrit/issues/detail?id=1142
Re: Sysadmin report on the modernization of our infrastructure
On Wednesday 21 January 2015 17:12:07 Ben Cooksley wrote: Hi all, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. It contains a detailed summary of what is broken with our existing systems, why change is necessary and an evaluation of the options we considered. We have also made a proposal based on our evaluations and the wishlist of functionality drawn up the community. Please find it attached - and let us know what you think. Feedback is welcome. Hello Ben, thanks for this thorough explanation, it fills in quite some gaps that I was not aware of. Phabricator does look interesting, and I'd be willing to experiment with it as well. But, and this is a big but, there are some issues I have with the proposal: a) the scratch repo functionality not being present yet. b) https://secure.phabricator.com/T4333 -- this is a serious blocker Does this mean we will wait until upstream has this implemented before we start moving KDE? I'd dislike doing a move, or even a test-run, before all required functionality exists. For those who want to try out Phabricator, I found https://showoff.phab.io/ Furthermore, some open questions from my side, regarding Phabricator: - is it really trivial to implement commit hooks, such as closing bugs in bugzilla, with herald? With the demo above, it is not clear how to do this. I can only Block Change With Message in a commit filter. - is it possible, and if so - how easy is it, to add bots to the review system? i.e. something like a nitpicker bot similar to the Qt Sanity Bot. The herald demo above does not seem to support running scripts which go beyond the simple point-and-click GUI. - Could, eventually, our CI be integrated? Such that, when a change is landed which breaks the build, it's automatically reverted and/or a comment added to the review page? There seems to be a build plan for phabricator - what is that? - will reviews be enforced, or will they stay opt-in as we have it in the current KDE workflow? - again a herald-related question: will the IRC bots continue to work which announce new commits? - is there still an ATOM/RSS feed of commits? I could not find that in the phabricator demo (note: must work without prior login) - can common herald rules be offered to users to save them some work? I.e. a simple button to get an email for every kdevelop or frameworks or kdepim or ... related review/commit/issue/... the more git repos we add, the bigger this issue becomes - you said projects.kde.org required custom patches, e.g. for adding support for selecting the i18n branches. how is this going to be different in Phabricator? what existing functionality there will supplement this, thereby obsoleting custom patches? - you also mentioned the issues around Gitolite metadata generation, kde_projects.xml etc. - how will Phabricator address these issues? Overall, I'd welcome if a simple dummy phabricator instance could be set up such that we all can play around with the features it provides, including pushing changes and trying out arc. Bye -- Milian Wolff m...@milianw.de http://milianw.de
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 4:28 AM, Thomas Lübking thomas.luebk...@gmail.com wrote: Some open questions: - Do we have access to Qt's (ie. KDE's major upstream) decision process (reasoning) towards gerrit? Our prior requests to be informed when new Qt repositories are setup were ignored, despite being made on more the one occasion. This means that our mirror becomes unusable until sysadmin is informed by someone (or our CI system) that Qt 5 can't be built. We therefore did not attempt to contact them. - The major concern about gerrit seems about scratch repos and I think I found a quite old bug/request on this which might cross trivial approaches w/ scripts? [1] Otoh, it seems Phabricator doesn't support scratch repos right now either. Phabricator requires only a minor modification to support them. - The lack of non-repo driven (web interface) patch supplies in gerrit does not seem to have been addressed in the document, but there've been many calls for such. So what's the state on this? Sorry, not sure I understand this. Phabricator certainly does support uploading patches through the web interface. - Phabricator is suggested to have integration w/ a bugtracker - Can gerrit be easily integrated w/ bugzilla (or other trackers), notably by cross-referencing the review (once! ;-) for the BUG keyword? (link and tag that this bug is being worked on - tag must be removed in case the patch is abandoned) - and of course closing as well. You'll need to ask one of the people more familiar with Gerrit about that i'm afraid. Note that Phabricator's integration between it's components is deep - so notes are posted on the task regarding the review, and a note is posted on the review regarding the task and one can see the status of both (open, closed, etc) from those notes. Cheers, Thomas Thanks, Ben [1] https://code.google.com/p/gerrit/issues/detail?id=1142
Re: Sysadmin report on the modernization of our infrastructure
On Wed, Jan 21, 2015 at 11:44 PM, Jan Kundrát j...@kde.org wrote: Hi Ben, Hi Jan, thanks for your proposal. A couple of points which I'm missing from the text, and a few further questions as well: Certainly. 1) A detailed list of the issues which a competing proposal would have to address. Some glimpse of this is in the last paragraph, but that's just a very high-level description. Could you please provide a list of functionality that has to be supported by a replacement, so that we can work out how to get there? The list in question was drawn from my summary mail which was sent to the previous thread. Not everything in that list is a requirement (as no software exists which can meet all of them). 2) It is unclear to me whether you plan to use Gitolite in future or not. At first, the proposal says that there are inherent scaling issues with it and that the replication is beyond its scaling limits, yet at the end of the document you say that a replacement has to support Gitolite metadata generation. I do not understand that. Phabricator will be replacing Gitolite in our proposed setup. There are no scaling issues with Gitolite (it handles things very well), the scaling issues were with Gitorious, and are with Chiliproject and our anongit scripts. With regards to the metadata - I think this was an unintentional reference to the Gitolite configuration generator. If your solution uses something else of course, you would need to build the appropriate logic for that instead. 3) The idea of rolling out Phabricator is not specified, so it's hard to get a proper understanding of what exactly will have to be done. What changes in workflow are going to be needed? What custom tooling will have to be written? How is Phabricator going to play with the rest of the infrastructure? What pieces of the infrastructure will actually remain? There would be no changes in workflow as such. The only way to properly test a tool though is to actually put it into use for a while - and thus find out whether it is able to fit our workflows. In terms of custom tooling, we will need to integrate it with Jenkins and set it up to receive keys from Identity. Due to the way it works with SSH keys this will be a fairly trivial process as our existing systems which auto-sync keys for svn.kde.org and git.kde.org can be almost entirely reused. 4) You have indicated some (pretty important to me) limitations of Phabricator with a remark that it's a fast moving software, we might get there eventually. I think that SW should be evaluated based on functionality present right now in the development version and not based on opened feature requests. We've been promising e.g. support for multiple accounts in Trojita for many years already, and it didn't make it happen. Our evaluation was conducted on the basis of functionality currently present in it. Meeting the requirements of the list was done on a best-effort basis. 5) You're indicating a requirement for scratch repos to be present from the very beginning, yet you acknowledge that Phabricator won't have it for at least six months. There are two paths for scratch repositories: 1) A solution we can implement right now with a minor (20 lines of code) change which will permit developers to create repositories with a certain naming schema. 2) Wait for namespaces, which is what is 6 months off at the earliest. We're likely to go for #1, and possibly migrate to #2 when upstream implements that. 6) The discussion focuses in highlighting Phabricator's benefits, which is understandable from your point of view. However, much of the same things can be said about Gerrit as well, especially its backing by a well-known player, adoption by multiple companies and multinational corporations (SonyErisccon, SAP, KitWare,...) as well as by many FLOSS projects (Qt, Andorid, LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod, Tizen, and a ton of others). Its scaleability has also been verified by real-world use over many years (for far longer than Phabricator). Now coming to Gerrit and its analyzis: 7) It is possible to have scratch repositories with Gerrit, but it's true that it will require a simple tool which verifies user's request and executes an API call with proper credentials. We're speaking about one trivial webapp or a shell script here. This is a separate application, and would therefore require separate maintenance correct? (Plus people have to find it) How would developers delete no longer used scratch repositories? 8) There is no need for any modifications to Gerrit as your text implies. What is running and integrated into KDE right now is an unpatched release straight from upstream, with no custom plugins. Thiago indicated that for sane mail handling one wants the patches Qt currently has. Is this not the case? 9) I do not know what effort for integration with KDE Indentity you're referring to. It's a simple LDAP
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 3:33 AM, Milian Wolff m...@milianw.de wrote: On Wednesday 21 January 2015 17:12:07 Ben Cooksley wrote: Hi all, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. It contains a detailed summary of what is broken with our existing systems, why change is necessary and an evaluation of the options we considered. We have also made a proposal based on our evaluations and the wishlist of functionality drawn up the community. Please find it attached - and let us know what you think. Feedback is welcome. Hello Ben, Hi Milian, thanks for this thorough explanation, it fills in quite some gaps that I was not aware of. Not a problem. Phabricator does look interesting, and I'd be willing to experiment with it as well. But, and this is a big but, there are some issues I have with the proposal: Of course. a) the scratch repo functionality not being present yet. b) https://secure.phabricator.com/T4333 -- this is a serious blocker Sorry, need to clarify here - scratch repository functionality can be provided using a ~20-30 line patch which would allow developers to freely create repositories with a given name. In terms of the 'arc land' issue we plan to work with the necessary folks upstream to get that implemented as soon as possible. Does this mean we will wait until upstream has this implemented before we start moving KDE? I'd dislike doing a move, or even a test-run, before all required functionality exists. We'll work on providing a patch to upstream should need be to get that issue out of the way. For those who want to try out Phabricator, I found https://showoff.phab.io/ Furthermore, some open questions from my side, regarding Phabricator: - is it really trivial to implement commit hooks, such as closing bugs in bugzilla, with herald? With the demo above, it is not clear how to do this. I can only Block Change With Message in a commit filter. Phabrictor supports using regular Git hooks as well, so our existing ones can simply be dropped in place. - is it possible, and if so - how easy is it, to add bots to the review system? i.e. something like a nitpicker bot similar to the Qt Sanity Bot. The herald demo above does not seem to support running scripts which go beyond the simple point-and-click GUI. It is certainly possible, assuming the bot exists. Please see http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/ - Could, eventually, our CI be integrated? Such that, when a change is landed which breaks the build, it's automatically reverted and/or a comment added to the review page? There seems to be a build plan for phabricator - what is that? Build plans are part of it's integrated Harbormaster CI system. We could either utilise it to trigger Jenkins (see http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html) or use the method documented above to trigger Jenkins. - will reviews be enforced, or will they stay opt-in as we have it in the current KDE workflow? They will remain opt-in. - again a herald-related question: will the IRC bots continue to work which announce new commits? Our existing IRC Bot, pursuivant, will be unaffected as it is fed by our current commit hooks - which we can continue to use. - is there still an ATOM/RSS feed of commits? I could not find that in the phabricator demo (note: must work without prior login) I don't think so - please see https://secure.phabricator.com/T1928 People could replace this with Herald rules and receive the notices by email instead though. - can common herald rules be offered to users to save them some work? I.e. a simple button to get an email for every kdevelop or frameworks or kdepim or ... related review/commit/issue/... the more git repos we add, the bigger this issue becomes You mean some kind of template? It does support global rules, but they're quite different in nature. It is quite likely users would have to add the repositories they're interested one by one to their own Herald rule. - you said projects.kde.org required custom patches, e.g. for adding support for selecting the i18n branches. how is this going to be different in Phabricator? what existing functionality there will supplement this, thereby obsoleting custom patches? - you also mentioned the issues around Gitolite metadata generation, kde_projects.xml etc. - how will Phabricator address these issues? We will be shifting the generation of kde_projects.xml off to a separate script which doesn't need to be updated as aggressively. This will allow us to only regenerate the file when necessary and will make it independent except for retrieving a list of repositories (which can be done using the Conduit API). As for i18n branches, i'll be discussing this with Albert, etc. to determine the future of this functionality. Overall, I'd welcome if a simple dummy phabricator instance could
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 12:07 AM, Martin Klapetek martin.klape...@gmail.com wrote: On Wed, Jan 21, 2015 at 5:12 AM, Ben Cooksley bcooks...@kde.org wrote: Hi all, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. It contains a detailed summary of what is broken with our existing systems, why change is necessary and an evaluation of the options we considered. We have also made a proposal based on our evaluations and the wishlist of functionality drawn up the community. Please find it attached - and let us know what you think. Feedback is welcome. Thanks a lot Ben for this, very comprehensive document. One question about Phabricator - in the discussions before the RB web UI and uploading patches via web was mentioned a lot, will uploading diff through web interface still be possible via Phabricator or is the Arcanist thing a must? This isn't clear from the report. Sorry about the confusion there. You can still upload patches through the web interface using Phabricator. Cheers -- Martin Klapetek | KDE Developer Thanks, Ben
Re: Sysadmin report on the modernization of our infrastructure
Thanks Ben for the right proposal. During the process when the original thread becomes a hell of bickering people defending his point instead of a concise approach, i almost loose the faith on results. I personally tested several of the tools on the last job, where i managed internally the buildsystem, so i vow for phabricator as a good decision. I really don't like gerrit, and my experience with it sometimes remove me from want to do things. Phabricator is the obvious system to go. We trusted our sysadmins decisions for years, would not be different now. Helio On Wed, Jan 21, 2015 at 9:07 AM, Martin Klapetek martin.klape...@gmail.com wrote: On Wed, Jan 21, 2015 at 5:12 AM, Ben Cooksley bcooks...@kde.org wrote: Hi all, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. It contains a detailed summary of what is broken with our existing systems, why change is necessary and an evaluation of the options we considered. We have also made a proposal based on our evaluations and the wishlist of functionality drawn up the community. Please find it attached - and let us know what you think. Feedback is welcome. Thanks a lot Ben for this, very comprehensive document. One question about Phabricator - in the discussions before the RB web UI and uploading patches via web was mentioned a lot, will uploading diff through web interface still be possible via Phabricator or is the Arcanist thing a must? This isn't clear from the report. Cheers -- Martin Klapetek | KDE Developer
Re: Sysadmin report on the modernization of our infrastructure
On Mittwoch, 21. Januar 2015 20:41:18 CEST, Ben Cooksley wrote: - Do we have access to Qt's (ie. KDE's major upstream) decision process (reasoning) towards gerrit? Our prior requests to be informed when new Qt repositories are setup Sorry, I was probably ambiguous here. What I meant is: Do we know why Qt chose gerrit? - The lack of non-repo driven (web interface) patch supplies in gerrit does not seem to have been addressed in the document, but there've been many calls for such. So what's the state on this? Sorry, not sure I understand this. Phabricator certainly does support uploading patches through the web interface. Yes. The question was: since this seemed a major issue in the previous thread, but was not mentioned as problem w/ gerrit in the report - does that mean the webinterface issue has been addressed (by either gerrit supporting such or the requirement is gone/replaced) You'll need to ask one of the people more familiar with Gerrit about that i'm afraid. My mail was indeed addressed to any interested reader, not particularily you ;-) so notes are posted on the task regarding the review, and a note is posted on the review regarding the task and one can see the status of both (open, closed, etc) from those notes. That does hopefully not imply one would get two mails for every comment (one for being cc'd on the bug and one for the RR), does it? Cheers, Thomas
Re: Sysadmin report on the modernization of our infrastructure
When the test system is in place I would like to start my Jenkins integration. So let me know! Thanks Scarlett
Re: Sysadmin report on the modernization of our infrastructure
On Mittwoch, 21. Januar 2015 20:55:54 CEST, Jan Kundrát wrote: The bug you linked to is about something else than scratch repos as far as I see. The bug cooked up for asking google about gerrit and scratch repos. The problem is that pushing into any branch would close a review - I can only assume it was linked in the mail thread I found because a similar issue would affect clones etc. (ie. likely when the change-id gets pushed anywhere) My uninformed guess would be to handle the change-id smarter, ie. bind it to a branch (and pot. repo) There's a tool for that, https://tools.wmflabs.org/gerrit-patch-uploader/ . Thanks - looks actually nice (and supports dumb diffs ;-) Sicne I've no mediawiki account: do you know whether it already allows ppl. to access (all) their patches w/o modifications (ie. like the RB dashboard) as well? - Can gerrit be easily integrated w/ bugzilla (or other Yes, see my other mail once it clears the moderation queue. Yupp, thanks. Sounds feature complete (and we're not bound to one bugtracker - though i don't know how good Pahbricator is on this) Cheers, Thomas
Re: Sysadmin report on the modernization of our infrastructure
Please read Bens article again. We do this currently and its not working. This is what needs to be replaced. Phabricator seems to support this, or so they say, **and** is does what Gerrit does. So why not use that and have everything integrated? It's not as simple as picking a WWW git browser, it must be integrated into the rest of the system. And it must be easy to maintain etc. pp. Really, just reread the report. The report mentions problems with the current Git replication. I'm also aware of the technical nature of these things beyond what was said in the report because I did talk to Nicolas and Ben about them. The current setup is slow, pull-based, unreliable, and can only work once an hour on an overseas node due to network latencies and systems' throughput. Fair enough. Compared to that, I know about people using Gerrit's replication with a world-wide, distributed network of Git mirrors on multiple continents. What I'm proposing is simply using what works well for these people and use that for scaleable repository browsing as well. There is no need to write any custom scripts. Just deploy as many mirrors as we might need, and add an arbitrary read-only WWW git browser to them, with no extra configuration required -- just serve all the repos in a read-only manner. Gerrit's replication is not stupid, it does understand the concept of queues and automated retries, so it actually *works* and handles bandwidth-limited or latency-bound nodes fine. If one was to use Phabricator, there will still be the same set of problems with git browser having to scale to sustain automated web crawlers and what not. We will also still need N machines to bring actual content close to the users. Nothing changes by using a single application here, really. The problem is that replication needs improvements, so let's improve the replication. Gerrit can do that. What does Phabricator use for replication, and how does it deal with crappy network? Scripting stuff means its not configurable to users, which is extremely important here. And no, writing a script, maintaining it, and adding a config UI on top of it if is explicitly **not** what the sysadmins are looking for. They want to cut down on tools and maintenance burden. OK, I agree that a perfect way is to add this to Gerrit's existing notifications and contributing that patch upstream -- that already includes the whole infrastructure, up to and including the pretty UI. Let's do this; I'll write that patch and push it upstream once I know that this is not going to be a wasted work (see my first mail in this thread). Note: You removed my marker here, that the below is nice to have. Please keep it in, it's important. I never said that this stuff is of utmost importance. I remove those parts of the mail which I mostly agree with; I find that better for readability. Sorry for confusion. As you said, it's important to assign a proper importance to various features; we're in total agreement here. It's true that there's no git browser where you could attach notes to a particular line and open a ticket for that. Do we need such a feature, and if so, how do we intend to use it? We currently have this in the form of the kde-commits mailing list. It is an extremely useful feature of the KDE community. What we get with Phabricator is that, just so much better. Fair enough, the problem with what we have right now is that nobody is guaranteed to read these and to follow up on them, to paraphrase someone (you, maybe?) from the previous iteration of this conversation. Improving that is a nice bonus, so it's indeed cool to be able to create tickets/issues by a single click when one browses a git repo. It's nice that Phabricator can do that. Do we however actually have some people doing this project-wide code auditing *and* not sending patches at the same time? I'm just wondering whether this fancy feature would be used that often, and whether it could be better to do this as regular code review instead. And I also understand that you've listed it in the nice bonuses section. Having an external tool like our current kanban board on todo.kde.org means it's not integrated with the rest. No easy way to link to reviews, branches, issues, etc. pp. Phabricator gives us all of this, for free! If something is bundled, it doesn't mean that it's free. One of the costs that you pay is that you cannot switch to a better alternative because everything and everybody's workflow assumes that you're using the built-in solution. Well, and I disagree here. Having it all integrated will mean we eventually have a GitHub clone which makes it trivial to close issues or reference stuff from the wiki and have stuff updated there as needed. And remember, I said that this stuff is *nice to have*. It's not the important reason why we should use Phabricator, it's just additional sugar on top which you won't have with
Re: Sysadmin report on the modernization of our infrastructure
On Wednesday, 21 January 2015 16:28:20 CEST, Thomas Lübking wrote: - The major concern about gerrit seems about scratch repos and I think I found a quite old bug/request on this which might cross trivial approaches w/ scripts? [1] Otoh, it seems Phabricator doesn't support scratch repos right now either. The bug you linked to is about something else than scratch repos as far as I see. - The lack of non-repo driven (web interface) patch supplies in gerrit does not seem to have been addressed in the document, but there've been many calls for such. So what's the state on this? There's a tool for that, https://tools.wmflabs.org/gerrit-patch-uploader/ . - Phabricator is suggested to have integration w/ a bugtracker - Can gerrit be easily integrated w/ bugzilla (or other trackers), notably by cross-referencing the review (once! ;-) for the BUG keyword? (link and tag that this bug is being worked on - tag must be removed in case the patch is abandoned) - and of course closing as well. Yes, see my other mail once it clears the moderation queue. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Sysadmin report on the modernization of our infrastructure
On 01/21/2015 10:18 PM, Ivan Čukić wrote: Hi, I'm volunteering KActivities for the test. Konversation, too. Cheerio, Ivan Cheers, Eike
Re: Sysadmin report on the modernization of our infrastructure
On Wednesday 21 January 2015 11:44:00 Jan Kundrát wrote: Hi Ben, thanks for your proposal. A couple of points which I'm missing from the text, and a few further questions as well: 1) A detailed list of the issues which a competing proposal would have to address. Some glimpse of this is in the last paragraph, but that's just a very high-level description. Could you please provide a list of functionality that has to be supported by a replacement, so that we can work out how to get there? Yes, a simple bullet point list would be useful. I'll list some stuff below also. 2) It is unclear to me whether you plan to use Gitolite in future or not. At first, the proposal says that there are inherent scaling issues with it and that the replication is beyond its scaling limits, yet at the end of the document you say that a replacement has to support Gitolite metadata generation. I do not understand that. Agreed. 3) The idea of rolling out Phabricator is not specified, so it's hard to get a proper understanding of what exactly will have to be done. What changes in workflow are going to be needed? What custom tooling will have to be written? How is Phabricator going to play with the rest of the infrastructure? What pieces of the infrastructure will actually remain? snip 5) You're indicating a requirement for scratch repos to be present from the very beginning, yet you acknowledge that Phabricator won't have it for at least six months. Agreed. 6) The discussion focuses in highlighting Phabricator's benefits, which is understandable from your point of view. However, much of the same things can be said about Gerrit as well, especially its backing by a well-known player, adoption by multiple companies and multinational corporations (SonyErisccon, SAP, KitWare,...) as well as by many FLOSS projects (Qt, Andorid, LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod, Tizen, and a ton of others). Its scaleability has also been verified by real-world use over many years (for far longer than Phabricator). Imo, quite the contrary. It concentrates on the issues the admins have with their current setup, and then shows how Phabricator could help with that. Now coming to Gerrit and its analyzis: snip 9) I do not know what effort for integration with KDE Indentity you're referring to. It's a simple LDAP server, and the entire integration was trivial. It's really just a matter of configuring Gerrit to talk to LDAP, which I expect to be also the case with Phabricator. I don't see where this is mentioned in regard to Gerrit. I can only find LDAP being mentioned when talking about the status quo for KDE, which does not include Gerrit. 4) You have indicated some (pretty important to me) limitations of Phabricator with a remark that it's a fast moving software, we might get there eventually. I think that SW should be evaluated based on functionality present right now in the development version and not based on opened feature requests. We've been promising e.g. support for multiple accounts in Trojita for many years already, and it didn't make it happen. This belongs to the point below, imo. Or what are you referring to? If you do mean the git integration, then yes, maybe it's important to you, but it's really not important in the big picture, imo. And do note, I'd personally also prefer gerrit for reviewing. 10) Re Phabricator's support for submitting and retrieving changes via pure git, I believe you're overly optimistic in this regard. This is a pretty fundamental design decision which can only be retroactively fitted with asignificant amount of pain. As far as I understand it, we (KDE) won't jump to Phabricator in a matter of days, rather it will take a few more months before we even start (correct?). There are still some other blockers upstream, which will be resolved. Lets see how this works out. Generally, I don't think its impossible to add this to Phabricator. So lets not be overly pessimistic, nor optimistic. Maybe this feature is added, maybe not. While I personally would also love to have it, it's a minor detail compared to the rest of the feature set, imo. snip 12) WikiMedia uses Phabricator for issue tracking and nothing else -- it's just a Bugzilla replacement for them. According to publicly available information, their reasons for choosing Phabricator had everything to do with the PHP as a language known to all of their developers. They still use Gerrit for git repo management and code review. See http://www.mediawiki.org/wiki/Phabricator, what Jan says is true. But WikiMedia is in the process of migrating away from Gerrit, so what Ben says is also kinda true ;-) Now to the following: 7) It is possible to have scratch repositories with Gerrit, but it's true that it will require a simple tool which verifies user's request and executes an API call with proper credentials. We're speaking about one trivial webapp or a shell
Re: Sysadmin report on the modernization of our infrastructure
On Wednesday, 21 January 2015 15:54:52 CEST, Milian Wolff wrote: 6) The discussion focuses in highlighting Phabricator's benefits, which is understandable from your point of view. However, much of the same things can be said about Gerrit as well, especially its backing by a well-known player, adoption by multiple companies and multinational corporations (SonyErisccon, SAP, KitWare,...) as well as by many FLOSS projects (Qt, Andorid, LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod, Tizen, and a ton of others). Its scaleability has also been verified by real-world use over many years (for far longer than Phabricator). Imo, quite the contrary. It concentrates on the issues the admins have with their current setup, and then shows how Phabricator could help with that. Right, I should have said the proposal to use Phabricator and the reasons for using that particular tool focus on highlighting Phabricator benefits Sorry for not being clear, I appreciate the value of describing the pain points of the legacy setup (and I woud appreciate even more details to be able to offer a better alternative). 9) I do not know what effort for integration with KDE Indentity you're referring to. It's a simple LDAP server, and the entire integration was trivial. It's really just a matter of configuring Gerrit to talk to LDAP, which I expect to be also the case with Phabricator. I don't see where this is mentioned in regard to Gerrit. I can only find LDAP being mentioned when talking about the status quo for KDE, which does not include Gerrit. The part which I was commenting on is the following paragraph: As a result of this it would be necessary to combine several different components to produce a complete Git solution. This would require further effort to integrate them with both each other and parts of KDE infrastructure such as Identity. Even after such effort is completed a certain degree of synchronisation between the tools will need to be maintained, such as registering repositories in both the code hosting tool and Gerrit. There was no extra work to integrate Gerrit with KDE's Identity, and I expect most of other tools which we might need to use to have LDAP support out of box because that's an industry standard for identity management. KDE Identity == LDAP, in this context. The paragraph above also assumes that Gerrit would not be used as a primary code hosting place. There's no reason for that, so the raised conclusions (this will require syncing) are not true. 4) You have indicated some (pretty important to me) limitations of Phabricator with a remark that it's a fast moving software, we might get there eventually. I think that SW should be evaluated based on functionality present right now in the development version and not based on opened feature requests. We've been promising e.g. support for multiple accounts in Trojita for many years already, and it didn't make it happen. This belongs to the point below, imo. Or what are you referring to? When I read the proposal, I see some enthusiasm for Phabricator's swift development pace. That's good, but at the same time it isn't an answer to a lack of features. Here are some of the relevant bits: - CI integration - scratch repos - clustering for scaleability - preserving author metadata - direct git access to patches under review - patch upload via git What I'm saying is that an opened feature request and a rapid speed of development are not something to gurantee that these will be ready in a month, or even in a couple of years. To my knowledge, here are some things that Gerrit does not provide, but Phabricator potentially provides: Yes, Gerrit doesn't include wiki pages, issue tracker, Kanban planner and various calendars. I don't necessarily see that as a drawback. Do we want to migrate wikis and Bugzilla? If yes, I can understand that Phabricator might be a compeling tool, but so far the proposal was limited to just revamping the git hosting and code review. - a project overview with a code browser, project meta data etc. pp. and a list of commits inside a repository. Qt still uses gitorious for this, afaik Right, we will have to pick one (or multiple) of the WWW git browsers out there. If I was designing such a service, I would let it run from a dedicated VM and have Gerrit replicate stuff to these servers. Scaling up a stateless, read-only service such as cgit/gitweb/... is very easy. - the ability to get notified about new commits in a project. (this is different from new reviews) Gerrit has hooks for triggering this (ref-updated), and it's easy to send e-mails from that context. There are scripts for that in git's contrib/. @sysadmins, how is that tackled by Phabricator? Also, if a proper code review was enforced, there would be no need for this. - Apparently the anongit problem, but Ben would need to fill in more details here. We are already using a stock
Re: Sysadmin report on the modernization of our infrastructure
Hi, I'm volunteering KActivities for the test. Cheerio, Ivan
Re: Sysadmin report on the modernization of our infrastructure
On Wednesday 21 January 2015 17:53:51 Jan Kundrát wrote: On Wednesday, 21 January 2015 15:54:52 CEST, Milian Wolff wrote: snip 9) I do not know what effort for integration with KDE Indentity you're referring to. It's a simple LDAP server, and the entire integration was trivial. It's really just a matter of configuring Gerrit to talk to LDAP, which I expect to be also the case with Phabricator. I don't see where this is mentioned in regard to Gerrit. I can only find LDAP being mentioned when talking about the status quo for KDE, which does not include Gerrit. The part which I was commenting on is the following paragraph: As a result of this it would be necessary to combine several different components to produce a complete Git solution. This would require further effort to integrate them with both each other and parts of KDE infrastructure such as Identity. Even after such effort is completed a certain degree of synchronisation between the tools will need to be maintained, such as registering repositories in both the code hosting tool and Gerrit. There was no extra work to integrate Gerrit with KDE's Identity, and I expect most of other tools which we might need to use to have LDAP support out of box because that's an industry standard for identity management. KDE Identity == LDAP, in this context. The paragraph above also assumes that Gerrit would not be used as a primary code hosting place. There's no reason for that, so the raised conclusions (this will require syncing) are not true. OK, I see. Maybe Ben and Jeff where not aware of this. Or they mean something different - lets wait for them to clarify. 4) You have indicated some (pretty important to me) limitations of Phabricator with a remark that it's a fast moving software, we might get there eventually. I think that SW should be evaluated based on functionality present right now in the development version and not based on opened feature requests. We've been promising e.g. support for multiple accounts in Trojita for many years already, and it didn't make it happen. This belongs to the point below, imo. Or what are you referring to? When I read the proposal, I see some enthusiasm for Phabricator's swift development pace. That's good, but at the same time it isn't an answer to a lack of features. Here are some of the relevant bits: - CI integration - scratch repos - clustering for scaleability - preserving author metadata - direct git access to patches under review - patch upload via git What I'm saying is that an opened feature request and a rapid speed of development are not something to gurantee that these will be ready in a month, or even in a couple of years. I agree with you. At the same time, many things are missing in Gerrit - so it's not different there. To my knowledge, here are some things that Gerrit does not provide, but Phabricator potentially provides: Yes, Gerrit doesn't include wiki pages, issue tracker, Kanban planner and various calendars. I don't necessarily see that as a drawback. Do we want to migrate wikis and Bugzilla? If yes, I can understand that Phabricator might be a compeling tool, but so far the proposal was limited to just revamping the git hosting and code review. You are misinterpreting this. Kanban, wiki and issue tracker are optional things that are nice to have, but they are not the important stuff (see below). - a project overview with a code browser, project meta data etc. pp. and a list of commits inside a repository. Qt still uses gitorious for this, afaik Right, we will have to pick one (or multiple) of the WWW git browsers out there. If I was designing such a service, I would let it run from a dedicated VM and have Gerrit replicate stuff to these servers. Scaling up a stateless, read-only service such as cgit/gitweb/... is very easy. Please read Bens article again. We do this currently and its not working. This is what needs to be replaced. Phabricator seems to support this, or so they say, **and** is does what Gerrit does. So why not use that and have everything integrated? It's not as simple as picking a WWW git browser, it must be integrated into the rest of the system. And it must be easy to maintain etc. pp. Really, just reread the report. - the ability to get notified about new commits in a project. (this is different from new reviews) Gerrit has hooks for triggering this (ref-updated), and it's easy to send e-mails from that context. There are scripts for that in git's contrib/. Scripting stuff means its not configurable to users, which is extremely important here. And no, writing a script, maintaining it, and adding a config UI on top of it if is explicitly **not** what the sysadmins are looking for. They want to cut down on tools and maintenance burden. @sysadmins, how is that tackled by Phabricator? Herald. Trivial there, no scripting required whatsoever.
Re: Sysadmin report on the modernization of our infrastructure
On 01/21/2015 05:12 AM, Ben Cooksley wrote: Hi all, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. As someone on the original git infra team, I'm impressed - thanks for your hard work. Cheers, Ben Cooksley KDE Sysadmin Cheers, Eike
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 5:53 AM, Jan Kundrát j...@kde.org wrote: On Wednesday, 21 January 2015 15:54:52 CEST, Milian Wolff wrote: 6) The discussion focuses in highlighting Phabricator's benefits, which is understandable from your point of view. However, much of the same things can be said about Gerrit as well, especially its backing by a well-known player, adoption by multiple companies and multinational corporations (SonyErisccon, SAP, KitWare,...) as well as by many FLOSS projects (Qt, Andorid, LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod, Tizen, and a ton of others). Its scaleability has also been verified by real-world use over many years (for far longer than Phabricator). Imo, quite the contrary. It concentrates on the issues the admins have with their current setup, and then shows how Phabricator could help with that. Right, I should have said the proposal to use Phabricator and the reasons for using that particular tool focus on highlighting Phabricator benefits Sorry for not being clear, I appreciate the value of describing the pain points of the legacy setup (and I woud appreciate even more details to be able to offer a better alternative). What sort of information are you after exactly? The detailed part of the report lays out the problems fairly well. 9) I do not know what effort for integration with KDE Indentity you're referring to. It's a simple LDAP server, and the entire integration was trivial. It's really just a matter of configuring Gerrit to talk to LDAP, which I expect to be also the case with Phabricator. I don't see where this is mentioned in regard to Gerrit. I can only find LDAP being mentioned when talking about the status quo for KDE, which does not include Gerrit. The part which I was commenting on is the following paragraph: As a result of this it would be necessary to combine several different components to produce a complete Git solution. This would require further effort to integrate them with both each other and parts of KDE infrastructure such as Identity. Even after such effort is completed a certain degree of synchronisation between the tools will need to be maintained, such as registering repositories in both the code hosting tool and Gerrit. There was no extra work to integrate Gerrit with KDE's Identity, and I expect most of other tools which we might need to use to have LDAP support out of box because that's an industry standard for identity management. KDE Identity == LDAP, in this context. The paragraph above also assumes that Gerrit would not be used as a primary code hosting place. There's no reason for that, so the raised conclusions (this will require syncing) are not true. Please see my earlier mail. You need to draw SSH keys from Identity. 4) You have indicated some (pretty important to me) limitations of Phabricator with a remark that it's a fast moving software, we might get there eventually. I think that SW should be evaluated based on functionality present right now in the development version and not based on opened feature requests. We've been promising e.g. support for multiple accounts in Trojita for many years already, and it didn't make it happen. This belongs to the point below, imo. Or what are you referring to? When I read the proposal, I see some enthusiasm for Phabricator's swift development pace. That's good, but at the same time it isn't an answer to a lack of features. Here are some of the relevant bits: - CI integration We've addressed that. It can be done with little fuss. - scratch repos Also doable, once again with minimal fuss. - clustering for scaleability Not sure what you mean here. The folks behind Phabricator are building a hosted solution called Phacility, so it will be scalable. Components of it, given the right setup, can run on multiple machines. - preserving author metadata Being worked on upstream. Only a problem when you use arc land and will likely be resolved by the time the testers are finished with it. - direct git access to patches under review - patch upload via git I appreciate this is important to you, but it isn't as critical to others. What I'm saying is that an opened feature request and a rapid speed of development are not something to gurantee that these will be ready in a month, or even in a couple of years. To my knowledge, here are some things that Gerrit does not provide, but Phabricator potentially provides: Yes, Gerrit doesn't include wiki pages, issue tracker, Kanban planner and various calendars. I don't necessarily see that as a drawback. Do we want to migrate wikis and Bugzilla? If yes, I can understand that Phabricator might be a compeling tool, but so far the proposal was limited to just revamping the git hosting and code review. Others who are interested in the trial have indicated an interest in using these components to an extent. - a project overview with a code
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 3:54 AM, Milian Wolff m...@milianw.de wrote: On Wednesday 21 January 2015 11:44:00 Jan Kundrát wrote: Hi Ben, thanks for your proposal. A couple of points which I'm missing from the text, and a few further questions as well: 1) A detailed list of the issues which a competing proposal would have to address. Some glimpse of this is in the last paragraph, but that's just a very high-level description. Could you please provide a list of functionality that has to be supported by a replacement, so that we can work out how to get there? Yes, a simple bullet point list would be useful. I'll list some stuff below also. 2) It is unclear to me whether you plan to use Gitolite in future or not. At first, the proposal says that there are inherent scaling issues with it and that the replication is beyond its scaling limits, yet at the end of the document you say that a replacement has to support Gitolite metadata generation. I do not understand that. Agreed. 3) The idea of rolling out Phabricator is not specified, so it's hard to get a proper understanding of what exactly will have to be done. What changes in workflow are going to be needed? What custom tooling will have to be written? How is Phabricator going to play with the rest of the infrastructure? What pieces of the infrastructure will actually remain? snip 5) You're indicating a requirement for scratch repos to be present from the very beginning, yet you acknowledge that Phabricator won't have it for at least six months. Agreed. 6) The discussion focuses in highlighting Phabricator's benefits, which is understandable from your point of view. However, much of the same things can be said about Gerrit as well, especially its backing by a well-known player, adoption by multiple companies and multinational corporations (SonyErisccon, SAP, KitWare,...) as well as by many FLOSS projects (Qt, Andorid, LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod, Tizen, and a ton of others). Its scaleability has also been verified by real-world use over many years (for far longer than Phabricator). Imo, quite the contrary. It concentrates on the issues the admins have with their current setup, and then shows how Phabricator could help with that. Now coming to Gerrit and its analyzis: snip 9) I do not know what effort for integration with KDE Indentity you're referring to. It's a simple LDAP server, and the entire integration was trivial. It's really just a matter of configuring Gerrit to talk to LDAP, which I expect to be also the case with Phabricator. I don't see where this is mentioned in regard to Gerrit. I can only find LDAP being mentioned when talking about the status quo for KDE, which does not include Gerrit. 4) You have indicated some (pretty important to me) limitations of Phabricator with a remark that it's a fast moving software, we might get there eventually. I think that SW should be evaluated based on functionality present right now in the development version and not based on opened feature requests. We've been promising e.g. support for multiple accounts in Trojita for many years already, and it didn't make it happen. This belongs to the point below, imo. Or what are you referring to? If you do mean the git integration, then yes, maybe it's important to you, but it's really not important in the big picture, imo. And do note, I'd personally also prefer gerrit for reviewing. 10) Re Phabricator's support for submitting and retrieving changes via pure git, I believe you're overly optimistic in this regard. This is a pretty fundamental design decision which can only be retroactively fitted with asignificant amount of pain. As far as I understand it, we (KDE) won't jump to Phabricator in a matter of days, rather it will take a few more months before we even start (correct?). There are still some other blockers upstream, which will be resolved. Lets see how this works out. Generally, I don't think its impossible to add this to Phabricator. So lets not be overly pessimistic, nor optimistic. Maybe this feature is added, maybe not. While I personally would also love to have it, it's a minor detail compared to the rest of the feature set, imo. That is correct - we have to evaluate it first, and no migration would be rushed through in a matter of days. Things have to be planned for, prepared, tested and pre-flight migrations run before the final production migration is done. snip 12) WikiMedia uses Phabricator for issue tracking and nothing else -- it's just a Bugzilla replacement for them. According to publicly available information, their reasons for choosing Phabricator had everything to do with the PHP as a language known to all of their developers. They still use Gerrit for git repo management and code review. See http://www.mediawiki.org/wiki/Phabricator, what Jan says is true. But WikiMedia is in the process of migrating away from
Re: State of kdesvn?
El Dimarts, 20 de gener de 2015, a les 20:14:39, Christian Ehrlicher va escriure: Hello, I recently tried to fix some bugs in kdesvn but got stuck because there is nobody who can review my patches. Rajko Albrecht as the original maintainer stopped the development of kdesvn sometime in 2013 and the other two project members (David Faure and Christophe Giboudeaux) have no time to dig deep into the subversion api for a proper review. Therefore I'm asking for help and suggestions how to go further. Maybe there are some people on this list who are still working with subversion and kdesvn and are interested in the further development of kdesvn. It'd be easier if you had written some of the urls for reviewboard here, so we could have clicked on them. So where are the patches that need review? Cheers, Albert Cheers, Christian
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 8:05 AM, Jan Kundrát j...@kde.org wrote: Please read Bens article again. We do this currently and its not working. This is what needs to be replaced. Phabricator seems to support this, or so they say, **and** is does what Gerrit does. So why not use that and have everything integrated? It's not as simple as picking a WWW git browser, it must be integrated into the rest of the system. And it must be easy to maintain etc. pp. Really, just reread the report. The report mentions problems with the current Git replication. I'm also aware of the technical nature of these things beyond what was said in the report because I did talk to Nicolas and Ben about them. The current setup is slow, pull-based, unreliable, and can only work once an hour on an overseas node due to network latencies and systems' throughput. Fair enough. Compared to that, I know about people using Gerrit's replication with a world-wide, distributed network of Git mirrors on multiple continents. What I'm proposing is simply using what works well for these people and use that for scaleable repository browsing as well. There is no need to write any custom scripts. Just deploy as many mirrors as we might need, and add an arbitrary read-only WWW git browser to them, with no extra configuration required -- just serve all the repos in a read-only manner. Gerrit's replication is not stupid, it does understand the concept of queues and automated retries, so it actually *works* and handles bandwidth-limited or latency-bound nodes fine. If one was to use Phabricator, there will still be the same set of problems with git browser having to scale to sustain automated web crawlers and what not. We will also still need N machines to bring actual content close to the users. Nothing changes by using a single application here, really. The problem is that replication needs improvements, so let's improve the replication. Gerrit can do that. What does Phabricator use for replication, and how does it deal with crappy network? Automated web crawlers are told to get lost. We publish robots.txt files which block bots, and for bots caught ignoring that follow up with various forms of blocks. Phabricator itself doesn't provide replication - we'll be building that to match our needs (which includes repository removal and addition, which I imagine you'll need to add to Gerrit as well). Scripting stuff means its not configurable to users, which is extremely important here. And no, writing a script, maintaining it, and adding a config UI on top of it if is explicitly **not** what the sysadmins are looking for. They want to cut down on tools and maintenance burden. OK, I agree that a perfect way is to add this to Gerrit's existing notifications and contributing that patch upstream -- that already includes the whole infrastructure, up to and including the pretty UI. Let's do this; I'll write that patch and push it upstream once I know that this is not going to be a wasted work (see my first mail in this thread). Note: You removed my marker here, that the below is nice to have. Please keep it in, it's important. I never said that this stuff is of utmost importance. I remove those parts of the mail which I mostly agree with; I find that better for readability. Sorry for confusion. As you said, it's important to assign a proper importance to various features; we're in total agreement here. It's true that there's no git browser where you could attach notes to a particular line and open a ticket for that. Do we need such a feature, and if so, how do we intend to use it? We currently have this in the form of the kde-commits mailing list. It is an extremely useful feature of the KDE community. What we get with Phabricator is that, just so much better. Fair enough, the problem with what we have right now is that nobody is guaranteed to read these and to follow up on them, to paraphrase someone (you, maybe?) from the previous iteration of this conversation. Improving that is a nice bonus, so it's indeed cool to be able to create tickets/issues by a single click when one browses a git repo. It's nice that Phabricator can do that. Do we however actually have some people doing this project-wide code auditing *and* not sending patches at the same time? I'm just wondering whether this fancy feature would be used that often, and whether it could be better to do this as regular code review instead. And I also understand that you've listed it in the nice bonuses section. Having an external tool like our current kanban board on todo.kde.org means it's not integrated with the rest. No easy way to link to reviews, branches, issues, etc. pp. Phabricator gives us all of this, for free! If something is bundled, it doesn't mean that it's free. One of the costs that you pay is that you cannot switch to a better alternative because everything and everybody's workflow assumes that you're
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 10:48 AM, Thomas Lübking thomas.luebk...@gmail.com wrote: On Mittwoch, 21. Januar 2015 20:41:18 CEST, Ben Cooksley wrote: - Do we have access to Qt's (ie. KDE's major upstream) decision process (reasoning) towards gerrit? Our prior requests to be informed when new Qt repositories are setup Sorry, I was probably ambiguous here. What I meant is: Do we know why Qt chose gerrit? Unfortunately not. Someone who knows the right people or is familiar with their process will have to fill us in on the details behind their decision. - The lack of non-repo driven (web interface) patch supplies in gerrit does not seem to have been addressed in the document, but there've been many calls for such. So what's the state on this? Sorry, not sure I understand this. Phabricator certainly does support uploading patches through the web interface. Yes. The question was: since this seemed a major issue in the previous thread, but was not mentioned as problem w/ gerrit in the report - does that mean the webinterface issue has been addressed (by either gerrit supporting such or the requirement is gone/replaced) You'll need to ask one of the people more familiar with Gerrit about that i'm afraid. My mail was indeed addressed to any interested reader, not particularily you ;-) so notes are posted on the task regarding the review, and a note is posted on the review regarding the task and one can see the status of both (open, closed, etc) from those notes. That does hopefully not imply one would get two mails for every comment (one for being cc'd on the bug and one for the RR), does it? I guess it would depend on your Herald settings - we would need to test it's behaviour here. From my understanding you can have metadata changes (like associated commits) not trigger emails. Cheers, Thomas Thanks, Ben
Re: Sysadmin report on the modernization of our infrastructure
On Mittwoch, 21. Januar 2015 22:47:20 CEST, Ben Cooksley wrote: - CI integration We've addressed that. It can be done with little fuss. Do you have any hard data on the little fuss? I googled for CI and saw quite some we need such comments over the last two years and also some homebrew script to integrate travis, but it didn't sound like put this 10 LOC sippet here and be happy or so and apparently is not a trivial issue. Would Scarletts efforts (you're pointing them?) be upstreamed to Phabricator? - clustering for scaleability Not sure what you mean here. I'd say https://secure.phabricator.com/book/phabricator/article/cluster/ That looks scary. On how many machines do we run atm? (combined, ie. git, RB, and bugzilla, in case) I appreciate this is important to you, but it isn't as critical to others. It has it's charme to not have to understand and use a second D/VCS and switch around between git and arc logics. The report seemed to suggest this was in production? Cheers, Thomas
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 10:39 AM, Jan Kundrát j...@kde.org wrote: On Wednesday, 21 January 2015 20:07:21 CEST, Ben Cooksley wrote: 1) A detailed list of the issues which a competing proposal would have to address. Some glimpse of this is in the last paragraph, but that's just a very high-level description. Could you please provide a list of functionality that has to be supported by a replacement, so that we can work out how to get there? The list in question was drawn from my summary mail which was sent to the previous thread. Not everything in that list is a requirement (as no software exists which can meet all of them). Ben, it just isn't clear to me what parts of the system are free for replacement and which parts are set in stone to remain as-is. Maybe we don't know yet; that would be fine as well. However, as I'm reading the rest of the text, it seems that some people are looking forward to migrate away from Buzgilla, or to use Phabricator for task management as well. There are also options for getting away with the current set of git hooks, or at least some of them. We don't entirely know yet. Part of the fun of considering migrations like this really. The less we have to maintain though, the better. 3) The idea of rolling out Phabricator is not specified, so it's hard to get a proper understanding of what exactly will have to be done. What changes in workflow are going to be needed? What custom tooling will have to be written? How is Phabricator going to play with the rest of the infrastructure? What pieces of the infrastructure will actually remain? There would be no changes in workflow as such. The only way to properly test a tool though is to actually put it into use for a while - and thus find out whether it is able to fit our workflows. In terms of custom tooling, we will need to integrate it with Jenkins and set it up to receive keys from Identity. Due to the way it works with SSH keys this will be a fairly trivial process as our existing systems which auto-sync keys for svn.kde.org and git.kde.org can be almost entirely reused. Let me ask in a different way, then: - How are you to integrate with CI? Using either http://www.guywarner.com/2014/06/part-2-integrating-phabricator-and.html or http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/ or a variation thereof. - How are you to plug into Bugzilla? Yet to be determined. People have linked Phabricator with JIRA, so similar integration should be achievable for Bugzilla. - How are you going to tie with Jenkins in order to automatically create build jobs/profiles? Jenkins will handle this, using the Job DSL plugin in all likelihood. Jobs would be created based on information provided by the Phabricator Conduit API. - How are you going to send commit e-mails? How are people going to filter them or to subscribe to various projects of interest? Commit emails could either be sent by our existing hooks, or we could migrate to Herald and customise it's template to fit what we need if necessary. People would filter them / subscribe to them through Herald. - How are IRC notifications going to be supported? For commits or reviews? Probably the same mechanism we have at the moment to be honest - commit hooks feeding a Irker bot. I do know that Phabricator does have an IRC bot around which does integrate with it using the Conduit API. I'm sure there are more; the above is just to show what I'm asking about when I speak about integration. 7) It is possible to have scratch repositories with Gerrit, but it's true that it will require a simple tool which verifies user's request and executes an API call with proper credentials. We're speaking about one trivial webapp or a shell script here. This is a separate application, and would therefore require separate maintenance correct? Here's the script that I'm talking about, in pseudocode: if not check_ldap_group($user, developers): die you are not a KDE developer if not regexp_match($proj, '[-a-zA-Z0-9_]': die invalid project name if operation == delete: return `ssh bot@gerrit delete --yes-really-delete --force \ scratch/$user/$proj` if operation != create: die invalid operation return `ssh bot@gerrit create-project --owner $user \ --parent sysadmin/gerrit-user-scratch-projects \ scratch/$user/$proj` That's 9 lines prior to line-wrapping for mail, including error handling. As of maintenance, what's your estimate of the required time to maintain this over the next 10 years? Doesn't seem too high, although I don't see how that would be made web accessible - which might be the hard and costly part maintenance wise. (You have to deal with security issues too as you are in a separate web application, so you need to authenticate the developer first). As a nice bonus, Gerrit supports enforcing quotas for number of per-user repositories as well as the consumed disk space;
Re: Sysadmin report on the modernization of our infrastructure
On Wednesday, 21 January 2015 20:07:21 CEST, Ben Cooksley wrote: 1) A detailed list of the issues which a competing proposal would have to address. Some glimpse of this is in the last paragraph, but that's just a very high-level description. Could you please provide a list of functionality that has to be supported by a replacement, so that we can work out how to get there? The list in question was drawn from my summary mail which was sent to the previous thread. Not everything in that list is a requirement (as no software exists which can meet all of them). Ben, it just isn't clear to me what parts of the system are free for replacement and which parts are set in stone to remain as-is. Maybe we don't know yet; that would be fine as well. However, as I'm reading the rest of the text, it seems that some people are looking forward to migrate away from Buzgilla, or to use Phabricator for task management as well. There are also options for getting away with the current set of git hooks, or at least some of them. 3) The idea of rolling out Phabricator is not specified, so it's hard to get a proper understanding of what exactly will have to be done. What changes in workflow are going to be needed? What custom tooling will have to be written? How is Phabricator going to play with the rest of the infrastructure? What pieces of the infrastructure will actually remain? There would be no changes in workflow as such. The only way to properly test a tool though is to actually put it into use for a while - and thus find out whether it is able to fit our workflows. In terms of custom tooling, we will need to integrate it with Jenkins and set it up to receive keys from Identity. Due to the way it works with SSH keys this will be a fairly trivial process as our existing systems which auto-sync keys for svn.kde.org and git.kde.org can be almost entirely reused. Let me ask in a different way, then: - How are you to integrate with CI? - How are you to plug into Bugzilla? - How are you going to tie with Jenkins in order to automatically create build jobs/profiles? - How are you going to send commit e-mails? How are people going to filter them or to subscribe to various projects of interest? - How are IRC notifications going to be supported? I'm sure there are more; the above is just to show what I'm asking about when I speak about integration. 7) It is possible to have scratch repositories with Gerrit, but it's true that it will require a simple tool which verifies user's request and executes an API call with proper credentials. We're speaking about one trivial webapp or a shell script here. This is a separate application, and would therefore require separate maintenance correct? Here's the script that I'm talking about, in pseudocode: if not check_ldap_group($user, developers): die you are not a KDE developer if not regexp_match($proj, '[-a-zA-Z0-9_]': die invalid project name if operation == delete: return `ssh bot@gerrit delete --yes-really-delete --force \ scratch/$user/$proj` if operation != create: die invalid operation return `ssh bot@gerrit create-project --owner $user \ --parent sysadmin/gerrit-user-scratch-projects \ scratch/$user/$proj` That's 9 lines prior to line-wrapping for mail, including error handling. As of maintenance, what's your estimate of the required time to maintain this over the next 10 years? As a nice bonus, Gerrit supports enforcing quotas for number of per-user repositories as well as the consumed disk space; there's a plugin for that. (Plus people have to find it) I'll be happy to include a nice page in Gerrit's top menu saying Create project which will explain how to request official repositories as well as how to request regular projects from sysadmins :). How would developers delete no longer used scratch repositories? In the same manner as creating them, see above. It's also possible to have a cronjob querying the date of last change, etc. Thiago indicated that for sane mail handling one wants the patches Qt currently has. Is this not the case? I don't know what is or is not sane, but [1] is a random example of what we have right now. Is that sane enough? E-mails that I'm receiving from Qt's own Gerrit look the same, but maybe I'm missing some crucial difference. 9) I do not know what effort for integration with KDE Indentity you're referring to. It's a simple LDAP server, and the entire integration was trivial. It's really just a matter of configuring Gerrit to talk to LDAP, which I expect to be also the case with Phabricator. The integration in question I'm referring to are SSH keys, and ensuring details are automatically updated from LDAP when they change. If i'm not wrong, your current solution requires keys to be uploaded a second time. Yes. This is a matter of one command for each event: `cat $key | ssh bot@gerrit set-account $user --add-ssh-key -` ...and an appropriate
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 11:39 AM, Thomas Lübking thomas.luebk...@gmail.com wrote: On Mittwoch, 21. Januar 2015 22:47:20 CEST, Ben Cooksley wrote: - CI integration We've addressed that. It can be done with little fuss. Do you have any hard data on the little fuss? I googled for CI and saw quite some we need such comments over the last two years and also some homebrew script to integrate travis, but it didn't sound like put this 10 LOC sippet here and be happy or so and apparently is not a trivial issue. Reviewboard does not make it trivial to create such an integration at this time. Please read the report where this is explained in full. Phabricator does make it easier, please see the links i've posted to the mailing list. Would Scarletts efforts (you're pointing them?) be upstreamed to Phabricator? Scarlett's efforts are KDE specific and relate exclusively to Jenkins. We would provide information to the Phabricator community on how we achieved our integration of course. - clustering for scaleability Not sure what you mean here. I'd say https://secure.phabricator.com/book/phabricator/article/cluster/ That looks scary. On how many machines do we run atm? (combined, ie. git, RB, and bugzilla, in case) Git runs on the physical host Kater (i7-2600 w/ 32gb RAM). It shares this with Identity, CiviCRM and Jenkins, as well as our IRC services. The machine isn't heavily utilised except at certain times - but that is caused by scaling issues in the anongit network. At all other times the machine is fairly idle. Reviewboard runs on the physical host Tesla (i7-950 w/ 8gb RAM). It shares this with reports.kde.org, an Owncloud instance and some other minor services. Once again, the machine isn't heavily utilised except when someone makes a large number of commits or sends a stack of emails, which generates a bit of load as reports.kde.org processes them (design is slightly suboptimal here). Bugzilla runs on the physical host Katze (i7-3770 w/ 32gb RAM). It shares this with the Dot, Forum, Wikis as well as all our other CMS based sites including krita.org, calligra.org, kdevelop.org and kdenlive.org. It also hosts todo.kde.org and paste.kde.org. This machine gets a little busy during the EU daytime due to these other sites - Bugzilla doesn't present a load issue (it is a memory hog due to the Perl and MySQL caching needed to keep it performant when queried, but is itself a very low hit rate site). Quickgit runs on a donated container, on the anongit node Serenity. It shares this with nothing else. There are no load issues here. Chiliproject is extremely suboptimal, and a massive resource hog. It runs on a separate machine again. I don't expect there to be any issues requiring us to cluster or move beyond one server hosting Phabricator. I appreciate this is important to you, but it isn't as critical to others. It has it's charme to not have to understand and use a second D/VCS and switch around between git and arc logics. The report seemed to suggest this was in production? What was in production? The hack allowing for reviews to be posted using Git itself? It isn't at this time... Cheers, Thomas Thanks, Ben
Re: Sysadmin report on the modernization of our infrastructure
On Wednesday, 21 January 2015 23:57:07 CEST, Ben Cooksley wrote: Using either http://www.guywarner.com/2014/06/part-2-integrating-phabricator-and.html or http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/ or a variation thereof. That is quite some custom code that one has to maintain, though. Commit emails could either be sent by our existing hooks, or we could migrate to Herald and customise it's template to fit what we need if necessary. People would filter them / subscribe to them through Herald. How would they subcribe via Herald if it was done via the existing hooks? Doesn't seem too high, although I don't see how that would be made web accessible - which might be the hard and costly part maintenance wise. (You have to deal with security issues too as you are in a separate web application, so you need to authenticate the developer first). Well, Apache's mod_authnz_ldap and a Require group developers stanza makes this really easy. Just look up $user from an appropriate env var provided by the web server. Where is the problem? Our existing solution is triggered on change events in LDAP and causes all SSH keys to be re-read and a new ~/.ssh/authorized_keys file to be written out. You can't rely on OpenLDAP stating the addition/removals properly when using the syncrepl interface, at least in my experience. In this way we avoid dependence on the Identity web application. A quick dirty approach: `ssh bot@gerrit set-account $user --remove-ssh-keys ALL` `ssh bot@gerrit set-account $user --add-ssh-key - authorized_keys` A better and race-free code would have to invoke `comm` in addition to that, and only add/remove keys which has to be removed or added. That's left as an excercise for the reader, it's easy enough. Or, to avoid relying on a local state altogether, just issue a REST call for SSH key retrieval and base a decision on that. It's gonna be like 10 lines of custom code. Cheers, Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Sysadmin report on the modernization of our infrastructure
On Wednesday, 21 January 2015 23:12:33 CEST, Thomas Lübking wrote: The bug cooked up for asking google about gerrit and scratch repos. The problem is that pushing into any branch would close a review - I can only assume it was linked in the mail thread I found because a similar issue would affect clones etc. (ie. likely when the change-id gets pushed anywhere) My uninformed guess would be to handle the change-id smarter, ie. bind it to a branch (and pot. repo) IMHO, a pretty straightforward option is to close bugzilla entries once a change is approved and lands in an appropriate branch, and that's easy with Gerrit's its-bugzilla plugin. It is possible to specify which branches are appropriate, of course. Thanks - looks actually nice (and supports dumb diffs ;-) Sicne I've no mediawiki account: do you know whether it already allows ppl. to access (all) their patches w/o modifications (ie. like the RB dashboard) as well? Yes. Their instance doesn't keep a secondary index which limits searching, but a proper query would be something like owner:y...@example.org OR comment:'uploaded by y...@example.org'. That works on our Gerrit. Jan -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: Sysadmin report on the modernization of our infrastructure
On Thu, Jan 22, 2015 at 3:03 PM, Jan Kundrát j...@kde.org wrote: On Wednesday, 21 January 2015 23:57:07 CEST, Ben Cooksley wrote: Using either http://www.guywarner.com/2014/06/part-2-integrating-phabricator-and.html or http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/ or a variation thereof. That is quite some custom code that one has to maintain, though. Doesn't look like an excessive amount to me, once you strip out the comments (which all code should have anyway). Some of it also governs how Jenkins reports back to Phabricator - allowing to provide fine grained commentary on test success or failure. We could probably extend it to report on cppcheck or code coverage differences if someone wanted to even. I see our control over that as a plus point - so developers don't have to go look somewhere else to see the exact nature of the failure (whether build or test related). Commit emails could either be sent by our existing hooks, or we could migrate to Herald and customise it's template to fit what we need if necessary. People would filter them / subscribe to them through Herald. How would they subcribe via Herald if it was done via the existing hooks? Herald can generate mails itself and send them. While you wouldn't get the same commit mails unless we change the template Herald uses, you will get the commit mails. Doesn't seem too high, although I don't see how that would be made web accessible - which might be the hard and costly part maintenance wise. (You have to deal with security issues too as you are in a separate web application, so you need to authenticate the developer first). Well, Apache's mod_authnz_ldap and a Require group developers stanza makes this really easy. Just look up $user from an appropriate env var provided by the web server. Where is the problem? It's a separate web application to maintain. I'll admit that does solve the authentication issue nicely, except for the part where developers can't log out unless they close their browser. Our existing solution is triggered on change events in LDAP and causes all SSH keys to be re-read and a new ~/.ssh/authorized_keys file to be written out. You can't rely on OpenLDAP stating the addition/removals properly when using the syncrepl interface, at least in my experience. In this way we avoid dependence on the Identity web application. A quick dirty approach: `ssh bot@gerrit set-account $user --remove-ssh-keys ALL` `ssh bot@gerrit set-account $user --add-ssh-key - authorized_keys` A better and race-free code would have to invoke `comm` in addition to that, and only add/remove keys which has to be removed or added. That's left as an excercise for the reader, it's easy enough. Or, to avoid relying on a local state altogether, just issue a REST call for SSH key retrieval and base a decision on that. It's gonna be like 10 lines of custom code. Cheers, Jan Regards, Ben -- Trojitá, a fast Qt IMAP e-mail client -- http://trojita.flaska.net/
Re: State of kdesvn?
Am 21.01.2015 um 23:14 schrieb Albert Astals Cid: El Dimarts, 20 de gener de 2015, a les 20:14:39, Christian Ehrlicher va escriure: Hello, I recently tried to fix some bugs in kdesvn but got stuck because there is nobody who can review my patches. Rajko Albrecht as the original maintainer stopped the development of kdesvn sometime in 2013 and the other two project members (David Faure and Christophe Giboudeaux) have no time to dig deep into the subversion api for a proper review. Therefore I'm asking for help and suggestions how to go further. Maybe there are some people on this list who are still working with subversion and kdesvn and are interested in the further development of kdesvn. It'd be easier if you had written some of the urls for reviewboard here, so we could have clicked on them. I'm looking more for a long-term solution as there is a lot of work to do to port it e.g. to Qt5/KF5. I think it wouldn't be good when I sent every review request to kcd... :) So where are the patches that need review? https://reviewboard.kde.org/r/121814/ - bugreport from ubuntu https://reviewboard.kde.org/r/121990/ - krazy2 fixes https://reviewboard.kde.org/r/121986/ - bugreport from kdebugs Thx, Christian
Re: Sysadmin report on the modernization of our infrastructure
Hi, I am all for the proposal -- though I would like to use Phabricator for issue tracking as well, actually. I would also like to propose Calligra/Krita as one of the test projects. On Wednesday 21 January 2015 Jan 17:12:07 Ben Cooksley wrote: Hi all, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. It contains a detailed summary of what is broken with our existing systems, why change is necessary and an evaluation of the options we considered. We have also made a proposal based on our evaluations and the wishlist of functionality drawn up the community. Please find it attached - and let us know what you think. Feedback is welcome. Cheers, Ben Cooksley KDE Sysadmin -- Boudewijn Rempt http://www.valdyas.org, http://www.krita.org
Re: Sysadmin report on the modernization of our infrastructure
On Wed, Jan 21, 2015 at 5:12 AM, Ben Cooksley bcooks...@kde.org wrote: Hi all, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. It contains a detailed summary of what is broken with our existing systems, why change is necessary and an evaluation of the options we considered. We have also made a proposal based on our evaluations and the wishlist of functionality drawn up the community. Please find it attached - and let us know what you think. Feedback is welcome. Thanks a lot Ben for this, very comprehensive document. One question about Phabricator - in the discussions before the RB web UI and uploading patches via web was mentioned a lot, will uploading diff through web interface still be possible via Phabricator or is the Arcanist thing a must? This isn't clear from the report. Cheers -- Martin Klapetek | KDE Developer
Re: Sysadmin report on the modernization of our infrastructure
Hi Ben, thanks for your proposal. A couple of points which I'm missing from the text, and a few further questions as well: 1) A detailed list of the issues which a competing proposal would have to address. Some glimpse of this is in the last paragraph, but that's just a very high-level description. Could you please provide a list of functionality that has to be supported by a replacement, so that we can work out how to get there? 2) It is unclear to me whether you plan to use Gitolite in future or not. At first, the proposal says that there are inherent scaling issues with it and that the replication is beyond its scaling limits, yet at the end of the document you say that a replacement has to support Gitolite metadata generation. I do not understand that. 3) The idea of rolling out Phabricator is not specified, so it's hard to get a proper understanding of what exactly will have to be done. What changes in workflow are going to be needed? What custom tooling will have to be written? How is Phabricator going to play with the rest of the infrastructure? What pieces of the infrastructure will actually remain? 4) You have indicated some (pretty important to me) limitations of Phabricator with a remark that it's a fast moving software, we might get there eventually. I think that SW should be evaluated based on functionality present right now in the development version and not based on opened feature requests. We've been promising e.g. support for multiple accounts in Trojita for many years already, and it didn't make it happen. 5) You're indicating a requirement for scratch repos to be present from the very beginning, yet you acknowledge that Phabricator won't have it for at least six months. 6) The discussion focuses in highlighting Phabricator's benefits, which is understandable from your point of view. However, much of the same things can be said about Gerrit as well, especially its backing by a well-known player, adoption by multiple companies and multinational corporations (SonyErisccon, SAP, KitWare,...) as well as by many FLOSS projects (Qt, Andorid, LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod, Tizen, and a ton of others). Its scaleability has also been verified by real-world use over many years (for far longer than Phabricator). Now coming to Gerrit and its analyzis: 7) It is possible to have scratch repositories with Gerrit, but it's true that it will require a simple tool which verifies user's request and executes an API call with proper credentials. We're speaking about one trivial webapp or a shell script here. 8) There is no need for any modifications to Gerrit as your text implies. What is running and integrated into KDE right now is an unpatched release straight from upstream, with no custom plugins. 9) I do not know what effort for integration with KDE Indentity you're referring to. It's a simple LDAP server, and the entire integration was trivial. It's really just a matter of configuring Gerrit to talk to LDAP, which I expect to be also the case with Phabricator. 10) Re Phabricator's support for submitting and retrieving changes via pure git, I believe you're overly optimistic in this regard. This is a pretty fundamental design decision which can only be retroactively fitted with asignificant amount of pain. 11) While the Gerrit trial has been running for a few months, being used by real KDE projects and generated actual feedback and tweaks, there's been no trial of Phabricator so far. In my opinion, it is premature to have a plan for migration to a particular tool prior to having verified said tool in production environment. In this regard, your proposal effectively discusses throwing away the results we got with Gerrit, and fails to provide some rationale for that. Indeed, the question is where is Phabricator better than Gerrit, and I propose to focus on this aspect in future. 12) WikiMedia uses Phabricator for issue tracking and nothing else -- it's just a Bugzilla replacement for them. According to publicly available information, their reasons for choosing Phabricator had everything to do with the PHP as a language known to all of their developers. They still use Gerrit for git repo management and code review. So given that we're about to rebuild the whole Git infrastructure anyway, the counter-proposal is to build it around Gerrit this time. Based on what I know about KDE's infrastructure, Gerrit can fill in these roles without any problem. I would like to work with you towards that goal; are you interested? Finally, I would like to say that I appreciate the work of the sysadmin team. In future though, I'd love to see a bit more transparency of the entire process. Right now it isn't clear to me whether investing a few additional man-months of my time towards working with Gerrit has any merit, or whether it's been already decided that the KDE's future is with Phabricator. I don't
Re: Sysadmin report on the modernization of our infrastructure
On Wed, Jan 21, 2015 at 7:50 PM, Luca Beltrame lbeltr...@kde.org wrote: Ben Cooksley wrote: Hello Ben, Hi Luca, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. I know I'm much less into coding than other people here (which are probably far more knowledgeable), but in general the proposal seems very sane to me. Assuming this proposal is accepted by the community at large, how can the community help as well? Meaning, is there anything that can be done, given that sysadmin resources are thin and already spread out? We can probably come up with some items I think. Items like determining the rewrite rules for things like Quickgit, WebSVN and Chiliproject would definitely help ease matters. We'd also need to consider a migration path for existing Git project subscriptions into Herald. This isn't a complete list of course, I imagine there will be more. That question aside, I have a question on the proposal itself: KDE uses both Git and SVN, so my question is what will be done to ensure smooth transition also of the SVN side (as the hooks are very old, IIRC, and do a lot of things). We should be able to use our existing Subversion hooks, much like our existing Git ones should also be usable from what I understand. Also, can Phabricator be configured to do what our hooklib.py does here? In particular license checks,source checks such as eol and so on? I've yet to look into the complete power of Herald and it's pre-commit powers, but it is certainly possible that it is capable of doing some of this, yes. Lastly, I assume that there is no plan to use the issue management of Phabricator for now, with the exact rationale of when Redmine was chosen (need to import existing database, size of said database...). I have no issue with it, but perhaps it should still be pointed out. At least in terms of bugs.kde.org - there isn't a plan to use that at the moment. We would need to consider how to prepare Dr Konqi for that, and how to ease the migration there which is probably another project of work. The Wikimedia migration of their instance (about 80,000 bugs) took quite a while from what I understand - and we have ~337,000 bugs in our system at the moment (composed of 1.49 million comments). We do have Kanboard (todo.kde.org) though, for which migration might make sense and should be fairly easy to achieve. Thanks for all your work. Thanks, Ben -- Luca Beltrame - KDE Forums team KDE Science supporter GPG key ID: 6E1A4E79
Re: Review Request 121831: libksysguard: process.h: encapsulate private fields
On Jan. 19, 2015, 6:24 nachm., Aleix Pol Gonzalez wrote: processcore/process.h, line 103 https://git.reviewboard.kde.org/r/121831/diff/2/?file=342813#file342813line103 Maybe you want to prefix the attribute variables with m_ then? Gregor Mi wrote: Thanks for the hint. I will move this member and the ones below it also to the ProcessPrivate class. There the m_ can (should?) be omitted, can't it? Please dont prefix with m_. Is is quite common that member variables m_something become d-something when hidden behind a d-pointer. Having d-m_ would communicate this twice. - Dominik --- This is an automatically generated e-mail. To reply, visit: https://git.reviewboard.kde.org/r/121831/#review74353 --- On Jan. 19, 2015, 9:37 nachm., Gregor Mi wrote: --- This is an automatically generated e-mail. To reply, visit: https://git.reviewboard.kde.org/r/121831/ --- (Updated Jan. 19, 2015, 9:37 nachm.) Review request for KDE Base Apps, Dominik Haumann, Eike Hein, and John Tapsell. Repository: libksysguard Description --- In process.h there are several public fields. I propose to encapsulate the private fields with getter methods. I implemented it exemplary for the fields 'login', 'uid' and 'euid'. (In a separate commit I would add the .reviewboardrc file) What is the current policy on using small C++ macros as done in this RR? Use it (code is more compact and readable) or don't use it (better for debugging)? Diffs - .reviewboardrc PRE-CREATION CMakeLists.txt cefc86f12be684e195bd148641483e9e1734e636 processcore/process.h b6695c0ed301dc5f0fad8ba847da811f19ebfd9a processcore/process.cpp a38b8be71da1a51cb87f636664ebac817b1d20ab processcore/processes.cpp 6c0effc903b7792a078e68d2ac6f7ac29dbbcc31 processcore/processes_atop_p.cpp 24c76e3e35f62bd8e9e705ad32cc11cbd3662601 processcore/processes_linux_p.cpp 898d4fa491873fe95a8b32a5c1b85642b2e46ad5 processui/ProcessFilter.cpp ec520593fb67c777d56817f2493d40dc5ade0347 processui/ProcessModel.cpp 53bc041110c9cdb686fef783895104969b661889 processui/ksysguardprocesslist.cpp 4dc142e864d8353ceafc3a6735ffa81e48291420 processui/scripting.h 2445c0ab0d81af3283c0f6e9c5f349a3d70b0de9 Diff: https://git.reviewboard.kde.org/r/121831/diff/ Testing --- Compiles and runs. Data is still shown. No error. Thanks, Gregor Mi
Re: Review Request 121831: libksysguard: process.h: encapsulate private fields
---
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/121831/#review74463
---
I think it's a good idea to hide the impl behind a d-pointer.
It's yet unclear to me, whether the additional setter/getter calls are really a
performance issue. If required, we could save storing local variables by adding
reference-getters, see comments below. But I'd only do this if we have input
that this is really necessary.
Comments from the KSysGuard authors would be very much appreciated :-)
processcore/process.h
https://git.reviewboard.kde.org/r/121831/#comment51643
Is virtual needed here?
processcore/process.h
https://git.reviewboard.kde.org/r/121831/#comment51644
You can do it like this. Q_DECLARE_PRIVATE does something along these lines:
#define Q_DECLARE_PRIVATE(Class) \
inline Class##Private* d_func() { return
reinterpret_castClass##Private *(qGetPtrHelper(d_ptr)); } \
inline const Class##Private* d_func() const { \
return reinterpret_castconst Class##Private
*(qGetPtrHelper(d_ptr)); } \
friend class Class##Private;
Furtheri, Q_D looks like this:
#define Q_D(Class) Class##Private * const d = d_func()
So you can either use:
Q_D(Process);
d-...
or
d_ptr-...
or
d_func()-...
Personally, I prefer declaring a d-pointer directly, since it does not
require any macro an das such you can directly see what happens, without any
Q_D or d_func() magic.
So in essence: the way you do it is correct, it's just a matter of taste.
Right now, you always need the extra line Q_D(const Process); in the getters.
You could save these by either writing d_ptr-, or by going the pure d-route.
Would be a bit less code (which imho is good), but as said, matter of taste.
Please remove the comment :-)
processcore/process.cpp
https://git.reviewboard.kde.org/r/121831/#comment51646
Can you write:
KSysGuard::Process::Process()
: d_ptr(new ProcessPrivate())
{
clear();
}
Although the curly brace often is in the same line in this class, the kde
coding style is more the above.
processcore/process.cpp
https://git.reviewboard.kde.org/r/121831/#comment51647
Same here.
processcore/process.cpp
https://git.reviewboard.kde.org/r/121831/#comment51648
Yes, needed.
processcore/process.cpp
https://git.reviewboard.kde.org/r/121831/#comment51649
This line is wrong:
(d-tracerpid == d-tracerpid) is always true.
processcore/processes_linux_p.cpp
https://git.reviewboard.kde.org/r/121831/#comment51650
In theorey, if we wanted to avoid the local varialbes here, we could add
reference-getters, e.g.:
qlonglong process-ruid();
These reference getters could be used as parameters to store the data
directly in the desired varialbles, which would equal the current behavior.
Not sure it's worth it, would be cool to have input from true KSysGuard
developers here.
processcore/processes_linux_p.cpp
https://git.reviewboard.kde.org/r/121831/#comment51651
Don't you change behavior here?
Before, we just wrote into the varialbes.
Now, we use the setters, which also sets 'changes |= Process::Gids;'
Is that maybe an issue? I myself don't know the code well enough to see
this here.
- Dominik Haumann
On Jan. 19, 2015, 9:37 nachm., Gregor Mi wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://git.reviewboard.kde.org/r/121831/
---
(Updated Jan. 19, 2015, 9:37 nachm.)
Review request for KDE Base Apps, Dominik Haumann, Eike Hein, and John
Tapsell.
Repository: libksysguard
Description
---
In process.h there are several public fields. I propose to encapsulate the
private fields with getter methods. I implemented it exemplary for the fields
'login', 'uid' and 'euid'.
(In a separate commit I would add the .reviewboardrc file)
What is the current policy on using small C++ macros as done in this RR? Use
it (code is more compact and readable) or don't use it (better for debugging)?
Diffs
-
.reviewboardrc PRE-CREATION
CMakeLists.txt cefc86f12be684e195bd148641483e9e1734e636
processcore/process.h b6695c0ed301dc5f0fad8ba847da811f19ebfd9a
processcore/process.cpp a38b8be71da1a51cb87f636664ebac817b1d20ab
processcore/processes.cpp 6c0effc903b7792a078e68d2ac6f7ac29dbbcc31
processcore/processes_atop_p.cpp 24c76e3e35f62bd8e9e705ad32cc11cbd3662601
processcore/processes_linux_p.cpp 898d4fa491873fe95a8b32a5c1b85642b2e46ad5
processui/ProcessFilter.cpp ec520593fb67c777d56817f2493d40dc5ade0347
processui/ProcessModel.cpp
Re: Sysadmin report on the modernization of our infrastructure
Ben Cooksley wrote: Hello Ben, As promised in the earlier thread, i'd like to present the sysadmin report on the state of the infrastructure surrounding our code. I know I'm much less into coding than other people here (which are probably far more knowledgeable), but in general the proposal seems very sane to me. Assuming this proposal is accepted by the community at large, how can the community help as well? Meaning, is there anything that can be done, given that sysadmin resources are thin and already spread out? That question aside, I have a question on the proposal itself: KDE uses both Git and SVN, so my question is what will be done to ensure smooth transition also of the SVN side (as the hooks are very old, IIRC, and do a lot of things). Also, can Phabricator be configured to do what our hooklib.py does here? In particular license checks,source checks such as eol and so on? Lastly, I assume that there is no plan to use the issue management of Phabricator for now, with the exact rationale of when Redmine was chosen (need to import existing database, size of said database...). I have no issue with it, but perhaps it should still be pointed out. Thanks for all your work. -- Luca Beltrame - KDE Forums team KDE Science supporter GPG key ID: 6E1A4E79
