The purpose of this proposal is to encourage KDE, as it migrates to Git, to migrate to Gitorious.org as a hosting solution. Although I will not be able to attend the KDE SCM BoF at Akademy/GCDS, Johan Sørensen (the developer of Gitorious and founder of Gitorious.org) will be in attendance. I hope that people will read this proposal and consider its merit, then discuss the items presented here both on this list (for initial discussion, and to come up with further questions) and at the relevant SCM BoF when Johan can be present.
==Background== Gitorious is software that sits as middleware between Git and the user. On the Git side, it manages access control to repositories and contains macros and scripts that allow things like merge requests to be handled intuitively through the web. On the web side, it manages authentication, administrative control over users/groups/repositories, provides an interface to code browsing that exceeds the capabilities of GitWeb, and more. It's an excellent piece of software that is actively developed and ever-increasing in features. It is already serving many projects well, including Qt, and many of the workflow issues discovered as a result of Qt's transition to it are actively being fixed. The current plan for KDE Git hosting is to host an instance of Gitorious local to KDE's systems. Dirk Mueller has been tasked with looking into this. However, hosting Gitorious within KDE has some drawbacks. The most egregious is that it places burden on our system administrators, mainly by requiring constant merging of the Gitorious code to keep up-to-date. This increases the time commitments required by a sysadmin team that is already spread thin. It also means that support for modifications or fixes needed by KDE is likely to be thin, in a more "when I get to it" fashion by the Gitorious developers. In contrast, the most actively supported version of Gitorious is always going to be the instance on Gitorious.org. Gitorious is self-hosted, and used constantly by the main Gitorious developer, Johan Sørensen. In addition, problems encountered by KDE would most likely be affecting other projects as well, meaning that they would be likely to be fixed quickly on the production site. ==Proposal== KDE should choose Gitorious.org as its Git hosting solution. This proposal is organized as follows: First, explicit advantages are discussed. Second, possible drawbacks as identified by Dirk (who has looked into a Git migration) are presented. Third, concerns with the migration itself are discussed. Finally, further considerations are enumerated. ==Advantages== Using Gitorious.org as a hosting solution would bring numerous technical and community advantages. A1) The people managing our SCM instance are Git (and Gitorious) experts. This lessens the need for our own sysadmins to become Git experts simply to keep the site running. A2) Developers can share accounts between Qt and KDE, allowing for easier cross-library collaboration and patches. A3) Developers use the same accounts when comitting code to KDE and other (non-Qt) projects. This helps build community because it helps build cross-references between KDE and other projects, and show collaboration that is taking place. A4) Nokia pays for hosted support and a SLA from Gitorious. According to Thomas Zander, there is a possibility that they would be interested in contributing further to cover KDE's costs as well. This would ease our own hosting and support burden and would increase Gitorious' capacity. This doesn't preclude the possibility of us collaborating with Gitorious.org as described in the Migration and Capabilities section. A5) KDE and Gitorious (and Qt) benefit from cross-marketing and promotion. We'd be the most visible project on the site (most likely above Qt in terms of activity), and our involvement would help make Gitorious more visible and legitimate. A6) It makes KDE development look more community-related and less of a walled garden. It is obvious that many of these points are wins in terms of marketing. In fact, in discussions with members of the Marketing Working Group, I posed the question as to what the marketing advantages and disadvantages of following in Qt's footsteps and using Gitorious.org as our hosting provider would be, and many of the above points came directly from this conversation. ==Drawbacks== These are some of the drawbacks of using Gitoriorious.org as a hosting provider. When discussing the Git testing with Dirk, I broached the topic of using Gitorious.org instead of KDE hosting its own instance. The reasons I was told that it was not considered were as follows (verbatim): D1) "I think it is important to have full control over our core infrastructure to stay independent." While it is true that control over our core infrastructure helps ensure our independence, part of the draw of Git is that clones are identical copies of the repository. Should we ever actually need to switch hosting solutions, migration of the actual source code should be relatively simple -- pulling the repositories from one place and pushing them to another. Of course, this would entail a loss of the extra value provided by Gitorious.org. Since Gitorious is open-source we could still use it, but it may require some significant effort to regain the extra value built up on Gitorious.org. D2) "There is not much other reason than that we have the hardware and the hosting, so we can do it." This is provided for completeness, but there isn't much discussion necessary. ==Migration and Capabilities== I've discussed the possibility of migration with Johan to think of the issues that could arise and how they could be avoided. These are the migration concerns that I posed to Johan, and his advice/answers. S1) Commitfilter Although running arbitrary shellscripts on Gitorious.org as git hooks is not allowed for security reasons, Johan is planning to add support to POST the data of commits to URLs (it's in a branch that will be integrated soon). This will allow you to supply one or more callback URLs for a repository, so whenever someone pushes the details of the commit are POSTed via HTTP to the callback URL(s) as JSON. It includes items like the actual commit, who pushed it, and the before/after SHAs. This could be used (with IP-based access control for security) to still be able to provide commitfilter-like functionality (although commitfilter as it exists would likely need to be updated as a result of the SCM difference in the first place). S2) Capacity Capacity is a concern, as KDE would become the largest project both in terms of source code and activity. When these issues was posed to Johan, here was his answer: 'We should look into some kind of general mirroring facility in the long run (not really because of bandwidth used, but also more to speed up transfers on other continents). We do pay for SAN storage and bandwidth, but there's "plenty available" as long as we pay for it.' However, there are two ways this could be mitigated: * Johan said that extra capacity (bandwidth and disk space) is available for purchase in his datacenter, which will ensure capacity is not a problem if KDE/Nokia are providing monetary support in return for the hosting. Bandwidth for the site is currently costing them about $120USD/month. * The servers currently used by KDE for distributing code via Subversion could become a part of a wider Gitorious.org infrastructure, acting as local mirrors and distribution centers for the subset of KDE repositories and clones. This benefits both KDE and Gitorious.org by speeding up transfers to various parts of the world, increasing redundancy, and easing the burden on the main site. For instance, KDE could handle reads on the various KDE repository and clones, while pushes still go to the main server. This would significantly cut down on bandwidth at the main Gitorious.org site. Johan is working on getting an estimate for the cost of a SLA for KDE. S3) Scripty Chani Armitage did some wonderful work porting Scripty to Git. Part of the work includes support for different backends, so that the translators can still use SVN while projects slowly migrate. S4) Identity It is understandable if people would like our Git repositories to be available at git.kde.org. HTTP redirects/proxying or other techniques could be used to redirect (transparently or otherwise) git.kde.org to e.g. kde.gitorious.org. Johan provided some further information: 'It would be a similar approach such as how http://qt.gitorious.org is being done. In nutshell we wrap that site in a custom layout (markup+css+images) and projects are then associated with that site, which in turn means that they'll be presented in the same layout and domain.' S5) Testing the Waters Amarok has been chomping at the bit to switch to Git, and the Amarok developers are willing to be the guinea pigs. If it is decided that Gitorious.org would be a desirable hosting platform, Amarok will immediately port to using it and start ironing out any issues. TagLib would also be switching shortly afterwards. ==Further Considerations== These are further considerations, questions that may be asked that have been posed to Johan, or just relevant points about Gitorious. C1) Are we able to fix technical problems with Gitorious.org or get them fixed in a timely manner? Johan's answer: 'I think we could find some kind of arrangement here perhaps, but we'd have to look into how to solve practically it first, since there are few things worth keeping in mind: * We do have an SLA towards companies such as Nokia * We do have a privacy policy that would need to be governed * We do pay a good amount of money for system administration already (as a result of #1) ** It only goes so far when it comes to actual gitorious-the-application-itself things, but that would be the same situation with any potential KDE admin (at least to begin with) The upside is that apart from personal information supplied by users, there aren't any secret IP used or stored, since it is all under some kind of OSS license' What Johan is referring to is a suggestion by me that trusted KDE sysadmins could have access to admin Gitoroius.org as well. He is not opposed to the idea, but the exact parameters would need to be worked out (which may just entail ensuring that appropriate KDE sysadmins are versed in how the site works and can be trusted not to make any problems that happen *worse*). C2) What happens if Gitorious.org goes down, how long would it take to get it up and running again? Johan's answer: '"if there's a bug, how long does it take to fix?" :) In the case of hardware failures, we do have redundant hardware. We currently run Gitorious.org on two sun x4150's with 32GB of ram, connected to a SAN (over FiberChannel) where the actual Xen images and repository data etc is stored. The machines only utilitize 50% of the resources that can't be grown or shrunk, so we can move the images between them in case of hardware failure.' C3) How will account management and access control work with Gitorious.org? Do we have full control over that? Johan's answer: "The easiest way would probably be to create one or more teams, such as the one Thiago has created here: http://gitorious.org/+kde-developers (or use that). That would allow anyone with an administrator role within that team to add/remove/promote/demote members and all members would have commits rights to any projects or repositories either owned by that team, or where the team is added as committers to a repository. Whoever has commits rights to repositories is entirely governed by the owner of that repo, which can be either a single user or a team (in which case the admins of that team would have the ability to do admin type things on the repo or project)." To me, this sounds like an entirely reasonable method of managing committers. Established KDE developers could be part of the kde-developers group, SoC students could be part of a kde-soc-students group while SoC is going on, and those that continue comitting could be moved into kde-developers, etc. C4) What happens if the Gitorious.org maintainer goes mad and tries to harm KDE or some such thing? Johan's answer: 'There is more than one maintainer of Gitorious these days, and we (Shortcut, a company I co-founded, which manage on Gitorious.org) do have commercial interests and contracts with companies such as Qt software, doing anything harmful or destructive would be a, well, bad business move. Any agreement we come to should probably include some kind of exit strategy for KDE, meaning anything that isn't blocked by the privacy policy could get transfered back to KDE. And the actual Git repositories would be in more than one place anyway (as opposed to a single central SVN repository).' C5) Will 'push -f' be disabled? We can't have a public repository with that allowed. Johan's answer: 'Yes, we're refactoring the pre-receive hook handling quite a bit right now (for the web-hooks and new merge-request features, among other things) so it is something that'll get implemented within a week I think.' C6) What are some pros/cons vs. GitHub? Obviously being open-source is a killer feature, but it'd be good to know what is coming at Gitorious that GitHub doesn't have, since there are many things GitHub has that Gitorious doesn't. What's the Gitorious roadmap? Johan's answer: 'Well, what features do you miss the most from Github then? I think the fact that Gitorious pivots around projects as a whole, rather than individual repositories is a major advantage. Since it helps in keeping everyone moving towards to the same thing ("the project") in the long run, rather than encouraging lots of siloed development. Personally, I also think it works better for collaborating on large projects. I also think that the merge-requests on gitorious is a major advantage, rather than simply being a simple notification feature they can actually be used to implement workflow features such as code-review (with inline commenting) and pushing directly to a magic ref in the target repository, which creates a new merge request (or updates an existing one). The last two items are something we're working hard on right now, as part of our consulting work for Qt. We're also gearing up for some more design and UI improvements. I also think that the fact that Qt is there would be a major advantage for KDE.' I'd like to second something he said there. Being a user of both GitHub and Gitorious, I find that despite what GitHub advertises "Social Coding", the Gitorious paradigm feels much more social. On GitHub, they encourage you to "fork" projects (their word for what everyone else calls "clone" and have no mechanism for real collaboration on the site other than sending merge requests and sending notes. (There are some extras, like the ability to have commits go into chatrooms and such that help with collaboration.) You can have multiple people with commit rights to a project, but they're simply extra allowed SSH keys. In short, GitHub feels more oriented around individual users, and in a sense almost feels isolating, not social. Gitorious has a different feel in that not only are there teams, but teams can own projects. The team paradigm helps people feel like they are really part of a collaborative effort, especially when a team owns a project. It may not be a huge difference from GitHub in terms of the paradigm, but it really gives it a different feel in my experience, and one that makes it feel far more social than GitHub. ==Summary== Gitoroius.org as a hosting platform has many advantages and few, if any, real drawbacks. The developer of Gitorious would be thrilled to have KDE hosted there, and there are many indications that it would be a mutually beneficial relationship. It would ease account management and sysadmin burden while not significantly impacting our autonomy or identity, and would allow KDE and Qt collaboration and development to happen more easily. Johan has indicated that he would be willing to help us migrate and be willing to prioritize adding features or fixing bugs that we would discover or need short-term as we migrate; in addition, he is also happy to discuss having our help and input with system administration should we want it, which helps ensure that we have both a stake in the project and an exit plan should an unfortunate situation arise. Gitorious.org is the logical choice for hosting our Git repositories compared to hosting it ourselves, and should be given serious consideration.
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Kde-scm-interest mailing list [email protected] https://mail.kde.org/mailman/listinfo/kde-scm-interest
