Re: [Python-Dev] Source control tools
On Tue, Jun 13, 2006 at 10:27:17AM +0200, Alexander Schremmer wrote: Maybe you benchmarked a Tailor deficiency here, but Mercurial scales very well. People use it for work on the Linux kernel etc. Compared to that, Bazaar-NG seems to reach limits already when working on it's own code/repository. We happen to have switched to mercurial a month ago for all of our code and are happy with it. It scales well, as opposed to darcs. It is very similar to git. Mercurial is used for OpenSolaris if I recall correctly. -- Nicolas Chauvat logilab.fr - services en informatique avancée et gestion de connaissances ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
On Thu, Jun 15, 2006 at 10:33:49PM +0200, Alexander Schremmer wrote: On Thu, 15 Jun 2006 19:00:09 +0200, Jan Claeys wrote: Op di, 13-06-2006 te 10:27 +0200, schreef Alexander Schremmer: Bazaar-NG seems to reach limits already when working on it's own code/repository. Canonical uses bzr to develop launchpad.net, which is a little bit larger dan bzr itself, I suspect...? I don't think so, without having seen the Launchpad code. I assume that Launchpad has less comitters (closed source!) and therefore less change sets and less parallel branches. Actually, Launchpad's got twice as many lines of source (as measured by sloccount), nearly 10 times as many versioned files, and about twice as many revisions as bzr. We typically have 10-20 parallel branches going through our review process at any one time (branches are generally reviewed before they land on the trunk), and probably many others being worked on at any given time. Once I pulled the bzr changesets (1-3 months ago) and it needed 3 hours on a 900 MHz machine with a high-speed ( 50 MBit) internet connection (and it was CPU bound). Note that bzr has gained a lot of speed since then, though. That would have been when it was in weave format? The current knit format doesn't suffer the CPU problems in my experience. It's still very slow over a network because it does a very large number of round trips. There's work to greatly reduce that problem, by pipelining and by reducing the number of HTTP requests (by issuing one request with a range header with many ranges, rather than one request per range!). There are also plans to write a smart server. There's a big focus on performance improvements on the bzr list at the moment, and they seem to be making rapid progress. -Andrew. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
Op di, 13-06-2006 te 10:27 +0200, schreef Alexander Schremmer: Bazaar-NG seems to reach limits already when working on it's own code/repository. Canonical uses bzr to develop launchpad.net, which is a little bit larger dan bzr itself, I suspect...? -- Jan Claeys ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
On Thu, 15 Jun 2006 19:00:09 +0200, Jan Claeys wrote: Op di, 13-06-2006 te 10:27 +0200, schreef Alexander Schremmer: Bazaar-NG seems to reach limits already when working on it's own code/repository. Canonical uses bzr to develop launchpad.net, which is a little bit larger dan bzr itself, I suspect...? I don't think so, without having seen the Launchpad code. I assume that Launchpad has less comitters (closed source!) and therefore less change sets and less parallel branches. Once I pulled the bzr changesets (1-3 months ago) and it needed 3 hours on a 900 MHz machine with a high-speed ( 50 MBit) internet connection (and it was CPU bound). Note that bzr has gained a lot of speed since then, though. Kind regards, Alexander ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
On Mon, 12 Jun 2006 23:31:14 +0200, Thomas Wouters wrote: I did partial imports into Mercurial and Bazaar-NG, but I got interrupted and couldn't draw any conclusions -- although from looking at the implementation, I don't think they'd scale very well at the moment (but that could probably be fixed.) Maybe you benchmarked a Tailor deficiency here, but Mercurial scales very well. People use it for work on the Linux kernel etc. Compared to that, Bazaar-NG seems to reach limits already when working on it's own code/repository. Here is a paper comparing different DVCS for the FreeBSD ports tree (one of the largest CVS repositories that exists ;-)): http://www.keltia.net/BSDCan/slides.pdf Kind regards, Alexander ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
On 6/12/06, Guido van Rossum [EMAIL PROTECTED] wrote: Perhaps issues like these should motivate us to consider a differentsource control tool. There's a new crop of tools out that could solvethis by having multiple repositories that can be sync'ed with eachother. This sounds like an important move towards world peace! It would be an important move towards world peace, if it didn't inspire whole new SCM-holy-wars :-) I have a fair bit of experience with different SCM (VC, source control tool, however you want to call them) so I'll take this opportunity to toss up some observations. Not that switching to another SCM will really solve the issues you are referring to, but I happen to think switching to another SCM is a great idea :) Although I don't see an obvious candidate at the moment... I'll explain why. First of all, changing SCM means changing how everyone works. It's nothing like the CVS-Subversion switch, which changed very little in workflow. All the cool SCMs use 'real branches', and to get full benefit you have to switch your development to a 'branch oriented model', if you'll pardon the buzzwordyness. At XS4ALL we've used BitKeeper for a few years now, and while it really took a while for some of the developers to catch on, the branch thing makes parallel development *much* easier. If you haven't experienced it yourself, it's hard to see why it matters so much (except maybe in cases with extreme parallel development, like the Linux kernel), but it really does make life a lot easier in the long run, for programmers and release managers. Secondly, the way most of the 'less-linear' SCMs work is that everyone has their own repository. That is, in fact, what makes it so useful -- no need for central repository access, no need for a network connection for full operability, no need for write access to get your changes checked in (locally), and easy interchanging between repositories. With a large (history-rich) project like Python, though, those repositories can get a tad large. Most of the SCMs have ways to work around it (not downloading the full history, side-storing the full history in a separate branch, etc) but it's still an extra inconvenience. Now, me, I don't mind downloading a couple hundred megabytes to get a 25Mb sourcetree with full history, but I have a 1Gbit link to the python.org servers :) On the other hand, with most of the SCMs, you only download that full history once, and then locally branch off of that.The real reason I think we should consider other SCMs is because I fear what the history will look like when 3.0 is done. In Subversion, merges of branches don't preserve history -- you have to do it yourself. Given the way Subversion works, I don't think that'll really change; it's just not what Subversion is meant to do (and that's fine.) It does mean, however, that when we switch the trunk to 3.0, we have to decide between the history of the trunk or the history of the p3yk branch. We either merge the p3yk branch into the trunk, making a single huge checkin message explaining all the changes (or not), or we swap the trunk and the p3yk branch. The former means 'svn blamelog', for instance, will show a single revision and a single author for *all* p3yk changes. The latter means 'svn blamelog' will group the trunk changes into the merge commits you can already see on the python-3000-checkins list: a block of merges at a time, based on whenever I (or someone else) has the time to merge the trunk in. So, in that case, 'svn blamelog' will show *me* as author of all 2.5-to-2.7 changes, at a time the original change didn't go in, with log messages that are largely irrelevant ;-) And the mess gets bigger if part of p3yk or trunk's development is done in other branches -- svnmerge log messages hidden in svnmerge log messages. ugh. Before XS4ALL switched to BitKeeper, I spent quite a while examining different SCMs, but at the time, they just weren't at the stage of development you'd trust your company development to (not if you can afford BitKeeper anyway ;) After (re-)experiencing the pain that is Subversion/CVS-style branching with the p3yk branch and the manual trunk merges, I went ahead and checked out the current state of the alternatives. There are quite a few, now (Monotone, Darcs, Git/Cogito, Mercurial, Arch/tla/Bazaar, Bazaar-NG, Arx, CodeVille, SVK) and I haven't had time to give them all the in-depth examination they are worthy of, but so far it looks like only a few of them currently scale well enough to handle a large (history-rich) project like Python. Not that it's fair to expect them to scale well, mind you, given that most of them are only a few years old and most don't claim to be 1.0.Using 'tailor' ( http://www.darcs.net/DarcsWiki/Tailor ) I imported the Python sourcetree with full history into Darcs and Git. I also did a partial import into Monotone (history going back a few months) -- the Monotone import was a lot slower than the others, and I didn't feel like
Re: [Python-Dev] Source control tools
On 6/12/06, Thomas Wouters [EMAIL PROTECTED] wrote: On 6/12/06, Guido van Rossum [EMAIL PROTECTED] wrote: Perhaps issues like these should motivate us to consider a differentsource control tool. There's a new crop of tools out that could solvethis by having multiple repositories that can be sync'ed with eachother. This sounds like an important move towards world peace! It would be an important move towards world peace, if it didn't inspire whole new SCM-holy-wars :-) I have a fair bit of experience with different SCM (VC, source control tool, however you want to call them) so I'll take this opportunity to toss up some observations. Not that switching to another SCM will really solve the issues you are referring to, but I happen to think switching to another SCM is a great idea :) Although I don't see an obvious candidate at the moment... I'll explain why. First of all, changing SCM means changing how everyone works. It's nothing like the CVS-Subversion switch, which changed very little in workflow. All the cool SCMs use 'real branches', and to get full benefit you have to switch your development to a 'branch oriented model', if you'll pardon the buzzwordyness. At XS4ALL we've used BitKeeper for a few years now, and while it really took a while for some of the developers to catch on, the branch thing makes parallel development *much* easier. If you haven't experienced it yourself, it's hard to see why it matters so much (except maybe in cases with extreme parallel development, like the Linux kernel), but it really does make life a lot easier in the long run, for programmers and release managers. Secondly, the way most of the 'less-linear' SCMs work is that everyone has their own repository. That is, in fact, what makes it so useful -- no need for central repository access, no need for a network connection for full operability, no need for write access to get your changes checked in (locally), and easy interchanging between repositories. With a large (history-rich) project like Python, though, those repositories can get a tad large. Most of the SCMs have ways to work around it (not downloading the full history, side-storing the full history in a separate branch, etc) but it's still an extra inconvenience. Now, me, I don't mind downloading a couple hundred megabytes to get a 25Mb sourcetree with full history, but I have a 1Gbit link to the python.org servers :) On the other hand, with most of the SCMs, you only download that full history once, and then locally branch off of that. The real reason I think we should consider other SCMs is because I fear what the history will look like when 3.0 is done. In Subversion, merges of branches don't preserve history -- you have to do it yourself. Given the way Subversion works, I don't think that'll really change; it's just not what Subversion is meant to do (and that's fine.) It does mean, however, that when we switch the trunk to 3.0, we have to decide between the history of the trunk or the history of the p3yk branch. We either merge the p3yk branch into the trunk, making a single huge checkin message explaining all the changes (or not), or we swap the trunk and the p3yk branch. The former means 'svn blamelog', for instance, will show a single revision and a single author for *all* p3yk changes. The latter means 'svn blamelog' will group the trunk changes into the merge commits you can already see on the python-3000-checkins list: a block of merges at a time, based on whenever I (or someone else) has the time to merge the trunk in. So, in that case, 'svn blamelog' will show *me* as author of all 2.5-to-2.7 changes, at a time the original change didn't go in, with log messages that are largely irrelevant ;-) And the mess gets bigger if part of p3yk or trunk's development is done in other branches -- svnmerge log messages hidden in svnmerge log messages. ugh. Before XS4ALL switched to BitKeeper, I spent quite a while examining different SCMs, but at the time, they just weren't at the stage of development you'd trust your company development to (not if you can afford BitKeeper anyway ;) After (re-)experiencing the pain that is Subversion/CVS-style branching with the p3yk branch and the manual trunk merges, I went ahead and checked out the current state of the alternatives. There are quite a few, now (Monotone, Darcs, Git/Cogito, Mercurial, Arch/tla/Bazaar, Bazaar-NG, Arx, CodeVille, SVK) and I haven't had time to give them all the in-depth examination they are worthy of, but so far it looks like only a few of them currently scale well enough to handle a large (history-rich) project like Python. Not that it's fair to expect them to scale well, mind you, given that most of them are only a few years old and most don't claim to be 1.0.Using 'tailor' ( http://www.darcs.net/DarcsWiki/Tailor ) I imported the Python sourcetree with full history into Darcs and Git. I also did a partial import into Monotone (history going back a few months) -- the Monotone import was a lot
Re: [Python-Dev] Source control tools
On 6/12/06, Thomas Wouters [EMAIL PROTECTED] wrote: [*] Short? This whole mail was short! I can talk for hours about the benefit of proper branches and what kind of stuff is easier, better and more efficient with them. I can draw huge ASCII diagrams explaining the fundamental difference between CVS/SVN, BitKeeper/Arch/Darcs/Bazaar-NG/Mercurial and Monotone (yes, that's three groups), and I have powerpoint(!) sheets I used to give a presentation about how and why BitKeeper works at work. It's probably a bit off-topic here, though, so don't tempt me ;P Will you be at EuroPython? This might be a good OpenSpace topic if you haven't already secured your speaking slot. It so happens that I saw a talk about Mercurial at last week's baypiggies meeting and the author claimed that it's as fast as Git. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
On Mon, Jun 12, 2006 at 11:31:14PM +0200, Thomas Wouters wrote: First of all, changing SCM means changing how everyone works. Distributed branches is not the only requirement. There are also: -- subtree authorization (different access rights in different parts of the tree); in distributed SCMs this is solved by policies, not by tools, as far as I understand; -- web-based access (ViewCV or like); -- tracker integration (like Subversion with Trac); -- mail notification. Slightly offtopic: I am working for a company where developers work in different OS (Linux, w32, FreeBSD) and speak different languages (Russian, Latvian and English). Two features I really love in Subversion: svn:mime-type and svn:eol-style. The former allows to set character encoding for a file (useful for web-based access); the latter allow SVN to automatically convert line endings between different OSes, but it also allow to set a fixed line ending style for specific files. I don't know another SCM that supports such useful features. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
On 6/13/06, Oleg Broytmann [EMAIL PROTECTED] wrote: On Mon, Jun 12, 2006 at 11:31:14PM +0200, Thomas Wouters wrote: First of all, changing SCM means changing how everyone works. Distributed branches is not the only requirement. Oh, I know, no worries about that. There are also:-- subtree authorization (different access rights in different parts of the tree); in distributed SCMs this is solved by policies, not by tools, as far as I understand;Pretty much a no-brainer in most SCMs, yes: you need privileges to push certain changes to a repository, not to commit them locally. The receiving repository can make as complicated an authentication and authorization step as it wants. Monotone's approach to this is particularly enjoyable: it works with digital signatures, and you can (have to) tell Monotone who's checkins *you* trust ;) -- web-based access (ViewCV or like); -- tracker integration (like Subversion with Trac);-- mail notification.All of these are of course requirements before Python can switch to another SCM, but not for looking at them in the first place -- in most cases, it's not hard to add, if they don't have it already. And besides, while svn has trac integration, we don't actually use it (yet). Slightly offtopic: I am working for a company where developers work indifferent OS (Linux, w32, FreeBSD) and speak different languages (Russian, Latvian and English). Two features I really love in Subversion:svn:mime-type and svn:eol-style. The former allows to set characterencoding for a file (useful for web-based access); the latter allow SVN toautomatically convert line endings between different OSes, but it also allow to set a fixed line ending style for specific files. I don't knowanother SCM that supports such useful features.Those two I actually consider more important features than the three you mentioned above -- as they aren't as easily bolted-on. Thanks, I'll keep them in mind :) -- Thomas Wouters [EMAIL PROTECTED]Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Source control tools
Thomas Wouters [EMAIL PROTECTED] wrote: It would be an important move towards world peace, if it didn't inspire whole new SCM-holy-wars :-) I have a fair bit of experience with different SCM (VC, source control tool, however you want to call them) so I'll take this opportunity to toss up some observations. Not that switching to another SCM will really solve the issues you are referring to, but I happen to think switching to another SCM is a great idea :) Would you like to show me a comprehensive list of which procedures need to be improved, wrt the current workflow with SVN? My own experience is that SVN is pretty bare-bone, that is it provides powerful low-level primitives, but basically no facility to abstract these primitives into higher-level concepts. Basically, it makes many things possible, but few convenient. I believe that tools like svnmerge show that it is indeed possible to build upon SVN to construct higher-level layers, but there's quite some work to do. I would like also to note that SVK is sexy :) The real reason I think we should consider other SCMs is because I fear what the history will look like when 3.0 is done. In Subversion, merges of branches don't preserve history -- you have to do it yourself. Given the way Subversion works, I don't think that'll really change; Actually, Subversion is growing merge-tracking facilities, but it's a long way to what you'd like to happen with the Py3k history (and I don't think that's achievable whatsoever). Git, the 'low level SCM' developed for the Linux kernel, is incredibly fast in normal operation and does branches quite well. I suspect much of its speed is lost on non-Linux platforms, though, and I don't know how well it works on Windows ;) The higher-level part of GIT is written in a combination of bash and perl, with richful usage of textuils, coreutils and whatnot. It's basically unportable by design to native Windows. I guess the only sane approach is to use it under Cygwin. IMO this is a big no-go for real Windows developers. Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
