Re: distributed scm+freebsd svn?
This is amazing! Have you considered an article for publication based on your experiences? Perhaps a handbook article? This will be very helpful going forward. (Although, I do kind of wish it was a instructions fits on the back of a napkin kind of operation.) :) thanks again! -Alfred * Giorgos Keramidas keram...@freebsd.org [090802 18:32] wrote: On Sun, 26 Jul 2009 16:15:34 -0700, Alfred Perlstein alf...@freebsd.org wrote: Hello hackers, Does anyone here use one of the distributed SCMs to manage contributions to FreeBSD in an easy manner? Hi Alfred, Yes, I do that. Any pointers to a setup you have? I thought git was supposed to make this easy, but going over the docs leaves me with a lot of questions. Git is a wonderful system but it's UI and documentation often make me want to scream bad things. My own suggestion is to go with Mercurial, because it's command set looks a *lot* like CVS or Subversion, it's often as fast or even faster than Git, and it doesn't seem as 'confusing' as Git. More details below... I'm hoping to be able to basically: sync into my distributed repo. allow a third party access to it. easily commit upstream back into svn from a branch in my distributed scm. I use a local Mercurial repository for my own patches. It seems to support most of the things I want to do, i.e.: * Keep a clean `/hg/bsd/head' workspace and pull full changesets into that from our svn repository * Support incremental updates of `/hg/bsd/head'. * Easily clone my `/hg/bsd/head' to one or more `feature' branches. * Allow others to pull from `head' as a read-only source over http or ssh. The /head branch has a huge history that I don't really want to keep around in every clone. So I started my conversion from 2007-12-31 and I keep updating it with the `hg convert ...' command wrapped in a small shell script: $ cat -n /hg/bsd/pull-head.sh 1 #!/bin/sh 2 3 set -e 4 hg convert \ 5 --config convert.svn.startrev='175021' \ 6 --config convert.svn.trunk='head' \ 7 --config convert.svn.branches='' \ 8 --config convert.svn.tags='' \ 9 file:///home/svn/base/ \ 10 /hg/bsd/head You can use the webdav http://svn.freebsd.org/base/ or an SSH tunneled URI to access to Subversion repository, but I keep a local mirror of the Subversion repository too, so I prefer that. Typical Mercurial-based Workflow 1. Pull subversion commits into the 'head' workspace. 2. Pull these changes from 'head' to my working tree. 3. Merge the changes with the local patches of the working tree. 4. Extract one or more patches for committing to Subversion 5. Rinse, leather, repeat... Pulling the latest commits from Subversion -- The first step is the easiest bit. I just run `/hg/bsd/pull-head.sh'. This requires an installed copy of the Python bindings of Subversion [devel/py-subversion] and the `convert' extension enabled in my ~/.hgrc file with: [extensions] convert = A sample run of `pull-head.sh' looks like this: keram...@kobe:/hg/bsd$ time ./pull-head.sh scanning source... sorting... converting... 1 Many network stack subsystems use a single global data structure to hold 0 Add padding to struct inpcb, missed during our padding sweep earlier in 3.306 real 1.809 user 0.619 sys keram...@kobe:/hg/bsd$ This is reasonably fast, but it does come with an important caveat. It's not terribly important for my own work, but it *may* be for yours: The Python bindings of Subversion do not support svn:keywords, so all our manually configured '$FreeBSD$' stuff is unexpanded in the converted tree. Mergemaster may cause various levels of fun and amusement if you mix, match and alternate between svn-based and mercurial-based workspaces often! At this point, after pull-head.sh has finished running, the most recent commit in the head/.hg/ workspace state is the last commit by rwatson: keram...@kobe:/hg/bsd/head$ hg log --limit 1 changeset: 12589:8ce7c7a0b804 branch: head tag: tip user:rwatson date:Sun Aug 02 22:47:08 2009 + summary: Add padding to struct inpcb, missed during our padding sweep earlier in keram...@kobe:/hg/bsd/head$ This clone/workspace is my 'clean' slate, and it only contains an `.hg' data store. No checkout or other workspace contents: keram...@kobe:/hg/bsd/head$ ls -la total 6 drwxr-xr-x 3 keramida users - 512 Nov 10 2008 . drwxr-xr-x 8 keramida users - 512 Aug 3 02:36 .. drwxr-xr-x 3 keramida users - 512 Aug 3 02:36 .hg
Re: distributed scm+freebsd svn?
On Sun, 26 Jul 2009 16:15:34 -0700, Alfred Perlstein alf...@freebsd.org wrote: Hello hackers, Does anyone here use one of the distributed SCMs to manage contributions to FreeBSD in an easy manner? Hi Alfred, Yes, I do that. Any pointers to a setup you have? I thought git was supposed to make this easy, but going over the docs leaves me with a lot of questions. Git is a wonderful system but it's UI and documentation often make me want to scream bad things. My own suggestion is to go with Mercurial, because it's command set looks a *lot* like CVS or Subversion, it's often as fast or even faster than Git, and it doesn't seem as 'confusing' as Git. More details below... I'm hoping to be able to basically: sync into my distributed repo. allow a third party access to it. easily commit upstream back into svn from a branch in my distributed scm. I use a local Mercurial repository for my own patches. It seems to support most of the things I want to do, i.e.: * Keep a clean `/hg/bsd/head' workspace and pull full changesets into that from our svn repository * Support incremental updates of `/hg/bsd/head'. * Easily clone my `/hg/bsd/head' to one or more `feature' branches. * Allow others to pull from `head' as a read-only source over http or ssh. The /head branch has a huge history that I don't really want to keep around in every clone. So I started my conversion from 2007-12-31 and I keep updating it with the `hg convert ...' command wrapped in a small shell script: $ cat -n /hg/bsd/pull-head.sh 1 #!/bin/sh 2 3 set -e 4 hg convert \ 5 --config convert.svn.startrev='175021' \ 6 --config convert.svn.trunk='head' \ 7 --config convert.svn.branches='' \ 8 --config convert.svn.tags='' \ 9 file:///home/svn/base/ \ 10 /hg/bsd/head You can use the webdav http://svn.freebsd.org/base/ or an SSH tunneled URI to access to Subversion repository, but I keep a local mirror of the Subversion repository too, so I prefer that. Typical Mercurial-based Workflow 1. Pull subversion commits into the 'head' workspace. 2. Pull these changes from 'head' to my working tree. 3. Merge the changes with the local patches of the working tree. 4. Extract one or more patches for committing to Subversion 5. Rinse, leather, repeat... Pulling the latest commits from Subversion -- The first step is the easiest bit. I just run `/hg/bsd/pull-head.sh'. This requires an installed copy of the Python bindings of Subversion [devel/py-subversion] and the `convert' extension enabled in my ~/.hgrc file with: [extensions] convert = A sample run of `pull-head.sh' looks like this: keram...@kobe:/hg/bsd$ time ./pull-head.sh scanning source... sorting... converting... 1 Many network stack subsystems use a single global data structure to hold 0 Add padding to struct inpcb, missed during our padding sweep earlier in 3.306 real 1.809 user 0.619 sys keram...@kobe:/hg/bsd$ This is reasonably fast, but it does come with an important caveat. It's not terribly important for my own work, but it *may* be for yours: The Python bindings of Subversion do not support svn:keywords, so all our manually configured '$FreeBSD$' stuff is unexpanded in the converted tree. Mergemaster may cause various levels of fun and amusement if you mix, match and alternate between svn-based and mercurial-based workspaces often! At this point, after pull-head.sh has finished running, the most recent commit in the head/.hg/ workspace state is the last commit by rwatson: keram...@kobe:/hg/bsd/head$ hg log --limit 1 changeset: 12589:8ce7c7a0b804 branch: head tag: tip user:rwatson date:Sun Aug 02 22:47:08 2009 + summary: Add padding to struct inpcb, missed during our padding sweep earlier in keram...@kobe:/hg/bsd/head$ This clone/workspace is my 'clean' slate, and it only contains an `.hg' data store. No checkout or other workspace contents: keram...@kobe:/hg/bsd/head$ ls -la total 6 drwxr-xr-x 3 keramida users - 512 Nov 10 2008 . drwxr-xr-x 8 keramida users - 512 Aug 3 02:36 .. drwxr-xr-x 3 keramida users - 512 Aug 3 02:36 .hg keram...@kobe:/hg/bsd/head$ du -sh . 243M. keram...@kobe:/hg/bsd/head$ It does however contains separate changesets for each subversion commit, so I can browse the (local only now) history with a fair amount of speed. Pulling these changes in my personal workspace -- The second step is to pull the latest versions in my personal workspace at `/hg/bsd/src': keram...@kobe:/home/keramida$ cd /hg/bsd/src 1) keram...@kobe:/hg/bsd/src$ hg
Re: distributed scm+freebsd svn?
Hello, I think the most important thing (other than compatibility) is that you use something that you work well with. Sure a lot of these VCS systems have advantages over the other; however, space is cheap -- git packs vs. mercurials way of packing is a minor detail compared to how useful the vcs is to you. That said I've used git, hg, and svn for freebsd development and use them extensively all over. In fact I've just finished a port of hg and hg-git (allows you to use hg to maintain a git repository) and find that its a very nice way to control all of my repositories with one program. I think the only real way to do development that you plan to try to merge is with freebsd-subversion; however, I'm sure there are people with varying views of this sentiment. Both git and hg have svn integration, so it really is up to you -- and if some functionality is missing -- you can always add it ;) have a good day and respectfully, james++ On Tue, Jul 28, 2009 at 01:48, Matthew D. Fullerfulle...@over-yonder.net wrote: On Mon, Jul 27, 2009 at 01:17:34PM +0300 I heard the voice of Andriy Gapon, and lo! it spake thus: P.S. I am looking for a distributed solution (mercurial, bazaar?) that won't take away what I have with git, but would correctly work with svn mergeinfo. I use bazaar, but I don't use the svn integration. AIUI, bzr-svn will _write_ mergeinfo records for merges it creates and pushes into svn, but I don't know whether it reads existing ones. And a great deal of the existing mergeinfo in the BSD repo will be for cherrypicked revs, which aren't recorded anyway, so I don't know how much it would really gain. Vendor branch imports, maybe, but MFC's are right out. -- Matthew Fuller (MF4839) | fulle...@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: distributed scm+freebsd svn?
On Mon, Jul 27, 2009 at 01:17:34PM +0300 I heard the voice of Andriy Gapon, and lo! it spake thus: P.S. I am looking for a distributed solution (mercurial, bazaar?) that won't take away what I have with git, but would correctly work with svn mergeinfo. I use bazaar, but I don't use the svn integration. AIUI, bzr-svn will _write_ mergeinfo records for merges it creates and pushes into svn, but I don't know whether it reads existing ones. And a great deal of the existing mergeinfo in the BSD repo will be for cherrypicked revs, which aren't recorded anyway, so I don't know how much it would really gain. Vendor branch imports, maybe, but MFC's are right out. -- Matthew Fuller (MF4839) | fulle...@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: distributed scm+freebsd svn?
On Sun, 26 Jul 2009 16:15:34 -0700 Alfred Perlstein alf...@freebsd.org mentioned: Hello hackers, Does anyone here use one of the distributed SCMs to manage contributions to FreeBSD in an easy manner? Any pointers to a setup you have? I thought git was supposed to make this easy, but going over the docs leaves me with a lot of questions. I'm hoping to be able to basically: sync into my distributed repo. allow a third party access to it. easily commit upstream back into svn from a branch in my distributed scm. Hi, Alfred! As far as I know there's a lot of people in the communtity use distributed SCMs to work with FreeBSD repo: both git and mercurial are popular. I only used mercurial so I can share my experience only on this SCM. Since I started using it (about 2 years ago) I explored a number of possible ways to keep my repo in sync with FreeBSD one to make merges easier starting from hourly syncs by simple cvsup + hg commit (outlined at wiki) and ending by full repository conversions. Luckily enough after switch to subversion it is now possibleto convert the full repo with all history both to git and hg without much problem. Currently I'm using the Hg convert extentsion bundled into the default mercurial install. I made the result of conversion available here: http://repoman.deglitch.com/bsdhg/FreeBSD/base/ It is updated regularly (every 15 minutes) from the local svn mirror. The size of converted repo is about 885 MiB (with all known branches and tags and entire history). Some points to note if you will try do the initial conversion yourself: 1) Use memory disk for repository storage. That will spedup things a lot (an order of magnitude). 2) Hg convert extension expect the default svn layout (i.e. trunk/ for head, tags/ for tags and branches/ for branches). FreeBSD doesn't follow this layout so I had to tweak the application by hand to allow it to found our branches. It is possible to use command line switches to point it to your trunk, tags and branches location, but as we effectively have two root-level folders for branches (stable/ and releng/) it doesn't work well, so I had to add yet one switch to point to another branches folder. 3) Using local svn mirror is usually preferrable for performance matters. For development I usually do a branch clone (either from head branch, or some stable branch depending what I'm going to work on) by hg clone command and then do my work in that branch (ordiary commit/pull/push workflow, nothing specific). Whenever I need to sync with head I do a pull from the parent branch (e.g. hg pull -r head ssh://s...@repoman.deglitch.com/bsdhg/FreeBSD/base/) and then hg merge -r head. This works like a charm for any kind of work I do. When there's a need to get a diff between the stock FreeBSD tree and you tree you can achieve this by using hg diff -r youbranch -r head. I do not perform commits directly from mercurial, but there're extensions to do this, afaik. This just doesn't sound useful much for me. Access to 3rd party users can be granted the same way it is done in other source control systems. I prefer the plain ssh access for this task by configuring a jail with repository access and creating users accounts with limited access and the special wrapper instead of login shell which allows execution of the limited set of commands. Today such kind of shell is already included into the distribution so there's no need to hack up your own version. -- Stanislav Sedov ST4096-RIPE pgp4Gu0KbYGcA.pgp Description: PGP signature
Re: distributed scm+freebsd svn?
on 27/07/2009 02:15 Alfred Perlstein said the following: Hello hackers, Does anyone here use one of the distributed SCMs to manage contributions to FreeBSD in an easy manner? Any pointers to a setup you have? I thought git was supposed to make this easy, but going over the docs leaves me with a lot of questions. I'm hoping to be able to basically: sync into my distributed repo. allow a third party access to it. easily commit upstream back into svn from a branch in my distributed scm. I am using git for all of the above. It works quite satisfactory, but there is one major annoyance - git-svn doesn't understand svn mergeinfo. As such merges done in svn are not recorded in git. And MFC done in git (e.g. via cherry-pick) won't get propagated as svn merge on dcommit. Some info on my setup. I have 3 git repositories: - main one (svn), it is used for git-svn syncing and it tracks head, stable/7 and stable/6 svn branches; - stable7, used for stable/7 development; - head, used for head/current development; I think of stable7 and head repositories as heavy-wait branches. In these repositories I have a master branch where I track svn through svn repository and I also have light-weight git branches for topic development. Typical work-cycle (e.g. for head): 1. develop in a topic branch of head repository 2. once development goal is achieved merge changes to master branch 3. rebase changes so that all commits are on top of all svn commits (and not interlaced with them) 4. further improve local commits, e.g. split, merge, change commit messages, etc 5. test and review 6. push changes to the corresponding branch of svn repository 7. dcommit in svn repository, changes go to svn This works quite well for head development and also for local changes that are not meant to go to svn. Unfortunately, as I said in the beginning, I have to use svn for doing official (svn) MFCs. Local MFCs via git always stay local. P.S. I am looking for a distributed solution (mercurial, bazaar?) that won't take away what I have with git, but would correctly work with svn mergeinfo. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
distributed scm+freebsd svn?
Hello hackers, Does anyone here use one of the distributed SCMs to manage contributions to FreeBSD in an easy manner? Any pointers to a setup you have? I thought git was supposed to make this easy, but going over the docs leaves me with a lot of questions. I'm hoping to be able to basically: sync into my distributed repo. allow a third party access to it. easily commit upstream back into svn from a branch in my distributed scm. -- - Alfred Perlstein VMOA #5191, 03 vmax, 92 gs500, ch250 - FreeBSD ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
