Re: [hwloc-devel] git / nightly builds
Jeff, I am watching with intense interest, as GASNet will be moving to git on Nov 1. Could you give me a pointer to where I can get copies all the scripts that you guys use for nightly tarballs, commit emails and such? I'll want to have look after you have all the wrinkles ironed out to your satisfaction. FWIW: a viewpoint from somebody who has yet to actually try to implement his ideas: We have PLANNED to script our nightlies to be named with a date stamp and 6 or 8 chars of the SHA1. The format would be something like PROJECT-BRANCH-20130928-12abcdef.tar.gz Where BRANCH=v. for the UPCOMING release in most scenarios (but could be a named feature branch like "oshmem") That way directory listings would sort by branch and date (simple for mere humans) while the SHA1 would allow fetching the corresponding version from git. The full SHA1 would, of course, also be in a file in the tarball (actual filename TBD). I don't think we consider "git describe" to be a useful naming mechanism for nightlies, though for other snapshots it might be useful. For RCs, we pan to tag something like "1.7.3RC" at the start of the RC cycle to get "git describe" to give names containing "1.7.3RC--" which make some sense at a glance, even though N is incremented with every commit and may grow much higher than a contentional RC number would. Again, "1.7.3" in this example is the UPCOMING release rather than the previous one. For a developer's "make dist" from a developer's clone, however, I think we agree "git describe" is as good as anything else readily available. So, in short: I think our plan aligns with yours on scenarios #1 ("make dist" by Jane Developer) and #3 (official releases), but we wanted something more people-friendly for #2 (nightly tarballs). -Paul On Sat, Sep 28, 2013 at 4:30 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com > wrote: > On Sep 28, 2013, at 4:41 AM, Brice Goglin <brice.gog...@inria.fr> wrote: > > > I am having lots of problems when switching the regression testing stuff > > (jenkins) to git, mostly because of make dist. Actually it worked 2 days > > ago, no it breaks because hwloc-unknown.tar.* remain after make > distcheck. > > This means the versions junk still isn't right yet. :-\ > > > 1) There's something I don't understand in the dist scripts. > > config/distscript.csh is only called the top-level Makefile.am, with 4th > > argument = HWLOC_SVN_R, which is never set. So we always fallback to > > git-describe. When building from a tarball, you get "unknown". That > > seems to break make distcheck. We need to pass something in that > > argument, but I don't see what. > > Good catch; sorry about that. HWLOC_SVN_R no longer exists (as you > noted). I just removed that 4th argument to distscript.csh. Now, > distscript (on master and v1.7) only edits VERSION if snapshot=1 and > snapshot_version is empty (i.e., if you do "make dist" in a git clone > instead of running contrib/nightly/make_nightly_snapshot, which will edit > VERSION before running "make distcheck"). > > > 2) VPATH dist support is more fragile than before (I always build under > > $srcdir/build). In the past, we could do a VPATH make dist by just > > symlinking srcdir/doc/doxygen-doc to builddir/doc/doxygen-doc. This now > > generates "unknown" tarballs because we check for .git existence > > explicitly. I fixed my own case by checking for ../.git as well but > > that's not satisfying. > > Mmm. I preserved your ../.git check in > https://github.com/open-mpi/hwloc/commit/c2b7f3d3c713feadb1d2c5a7ccd747cb6673d249 > . > > I don't think I knew/realized you were doign VPATH dist builds by doing > the sym link. > > > It looks like we can fix this by checking for $srcdir/.git instead. If > > we want VPATH support where $builddir isn't a child of $srcdir, we'll > > have to set GIT_DIR=$srcdir/.git before calling git-describe too. > > > > I think this is becoming way too complicated. Nobody won't be able to > > maintain that code in a couple years. Worse, what if you leave Cisco and > > stop working on hwloc one day? In my other projects, I would just > > override the VERSION makefile variable on the make command-line to > > change the tarball name (you won't get that VERSION in lstopo --version, > > but you would still see 1.8a1 from configure). We should rethink what we > > actually need here. > > Yes, these are good points. I agree: the system is too complicated right > now. :-\ > > Let's go through the use cases of what we want: > > 1. "make dist" in a developer's git clone. This should make a "hwloc- describe>.tar.*. > >
Re: [hwloc-devel] nightly snapshot tarballs now available
Le 27/09/2013 23:11, Brice Goglin a écrit : > A couple things: > * Did you rebuild the secondary git repo of the debian/ "submodule"? We > said we would store it on the side of the main hwloc repo. It can go > under github too, but no need to integrate it with trac or so. I've updated the wiki and the website. The only things that we're still missing as far as I know are: * the debian repository as explained above (by the way, beware that branches are not under branches/ in there) * some missing branches on github (mainly the cuda branch) Brice > * I am looking at updating > http://www.open-mpi.org/projects/hwloc/svn.php, I will likely remove a > lot of things from there > * what about > https://git.open-mpi.org/trac/hwloc/wiki/TracSVNCommitMessages, is > everything valid after a s/SVN/GIT/ ? > * Do you see any other page to update? > > Brice > > ___ > hwloc-devel mailing list > hwloc-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
Re: [hwloc-devel] commit messages
Done at https://git.open-mpi.org/trac/hwloc/wiki/CommitMessages I also updated some references to SVN in other wiki pages. Brice Le 28/09/2013 14:19, Jeff Squyres (jsquyres) a écrit : > This sounds good. Can you add this to the trac wiki? > > (e.g., MPICH did something like this when they converted to git) > > > On Sep 28, 2013, at 8:15 AM, Brice Goglinwrote: > >> Let's discuss some rules for commit messages. I used svn propedit often >> in the past, that's not possible anymore once pushed to the main git repo. >> >> 1) Obviously we should follow the commit log convention: >> >> one short line description (less than 80 chars) >> >> full description >> >> >> 2) When backporting patches between public branches, use git cherry-pick >> *-x* so that the old commit ID is recorded is the new commit log. If >> you're working in your private branches, this may not be needed (if the >> old commit ID may change before you actually push it). >> >> 3) Configure your username and email properly before commtting. >> >> git config user.email >> git config user.name "First Last" >> >> This goes into .git/config. Add --global to make these global for all >> your git repos (goes into ~/.gitconfig) >> >> What else? >> >> Brice >> >> ___ >> hwloc-devel mailing list >> hwloc-de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel >
Re: [hwloc-devel] commit messages
I pinged IU. We ran into this problem during the svn->git transition, but I thought DongInn fixed it. Sorry... On Sep 28, 2013, at 8:23 AM, Brice Goglinwrote: > I can't login on Trac. Requested a new password, doesn't work either. > > Brice > > > Le 28/09/2013 14:19, Jeff Squyres (jsquyres) a écrit : >> This sounds good. Can you add this to the trac wiki? >> >> (e.g., MPICH did something like this when they converted to git) >> >> >> On Sep 28, 2013, at 8:15 AM, Brice Goglin wrote: >> >>> Let's discuss some rules for commit messages. I used svn propedit often >>> in the past, that's not possible anymore once pushed to the main git repo. >>> >>> 1) Obviously we should follow the commit log convention: >>> >>> one short line description (less than 80 chars) >>> >>> full description >>> >>> >>> 2) When backporting patches between public branches, use git cherry-pick >>> *-x* so that the old commit ID is recorded is the new commit log. If >>> you're working in your private branches, this may not be needed (if the >>> old commit ID may change before you actually push it). >>> >>> 3) Configure your username and email properly before commtting. >>> >>> git config user.email >>> git config user.name "First Last" >>> >>> This goes into .git/config. Add --global to make these global for all >>> your git repos (goes into ~/.gitconfig) >>> >>> What else? >>> >>> Brice >>> >>> ___ >>> hwloc-devel mailing list >>> hwloc-de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel >> > > ___ > hwloc-devel mailing list > hwloc-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-devel] commit messages
I can't login on Trac. Requested a new password, doesn't work either. Brice Le 28/09/2013 14:19, Jeff Squyres (jsquyres) a écrit : > This sounds good. Can you add this to the trac wiki? > > (e.g., MPICH did something like this when they converted to git) > > > On Sep 28, 2013, at 8:15 AM, Brice Goglinwrote: > >> Let's discuss some rules for commit messages. I used svn propedit often >> in the past, that's not possible anymore once pushed to the main git repo. >> >> 1) Obviously we should follow the commit log convention: >> >> one short line description (less than 80 chars) >> >> full description >> >> >> 2) When backporting patches between public branches, use git cherry-pick >> *-x* so that the old commit ID is recorded is the new commit log. If >> you're working in your private branches, this may not be needed (if the >> old commit ID may change before you actually push it). >> >> 3) Configure your username and email properly before commtting. >> >> git config user.email >> git config user.name "First Last" >> >> This goes into .git/config. Add --global to make these global for all >> your git repos (goes into ~/.gitconfig) >> >> What else? >> >> Brice >> >> ___ >> hwloc-devel mailing list >> hwloc-de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel >
Re: [hwloc-devel] commit messages
This sounds good. Can you add this to the trac wiki? (e.g., MPICH did something like this when they converted to git) On Sep 28, 2013, at 8:15 AM, Brice Goglinwrote: > Let's discuss some rules for commit messages. I used svn propedit often > in the past, that's not possible anymore once pushed to the main git repo. > > 1) Obviously we should follow the commit log convention: > > one short line description (less than 80 chars) > > full description > > > 2) When backporting patches between public branches, use git cherry-pick > *-x* so that the old commit ID is recorded is the new commit log. If > you're working in your private branches, this may not be needed (if the > old commit ID may change before you actually push it). > > 3) Configure your username and email properly before commtting. > > git config user.email > git config user.name "First Last" > > This goes into .git/config. Add --global to make these global for all > your git repos (goes into ~/.gitconfig) > > What else? > > Brice > > ___ > hwloc-devel mailing list > hwloc-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-devel] commit messages
Let's discuss some rules for commit messages. I used svn propedit often in the past, that's not possible anymore once pushed to the main git repo. 1) Obviously we should follow the commit log convention: one short line description (less than 80 chars) full description 2) When backporting patches between public branches, use git cherry-pick *-x* so that the old commit ID is recorded is the new commit log. If you're working in your private branches, this may not be needed (if the old commit ID may change before you actually push it). 3) Configure your username and email properly before commtting. git config user.emailgit config user.name "First Last" This goes into .git/config. Add --global to make these global for all your git repos (goes into ~/.gitconfig) What else? Brice
Re: [hwloc-devel] git / nightly builds
Le 28/09/2013 13:30, Jeff Squyres (jsquyres) a écrit : > Good catch; sorry about that. HWLOC_SVN_R no longer exists (as you > noted). I just removed that 4th argument to distscript.csh. Now, > distscript (on master and v1.7) only edits VERSION if snapshot=1 and > snapshot_version is empty (i.e., if you do "make dist" in a git clone > instead of running contrib/nightly/make_nightly_snapshot, which will > edit VERSION before running "make distcheck"). Thanks, things look better now. > Yes, these are good points. I agree: the system is too complicated right > now. :-\ > > Let's go through the use cases of what we want: > > 1. "make dist" in a developer's git clone. This should make a "hwloc- describe>.tar.*. This is actually the critical point, see below. > 2. make a nightly snapshot tarball. The more I think about this, the more I > think it's the same as #1, right? Yes, that's why I didn't understand why the create_tarball script also modified VERSION to add git describe. These changes should be either entirely outside of make dist (if we want git describe for nightly but not for make dist) or entirely inside make dist (if we want for both). > 3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*". > > Are these the three (or two, if #2 is the same as #1) use cases we want? If > so, I can see about making the code simpler -- e.g., I didn't know about > overriding the VERSION Makefile macro trick... Changing VERSION on the command-line doesn't change the lstopo --version, so it may not be what we really want. Also, if changing the suffix is just a sed on VERSION file before autogen or configure, it's fine too. This all depends on what we really want for (1). * If we don't do (1), we can remove tons of lines of code from the configury and just have the nightly and release scripts modify VERSION before running autogen. Easy. * If we do (1), that needs much more work. I actually don't care much about (1), I am used to tarballs without the SVN revision suffix (not sure why I didn't always get that suffix). I agree that it's convenient to have the suffix for developer builds (when you want to compare several of them, when you distribute that tarball for some reason, etc). But maybe the nightly script is enough for these cases? Does it work with locally modified trees? Or trees that contain additional commits? By the way, maybe move that script back from nightly to dist. > Yeah, I'm a bit disappointed by the github email service hook (the config is > here: https://github.com/open-mpi/hwloc/settings/hooks; scroll down to > "Email"). There's *very* little configuration available: > > - the address to send to > - an email address secret > - what address to send from > > There's no option for diffs (!), and no option to customize the mail/subject. > :-\ > > Do you have a favorite git commit emailing script? We can probably use the > generic github "WebHook URLs" hook (at the top of the list) and host the > diff-emailing script at IU (or anywhere, actually). > I use the attached one for Open-MX and KNEM. Not sure I tried many of them, but this one worked fine so far. It generates messages such as: http://lists.gforge.inria.fr/pipermail/knem-commits/2013-July/000465.html I don't think it truncates the diff yet. We may want some separators between commits too. All this shouldn't be hard to configure. Brice #!/bin/sh # # Copyright (c) 2007 Andy Parkins # # An example hook script to mail out commit update information. This hook # sends emails listing new revisions to the repository introduced by the # change being reported. The rule is that (for branch updates) each commit # will appear on one email and one email only. # # This hook is stored in the contrib/hooks directory. Your distribution # will have put this somewhere standard. You should make this script # executable then link to it in the repository you would like to use it in. # For example, on debian the hook is stored in # /usr/share/doc/git-core/contrib/hooks/post-receive-email: # # chmod a+x post-receive-email # cd /path/to/your/repository.git # ln -sf /usr/share/doc/git-core/contrib/hooks/post-receive-email hooks/post-receive # # This hook script assumes it is enabled on the central repository of a # project, with all users pushing only to it and not between each other. It # will still work if you don't operate in that style, but it would become # possible for the email to be from someone other than the person doing the # push. # # To help with debugging and use on pre-v1.5.1 git servers, this script will # also obey the interface of hooks/update, taking its arguments on the # command line. Unfortunately, hooks/update is called once for each ref. # To avoid firing one email per ref, this script just prints its output to # the screen when used in this mode. The output can then be redirected if # wanted. # # Config # -- # hooks.mailinglist # This is the list that all pushes will go to; leave it blank to
Re: [hwloc-devel] git / nightly builds
On Sep 28, 2013, at 4:41 AM, Brice Goglinwrote: > I am having lots of problems when switching the regression testing stuff > (jenkins) to git, mostly because of make dist. Actually it worked 2 days > ago, no it breaks because hwloc-unknown.tar.* remain after make distcheck. This means the versions junk still isn't right yet. :-\ > 1) There's something I don't understand in the dist scripts. > config/distscript.csh is only called the top-level Makefile.am, with 4th > argument = HWLOC_SVN_R, which is never set. So we always fallback to > git-describe. When building from a tarball, you get "unknown". That > seems to break make distcheck. We need to pass something in that > argument, but I don't see what. Good catch; sorry about that. HWLOC_SVN_R no longer exists (as you noted). I just removed that 4th argument to distscript.csh. Now, distscript (on master and v1.7) only edits VERSION if snapshot=1 and snapshot_version is empty (i.e., if you do "make dist" in a git clone instead of running contrib/nightly/make_nightly_snapshot, which will edit VERSION before running "make distcheck"). > 2) VPATH dist support is more fragile than before (I always build under > $srcdir/build). In the past, we could do a VPATH make dist by just > symlinking srcdir/doc/doxygen-doc to builddir/doc/doxygen-doc. This now > generates "unknown" tarballs because we check for .git existence > explicitly. I fixed my own case by checking for ../.git as well but > that's not satisfying. Mmm. I preserved your ../.git check in https://github.com/open-mpi/hwloc/commit/c2b7f3d3c713feadb1d2c5a7ccd747cb6673d249. I don't think I knew/realized you were doign VPATH dist builds by doing the sym link. > It looks like we can fix this by checking for $srcdir/.git instead. If > we want VPATH support where $builddir isn't a child of $srcdir, we'll > have to set GIT_DIR=$srcdir/.git before calling git-describe too. > > I think this is becoming way too complicated. Nobody won't be able to > maintain that code in a couple years. Worse, what if you leave Cisco and > stop working on hwloc one day? In my other projects, I would just > override the VERSION makefile variable on the make command-line to > change the tarball name (you won't get that VERSION in lstopo --version, > but you would still see 1.8a1 from configure). We should rethink what we > actually need here. Yes, these are good points. I agree: the system is too complicated right now. :-\ Let's go through the use cases of what we want: 1. "make dist" in a developer's git clone. This should make a "hwloc-.tar.*. 2. make a nightly snapshot tarball. The more I think about this, the more I think it's the same as #1, right? 3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*". Are these the three (or two, if #2 is the same as #1) use cases we want? If so, I can see about making the code simpler -- e.g., I didn't know about overriding the VERSION Makefile macro trick... > For instance if we can simpify things by not > building 1.8-final when we build 1.8-rcX, that'd be fine with me. I think that part is actually fairly simple; it just runs "make dist", strips out the greek value from VERSION, and runs "make dist" again. > Random other questions: > * where do you configure commit emails? can we drop/change the > open-mpi/hwloc subject prefix? I removed the hwloc-svn prefix from > mailman to avoid double-prefixing for now > * can we get commit diffs in email, with some truncation limit to 50kB > or so? Yeah, I'm a bit disappointed by the github email service hook (the config is here: https://github.com/open-mpi/hwloc/settings/hooks; scroll down to "Email"). There's *very* little configuration available: - the address to send to - an email address secret - what address to send from There's no option for diffs (!), and no option to customize the mail/subject. :-\ Do you have a favorite git commit emailing script? We can probably use the generic github "WebHook URLs" hook (at the top of the list) and host the diff-emailing script at IU (or anywhere, actually). -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [hwloc-devel] git / nightly builds
Jeff, I am having lots of problems when switching the regression testing stuff (jenkins) to git, mostly because of make dist. Actually it worked 2 days ago, no it breaks because hwloc-unknown.tar.* remain after make distcheck. 1) There's something I don't understand in the dist scripts. config/distscript.csh is only called the top-level Makefile.am, with 4th argument = HWLOC_SVN_R, which is never set. So we always fallback to git-describe. When building from a tarball, you get "unknown". That seems to break make distcheck. We need to pass something in that argument, but I don't see what. 2) VPATH dist support is more fragile than before (I always build under $srcdir/build). In the past, we could do a VPATH make dist by just symlinking srcdir/doc/doxygen-doc to builddir/doc/doxygen-doc. This now generates "unknown" tarballs because we check for .git existence explicitly. I fixed my own case by checking for ../.git as well but that's not satisfying. It looks like we can fix this by checking for $srcdir/.git instead. If we want VPATH support where $builddir isn't a child of $srcdir, we'll have to set GIT_DIR=$srcdir/.git before calling git-describe too. I think this is becoming way too complicated. Nobody won't be able to maintain that code in a couple years. Worse, what if you leave Cisco and stop working on hwloc one day? In my other projects, I would just override the VERSION makefile variable on the make command-line to change the tarball name (you won't get that VERSION in lstopo --version, but you would still see 1.8a1 from configure). We should rethink what we actually need here. For instance if we can simpify things by not building 1.8-final when we build 1.8-rcX, that'd be fine with me. Random other questions: * where do you configure commit emails? can we drop/change the open-mpi/hwloc subject prefix? I removed the hwloc-svn prefix from mailman to avoid double-prefixing for now * can we get commit diffs in email, with some truncation limit to 50kB or so? Le 27/09/2013 15:36, Jeff Squyres (jsquyres) a écrit : > I'd say that git and the new Trac are now fully open for business. I might > still do some shifting around of tags (see below), but otherwise, I think > everything is just about ready. > > I'm revamping make_dist_script and the nightly build script; I should finally > be able to commit those today, if all goes well. However, I had to make some > changes to VERSION and some other surrounding infrastructure to make it all > work. Specifically: git just does some things *differently* than SVN, so it > required some changes in the way that hwloc does things, infrastructure-wise. > > > Two things: > > 1. I'm not excited about back-porting these changes all the way back to > hwloc-1.0 just so that we can make nightly tarballs for these branches which > aren't used any more. I'm thinking that I should definitely make these > changes for master and the v1.7 branch. Should I also do the v1.6 branch? > > I'm thinking that back-porting further is useless; we should just stop making > nightlies for all the older branches. > > To clarify: with SVN, we still checked all the older branches every night and > would make a new tarball if there was ever a change. We never made changes > in those older branches, so we never made new nightlies. But we *would > have*, if a change had every been committed on those branches. > > 2. With SVN, adding the r number in the tarball version was sufficient to > observe ordering of the tarballs. For example: > >hwloc-1.8r1234.tar.bz2 >hwloc-1.8r1240.tar.bz2 >hwloc-1.8r1255.tar.bz2 >...etc. > > With git, we only have a hash number. So there's no obvious ordering of the > tarballs. For example: > >hwloc-1.8git11223344.tar.bz2 >hwloc-1.8git00aa3344.tar.bz2 >hwloc-1.8git99aa2277.tar.bz2 >...etc. > > However, there is "git describe" functionality which, in our case, can show > you how many commits you are beyond a given tag. For example, I added a > "dev" tag on the master branch for the first pure-git commit (i.e., 1 beyond > the last SVN commit). For example: > > $ git clone g...@github.com:open-mpi/hwloc.git > ...clone completes successfully... > $ cd hwloc > $ git checkout master > $ git describe --always > dev-3-g51efdd1 > > Where the "3" represents that we're 3 commits beyond the "dev" tag. The "g" > just means "git", and the "51efdd1" is the hash of that commit. > > Hence, if we use that as our version string, we get tarballs named something > like this: > > hwloc-dev-3-g51efdd1.tar.bz2 > hwloc-dev-10-gf50a385.tar.bz2 > ...etc. > > For the master branch, I think that's fine. However, note that ***THIS IS > DIFFERENT THAN WHAT WE HAVE PREVIOUSLY DONE ON RELEASE BRANCHES!*** > > For example, if you checkout the v1.7 branch: > > $ git checkout v1.7 > $ git describe --always > hwloc-1.7.2-4-g3a6f84c > > *** NOTE THE DIFFERENCE HERE: > > a) The