Re: [hwloc-devel] git / nightly builds

2013-10-02 Thread Jeff Squyres (jsquyres) 
Sorry for the delay in replying.


On Sep 29, 2013, at 10:32 AM, Brice Goglin  wrote:

> Now why do we still need a call to git-describe in get_version.sh? Isn't
> this script supposed to just read what distscript.csh did? (which would
> mean that "if test -z "$HWLOC_SNAPSHOT_VERSION" is useless). Or do you
> need that as a fallback for when we compile instead of doing make dist?
> In one case, we force the snapshot by modifying VERSION (make dist), in
> the other case we git describe at runtime (make). It would be good to
> merge these two cases somehow.

Basically, there's a (possibly artificial?) disparity between:

1. running "make dist" from a developer clone
2. pre-processing VERSION to put in the describe version and then running "make 
dist" (i.e., the make_*_tarball scripts)

Specifically, VERSION in the repo has:

snapshot=1
snapshot_version=

Ie., snapshot_version is blank.  Which is why get_version.sh will fill in the 
current "git describe" value if snapshot_version is blank.

But you're right -- this is putting "git describe" in two places.  What if 
VERSION in the repo has:

snapshot=1
snapshot_version=gitclone

And therefore get_version.sh always just uses the snapshot_version value 
(because it will never be blank).  

The downside of this is that "make dist" from a dev clone won't accurately 
represent the tree, but that's probably ok.  

*** If you're kosher with this, I'll remove that extra logic from 
get_version.sh.  Maybe I'll make it error if "snapshot_version" is empty, or 
something.

>> 2. contrib/nightly/make_snapshot_tarball:
>>   - Invoked via cron on the build machine
>> [snip]
> 
> Ok I didn't know that there was so website-specific things in that
> script. I assumed it was mainly a make distcheck (if so, I would have
> tried to reuse it in the regression testing tool).

K.  I think "make dist[check]" is the heart of everything (and the thing that 
is "re-used", so to speak everywhere); the rest is processing that we do for 
whatever reason that the tarball is being built.

Any other thoughts on how we can simplify things?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [hwloc-devel] git / nightly builds

2013-09-29 Thread Paul Hargrove 
On Sun, Sep 29, 2013 at 5:20 AM, Jeff Squyres (jsquyres)  wrote:

[snip]

> The only branch that didn't have any tags at all was master, so I just
> created a "dev" tag on master, and that was sufficient.


Right, we plan to use the gitflow model in which master contains released
code and a "develop" branch serves the role of the svn trunk or cvs head as
the target for 99% of commits.  Since as I understand it, "git describe"
counts from the most recent tag, if we used git describe for tarballs we
probably apply a tag to the develop branch which names the upcoming
release.  Perhaps "v1.7.3devel".
Even if master is your development branch, applying a more descriptive tag
is something you might consider.

[snip]

> That being said, when we tell users to get a nightly tarball (e.g., to get
> a bug fix), my experience is that they don't know/care about the nightly
> tarball numbering scheme: they always just get the most recent version.



The advantage isn't only for the user.
When somebody reports "I found a bug when trying nightly tarball
.tar.gz" it seems useful to me to (without leaving my email
client) know "that was 8 days ago and I *know* we fixed  since then".
 With just a hash I am likely to defer my response until I have a moment to
correlate the hash to a date.  This is NOT a big deal, just an observation
that having the date in the filename makes talking about "it" more
meaningful.

Who knows, maybe we'll just end up using git describe for its simplicity.

-Paul


-- 
Paul H. Hargrove  phhargr...@lbl.gov
Future Technologies Group
Computer and Data Sciences Department Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

Re: [hwloc-devel] git / nightly builds

2013-09-29 Thread Brice Goglin 
Le 29/09/2013 14:00, Jeff Squyres (jsquyres) a écrit :
> Agreed. Let's have distscript.csh be the one that sets the
> git-describe value in VERSION (more below) 

Ok thanks, much better.

Now why do we still need a call to git-describe in get_version.sh? Isn't
this script supposed to just read what distscript.csh did? (which would
mean that "if test -z "$HWLOC_SNAPSHOT_VERSION" is useless). Or do you
need that as a fallback for when we compile instead of doing make dist?
In one case, we force the snapshot by modifying VERSION (make dist), in
the other case we git describe at runtime (make). Itwould be good to
merge these two cases somehow.

> 2. contrib/nightly/make_snapshot_tarball:
>- Invoked via cron on the build machine
>- Very specifically written to interact with the web download tree
>- Generally does the following:
>  a) Gets a new git clone (into a unique directory)
>  b) Compares output from "git describe ..." to latest_snapshot.txt
>  c) If they're the same, exit 0
> --> If there are no commits since last tarball, no need to do anything
>  d) Run autogen, configure, make distcheck.
>  e) Move resulting tarballs to the web download directory
>  e) Re-generate md5sums.txt/sha1sums.txt

Ok I didn't know that there was so website-specific things in that
script. I assumed it was mainly a make distcheck (if so, I would have
tried to reuse it in the regression testing tool).

> However, for the other cases, I think that doxygen is our main culprit for 
> complexity.  :-\  Meaning: I'm now not sure how to make them simpler...

Yes, there's likely no way to simplify that much.

Brice

Re: [hwloc-devel] git / nightly builds

2013-09-29 Thread Jeff Squyres (jsquyres) 
On Sep 28, 2013, at 4:47 PM, Paul Hargrove  wrote:

> I am watching with intense interest, as GASNet will be moving to git on Nov 1.
> Could you give me a pointer to where I can get copies all the scripts that 
> you guys use for nightly tarballs, commit emails and such?
> I'll want to have look after you have all the wrinkles ironed out to your 
> satisfaction.

FWIW, I think we're generally reviewing these scripts mainly because they seem 
to have become too complicated.  Moving to adapt them to use git was probably 
the catalyst for the review; not really the reason.

But the 3 scripts are located in the hwloc master branch (I just described them 
in my prior mail):

  config/distscript.csh
  contrib/nightly/make_snapshot_tarball
  contrib/dist/make_dist_tarball

> FWIW: a viewpoint from somebody who has yet to actually try to implement his 
> ideas:
> 
> We have PLANNED to script our nightlies to be named with a date stamp and 6 
> or 8 chars of the SHA1.
> The format would be something like PROJECT-BRANCH-20130928-12abcdef.tar.gz
> Where BRANCH=v. for the UPCOMING release in most scenarios (but 
> could be a named feature branch like "oshmem")
> That way directory listings would sort by branch and date (simple for mere 
> humans) while the SHA1 would allow fetching the corresponding version from 
> git.  The full SHA1 would, of course, also be in a file in the tarball 
> (actual filename TBD).

FWIW: I talked to Dave Goodell about this quite a bit before going with "git 
describe".  I think I mentioned that our prior tarballs used the SVN r number, 
which therefore made it quite easy to order the tarballs; the git hash 
obviously doesn't provide this ordering.

"git describe" provides a convenient mechanism, IMHO, and does all the work for 
you. It tells you exactly how many commits you are beyond the last tag on that 
branch.  hwloc's tags conveniently imply exactly which branch you're on, so it 
all worked out (i.e., each release series has its own branch -- the 1.7.x 
releases are on the v1.7 branch, the 1.6.x releases are on the v1.6 branch, and 
so on).

The only branch that didn't have any tags at all was master, so I just created 
a "dev" tag on master, and that was sufficient.

Two other minor points contributed to my decision:

1. Dave indicated that both MPICH and other open source projects use the "git 
describe" scheme for their nightly tarballs
2. Using "git describe", I didn't have to muck with the date, branch, etc.

> I don't think we consider "git describe" to be a useful naming mechanism for 
> nightlies, though for other snapshots it might be useful.
> For RCs, we pan to tag something like "1.7.3RC" at the start of the RC cycle 
> to get "git describe" to give names containing "1.7.3RC--" which 
> make some sense at a glance, even though N is incremented with every commit 
> and may grow much higher than a contentional RC number would.  Again, "1.7.3" 
> in this example is the UPCOMING release rather than the previous one.
> 
> For a developer's "make dist" from a developer's clone, however, I think we 
> agree "git describe" is as good as anything else readily available.
> 
> So, in short: I think our plan aligns with yours on scenarios #1 ("make dist" 
> by Jane Developer) and #3 (official releases), but we wanted something more 
> people-friendly for #2 (nightly tarballs).

Fair enough.

That being said, when we tell users to get a nightly tarball (e.g., to get a 
bug fix), my experience is that they don't know/care about the nightly tarball 
numbering scheme: they always just get the most recent version.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [hwloc-devel] git / nightly builds

2013-09-29 Thread Jeff Squyres (jsquyres) 
On Sep 28, 2013, at 8:12 AM, Brice Goglin  wrote:

>> Let's go through the use cases of what we want:
>> 
>> 1. "make dist" in a developer's git clone.  This should make a "hwloc-> describe>.tar.*.
> 
> This is actually the critical point, see below.
> 
>> 2. make a nightly snapshot tarball.  The more I think about this, the more I 
>> think it's the same as #1, right?
> 
> Yes, that's why I didn't understand why the create_tarball script also
> modified VERSION to add git describe. These changes should be either
> entirely outside of make dist (if we want git describe for nightly but
> not for make dist) or entirely inside make dist (if we want for both).

Agreed.  Let's have distscript.csh be the one that sets the git-describe value 
in VERSION (more below).

>> 3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*".
>> 
>> Are these the three (or two, if #2 is the same as #1) use cases we want?  If 
>> so, I can see about making the code simpler -- e.g., I didn't know about 
>> overriding the VERSION Makefile macro trick...
> 
> Changing VERSION on the command-line doesn't change the lstopo
> --version, so it may not be what we really want. Also, if changing the
> suffix is just a sed on VERSION file before autogen or configure, it's
> fine too.
> 
> This all depends on what we really want for (1).
> * If we don't do (1), we can remove tons of lines of code from the
> configury and just have the nightly and release scripts modify VERSION
> before running autogen. Easy.
> * If we do (1), that needs much more work.

I'm not sure I agree.  Let me review the 3 scripts and what each does (and why):

1. config/distscript.csh:
   - Called by "make dist[check]"
   - Does 3 main things:
 a) Sets git-describe value in VERSION file (but only if snapshot=1)
 b) Generates/copies doxygen output files to the dist tree
--> We wanted to not require users to have doxygen
--> We also wanted to ship final doxygen output in tarballs
--> This created quite a bit of complexity in itself
 c) Get new config.guess and config.sub files from git.savannah.gnu.org
--> This code is from when GNU Autotools releases were few and far 
between; it may not be necessary any more.  We might well be able to just 
remove this code and still be fine.

2. contrib/nightly/make_snapshot_tarball:
   - Invoked via cron on the build machine
   - Very specifically written to interact with the web download tree
   - Generally does the following:
 a) Gets a new git clone (into a unique directory)
 b) Compares output from "git describe ..." to latest_snapshot.txt
 c) If they're the same, exit 0
--> If there are no commits since last tarball, no need to do anything
 d) Run autogen, configure, make distcheck.
 e) Move resulting tarballs to the web download directory
 e) Re-generate md5sums.txt/sha1sums.txt

3. contrib/dist/make_dist_tarball:
   - Invoked manually by a human when making releases
   - Generally does the following:
 a) Check versions of GNU Autotools
--> To ensure to use *the same* versions of the Autotools for an entire 
release series
 b) Update VERSION file with the release date
 c) Run autogen + configure, build doxygen docs, build README, run make 
distcheck
 d) Remove "greek" value from VERSION and run c) again

> I actually don't care much about (1), I am used to tarballs without the
> SVN revision suffix (not sure why I didn't always get that suffix). I
> agree that it's convenient to have the suffix for developer builds (when
> you want to compare several of them, when you distribute that tarball
> for some reason, etc). But maybe the nightly script is enough for these
> cases? Does it work with locally modified trees? Or trees that contain
> additional commits?

No, it gets a fresh clone every night.  The intent was that this script runs in 
an automated fashion, and sometimes the build fails.  So it leaves the failed 
clone/build in a place where a human can go post-mortem the tree and figure out 
why the build failed.

But to be fair, this hasn't been a big issue for hwloc (it has with Open MPI -- 
e.g., "make dist" works for developers, but it failed on the build machine.  So 
it was really helpful to be able to login as mpiteam@build_machine and go look 
into the build tree and see WTF happened).

It also specifically interacts with the web download tree.  To be clear: this 
script is all about deciding whether to make a new snapshot, and if so, run 
"make dist" to do so, and then copies to results to the web tree.

The core of this is:

- *all* use cases run "make dist"; config/distscript.csh is used by all of them
- make_dist_tarball is some additional accounting surrounding "make dist"
- make_snapshot_tarball is some additional (different) accounting surrounding 
"make dist"

All that being said, I think 2 immediate improvements/simplifications are:

- have make_snapshot_tarball not set "git

Re: [hwloc-devel] git / nightly builds

2013-09-28 Thread Paul Hargrove 
Jeff,

I am watching with intense interest, as GASNet will be moving to git on Nov
1.
Could you give me a pointer to where I can get copies all the scripts that
you guys use for nightly tarballs, commit emails and such?
I'll want to have look after you have all the wrinkles ironed out to your
satisfaction.

FWIW: a viewpoint from somebody who has yet to actually try to implement
his ideas:

We have PLANNED to script our nightlies to be named with a date stamp and 6
or 8 chars of the SHA1.
The format would be something like PROJECT-BRANCH-20130928-12abcdef.tar.gz
Where BRANCH=v. for the UPCOMING release in most scenarios
(but could be a named feature branch like "oshmem")
That way directory listings would sort by branch and date (simple for mere
humans) while the SHA1 would allow fetching the corresponding version from
git.  The full SHA1 would, of course, also be in a file in the tarball
(actual filename TBD).

I don't think we consider "git describe" to be a useful naming mechanism
for nightlies, though for other snapshots it might be useful.
For RCs, we pan to tag something like "1.7.3RC" at the start of the RC
cycle to get "git describe" to give names containing "1.7.3RC--"
which make some sense at a glance, even though N is incremented with every
commit and may grow much higher than a contentional RC number would.
 Again, "1.7.3" in this example is the UPCOMING release rather than the
previous one.

For a developer's "make dist" from a developer's clone, however, I think we
agree "git describe" is as good as anything else readily available.

So, in short: I think our plan aligns with yours on scenarios #1 ("make
dist" by Jane Developer) and #3 (official releases), but we wanted
something more people-friendly for #2 (nightly tarballs).

-Paul


On Sat, Sep 28, 2013 at 4:30 AM, Jeff Squyres (jsquyres)  wrote:

> On Sep 28, 2013, at 4:41 AM, Brice Goglin  wrote:
>
> > I am having lots of problems when switching the regression testing stuff
> > (jenkins) to git, mostly because of make dist. Actually it worked 2 days
> > ago, no it breaks because hwloc-unknown.tar.* remain after make
> distcheck.
>
> This means the versions junk still isn't right yet.  :-\
>
> > 1) There's something I don't understand in the dist scripts.
> > config/distscript.csh is only called the top-level Makefile.am, with 4th
> > argument = HWLOC_SVN_R, which is never set. So we always fallback to
> > git-describe. When building from a tarball, you get "unknown". That
> > seems to break make distcheck. We need to pass something in that
> > argument, but I don't see what.
>
> Good catch; sorry about that.  HWLOC_SVN_R no longer exists (as you
> noted).  I just removed that 4th argument to distscript.csh.  Now,
> distscript (on master and v1.7) only edits VERSION if snapshot=1 and
> snapshot_version is empty (i.e., if you do "make dist" in a git clone
> instead of running contrib/nightly/make_nightly_snapshot, which will edit
> VERSION before running "make distcheck").
>
> > 2) VPATH dist support is more fragile than before (I always build under
> > $srcdir/build). In the past, we could do a VPATH make dist by just
> > symlinking srcdir/doc/doxygen-doc to builddir/doc/doxygen-doc. This now
> > generates "unknown" tarballs because we check for .git existence
> > explicitly. I fixed my own case by checking for ../.git as well but
> > that's not satisfying.
>
> Mmm.  I preserved your ../.git check in
> https://github.com/open-mpi/hwloc/commit/c2b7f3d3c713feadb1d2c5a7ccd747cb6673d249
> .
>
> I don't think I knew/realized you were doign VPATH dist builds by doing
> the sym link.
>
> > It looks like we can fix this by checking for $srcdir/.git instead. If
> > we want VPATH support where $builddir isn't a child of $srcdir, we'll
> > have to set GIT_DIR=$srcdir/.git before calling git-describe too.
> >
> > I think this is becoming way too complicated. Nobody won't be able to
> > maintain that code in a couple years. Worse, what if you leave Cisco and
> > stop working on hwloc one day? In my other projects, I would just
> > override the VERSION makefile variable on the make command-line to
> > change the tarball name (you won't get that VERSION in lstopo --version,
> > but you would still see 1.8a1 from configure). We should rethink what we
> > actually need here.
>
> Yes, these are good points.  I agree: the system is too complicated right
> now.  :-\
>
> Let's go through the use cases of what we want:
>
> 1. "make dist" in a developer's git clone.  This should make a "hwloc- describe>.tar.*.
>
> 2. make a nightly snapshot tarball.  The more I think about this, the more
> I think it's the same as #1, right?
>
> 3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*".
>
> Are these the three (or two, if #2 is the same as #1) use cases we want?
>  If so, I can see about making the code simpler -- e.g., I didn't know
> about overriding the VERSION Makefile macro trick...
>
> > For instance if

Re: [hwloc-devel] git / nightly builds

2013-09-28 Thread Brice Goglin 
Le 28/09/2013 13:30, Jeff Squyres (jsquyres) a écrit :
> Good catch; sorry about that. HWLOC_SVN_R no longer exists (as you
> noted). I just removed that 4th argument to distscript.csh. Now,
> distscript (on master and v1.7) only edits VERSION if snapshot=1 and
> snapshot_version is empty (i.e., if you do "make dist" in a git clone
> instead of running contrib/nightly/make_nightly_snapshot, which will
> edit VERSION before running "make distcheck").

Thanks, things look better now.

> Yes, these are good points.  I agree: the system is too complicated right 
> now.  :-\
>
> Let's go through the use cases of what we want:
>
> 1. "make dist" in a developer's git clone.  This should make a "hwloc- describe>.tar.*.

This is actually the critical point, see below.

> 2. make a nightly snapshot tarball.  The more I think about this, the more I 
> think it's the same as #1, right?

Yes, that's why I didn't understand why the create_tarball script also
modified VERSION to add git describe. These changes should be either
entirely outside of make dist (if we want git describe for nightly but
not for make dist) or entirely inside make dist (if we want for both).

> 3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*".
>
> Are these the three (or two, if #2 is the same as #1) use cases we want?  If 
> so, I can see about making the code simpler -- e.g., I didn't know about 
> overriding the VERSION Makefile macro trick...

Changing VERSION on the command-line doesn't change the lstopo
--version, so it may not be what we really want. Also, if changing the
suffix is just a sed on VERSION file before autogen or configure, it's
fine too.

This all depends on what we really want for (1).
* If we don't do (1), we can remove tons of lines of code from the
configury and just have the nightly and release scripts modify VERSION
before running autogen. Easy.
* If we do (1), that needs much more work.

I actually don't care much about (1), I am used to tarballs without the
SVN revision suffix (not sure why I didn't always get that suffix). I
agree that it's convenient to have the suffix for developer builds (when
you want to compare several of them, when you distribute that tarball
for some reason, etc). But maybe the nightly script is enough for these
cases? Does it work with locally modified trees? Or trees that contain
additional commits? By the way, maybe move that script back from nightly
to dist.

> Yeah, I'm a bit disappointed by the github email service hook (the config is 
> here: https://github.com/open-mpi/hwloc/settings/hooks; scroll down to 
> "Email").  There's *very* little configuration available:
>
> - the address to send to
> - an email address secret
> - what address to send from
>
> There's no option for diffs (!), and no option to customize the mail/subject. 
>  :-\
>
> Do you have a favorite git commit emailing script?  We can probably use the 
> generic github "WebHook URLs" hook (at the top of the list) and host the 
> diff-emailing script at IU (or anywhere, actually).
>

I use the attached one for Open-MX and KNEM. Not sure I tried many of
them, but this one worked fine so far. It generates messages such as:

http://lists.gforge.inria.fr/pipermail/knem-commits/2013-July/000465.html
I don't think it truncates the diff yet. We may want some separators
between commits too. All this shouldn't be hard to configure.

Brice

#!/bin/sh
#
# Copyright (c) 2007 Andy Parkins
#
# An example hook script to mail out commit update information.  This hook
# sends emails listing new revisions to the repository introduced by the
# change being reported.  The rule is that (for branch updates) each commit
# will appear on one email and one email only.
#
# This hook is stored in the contrib/hooks directory.  Your distribution
# will have put this somewhere standard.  You should make this script
# executable then link to it in the repository you would like to use it in.
# For example, on debian the hook is stored in
# /usr/share/doc/git-core/contrib/hooks/post-receive-email:
#
#  chmod a+x post-receive-email
#  cd /path/to/your/repository.git
#  ln -sf /usr/share/doc/git-core/contrib/hooks/post-receive-email 
hooks/post-receive
#
# This hook script assumes it is enabled on the central repository of a
# project, with all users pushing only to it and not between each other.  It
# will still work if you don't operate in that style, but it would become
# possible for the email to be from someone other than the person doing the
# push.
#
# To help with debugging and use on pre-v1.5.1 git servers, this script will
# also obey the interface of hooks/update, taking its arguments on the
# command line.  Unfortunately, hooks/update is called once for each ref.
# To avoid firing one email per ref, this script just prints its output to
# the screen when used in this mode.  The output can then be redirected if
# wanted.
#
# Config
# --
# hooks.mailinglist
#   This is the list that all pushes will go to; leave it blank to

Re: [hwloc-devel] git / nightly builds

2013-09-28 Thread Jeff Squyres (jsquyres) 
On Sep 28, 2013, at 4:41 AM, Brice Goglin  wrote:

> I am having lots of problems when switching the regression testing stuff
> (jenkins) to git, mostly because of make dist. Actually it worked 2 days
> ago, no it breaks because hwloc-unknown.tar.* remain after make distcheck.

This means the versions junk still isn't right yet.  :-\

> 1) There's something I don't understand in the dist scripts.
> config/distscript.csh is only called the top-level Makefile.am, with 4th
> argument = HWLOC_SVN_R, which is never set. So we always fallback to
> git-describe. When building from a tarball, you get "unknown". That
> seems to break make distcheck. We need to pass something in that
> argument, but I don't see what.

Good catch; sorry about that.  HWLOC_SVN_R no longer exists (as you noted).  I 
just removed that 4th argument to distscript.csh.  Now, distscript (on master 
and v1.7) only edits VERSION if snapshot=1 and snapshot_version is empty (i.e., 
if you do "make dist" in a git clone instead of running 
contrib/nightly/make_nightly_snapshot, which will edit VERSION before running 
"make distcheck").

> 2) VPATH dist support is more fragile than before (I always build under
> $srcdir/build). In the past, we could do a VPATH make dist by just
> symlinking srcdir/doc/doxygen-doc to builddir/doc/doxygen-doc. This now
> generates "unknown" tarballs because we check for .git existence
> explicitly. I fixed my own case by checking for ../.git as well but
> that's not satisfying.

Mmm.  I preserved your ../.git check in 
https://github.com/open-mpi/hwloc/commit/c2b7f3d3c713feadb1d2c5a7ccd747cb6673d249.

I don't think I knew/realized you were doign VPATH dist builds by doing the sym 
link.

> It looks like we can fix this by checking for $srcdir/.git instead. If
> we want VPATH support where $builddir isn't a child of $srcdir, we'll
> have to set GIT_DIR=$srcdir/.git before calling git-describe too.
> 
> I think this is becoming way too complicated. Nobody won't be able to
> maintain that code in a couple years. Worse, what if you leave Cisco and
> stop working on hwloc one day? In my other projects, I would just
> override the VERSION makefile variable on the make command-line to
> change the tarball name (you won't get that VERSION in lstopo --version,
> but you would still see 1.8a1 from configure). We should rethink what we
> actually need here.

Yes, these are good points.  I agree: the system is too complicated right now.  
:-\

Let's go through the use cases of what we want:

1. "make dist" in a developer's git clone.  This should make a "hwloc-.tar.*.

2. make a nightly snapshot tarball.  The more I think about this, the more I 
think it's the same as #1, right?

3. make a release tarball, "hwloc-major.minor.releasegreek.tar.*".

Are these the three (or two, if #2 is the same as #1) use cases we want?  If 
so, I can see about making the code simpler -- e.g., I didn't know about 
overriding the VERSION Makefile macro trick...

> For instance if we can simpify things by not
> building 1.8-final when we build 1.8-rcX, that'd be fine with me.

I think that part is actually fairly simple; it just runs "make dist", strips 
out the greek value from VERSION, and runs "make dist" again.

> Random other questions:
> * where do you configure commit emails? can we drop/change the
> open-mpi/hwloc subject prefix? I removed the hwloc-svn prefix from
> mailman to avoid double-prefixing for now
> * can we get commit diffs in email, with some truncation limit to 50kB
> or so?

Yeah, I'm a bit disappointed by the github email service hook (the config is 
here: https://github.com/open-mpi/hwloc/settings/hooks; scroll down to 
"Email").  There's *very* little configuration available:

- the address to send to
- an email address secret
- what address to send from

There's no option for diffs (!), and no option to customize the mail/subject.  
:-\

Do you have a favorite git commit emailing script?  We can probably use the 
generic github "WebHook URLs" hook (at the top of the list) and host the 
diff-emailing script at IU (or anywhere, actually).

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [hwloc-devel] git / nightly builds

2013-09-28 Thread Brice Goglin 
Jeff,

I am having lots of problems when switching the regression testing stuff
(jenkins) to git, mostly because of make dist. Actually it worked 2 days
ago, no it breaks because hwloc-unknown.tar.* remain after make distcheck.

1) There's something I don't understand in the dist scripts.
config/distscript.csh is only called the top-level Makefile.am, with 4th
argument = HWLOC_SVN_R, which is never set. So we always fallback to
git-describe. When building from a tarball, you get "unknown". That
seems to break make distcheck. We need to pass something in that
argument, but I don't see what.

2) VPATH dist support is more fragile than before (I always build under
$srcdir/build). In the past, we could do a VPATH make dist by just
symlinking srcdir/doc/doxygen-doc to builddir/doc/doxygen-doc. This now
generates "unknown" tarballs because we check for .git existence
explicitly. I fixed my own case by checking for ../.git as well but
that's not satisfying.

It looks like we can fix this by checking for $srcdir/.git instead. If
we want VPATH support where $builddir isn't a child of $srcdir, we'll
have to set GIT_DIR=$srcdir/.git before calling git-describe too.


I think this is becoming way too complicated. Nobody won't be able to
maintain that code in a couple years. Worse, what if you leave Cisco and
stop working on hwloc one day? In my other projects, I would just
override the VERSION makefile variable on the make command-line to
change the tarball name (you won't get that VERSION in lstopo --version,
but you would still see 1.8a1 from configure). We should rethink what we
actually need here. For instance if we can simpify things by not
building 1.8-final when we build 1.8-rcX, that'd be fine with me.


Random other questions:
* where do you configure commit emails? can we drop/change the
open-mpi/hwloc subject prefix? I removed the hwloc-svn prefix from
mailman to avoid double-prefixing for now
* can we get commit diffs in email, with some truncation limit to 50kB
or so?





Le 27/09/2013 15:36, Jeff Squyres (jsquyres) a écrit :
> I'd say that git and the new Trac are now fully open for business.  I might 
> still do some shifting around of tags (see below), but otherwise, I think 
> everything is just about ready.
>
> I'm revamping make_dist_script and the nightly build script; I should finally 
> be able to commit those today, if all goes well.  However, I had to make some 
> changes to VERSION and some other surrounding infrastructure to make it all 
> work.  Specifically: git just does some things *differently* than SVN, so it 
> required some changes in the way that hwloc does things, infrastructure-wise. 
>  
>
> Two things:
>
> 1. I'm not excited about back-porting these changes all the way back to 
> hwloc-1.0 just so that we can make nightly tarballs for these branches which 
> aren't used any more.  I'm thinking that I should definitely make these 
> changes for master and the v1.7 branch.  Should I also do the v1.6 branch?  
>
> I'm thinking that back-porting further is useless; we should just stop making 
> nightlies for all the older branches.
>
> To clarify: with SVN, we still checked all the older branches every night and 
> would make a new tarball if there was ever a change.  We never made changes 
> in those older branches, so we never made new nightlies.  But we *would 
> have*, if a change had every been committed on those branches.
>
> 2. With SVN, adding the r number in the tarball version was sufficient to 
> observe ordering of the tarballs.  For example:
>
>hwloc-1.8r1234.tar.bz2
>hwloc-1.8r1240.tar.bz2
>hwloc-1.8r1255.tar.bz2
>...etc.
>
> With git, we only have a hash number.  So there's no obvious ordering of the 
> tarballs.  For example:
>
>hwloc-1.8git11223344.tar.bz2
>hwloc-1.8git00aa3344.tar.bz2
>hwloc-1.8git99aa2277.tar.bz2
>...etc.
>
> However, there is "git describe" functionality which, in our case, can show 
> you how many commits you are beyond a given tag.  For example, I added a 
> "dev" tag on the master branch for the first pure-git commit (i.e., 1 beyond 
> the last SVN commit).  For example:
>
> $ git clone g...@github.com:open-mpi/hwloc.git
> ...clone completes successfully...
> $ cd hwloc
> $ git checkout master
> $ git describe --always
> dev-3-g51efdd1
>
> Where the "3" represents that we're 3 commits beyond the "dev" tag.  The "g" 
> just means "git", and the "51efdd1" is the hash of that commit.
>
> Hence, if we use that as our version string, we get tarballs named something 
> like this:
>
> hwloc-dev-3-g51efdd1.tar.bz2
> hwloc-dev-10-gf50a385.tar.bz2
> ...etc.
>
> For the master branch, I think that's fine.  However, note that ***THIS IS 
> DIFFERENT THAN WHAT WE HAVE PREVIOUSLY DONE ON RELEASE BRANCHES!***
>
> For example, if you checkout the v1.7 branch:
>
> $ git checkout v1.7
> $ git describe --always
> hwloc-1.7.2-4-g3a6f84c
>
> *** NOTE THE DIFFERENCE HERE:
>
> a) The

Re: [hwloc-devel] git / nightly builds

2013-09-27 Thread Jeff Squyres (jsquyres) 
On Sep 27, 2013, at 9:43 AM, Brice Goglin  wrote:

> I don't think so. I am planning to only update the regression testing
> for v1.7 as well.

K.

> BUT what if the stable OMPI 1.6 needs a hwloc fix? Should we keep the
> hwloc v1.5 branch open?

E... let's cross that bridge when/if we need to.  :-)

>> 2. With SVN, adding the r number in the tarball version was sufficient to 
>> observe ordering of the tarballs.  For example:
>> [snip]
>> I think that this is ok (other projects use this "git describe" kind of 
>> behavior for their nightly snapshots), but this is a change from what we 
>> used to do, so I wanted to call it out specifically.
>> 
>> Are you guys ok with this?
> 
> That'll work for me.
> 
> What about when we are actually doing a release where we don't want a
> git-describe suffix ? Does the above apply to contrib/make_dist_tarball
> in general? Or only to when it runs at night (in make dist only?) ?

There are actually 2 scripts:

contrib/dist/make_dist_tarball
contrib/nightly/make_snapshot_tarball (currently called create_tarball.sh)

The former works largely as it did before: it'll make hwloc-.tar.*.

The latter is being heavily updated to basically use "git describe".

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [hwloc-devel] git / nightly builds

2013-09-27 Thread Samuel Thibault 
Jeff Squyres (jsquyres), le Fri 27 Sep 2013 15:36:33 +0200, a écrit :
> a) The last SVN nightly snapshot on the v1.7 branch was named 
>hwloc-1.7.3rc1r5779.tar.bz2.
> b) The first git nightly snapshot on the v1.7 branch will be named 
>hwloc-1.7.2-4-g3a6f84c.tar.bz2.
> 
> Note "1.7.3rc1..." vs. "1.7.2...".  I.e., the git name will say "we're X 
> commits beyond the 1.7.2 tag", but the old SVN name was "we're at this 
> *upcoming* version".

I'm fine with it.

Samuel

Re: [hwloc-devel] git / nightly builds

2013-09-27 Thread Brice Goglin 
Le 27/09/2013 15:36, Jeff Squyres (jsquyres) a écrit :
> I'd say that git and the new Trac are now fully open for business.  I might 
> still do some shifting around of tags (see below), but otherwise, I think 
> everything is just about ready.
>
> I'm revamping make_dist_script and the nightly build script; I should finally 
> be able to commit those today, if all goes well.  However, I had to make some 
> changes to VERSION and some other surrounding infrastructure to make it all 
> work.  Specifically: git just does some things *differently* than SVN, so it 
> required some changes in the way that hwloc does things, infrastructure-wise. 
>  
>
> Two things:
>
> 1. I'm not excited about back-porting these changes all the way back to 
> hwloc-1.0 just so that we can make nightly tarballs for these branches which 
> aren't used any more.  I'm thinking that I should definitely make these 
> changes for master and the v1.7 branch.  Should I also do the v1.6 branch?

I don't think so. I am planning to only update the regression testing
for v1.7 as well.

BUT what if the stable OMPI 1.6 needs a hwloc fix? Should we keep the
hwloc v1.5 branch open?

> 2. With SVN, adding the r number in the tarball version was sufficient to 
> observe ordering of the tarballs.  For example:
>
>hwloc-1.8r1234.tar.bz2
>hwloc-1.8r1240.tar.bz2
>hwloc-1.8r1255.tar.bz2
>...etc.
>
> With git, we only have a hash number.  So there's no obvious ordering of the 
> tarballs.  For example:
>
>hwloc-1.8git11223344.tar.bz2
>hwloc-1.8git00aa3344.tar.bz2
>hwloc-1.8git99aa2277.tar.bz2
>...etc.
>
> However, there is "git describe" functionality which, in our case, can show 
> you how many commits you are beyond a given tag.  For example, I added a 
> "dev" tag on the master branch for the first pure-git commit (i.e., 1 beyond 
> the last SVN commit).  For example:
>
> $ git clone g...@github.com:open-mpi/hwloc.git
> ...clone completes successfully...
> $ cd hwloc
> $ git checkout master
> $ git describe --always
> dev-3-g51efdd1
>
> Where the "3" represents that we're 3 commits beyond the "dev" tag.  The "g" 
> just means "git", and the "51efdd1" is the hash of that commit.
>
> Hence, if we use that as our version string, we get tarballs named something 
> like this:
>
> hwloc-dev-3-g51efdd1.tar.bz2
> hwloc-dev-10-gf50a385.tar.bz2
> ...etc.
>
> For the master branch, I think that's fine.  However, note that ***THIS IS 
> DIFFERENT THAN WHAT WE HAVE PREVIOUSLY DONE ON RELEASE BRANCHES!***
>
> For example, if you checkout the v1.7 branch:
>
> $ git checkout v1.7
> $ git describe --always
> hwloc-1.7.2-4-g3a6f84c
>
> *** NOTE THE DIFFERENCE HERE:
>
> a) The last SVN nightly snapshot on the v1.7 branch was named 
>hwloc-1.7.3rc1r5779.tar.bz2.
> b) The first git nightly snapshot on the v1.7 branch will be named 
>hwloc-1.7.2-4-g3a6f84c.tar.bz2.
>
> Note "1.7.3rc1..." vs. "1.7.2...".  I.e., the git name will say "we're X 
> commits beyond the 1.7.2 tag", but the old SVN name was "we're at this 
> *upcoming* version".
>
> I think that this is ok (other projects use this "git describe" kind of 
> behavior for their nightly snapshots), but this is a change from what we used 
> to do, so I wanted to call it out specifically.
>
> Are you guys ok with this?
>

That'll work for me.

What about when we are actually doing a release where we don't want a
git-describe suffix ? Does the above apply to contrib/make_dist_tarball
in general? Or only to when it runs at night (in make dist only?) ?

Brice

[hwloc-devel] git / nightly builds

2013-09-27 Thread Jeff Squyres (jsquyres) 
I'd say that git and the new Trac are now fully open for business.  I might 
still do some shifting around of tags (see below), but otherwise, I think 
everything is just about ready.

I'm revamping make_dist_script and the nightly build script; I should finally 
be able to commit those today, if all goes well.  However, I had to make some 
changes to VERSION and some other surrounding infrastructure to make it all 
work.  Specifically: git just does some things *differently* than SVN, so it 
required some changes in the way that hwloc does things, infrastructure-wise.  

Two things:

1. I'm not excited about back-porting these changes all the way back to 
hwloc-1.0 just so that we can make nightly tarballs for these branches which 
aren't used any more.  I'm thinking that I should definitely make these changes 
for master and the v1.7 branch.  Should I also do the v1.6 branch?  

I'm thinking that back-porting further is useless; we should just stop making 
nightlies for all the older branches.

To clarify: with SVN, we still checked all the older branches every night and 
would make a new tarball if there was ever a change.  We never made changes in 
those older branches, so we never made new nightlies.  But we *would have*, if 
a change had every been committed on those branches.

2. With SVN, adding the r number in the tarball version was sufficient to 
observe ordering of the tarballs.  For example:

   hwloc-1.8r1234.tar.bz2
   hwloc-1.8r1240.tar.bz2
   hwloc-1.8r1255.tar.bz2
   ...etc.

With git, we only have a hash number.  So there's no obvious ordering of the 
tarballs.  For example:

   hwloc-1.8git11223344.tar.bz2
   hwloc-1.8git00aa3344.tar.bz2
   hwloc-1.8git99aa2277.tar.bz2
   ...etc.

However, there is "git describe" functionality which, in our case, can show you 
how many commits you are beyond a given tag.  For example, I added a "dev" tag 
on the master branch for the first pure-git commit (i.e., 1 beyond the last SVN 
commit).  For example:

$ git clone g...@github.com:open-mpi/hwloc.git
...clone completes successfully...
$ cd hwloc
$ git checkout master
$ git describe --always
dev-3-g51efdd1

Where the "3" represents that we're 3 commits beyond the "dev" tag.  The "g" 
just means "git", and the "51efdd1" is the hash of that commit.

Hence, if we use that as our version string, we get tarballs named something 
like this:

hwloc-dev-3-g51efdd1.tar.bz2
hwloc-dev-10-gf50a385.tar.bz2
...etc.

For the master branch, I think that's fine.  However, note that ***THIS IS 
DIFFERENT THAN WHAT WE HAVE PREVIOUSLY DONE ON RELEASE BRANCHES!***

For example, if you checkout the v1.7 branch:

$ git checkout v1.7
$ git describe --always
hwloc-1.7.2-4-g3a6f84c

*** NOTE THE DIFFERENCE HERE:

a) The last SVN nightly snapshot on the v1.7 branch was named 
   hwloc-1.7.3rc1r5779.tar.bz2.
b) The first git nightly snapshot on the v1.7 branch will be named 
   hwloc-1.7.2-4-g3a6f84c.tar.bz2.

Note "1.7.3rc1..." vs. "1.7.2...".  I.e., the git name will say "we're X 
commits beyond the 1.7.2 tag", but the old SVN name was "we're at this 
*upcoming* version".

I think that this is ok (other projects use this "git describe" kind of 
behavior for their nightly snapshots), but this is a change from what we used 
to do, so I wanted to call it out specifically.

Are you guys ok with this?

(note: I'm still mucking with the final tag names, so some of the above may 
change slightly)

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/