Re: Request for stable, automated build process

[BrendaWang]

Dear all:
After one weeks great work.
Julien had release the new toolchain release.
please take a look of this Page.

http://wiki.openmoko.org/wiki/Toolchain



Bobby Martin ??:


From: Ian Darwin <[EMAIL PROTECTED] >
I did mention when I jumped in that I was talking about a slightly
different problem than what you seem to be trying to solve. That
said...

Storing your MD5s will let you know *if* you are repeating a
build. It will not (reasonably) let you repeat a build.


You completely misunderstood what I said. Storing the file name,
URL *and* its MD5 lets you be sure you are able to reproduce the
build. And it's the only way you can (well, we actually use SHA's
and RMD's in addition to MD5's).


If I misunderstood it, I still do. MD5 absolutely will not help you 
reproduce a build. It will only help you verify that your procedure 
(whatever it is) retrieved the same files. It won't help you find the 
correct file if the file you got was wrong.



It does not guarantee to build *exactly* the same binary, but you
can *never ever* make that guarantee in general because of changes
to shared libraries (in the Java world you have a greater chance,
but even then,
even the smallest point upgrade to the JDK could in theory change
the generated .class files), so there is really no point whatever
using MD5's on generated binaries. None. I have no idea why Hudson
bothers, if that's what it's using them for.


You need some way of identifying the file you want to build
*to the revision control system* (so you can download that
version) if you want repeatable builds.


In the case I'm talking about - where there are about 5,000
different external programs from about 2,500 different sources -
you don't even think about going to the revision control system (6
or 8 different services over 2,000 different hosts). You can ONLY
think in terms of
ftp/http download of version-numbered tarballs. Period.


No, you MOSTLY think in terms of ftp/http download. For the projects 
on which you're doing work (at least) you absolutely go to the 
revision control system.


Are we talking about the same thing? I'm talking about a repeatable 
build so I can do development work, modifying existing code. To do 
that, you'd better work with the revision control system. Even if you 
don't have submit access (which the vast majority of devs probably 
shouldn't have), in some cases you have a RCS that supports creating 
smart diffs that you can send to someone to review & submit trivially 
(e.g. git or darcs) and in other cases you almost certainly want to 
submit a patch from the latest possible revision.



MD5s sound nice to verify, if you don't trust your revision
control system (or perhaps the admins ;-)


See above for why revision control is completely irrelevant to us.


Who is 'us'?

Bobby

--
If it doesn't make you smile, you're doing something wrong.


___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
  



___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Bobby Martin]

> From: Ian Darwin <[EMAIL PROTECTED]>
> I did mention when I jumped in that I was talking about a slightly
> different problem than what you seem to be trying to solve. That said...
>
>  Storing your MD5s will let you know *if* you are repeating a build.  It
> > will not (reasonably) let you repeat a build.
> >
>
> You completely misunderstood what I said. Storing the file name, URL *and*
> its MD5 lets you be sure you are able to reproduce the build. And it's the
> only way you can (well, we actually use SHA's and RMD's in addition to
> MD5's).
>

If I misunderstood it, I still do.  MD5 absolutely will not help you
reproduce a build.  It will only help you verify that your procedure
(whatever it is) retrieved the same files.  It won't help you find the
correct file if the file you got was wrong.

>
> It does not guarantee to build *exactly* the same binary, but you can
> *never ever* make that guarantee in general because of changes to shared
> libraries (in the Java world you have a greater chance, but even then,
> even the smallest point upgrade to the JDK could in theory change the
> generated .class files), so there is really no point whatever using MD5's on
> generated binaries. None. I have no idea why Hudson bothers, if that's what
> it's using them for.
>

>  You need some way of identifying the file you want to build *to the
> > revision control system* (so you can download that version) if you want
> > repeatable builds.
> >
>
> In the case I'm talking about - where there are about 5,000 different
> external programs from about 2,500 different sources - you don't even think
> about going to the revision control system (6 or 8 different services over
> 2,000 different hosts). You can ONLY think in terms of
> ftp/http download of version-numbered tarballs. Period.
>

No, you MOSTLY think in terms of ftp/http download.  For the projects on
which you're doing work (at least) you absolutely go to the revision control
system.

Are we talking about the same thing?  I'm talking about a repeatable build
so I can do development work, modifying existing code.  To do that, you'd
better work with the revision control system.  Even if you don't have submit
access (which the vast majority of devs probably shouldn't have), in some
cases you have a RCS that supports creating smart diffs that you can send to
someone to review & submit trivially (e.g. git or darcs) and in other cases
you almost certainly want to submit a patch from the latest possible
revision.

>
>  MD5s sound nice to verify, if you don't trust your revision control
> > system (or perhaps the admins ;-)
> >
>
> See above for why revision control is completely irrelevant to us.
>

Who is 'us'?

Bobby

-- 
If it doesn't make you smile, you're doing something wrong.
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Ian Darwin]

On Mon, May 05, 2008 at 05:55:43PM -0400, Ian Darwin wrote:
> You completely misunderstood what I said. Storing the file name, URL 
> *and* its MD5 lets you be sure you are able to reproduce the build. And 

What I meant to say here was: "Storing the file name, URL
and the MD5 of the *source* tarball lets you be sure..."

Ian

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Ian Darwin]

I did mention when I jumped in that I was talking about a slightly 
different problem than what you seem to be trying to solve. That said...


Storing your MD5s will let you know *if* you are repeating a build.  It 
will not (reasonably) let you repeat a build.


You completely misunderstood what I said. Storing the file name, URL 
*and* its MD5 lets you be sure you are able to reproduce the build. And 
it's the only way you can (well, we actually use SHA's and RMD's in 
addition to MD5's).


It does not guarantee to build *exactly* the same binary, but you can 
*never ever* make that guarantee in general because of changes to shared 
libraries (in the Java world you have a greater chance, but even then,
even the smallest point upgrade to the JDK could in theory change the 
generated .class files), so there is really no point whatever using 
MD5's on generated binaries. None. I have no idea why Hudson bothers, if 
that's what it's using them for.


You need some way of identifying the file you want to build *to the 
revision control system* (so you can download that version) if you want 
repeatable builds.


In the case I'm talking about - where there are about 5,000 different 
external programs from about 2,500 different sources - you don't even 
think about going to the revision control system (6 or 8 different 
services over 2,000 different hosts). You can ONLY think in terms of

ftp/http download of version-numbered tarballs. Period.

MD5s sound nice to verify, if you don't trust your revision control 
system (or perhaps the admins ;-)


See above for why revision control is completely irrelevant to us.

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Bobby Martin]

From: Ian Darwin <[EMAIL PROTECTED]>

> 
> While I might not have worded it quite that way, I have considered writing
> a paper with the title "Maven Considered Harmful". But it would be too short
> to publish as a paper... The main problem is, as Hugo mentioned,
> dependencies can change without notice and break things on you.  You want
> repeatable builds? Write a build system that saves the complete name and MD5
> of every file, and checks every file that it downloads before using it,
> every time. The OpenBSD (Unix-like system) "ports" mechanism does this for
> all third-party software, and it therefore has repeatable builds. Maven does
> not.
>

Storing your MD5s will let you know *if* you are repeating a build.  It will
not (reasonably) let you repeat a build.

You need some way of identifying the file you want to build *to the revision
control system* (so you can download that version) if you want repeatable
builds.

That's why I've talked about dates for external build system, which someone
astutely pointed out could be an issue because of the many time zones
involved, and I believe they discussed some 'pin' notion in mtn.  I'm pretty
sure every RCS has some way of letting you get an identifier for the version
for each controlled file and retrieve that version later.

MD5s sound nice to verify, if you don't trust your revision control system
(or perhaps the admins ;-)

Bobby

-- 
If it doesn't make you smile, you're doing something wrong.
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[AVee]

On Sunday 04 May 2008 22:17, Ian Darwin wrote:
>
> You want repeatable builds? Write a build system that saves the
> complete name and MD5 of every file, and checks every file that it
> downloads before using it, every time. 

Pssst:
http://hudson.gotdns.com/wiki/display/HUDSON/Fingerprint

AVee
-- 
But such co-operation is of course based on the theory that, when you 
tie two stones together, the combination will float.
  -- Edsger Wybe Dijkstra


___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Ian Darwin]

Hugo Mills wrote:

On Sat, May 03, 2008 at 08:21:27PM -0400, Lally Singh wrote:

Just out of curiosity, would maven be completely out of the question?


Yes.


   Please, for the love of all that's holy, no.

   I work with maven in my job. It's the most horrible misbegotten
misdesigned piece of hideousness I've ever had the misfortune to work
with. I'm not alone in that opinion, either -- pretty much everyone I
know who's used it has the same opinion.


While I might not have worded it quite that way, I have considered 
writing a paper with the title "Maven Considered Harmful". But it would 
be too short to publish as a paper... The main problem is, as Hugo 
mentioned, dependencies can change without notice and break things on 
you.  You want repeatable builds? Write a build system that saves the 
complete name and MD5 of every file, and checks every file that it 
downloads before using it, every time. The OpenBSD (Unix-like system) 
"ports" mechanism does this for all third-party software, and it 
therefore has repeatable builds. Maven does not.


___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Lally Singh]

Huh, we've had the exact opposite experience over here.

The trick that's worked here is to recognize the top-level app is
little more than a shell for the plugins.  I download the source of
the plugins when there's a problem, and ping the maven listserv.  The
documentation needs help, a *lot* of help, but the code is workable.
Frankly, that level of work isn't any harder than debugging makefiles.
 In exchange, the benefits have been pretty solid.

But, yeah, it's easy for people to have either experience.  I just
wanted to provide my own datapoint.

In comparison to complex makefiles, I'm a lot happier on maven.

On Sat, May 3, 2008 at 9:03 PM, Hugo Mills <[EMAIL PROTECTED]> wrote:
> On Sat, May 03, 2008 at 08:21:27PM -0400, Lally Singh wrote:
>  > Just out of curiosity, would maven be completely out of the question?
>
>Please, for the love of all that's holy, no.
>
>I work with maven in my job. It's the most horrible misbegotten
>  misdesigned piece of hideousness I've ever had the misfortune to work
>  with. I'm not alone in that opinion, either -- pretty much everyone I
>  know who's used it has the same opinion.
>
>I recently spent some time tracking down a build failure in our
>  automated build system. It was eventually traceable to some component
>  of maven -- still ostensibly the same version as the previous week --
>  changing under our feet in an incompatible way, so it decided to
>  reject a configuration that had previously functioned correctly. This
>  is not the only problem we've had with it. It has a track record of
>  being awkward and/or broken in ways that are not obvious, and it is
>  almost completely opaque when you come to try and debug them.
>
>Please, please, stick to make.
>
>Hugo.
>
>  --
>  === Hugo Mills: [EMAIL PROTECTED] carfax.org.uk | darksatanic.net | 
> lug.org.uk ===
>   PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
>  --- My karma has run over my dogma. ---
>
> -BEGIN PGP SIGNATURE-
>  Version: GnuPG v1.4.6 (GNU/Linux)
>
>  iD8DBQFIHQtsIKyzvlFcI40RArezAKDTilX8eV6tsxwc8zY4kiBR+/4kEQCfXxNS
>  /0EbQ+2wi9M/i/z6JiSYwFs=
>  =CiYc
>  -END PGP SIGNATURE-
>
> ___
>  Openmoko community mailing list
>  community@lists.openmoko.org
>  http://lists.openmoko.org/mailman/listinfo/community
>
>



-- 
H. Lally Singh
Ph.D. Candidate, Computer Science
Virginia Tech

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Hugo Mills]

On Sat, May 03, 2008 at 08:21:27PM -0400, Lally Singh wrote:
> Just out of curiosity, would maven be completely out of the question?

   Please, for the love of all that's holy, no.

   I work with maven in my job. It's the most horrible misbegotten
misdesigned piece of hideousness I've ever had the misfortune to work
with. I'm not alone in that opinion, either -- pretty much everyone I
know who's used it has the same opinion.

   I recently spent some time tracking down a build failure in our
automated build system. It was eventually traceable to some component
of maven -- still ostensibly the same version as the previous week --
changing under our feet in an incompatible way, so it decided to
reject a configuration that had previously functioned correctly. This
is not the only problem we've had with it. It has a track record of
being awkward and/or broken in ways that are not obvious, and it is
almost completely opaque when you come to try and debug them.

   Please, please, stick to make.

   Hugo.

-- 
=== Hugo Mills: [EMAIL PROTECTED] carfax.org.uk | darksatanic.net | lug.org.uk 
===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
 --- My karma has run over my dogma. --- 


signature.asc
Description: Digital signature
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Lally Singh]

Just out of curiosity, would maven be completely out of the question?
Package up the dependent tarballs into maven projects, and list the OM
stack as depending on specific versions of them.  Maven can wrap up
the complexity of finding all the various packages for people who just
want to download & install.  If the process of converting a tarball
into a maven project can be automated (even if a few scripts are
necessary for different projects), then the real work need only be
done once per part.

It's also work that can get split up a bit amongst the community.

Would make continuous integration simple, and would act as a nice
wrapper around the current build system.  Hudson
 has tools for starting/stopping VMs,
etc as part of the test suite.

Also, being selfish for a moment, it'd be nice to have a build process
that'd port easily enough to Solaris & OS X :-)  It'd also enable some
IDEs to read the project for an easier dev cycle.

If anyone's interested in this line of consideration, I'd be willing
to do a little research into what's needed to set it up.

On Fri, May 2, 2008 at 6:12 PM, Holger Freyther <[EMAIL PROTECTED]> wrote:
> On Thursday 01 May 2008 18:01:38 Bobby Martin wrote:
>  > I sent this before to openmoko-devel and was greeted with a deafening
>  > silence, so I'm resending to a broader audience.  As evidenced by a recent
>  > post to openmoko-devel by saurabh gupta, one of the GSoC selectees, this
>  > really is a problem.  The longer it's put off, the more potential benefits
>  > (in the form of community contributors who give up rather than improve OM)
>  > are lost.
>
>  Well, the reason I didn't reply to the first mail is because it is touching
>  way too many topics to answer all of them in depth and finding a time slot to
>  do this.
>
>  I'm working on the april software update. I have my branch and it has a high
>  probablity of always building. Everything we built for the
>  openmoko-qtopia-x11-image (just for testing) is set to a fixed
>  version/srcdate. I can rebuild the whole distribution without internet
>  (because I don't have internet at home) and I have written a task for
>  base.bbclass to help me to lock down the built to versions.
>
>
>  The obvious downside is: To upgrade certain components I have to spent close
>  to a day to build, and test the result.
>
>  The good thing: I can make sure to have an always building image, which 
> allows
>  you to make phone calls, sms,...
>
>
>
>  > ---
>
> >-
>  >
>  > I really think OM is shortchanging themselves with the current build
>  > process.  As far as I can tell:
>  >
>  > * there is no way to run a build that has a high probability of working
>  > all the way through
>
>  Not true for my org.openmoko.zecke.april-update branch because everything is
>  locked down (I can build without internet if I have fetched all sources
>  before)
>
>
>
>  > * there is no way to identify the latest code that builds together
>
>  You know the SRCDATE, SRCREVS, tarballs you are going to use. The information
>  for a whole system is massive though.
>
>
>
>  > * there is no automated build process, nor even a stable script one can
>  > follow to do a complete build from source on one's local machine
>
>  Right, I have attached my Openembedded scripts and configuration I use to
>  build on my hardy system. You see it is really basic/simple.
>
>
>
>  > * it looks to me as if the code lives in at least three different kinds
>  > of repositories: svn, GIT, and mtn
>
>  Yes, we integrate a lot of different projects. They use tarballs on http/ftp,
>  cvs, hg, p4, mtn, git, svn. There is hardly anything we can change about
>  that.
>
>
>  > * I'm sure that there is a way to identify the code that went into a
>  > particular snapshot found on the web, but I don't know what it is
>
>  The ipk files should point to the recipe used to build the package and has 
> the
>  SRCDATE/SRCREV in the same. So it is easy to find out.
>
>
>  > * there is no way to identify which tasks definitely have someone
>  > reliable working on them, which are open issues & need attention they're
>  > not getting, and which have some casual developer looking at them
>
>
>  I agree, we don't have an bugmaster at Openmoko who could help on this. For
>  the april software update I mostly track the issues myself and pester the
>  people involved (oh already my 3rd task).
>
>
>  >
>  > I have done several very minor programs for my neo, which I would have much
>  > preferred to integrate into the real build process.  However, when I've
>  > tried the Mokomakefile process, it will run for hours and then die in some
>  > obscure way.  I spent a few hours over the course of several days debugging
>  > that process on my Ubuntu machine at home, before I realized there were
>  > version issues with the assembler that I didn't see any way to work around.
>
>

Re: Request for stable, automated build process

[Holger Freyther]

On Thursday 01 May 2008 18:01:38 Bobby Martin wrote:
> I sent this before to openmoko-devel and was greeted with a deafening
> silence, so I'm resending to a broader audience.  As evidenced by a recent
> post to openmoko-devel by saurabh gupta, one of the GSoC selectees, this
> really is a problem.  The longer it's put off, the more potential benefits
> (in the form of community contributors who give up rather than improve OM)
> are lost.

Well, the reason I didn't reply to the first mail is because it is touching 
way too many topics to answer all of them in depth and finding a time slot to 
do this.

I'm working on the april software update. I have my branch and it has a high 
probablity of always building. Everything we built for the 
openmoko-qtopia-x11-image (just for testing) is set to a fixed 
version/srcdate. I can rebuild the whole distribution without internet 
(because I don't have internet at home) and I have written a task for 
base.bbclass to help me to lock down the built to versions.


The obvious downside is: To upgrade certain components I have to spent close 
to a day to build, and test the result.

The good thing: I can make sure to have an always building image, which allows 
you to make phone calls, sms,...



> ---
>-
>
> I really think OM is shortchanging themselves with the current build
> process.  As far as I can tell:
>
> * there is no way to run a build that has a high probability of working
> all the way through

Not true for my org.openmoko.zecke.april-update branch because everything is 
locked down (I can build without internet if I have fetched all sources 
before)


> * there is no way to identify the latest code that builds together

You know the SRCDATE, SRCREVS, tarballs you are going to use. The information 
for a whole system is massive though.


> * there is no automated build process, nor even a stable script one can
> follow to do a complete build from source on one's local machine

Right, I have attached my Openembedded scripts and configuration I use to 
build on my hardy system. You see it is really basic/simple.


> * it looks to me as if the code lives in at least three different kinds
> of repositories: svn, GIT, and mtn

Yes, we integrate a lot of different projects. They use tarballs on http/ftp, 
cvs, hg, p4, mtn, git, svn. There is hardly anything we can change about 
that.

> * I'm sure that there is a way to identify the code that went into a
> particular snapshot found on the web, but I don't know what it is

The ipk files should point to the recipe used to build the package and has the 
SRCDATE/SRCREV in the same. So it is easy to find out.

> * there is no way to identify which tasks definitely have someone
> reliable working on them, which are open issues & need attention they're
> not getting, and which have some casual developer looking at them


I agree, we don't have an bugmaster at Openmoko who could help on this. For 
the april software update I mostly track the issues myself and pester the 
people involved (oh already my 3rd task).

>
> I have done several very minor programs for my neo, which I would have much
> preferred to integrate into the real build process.  However, when I've
> tried the Mokomakefile process, it will run for hours and then die in some
> obscure way.  I spent a few hours over the course of several days debugging
> that process on my Ubuntu machine at home, before I realized there were
> version issues with the assembler that I didn't see any way to work around.

Three things:
- Finding the balance between always stable and cutting edge, specially 
while 
in development is pretty hard. I have opted for locking everything down 
because I lack internet at home which is helpful for what you want as well.

- You can ask some one of the distro team to add your software to the 
build 
process, we are happy to do so.

- Most of us use debian/ubuntu systems so should be able to help you in 
getting it built.


> I just don't have the energy to spend a dozen hours debugging a process
> that will enable me to spend a few dozen hours doing development.  I would
> think that many other potential contributors fall in the same boat.
>
> I think that a few procedures could dramatically increase the development
> community's involvement in OM, resulting in faster bugfixes, more features
> added sooner, and greater visibility into the project status:

Yes, I agree. Ading poky's SDK management will help to easily build your 
software. If you use autotools in a proper way it should be easy to package 
them. I think our distro team is interested in helping here. The alternative 
would be to create a feed with user submitted packages but it has the 
downside of not easily recreating the package (e.g. due changing ABI, 
compilers, c libraries..., we have been through this with the Sharp Zaurus).



> There should also 

Re: Request for stable, automated build process

[Joachim Steiger]

> I haven't been on a project where we pulled a significant amount of code
> from repositories not under our control, and I agree that changes things
> slightly.  It seems to me that it's more reason, not less, for CI, though.
> (Really, I think you're already on the same track...)

atleast for QA reasons. yeah.

> Assuming that the outside repository doesn't already have a way to
> find the latest good build, just use a dated pull from the repositories
> outside your control, and record the "label" in a file.  Use the latest
> good build date for the external code as your "label" to use when
> building the code that *is* under your control.

well.. much easier than that:
we just need to know the head revision of the OE repo which we use to
build since all the revisions for regular packages should be fixed to
known good ones.
dates are a really bad idea as tag, since they break due to timezones.
only real revisions should be used for pinning.
you can also disable this pinning in OE with something called autorev,
which would for example get the svn fetcher to use head instead of a
pinned version.

afaik we only use that 'feature' for our own stuff, and not upstream
packages.

> You would probably want a separate CI process for each externally
> controlled repository, plus one for all the OM controlled code.
> 
> Also, certainly for external code you would need to build on a
> schedule, rather than restarting the build when new checkins
> happen.  Really, you should probably do the same for your
> internal code, too, though.  (With the caveat that you only
> build your internal code at the scheduled time if changes have
> been checked in).

this isn't really a concern due to the pinning i described above.
means without anybody changing the revision to be used of some upstream
code in OE it will stay the same forever. this enables incremental
builds being quite fast while oe still should know when to rebuild if
somebody updates the recipe with a new(higher) PR or a new version.

but i think our OE crowd can explain these details much better than i can.


-- 

Joachim Steiger
developer relations/support

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Bobby Martin]

On Fri, May 2, 2008 at 9:20 AM, Joachim Steiger <[EMAIL PROTECTED]> wrote:

> Michael 'Mickey' Lauer wrote:
> > This is all good and well, however there's one inherently problematic
> issue to
> > consider here. Continuous integration is very good and very possible
> (and we
> > need and will to use it to improve quality), however where it really
> shines
> > is, when the complete stack is under your control.
> >
> > Unfortunately (or rahter, luckily?), 80% of the Openmoko distribution
> actually
> > is not under Openmoko's control and we do not want to go and open a
> > repository and (effectively) fork all upstream projects.
>
> indeed.
> i thought of something more generic like bitbake invoking make check and
> put patches to add tests into the upstream packages into OE for now,
> with the goal for these to move upstream.
> after all, thats where tests belong and should be maintained.
> too bad the usual FOSS has a rather bad testcaseratio
>
> also i believe true CI will be problematic since the checkin rate is
> sometimes much higher than what buildhost could build. (cron triggered)
>
> --
>
> Joachim Steiger
> developer relations/support
>

I haven't been on a project where we pulled a significant amount of code
from repositories not under our control, and I agree that changes things
slightly.  It seems to me that it's more reason, not less, for CI, though.
(Really, I think you're already on the same track...)

Assuming that the outside repository doesn't already have a way to
find the latest good build, just use a dated pull from the repositories
outside your control, and record the "label" in a file.  Use the latest
good build date for the external code as your "label" to use when
building the code that *is* under your control.

You would probably want a separate CI process for each externally
controlled repository, plus one for all the OM controlled code.

Also, certainly for external code you would need to build on a
schedule, rather than restarting the build when new checkins
happen.  Really, you should probably do the same for your
internal code, too, though.  (With the caveat that you only
build your internal code at the scheduled time if changes have
been checked in).

-- 
If it doesn't make you smile, you're doing something wrong.
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Joachim Steiger]

Michael 'Mickey' Lauer wrote:
> This is all good and well, however there's one inherently problematic issue 
> to 
> consider here. Continuous integration is very good and very possible (and we 
> need and will to use it to improve quality), however where it really shines 
> is, when the complete stack is under your control.
> 
> Unfortunately (or rahter, luckily?), 80% of the Openmoko distribution 
> actually 
> is not under Openmoko's control and we do not want to go and open a 
> repository and (effectively) fork all upstream projects.

indeed.
i thought of something more generic like bitbake invoking make check and
put patches to add tests into the upstream packages into OE for now,
with the goal for these to move upstream.
after all, thats where tests belong and should be maintained.
too bad the usual FOSS has a rather bad testcaseratio

also i believe true CI will be problematic since the checkin rate is
sometimes much higher than what buildhost could build. (cron triggered)

-- 

Joachim Steiger
developer relations/support

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Joachim Steiger]

Bobby Martin wrote:
> What I mean by a label (sometimes called a tag) is that your build
> process uses
> your revision control systems' commands to apply a label to all files
> before the
> build process starts.  In svn, you use the svn copy command to tag
> files, like so:
> 
> svn copy file:///tmp/repos/test/trunk file:///tmp/repos/test/tags/building -m 
> "tag tree"
> 
> (see http://svnbook.red-bean.com/en/1.1/re07.html)
> 
> Unfortunately, I think OM uses three different repositories (svn, git,
> mtn) so you would
> issue at least three different commands to apply the label.  Then when
> you're done with
> the build, you would relabel it with a build-successful tag, e.g.:
> 
> svn copy file:///tmp/repos/test/tags/building 
> file:///tmp/repos/test/tags/build-success-2008-05-02-13-47 -m "tag tree"
> 
> (If the build happened at 13:47 on 2008/05/02)
> 
> Then you run your tests and relabel with test-success-2008-05-02-13-47
> if those pass.  When
> the build is done, you delete the building tag so you can re-use it for
> the next build.
> 
> You probably also want to tag those same files with build-success-latest
> and test-success-latest.
> That way people can pull that label from the repository and know what to
> expect.  You can also
> look at the labels and see when good builds happened and when things broke.

sounds nice, but i think due to the reasons you named it makes only very
limited sense to tag it in any repo. (besides that buildhost shouldnt
have write access. its executing scripts which 3rd partys released as
software packages so its not on my 'trusted' list)
after all we are building a complete distribution here, not only one
software project which would be clearly contained and propably only in
one repo.
also we use loads of upstream stuff from different svn, cvs, git repos
as well as tarballs via http or ftp.
in addition to that these are cached and again exported by the buildhost.

> A good CI system will also email some set list of people when the build
> is bad with the files that
> changed since the last good build (this set ideally including the list
> of people who checked
> those files in) so the build tends to get fixed quickly.

yeah. thats what i would like to have.

> Being able to retrieve the build products from good builds is obviously
> very important,
> but it's not necessarily very helpful for people doing development
> work.  For them,
> being able to get the very latest code from the project they're working
> on, and the
> latest "good" code from all projects it depends on, is typically what
> they want.

for that we manually cherry pick known to be usable builds after kevin
dean tested these.
they are at http://downloads.openmoko.org/recommended/

> I have used trac a bit, but probably not enough to be helpful without
> studying up on it.
> I wasn't aware it provided much Continuous Integration support.  I will
> do some research
> on it.

nice. its a plugin into trac, not a base feature, but thats how it
works. very simple skeleton with basic features, the rest is a plugin

> Thanks!
> Bobby

btw: we still want to hire a web_developer_ (needs to know python i say);)


-- 

Joachim Steiger
developer relations/support

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Michael 'Mickey' Lauer]

This is all good and well, however there's one inherently problematic issue to 
consider here. Continuous integration is very good and very possible (and we 
need and will to use it to improve quality), however where it really shines 
is, when the complete stack is under your control.

Unfortunately (or rahter, luckily?), 80% of the Openmoko distribution actually 
is not under Openmoko's control and we do not want to go and open a 
repository and (effectively) fork all upstream projects.

:M:

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Bobby Martin]

On Fri, May 2, 2008 at 12:02 AM, Joachim Steiger <[EMAIL PROTECTED]> wrote:

> Bobby Martin wrote:
> >
> >
> > On Thu, May 1, 2008 at 11:50 AM, Thomas Wood <[EMAIL PROTECTED]
> > > wrote:
> >
> > Well, there is buildhost.openmoko.org
> >  which builds images nightly. I've
> > been asking for the toolchain to be fixed and updated and I believe
> > Julian has this on his TODO list.
> >
> > Regards,
> >
> > Thomas
> >
> >
> > Thanks for pointing that out.
> >
> > Does it apply a label to builds that succeed?  Is there documentation of
> > what this hypothetical label might be, and also documentation of the
> > build process that buildhost.openmoko.org
> >  uses?
> >
> > (Hrm, reading over that, it looks like I'm trying to be sarcastic with
> > the thanks.  I'm definitely not!  The thanks, and the questions, are
> > real, not rhetorical.)
>
> i do not really get what you mean by labels.
> when buildhost successfully finished a build there is a new image.
> else, there is not.
> if it cannot compile all dependencies then it logically cannot package it.
>
> i think i know what you really want in the end.
> a full CI representation of what buildhost does build and what not.
>
> i have found http://bitten.edgewall.org/ recently and since we are
> currently in the process of moving to trac it is very interesting for me.
>
> whats missing is the glue between that and the openembedded build system.
> and to make it really usefull at all we would need unit tests for all
> 'packages', not only a 'built something' 'did report exit=!0'
>
> yes i want that. but let us finish work on getting trac first please ;)
>
> if you know your way around trac, python and OE i am happy about
> everyone who can help me there.


What I mean by a label (sometimes called a tag) is that your build process
uses
your revision control systems' commands to apply a label to all files before
the
build process starts.  In svn, you use the svn copy command to tag files,
like so:

svn copy file:///tmp/repos/test/trunk
file:///tmp/repos/test/tags/building -m "tag tree"

(see http://svnbook.red-bean.com/en/1.1/re07.html)

Unfortunately, I think OM uses three different repositories (svn, git, mtn)
so you would
issue at least three different commands to apply the label.  Then when
you're done with
the build, you would relabel it with a build-successful tag, e.g.:

svn copy file:///tmp/repos/test/tags/building
file:///tmp/repos/test/tags/build-success-2008-05-02-13-47 -m "tag
tree"

(If the build happened at 13:47 on 2008/05/02)

Then you run your tests and relabel with test-success-2008-05-02-13-47 if
those pass.  When
the build is done, you delete the building tag so you can re-use it for the
next build.

You probably also want to tag those same files with build-success-latest and
test-success-latest.
That way people can pull that label from the repository and know what to
expect.  You can also
look at the labels and see when good builds happened and when things broke.

A good CI system will also email some set list of people when the build is
bad with the files that
changed since the last good build (this set ideally including the list of
people who checked
those files in) so the build tends to get fixed quickly.

Being able to retrieve the build products from good builds is obviously very
important,
but it's not necessarily very helpful for people doing development work.
For them,
being able to get the very latest code from the project they're working on,
and the
latest "good" code from all projects it depends on, is typically what they
want.

I have used trac a bit, but probably not enough to be helpful without
studying up on it.
I wasn't aware it provided much Continuous Integration support.  I will do
some research
on it.

Thanks!
Bobby

-- 
If it doesn't make you smile, you're doing something wrong.
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[BrendaWang]

There are some documentation you can find on wiki.
Here is the links.
OpenEmbeded is used to make the distribution. BitBake is the useful 
command of OE, and being used to build and maintain a number of embedded 
Linux distributions. You can use this.

Now, we mainly use this.
http://wiki.openmoko.org/wiki/OpenEmbedded
http://wiki.openmoko.org/wiki/BitBake

You can use a prebuilt toolchain from the Openmoko project.
Now Julien is work on this. We are contact with OE , so maybe few days 
later, we can have new release.

http://wiki.openmoko.org/wiki/Toolchain

And from the page , about the MokoMakefile,
"MokoMakefile is developed by Rod Whitby  - it is 
not an official product of OpenMoko ", you can also get information form 
this.

http://wiki.openmoko.org/wiki/MokoMakefile#Requirements_for_building_Openmoko

These page might be helpful .
And it 's wiki. If you the information is too old , you can modify it .
And we can work out the solution together.

Brenda


Bobby Martin 提到:
On Thu, May 1, 2008 at 11:54 AM, Bobby Martin <[EMAIL PROTECTED] 
> wrote:




On Thu, May 1, 2008 at 11:50 AM, Thomas Wood
<[EMAIL PROTECTED] > wrote:

Well, there is buildhost.openmoko.org
 which builds images nightly. I've
been asking for the toolchain to be fixed and updated and I
believe
Julian has this on his TODO list.

Regards,

Thomas


Thanks for pointing that out.

Does it apply a label to builds that succeed? Is there
documentation of what this hypothetical label might be, and also
documentation of the build process that buildhost.openmoko.org
 uses?

(Hrm, reading over that, it looks like I'm trying to be sarcastic
with the thanks. I'm definitely not! The thanks, and the
questions, are real, not rhetorical.)


I should point out that if I knew the "right" way to run a build (I 
think there are three possible ways) and could actually get a build to 
go through at least once in a while, I would volunteer a server and 
the effort to set up continuous integration. My time is pretty 
restricted, so it might take me a fairly long time to get it all done, 
but I am willing to do the work if I can get answers to the basic 
questions.


Bobby

--
If it doesn't make you smile, you're doing something wrong.


___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community
  



___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Thomas Wood]

On Fri, 2008-05-02 at 07:02 +0200, Joachim Steiger wrote:

> i do not really get what you mean by labels.
> when buildhost successfully finished a build there is a new image.
> else, there is not.
> if it cannot compile all dependencies then it logically cannot package it.
> 
> i think i know what you really want in the end.
> a full CI representation of what buildhost does build and what not.

If you're using buildbot on the buildhost, it might be nice to enable
the waterfall output (http://buildbot.net/trac/wiki/ScreenShots) so that
users and developers can view the current progress/problems online.

We use buildbot in OpenedHand to build nightly images of Poky. We have
automatic notifications sent to a mailing list when a build problem
occurs, which includes the last person to check anything into svn for
the module that failed.

Regards,

Thomas

-- 
OpenedHand Ltd.

Unit R Homesdale Business Center / 216-218 Homesdale Road /
Bromley / BR1 2QZ / UK Tel: +44 (0)20 8819 6559

Expert Open Source For Consumer Devices - http://o-hand.com/



___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Joachim Steiger]

Bobby Martin wrote:
> 
> 
> On Thu, May 1, 2008 at 11:50 AM, Thomas Wood <[EMAIL PROTECTED]
> > wrote:
> 
> Well, there is buildhost.openmoko.org
>  which builds images nightly. I've
> been asking for the toolchain to be fixed and updated and I believe
> Julian has this on his TODO list.
> 
> Regards,
> 
> Thomas
> 
> 
> Thanks for pointing that out.
> 
> Does it apply a label to builds that succeed?  Is there documentation of
> what this hypothetical label might be, and also documentation of the
> build process that buildhost.openmoko.org
>  uses?
> 
> (Hrm, reading over that, it looks like I'm trying to be sarcastic with
> the thanks.  I'm definitely not!  The thanks, and the questions, are
> real, not rhetorical.)
> 
> -- 
> If it doesn't make you smile, you're doing something wrong.

i do not really get what you mean by labels.
when buildhost successfully finished a build there is a new image.
else, there is not.
if it cannot compile all dependencies then it logically cannot package it.

i think i know what you really want in the end.
a full CI representation of what buildhost does build and what not.

i have found http://bitten.edgewall.org/ recently and since we are
currently in the process of moving to trac it is very interesting for me.

whats missing is the glue between that and the openembedded build system.
and to make it really usefull at all we would need unit tests for all
'packages', not only a 'built something' 'did report exit=!0'

yes i want that. but let us finish work on getting trac first please ;)

if you know your way around trac, python and OE i am happy about
everyone who can help me there.

-- 

Joachim Steiger
Openmoko Central Services

___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Bobby Martin]

On Thu, May 1, 2008 at 11:54 AM, Bobby Martin <[EMAIL PROTECTED]>
wrote:

>
>
> On Thu, May 1, 2008 at 11:50 AM, Thomas Wood <[EMAIL PROTECTED]>
> wrote:
>
> > Well, there is buildhost.openmoko.org which builds images nightly. I've
> > been asking for the toolchain to be fixed and updated and I believe
> > Julian has this on his TODO list.
> >
> > Regards,
> >
> > Thomas
> >
>
> Thanks for pointing that out.
>
> Does it apply a label to builds that succeed?  Is there documentation of
> what this hypothetical label might be, and also documentation of the build
> process that buildhost.openmoko.org uses?
>
> (Hrm, reading over that, it looks like I'm trying to be sarcastic with the
> thanks.  I'm definitely not!  The thanks, and the questions, are real, not
> rhetorical.)
>

I should point out that if I knew the "right" way to run a build (I think
there are three possible ways) and could actually get a build to go through
at least once in a while, I would volunteer a server and the effort to set
up continuous integration.  My time is pretty restricted, so it might take
me a fairly long time to get it all done, but I am willing to do the work if
I can get answers to the basic questions.

Bobby

-- 
If it doesn't make you smile, you're doing something wrong.
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Bobby Martin]

On Thu, May 1, 2008 at 11:50 AM, Thomas Wood <[EMAIL PROTECTED]> wrote:

> Well, there is buildhost.openmoko.org which builds images nightly. I've
> been asking for the toolchain to be fixed and updated and I believe
> Julian has this on his TODO list.
>
> Regards,
>
> Thomas
>

Thanks for pointing that out.

Does it apply a label to builds that succeed?  Is there documentation of
what this hypothetical label might be, and also documentation of the build
process that buildhost.openmoko.org uses?

(Hrm, reading over that, it looks like I'm trying to be sarcastic with the
thanks.  I'm definitely not!  The thanks, and the questions, are real, not
rhetorical.)

-- 
If it doesn't make you smile, you're doing something wrong.
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Thomas Wood]

On Thu, 2008-05-01 at 11:29 -0500, Bobby Martin wrote:

> 
> I understand that your answer may well be the reason things are as
> they are, but I disagree strongly with the sentiment.
> 
> To my mind, the first step after you have an early prototype of a
> system that works is to set up a continuous integration server that
> automatically builds, runs automated tests, and labels appropriately.
> Without that, any work you do is shooting in the dark.  This is (of
> course) particularly true for a distributed project, where you can't
> just yell over the wall to someone that they broke the build.

Well, there is buildhost.openmoko.org which builds images nightly. I've
been asking for the toolchain to be fixed and updated and I believe
Julian has this on his TODO list.

Regards,

Thomas


-- 
OpenedHand Ltd.

Unit R Homesdale Business Center / 216-218 Homesdale Road /
Bromley / BR1 2QZ / UK Tel: +44 (0)20 8819 6559

Expert Open Source For Consumer Devices - http://o-hand.com/



___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Bobby Martin]

On Thu, May 1, 2008 at 11:18 AM, Thomas Wood <[EMAIL PROTECTED]> wrote:

> On Thu, 2008-05-01 at 11:01 -0500, Bobby Martin wrote:
> > I sent this before to openmoko-devel and was greeted with a deafening
> > silence, so I'm resending to a broader audience.  As evidenced by a
> > recent post to openmoko-devel by saurabh gupta, one of the GSoC
> > selectees, this really is a problem.  The longer it's put off, the
> > more potential benefits (in the form of community contributors who
> > give up rather than improve OM) are lost.
>
>
> OpenMoko (the software stack) is constantly changing and being updated.
> It is still in the very early stages of development and does not have a
> definite design specification or vision yet. I think until there are
> more concrete plans for the software strategy, you cannot expect a
> stable platform.
>
> I'm not disagreeing with you here, just pointing out why these problems
> arise.
>
> Regards,
>
> Thomas
>

I understand that your answer may well be the reason things are as they are,
but I disagree strongly with the sentiment.

To my mind, the first step after you have an early prototype of a system
that works is to set up a continuous integration server that automatically
builds, runs automated tests, and labels appropriately.  Without that, any
work you do is shooting in the dark.  This is (of course) particularly true
for a distributed project, where you can't just yell over the wall to
someone that they broke the build.

I've been several places where they did almost everything wrong in terms of
software development, but if they have a good continuous integration server
and automated test harness, and a culture of quick response to breaking the
build, it can work anyway.  I've also been on several projects where the
developers were technically brilliant and wrote great code, but it turned
into an unmaintainable mess because we couldn't forsee the consequences in
hundreds of thousands of lines of code when we make a seemingly innocuous
change to a low level subsystem.

Bobby
___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community

Re: Request for stable, automated build process

[Thomas Wood]

On Thu, 2008-05-01 at 11:01 -0500, Bobby Martin wrote:
> I sent this before to openmoko-devel and was greeted with a deafening
> silence, so I'm resending to a broader audience.  As evidenced by a
> recent post to openmoko-devel by saurabh gupta, one of the GSoC
> selectees, this really is a problem.  The longer it's put off, the
> more potential benefits (in the form of community contributors who
> give up rather than improve OM) are lost.


OpenMoko (the software stack) is constantly changing and being updated.
It is still in the very early stages of development and does not have a
definite design specification or vision yet. I think until there are
more concrete plans for the software strategy, you cannot expect a
stable platform.

I'm not disagreeing with you here, just pointing out why these problems
arise.

Regards,

Thomas

-- 
OpenedHand Ltd.

Unit R Homesdale Business Center / 216-218 Homesdale Road /
Bromley / BR1 2QZ / UK Tel: +44 (0)20 8819 6559

Expert Open Source For Consumer Devices - http://o-hand.com/



___
Openmoko community mailing list
community@lists.openmoko.org
http://lists.openmoko.org/mailman/listinfo/community