Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Jeremy Bicha]

On Wed, Oct 5, 2016 at 1:46 PM, Jeremy Bicha  wrote:
> And if parallel building is irrelevant, then why do you have the
> parallel= values set differently? (Although I agree that parallel=17
> compared to parallel=18 seems unlikely to find any bugs.)

I was wrong. gedit-plugins 3.22.0-1 builds fine on the Debian buildds
which use parallel=4 but one plugin loses its translations with -j17
and a different plugin with -j18.

We'll probably just set max-parallel=4 since -j4 seems to work fine
and the build doesn't take very long any way, unless someone proposes
a better fix.

Thanks,
Jeremy Bicha

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Jeremy Bicha]

On Fri, Aug 19, 2016 at 6:45 AM, Holger Levsen  wrote:
> - this aint a real world scenario for our use case, which is testing and
>   working on reproducible builds. IOW: this is just another area of QA
>   work, which I agree should probably be done, but it's out of scope for
>   our project and doing it would draw "human ressources".

I believe sbuild doesn't actually build parallel by default [1] so
yes, the build could have different results when built on a
developer's personal computer then when built on the buildds.

And if parallel building is irrelevant, then why do you have the
parallel= values set differently? (Although I agree that parallel=17
compared to parallel=18 seems unlikely to find any bugs.)

> - actual tests would run a *a lot* slower, thus we would see the results
>   were are interested in, at a later time, even more disconnected from
>   the actual upload
> - the overall results would become older

I don't know anything about your resources, but if you're ever all
caught up, I think it would be useful to run a test rebuild like I
suggested. Shouldn't things slow down once we start freezing for
stretch? And if you ever do find missing translations like I found by
chance, that's surely an RC bug.

I'm not suggesting you permanently disable parallel builds on one of
your builders, just once in a while if you can.

Jeremy

[1] https://wiki.debian.org/sbuild#Building_packages

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Holger Levsen]

On Fri, Aug 19, 2016 at 01:10:53PM +0100, Chris Lamb wrote:
> Only from me asking:
>  | So, just to be 100% clear, simply varying DEB_BUILD_OPTIONS="parallel=X"
>  | would not have discovered this gedit bug? 
> Where — in context — varying meant 17 vs. 18 instead of 1 vs. 18.

it wasn't clear to me that the resulting gedit package would be
reproducible. in fact, gedit with non parallel build currently still
aint reproducible on i386+armhf, maybe thats why.
 
> (Besides, we've seen enough strange packages to think that anything is
> possible…)

right. "so rare that's it's hardly worth testing and adding complexity…"


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Chris Lamb]

> > This entire email chain was prompted by such a case.
> 
> what makes you think so? this wasn't and isn't clear for me, neither
> from the mails nor from https://bugzilla.gnome.org/show_bug.cgi?id=770100 

Only from me asking:

 | So, just to be 100% clear, simply varying DEB_BUILD_OPTIONS="parallel=X"
 | would not have discovered this gedit bug? 

Where — in context — varying meant 17 vs. 18 instead of 1 vs. 18.

*shrugs*

(Besides, we've seen enough strange packages to think that anything is
possible…)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Holger Levsen]

On Fri, Aug 19, 2016 at 12:32:13PM +0100, Chris Lamb wrote:
> > yes, I cannot imagine there's something which is reproducible if build
> > in parallel but not if not.
> This entire email chain was prompted by such a case.

what makes you think so? this wasn't and isn't clear for me, neither
from the mails nor from https://bugzilla.gnome.org/show_bug.cgi?id=770100 


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Chris Lamb]

> > Right, so make all the pkg-gnome packages build reproducibly without
> > paralellism enabled (which you should do anyway) and then simply
> > build with parallel enabled.
> 
> yes, I cannot imagine there's something which is reproducible if build
> in parallel but not if not.

This entire email chain was prompted by such a case.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Holger Levsen]

Hi,

On Fri, Aug 19, 2016 at 12:58:24AM +0100, Chris Lamb wrote:
> > I understand that building with parallel disabled takes much longer
> > for many packages so I don't know if this is just a lack of resources
> It's more that we have two builds and I think (?) we would prefer to do
> different values of N > 1. So a different kind of resource problem.

this and there are more issues:

- quite a lot of packages force parallel builds in some way or another
  in their build systems, so these wouldnt be tested in this regard
- this aint a real world scenario for our use case, which is testing and
  working on reproducible builds. IOW: this is just another area of QA
  work, which I agree should probably be done, but it's out of scope for
  our project and doing it would draw "human ressources".
- actual tests would run a *a lot* slower, thus we would see the results
  were are interested in, at a later time, even more disconnected from
  the actual upload
- the overall results would become older

> It wouldn't be too much work to run such an test yourself or to use
> our infrastructure.

it's actually rather easy to build all packages in a loop… seriously, I
mean:

cd $path_to_svn
for i in $(find * -type d) ; do 
cd $i && debuild && cd ..
done

now you will have a lot of debs :) copy them away, apply whatever means
to enable or disable parallelism and run that loop again.

now do another loop, diff each pair of debs and if they are the same,
delete them. investigate the rest using diffoscope :)
 
> The problem is all about how you use the results; in the case where a
> package differed between builds you would not know whether this was
> due to the parallelism that was introduced or whether the package
> is inherently non-deterministic.

yup, but 
https://tests.reproducible-builds.org/debian/unstable/amd64/pkg_set_gnome.html
suggests most of the core gnome packages are actually reproducible
today…

> > Building everything in pkg-gnome svn individually with and without
> > parallel and then running diffoscope by hand to compare every binary
> > package is not much fun.
> 
> Right, so make all the pkg-gnome packages build reproducibly without
> paralellism enabled (which you should do anyway) and then simply
> build with parallel enabled.

yes, I cannot imagine there's something which is reproducible if build
in parallel but not if not.

So indeed, making packages build reproducible when build in parallel
will fix your original issue as well! :)


-- 
cheers,
Holger


signature.asc
Description: Digital signature
___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Chris Lamb]

> I understand that building with parallel disabled takes much longer
> for many packages so I don't know if this is just a lack of resources

It's more that we have two builds and I think (?) we would prefer to do
different values of N > 1. So a different kind of resource problem.

> I think it would be good to run at least periodically as code changes
> to ensure that parallel building hasn't broken anything
[..]
> I think it should probably be run over the entire archive since
> debhelper 10 will presumably be widely deployed.

It wouldn't be too much work to run such an test yourself or to use
our infrastructure.

The problem is all about how you use the results; in the case where a
package differed between builds you would not know whether this was
due to the parallelism that was introduced or whether the package
is inherently non-deterministic.

> Building everything in pkg-gnome svn individually with and without
> parallel and then running diffoscope by hand to compare every binary
> package is not much fun.

Right, so make all the pkg-gnome packages build reproducibly without
paralellism enabled (which you should do anyway) and then simply
build with parallel enabled.

If the SHA of any .deb is different, then you ipso facto have
identified an issue. No diffoscope inspection required.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Jeremy Bicha]

On Thu, Aug 18, 2016 at 5:47 PM, Chris Lamb  wrote:
> So, just to be 100% clear, simply varying DEB_BUILD_OPTIONS="parallel=X"
> would not have discovered this gedit bug? (Related, our list of variations
> between the builds can be viewed online[0])

I don't know, but the difference between parallel=17 and parallel=18
seems almost inconsequential compared to say, parallel=1 and
parallel=2. In other words, keep whatever high number is appropriate
for one build but completely disable parallel building (parallel=1)
for the other (or one of the other) builds.

I understand that building with parallel disabled takes much longer
for many packages so I don't know if this is just a lack of resources
issue.

> Ah, so this is mostly a one-off to justify that you would not break too
> many things before you enable it…? If so, I can think of a few hacky and
> temporary solutions.

I think it would be good to run at least periodically as code changes
to ensure that parallel building hasn't broken anything (similar to
how routine FTBFS testing is done [which now with reproducible builds
is done quite frequently]). I think it should probably be run over the
entire archive since debhelper 10 will presumably be widely deployed.

And no, it doesn't need to happen before some maintainer enables
parallel building; I think getting a bug report or a warning email or
whatever afterwards is fine.

> (Can you also clarify what you are saying would be arduous about "manually"
> diffoscoping? If the package is reproducible, then diffoscope would clearly
> return nothing…)

Building everything in pkg-gnome svn individually with and without
parallel and then running diffoscope by hand to compare every binary
package is not much fun. A script would make it easier but I don't
have a script for that. And maybe my problem is general and important
enough that a solution would be useful for all of Debian...

Thanks,
Jeremy

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Chris Lamb]

> I propose that one of the builds disable parallelism entirely

So, just to be 100% clear, simply varying DEB_BUILD_OPTIONS="parallel=X"
would not have discovered this gedit bug? (Related, our list of variations
between the builds can be viewed online[0])

> I want to go ahead and push parallel-building-enabled-by-default
> forward in pkg-gnome, but I (obviously) don't want to manually
> diffoscope the several hundred packages involved.

Ah, so this is mostly a one-off to justify that you would not break too
many things before you enable it…? If so, I can think of a few hacky and
temporary solutions.

(Can you also clarify what you are saying would be arduous about "manually"
diffoscoping? If the package is reproducible, then diffoscope would clearly
return nothing…)

 [0] https://tests.reproducible-builds.org/debian/index_variations.html


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Jeremy Bicha]

On Thu, Aug 18, 2016 at 4:18 PM, Chris Lamb  wrote:
> Very interesting bug; thanks for sharing.

I noticed that bug a while ago, so gedit in unstable/testing currently
explicitly opts out of parallel building.

> However, I may be misunderstanding your query as the tests we run as part
> of reproducible-builds.org(powered by jenkins.debian.net) against Debian
> pakages do actually vary the _level_ of parallelism right now. Perhaps you
> thinking about disabling parallelism entirely for one of the builds?

Yes, I propose that one of the builds disable parallelism entirely to
catch this sort of bug. Since parallel builds are becoming default in
more places, it would be great if we could find these bugs.

> If you feel like you want to have more control over the builds--for example,
> building your git HEAD on push--then I would first persue contributing to (or
> bribing Holger/another into…) supporting this on the jenkins.debian.net 
> server.
>
> This is because there are quite a few "boring" admin tasks that need doing
> such as keeping the chroots up-to-date, which you would just be tediously
> reimplementing, debugging and maintaining by yourself.

Setting up my own install of reproducible-builds.org sounds like a lot
more work than I was hoping for.

I want to go ahead and push parallel-building-enabled-by-default
forward in pkg-gnome, but I (obviously) don't want to manually
diffoscope the several hundred packages involved.

Thanks,
Jeremy

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds

Re: [Reproducible-builds] automated diffoscope for parallel build bugs

[Chris Lamb]

Dear Jeremy,

Thanks for getting in touch; glad you are finding diffoscope useful!

> https://bugzilla.gnome.org/770100

Very interesting bug; thanks for sharing.

> Going a step further, maybe reproducible-builds.org should do this
> check since it seems it would be relatively easy to implement there.

Indeed. The concept of reproducibility is indeed what you are after here
rather than "checking with diffoscope", if only because the former can
be done deterministically and without human intervention.

However, I may be misunderstanding your query as the tests we run as part
of reproducible-builds.org(powered by jenkins.debian.net) against Debian
pakages do actually vary the _level_ of parallelism right now. Perhaps you
thinking about disabling parallelism entirely for one of the builds?

If you feel like you want to have more control over the builds--for example,
building your git HEAD on push--then I would first persue contributing to (or
bribing Holger/another into…) supporting this on the jenkins.debian.net server.

This is because there are quite a few "boring" admin tasks that need doing
such as keeping the chroots up-to-date, which you would just be tediously
reimplementing, debugging and maintaining by yourself.

In addition, you would also need to ensure you were using our patched dpkg,
etc. which -- whilst going away Real Soon Now -- is not something we really
recommend or want others to be using.

I hope I understood your question and/or hope some of my reply helped. :)


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

___
Reproducible-builds mailing list
Reproducible-builds@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/reproducible-builds