Bug#835177: [PKG-Openstack-devel] Bug#835177: Bug#835177: aodh: FTBFS with eatmydata (build hangs)

[Santiago Vila]

On Thu, Oct 13, 2016 at 05:49:00PM +0200, Thomas Goirand wrote:
> On 10/13/2016 02:03 AM, Santiago Vila wrote:
>
> > No. If A "build-depends: B" and B "build-depends: C", that would be
> > three mirror pulses at most, and we have a mirror pulse every six
> > hours. Between the first and the third mirror pulse we would only have
> > 12 hours of "breakage", but we are talking about unstable, so that's
> > normal and expected.
> 
> I also need to sleep sometime, you know... :)

Well, it would be possible to use cron and upload in chunks automatically.

But I think the suggestion from Ondrej is even better: Upload everything
in source-only form at once and let the autobuilders do their job.

> [...]
> Though for the "propagate to testing", I need to find out a way to fix
> the current state of things, because right now, it's really done in a
> *very* messy way for which I'm not satisfied at all. We currently have
> half of Newton not yet migrated to Testing. Do you know if there's
> another way than just opening an RC bug on each and every package?

That's a very good question.

One possibility is to use the urgency field for the uploads.

For example, when uploading python-fixtures, using "high" instead of
"medium" would have reduced the time the packages are broken in
testing.

I think it is also possible to modify the urgency of an already
uploaded package by asking the release managers specifically.

For example, I think it would be possible to submit a bug against
release.debian.org saying "please modify the urgency for the following
source packages to be 20 days instead of the 5 days I used for the
uploads".

But maybe the thing that would help most to avoid the breakage
that happened in testing this time would be a britney script
which honors build-dependencies and not just dependencies.

Hopefully some FTPMaster some day will fix that. If not, the code for
"dak" is in git, so we would just need a little bit of time, patience,
and motivation...

Thanks.

Bug#835177: [PKG-Openstack-devel] Bug#835177: Bug#835177: aodh: FTBFS with eatmydata (build hangs)

[Thomas Goirand]

On 10/13/2016 02:03 AM, Santiago Vila wrote:
>> Then only, B can be built, which also
>> takes so long. Then only A can be built.
>>
>> All of this could take maybe 2 days.
> 
> No. If A "build-depends: B" and B "build-depends: C", that would be
> three mirror pulses at most, and we have a mirror pulse every six
> hours. Between the first and the third mirror pulse we would only have
> 12 hours of "breakage", but we are talking about unstable, so that's
> normal and expected.

I also need to sleep sometime, you know... :)

> They are still unbuildable in testing and they will remain unbuildable
> in testing for four additional days:
> 
> bandit
> [...]
> zaqar

All of that because I forgot to upload a single build-dependency:
python-fixtures. Sorry. Hopefully, I'll succeed convincing the current
package maintainers that everything from github.com/testing-cabal needs
to be maintained as a whole, by a single packaging team. Right now, it's
messy, with mock and fixtures maintained elsewhere.

> If you really care about doing "nice" things, I can think of many
> things a lot nicer than uploading 38 unbuildable source packages for
> unstable and then letting them to propagate to testing.

I do feel bad about it, and hopefully, I will have less time pressure
moving forward. I was clearly a bit late for Newton.

Though for the "propagate to testing", I need to find out a way to fix
the current state of things, because right now, it's really done in a
*very* messy way for which I'm not satisfied at all. We currently have
half of Newton not yet migrated to Testing. Do you know if there's
another way than just opening an RC bug on each and every package?

> The right thing is still to use a clean sid chroot or doing
> source-only uploads.

The right thing, IMO, is having a sid CI comparable to what I have for
Jessie. Yes, what you wrote works, but it would take too much time.

Cheers,

Thomas Goirand (zigo)

Bug#835177: [PKG-Openstack-devel] Bug#835177: Bug#835177: aodh: FTBFS with eatmydata (build hangs)

[Santiago Vila]

For the record, here is a tarball for the 38 unbuildable source
packages in testing I detected today related to OpenStack.

It could be that the intention was to avoid FTBFS, but this has been
the result.

We might better have FTBFS happen in buildd.debian.org in unstable,
as that's really the best way to ensure that packages are buildable.

Thanks.


build-logs.tar.gz
Description: application/gzip

Bug#835177: [PKG-Openstack-devel] Bug#835177: Bug#835177: aodh: FTBFS with eatmydata (build hangs)

[Santiago Vila]

On Thu, Oct 13, 2016 at 01:02:30AM +0200, Thomas Goirand wrote:

> The issue is inter-(build-)dependencies. Let's say we have package A
> that build-depends on B, which itself build-depends on C. We then have
> to do a source-only upload of C, wait for the next dak run, wait for it
> to be built, then installed in the master repository. Then it has to
> reach the mirrors of buildd machines (hint: packages propagate at
> different speed on each dak run, depending on the mirror configuration,
> Internet connectivity, and so on).

No. Please take a look at any recent build log, you will see something
like this:

Get:7 http://incoming.debian.org/debian-buildd buildd-unstable/contrib Sources 
[920 B]
Get:8 http://incoming.debian.org/debian-buildd buildd-unstable/non-free Sources 
[32 B]
Get:9 http://incoming.debian.org/debian-buildd buildd-unstable/main arm64 
Packages [213 kB]

I don't know the exact definition, but surely this "incoming" thing is
something you seem not to be taking in account at all.

> Then only, B can be built, which also
> takes so long. Then only A can be built.
> 
> All of this could take maybe 2 days.

No. If A "build-depends: B" and B "build-depends: C", that would be
three mirror pulses at most, and we have a mirror pulse every six
hours. Between the first and the third mirror pulse we would only have
12 hours of "breakage", but we are talking about unstable, so that's
normal and expected.

> [...]
> still much nicer than living unstable broken for days/weeks

No, 12 hours are not days or weeks, it's less than a single day.

Your theory is basically that "a little bit of cheating is ok".
Sorry, but that does not sound acceptable.

I didn't want to add to this discussion after the reply from Ondrej,
but since you insist, here is some data:

My autobuilder tried today to build in testing all the source packages
below, they were uploaded for unstable five days ago but they were
really unbuildable in unstable.

They are still unbuildable in testing and they will remain unbuildable
in testing for four additional days:

bandit
barbican
designate
glance
ironic-inspector
magnum
manila
python-cinderclient
python-congressclient
python-debtcollector
python-glance-store
python-heatclient
python-ironicclient
python-keystoneauth1
python-keystoneclient
python-keystonemiddleware
python-magnumclient
python-manilaclient
python-mistralclient
python-neutronclient
python-neutron-lib
python-novaclient
python-openstacksdk
python-osc-lib
python-oslo.concurrency
python-oslo.config
python-oslo.db
python-oslo.messaging
python-oslo.middleware
python-oslo.privsep
python-oslo.rootwrap
python-oslo.service
python-oslo.utils
python-oslo.vmware
python-pycadf
python-senlinclient
python-tooz
zaqar

If you really care about doing "nice" things, I can think of many
things a lot nicer than uploading 38 unbuildable source packages for
unstable and then letting them to propagate to testing.

IMHO, we should really have higher standards of quality.

The right thing is still to use a clean sid chroot or doing
source-only uploads.

Thanks.

Bug#835177: [PKG-Openstack-devel] Bug#835177: Bug#835177: aodh: FTBFS with eatmydata (build hangs)

[Thomas Goirand]

On 10/11/2016 08:13 PM, Ondrej Novy wrote:
> Hi,
> 
> 2016-10-11 18:58 GMT+02:00 Thomas Goirand  >:
> 
> It is impossible maintain 400+ interacting packages the way you would
> with your single pet package.
> 
> 
> I don't see any problem using source-only upload for all OS packages.
> buildd will wait for missed deps automatically and it's cross-check of
> your work.
> 
> The only way I see to fix this, is deploying the same kind of CI/CD we
> have designed in the OpenStack infrastructure, but using Sid, and
> 
> 
> this is not needed. We should use source-only upload, that's all. We
> already have something like CI/CD - buildd with source-only uploads +
> repro builds.

The issue is inter-(build-)dependencies. Let's say we have package A
that build-depends on B, which itself build-depends on C. We then have
to do a source-only upload of C, wait for the next dak run, wait for it
to be built, then installed in the master repository. Then it has to
reach the mirrors of buildd machines (hint: packages propagate at
different speed on each dak run, depending on the mirror configuration,
Internet connectivity, and so on). Then only, B can be built, which also
takes so long. Then only A can be built.

All of this could take maybe 2 days.

And that's not the only issue. In the OpenStack packages, there's cyclic
build-dependencies. We'd have to (temporarily) remove some unit tests
and build-dependencies for some packages to be buildable.

While you do all of this, as soon as package C is uploaded, maybe all of
OpenStack is broken (who knows? I wouldn't be surprised, I have seen
such things in the past...).

What I tried to achieve was uploading everything in a single Dak run, in
order to minimize the time when we're switching from Mitaka to Newton.
That's the best service I can do to both our users (so they get updates
at once if they run unstable), and to everyone that is contributing bug
reports, as it avoids FTBFS, runtime problems and such, just because we
don't have the correct working together set of packages.

So yeah, really, source-only uploads are nice, but it will never replace
a real CI, and it prevents mass-uploads. Yes, I did forget a
build-dependency, though that's still much nicer than living unstable
broken for days/weeks with the workflow you're both proposing.

If you still don't agree, let me know. I'd love to hear counter
arguments, or a solution if there's one.

Cheers,

Thomas Goirand (zigo)

Bug#835177: [PKG-Openstack-devel] Bug#835177: Bug#835177: aodh: FTBFS with eatmydata (build hangs)

[Ondrej Novy]

Hi,

2016-10-11 18:58 GMT+02:00 Thomas Goirand :

> It is impossible maintain 400+ interacting packages the way you would
> with your single pet package.
>

I don't see any problem using source-only upload for all OS packages.
buildd will wait for missed deps automatically and it's cross-check of your
work.

The only way I see to fix this, is deploying the same kind of CI/CD we
> have designed in the OpenStack infrastructure, but using Sid, and
>

this is not needed. We should use source-only upload, that's all. We
already have something like CI/CD - buildd with source-only uploads + repro
builds.

-- 
Best regards
 Ondřej Nový

Email: n...@ondrej.org
PGP: 3D98 3C52 EB85 980C 46A5  6090 3573 1255 9D1E 064B