Re: managing transitions

[Mattia Rizzolo]

On Tue, Oct 06, 2015 at 04:03:54PM +0200, Thomas Goirand wrote:
> On 10/06/2015 01:02 PM, Mattia Rizzolo wrote:
> > On Tue, Oct 06, 2015 at 08:39:48AM +, Brian May wrote:
> >> On Tue, 6 Oct 2015 at 18:46 Thomas Goirand  wrote:
> >>
> >>> This IMO is the same topic as having a Gerrit review system (and not
> >>> just Git) which could do tests on each change of a package even before
> >>> having them committed to our git.
> >>>
> >>
> >> Sounds like an interesting thing to discuss/test after we move to git...
> > 
> > Moreover, jenkins.debian.org is happening right about now, guess it
> > would help quite something (and I'd be happy to host such tests there).
> 
> I don't think a single machine is enough for this kind of workload.
> Imagine the upload of a new setuptools and rebuilding all packages
> build-depending on it. That's a lot of packages to rebuild in
> jenkins.d.o. I have offers from some cloud providers which we could use
> instead for this kind of job. We "only" need to do the work...

jenkins.d.o won't build anything but only keep the master node (without
builders, or maybe one or two to do maintenance stuff).

If what you would like to do is to "just" rebuild all rdeps, that's
really one of the thing I'd really like to do, it just takes time do all
the work (and atm I'm busy doing other stuff, if that wasn't clear) :)

To be precise, what I plan is to really improve the current
reproducible.d.n scripts and base everything on them.
I'd like to support both rebuilds on a modified toolchain on demand and
on all r-deps of a particular packages which uploads would act as a
trigger.
It just takes a shitload of time I don't currently have, as usual.
Currently there are means to ask for rebuild of stuff using a particular
packages (through the AWS credit, managed by lucas &
I-don't-remember-who), but to me it looks like it's not really used if
not for big packages (like, before gcc uploads), and the results are
somewhat difficult to access and understand and often used by a single
person, which sucks.

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  http://mapreri.org  : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: managing transitions (was: python-networkx_1.10-1_amd64.changes ACCEPTED into experimental)

[Barry Warsaw]

On Oct 06, 2015, at 08:39 AM, Brian May wrote:

>On Tue, 6 Oct 2015 at 18:46 Thomas Goirand  wrote:
>
>> This IMO is the same topic as having a Gerrit review system (and not
>> just Git) which could do tests on each change of a package even before
>> having them committed to our git.
>>
>
>Sounds like an interesting thing to discuss/test after we move to git...

I don't intend to bikeshed this, nor do I have time to do the work, so in our
do-it-ocracy any online review system would be a fantastic addition.  I'll
just point to GitLab community edition as a nice open source option in this
space.

Cheers,
-Barry

Re: managing transitions

[Ian Cordasco]

On Tue, Oct 6, 2015 at 1:11 PM, Barry Warsaw  wrote:
> On Oct 06, 2015, at 07:05 PM, Thomas Goirand wrote:
>
>>Interesting. It's the first time I hear about it, I thought it was just
>>closed source.
>
> The instance at gitlab.com is the non-free Enterprise Edition (EE).  EE has
> features we probably don't care about ayway.  The Community Edition (CE) is
> MIT/Expat.
>
>>Does it have test capabilities as well, so that we could
>>run tests on each new patch, like Gerrit does? Can we plug zuul and
>>nodepool to it?
>
> I don't know about plugging in zuul and nodepool, but the CE does have CI
> integration, which we use in the GNU Mailman project, and seems to work
> great.

It definitely can integrate with its own CI (which I believe is also
licensed similarly to GitLab CE) as well as Jenkins. Currently the
Flake8 project is hosted and developed on GitLab and uses integration
with vanilla Jenkins.

Re: managing transitions

[Barry Warsaw]

On Oct 06, 2015, at 07:05 PM, Thomas Goirand wrote:

>Interesting. It's the first time I hear about it, I thought it was just
>closed source.

The instance at gitlab.com is the non-free Enterprise Edition (EE).  EE has
features we probably don't care about ayway.  The Community Edition (CE) is
MIT/Expat.

>Does it have test capabilities as well, so that we could
>run tests on each new patch, like Gerrit does? Can we plug zuul and
>nodepool to it?

I don't know about plugging in zuul and nodepool, but the CE does have CI
integration, which we use in the GNU Mailman project, and seems to work
great.

Cheers,
-Barry

Re: managing transitions

[Barry Warsaw]

On Oct 06, 2015, at 03:12 PM, Fred Drake wrote:

>What CI tools are you using with GitLab CE?

We don't run CE; we use the hosted EE at gitlab.com.  But anyway, we have a
custom VM on which we run the GitLab runner software in a docker image.  This
runs our test suite in all supported Python 3s against both SQLite and
PostgreSQL.  At the time we set things up they didn't have shared runners, but
now they do so I don't think you even need to provision a VM.

http://doc.gitlab.com/ci/

Cheers,
-Barry


pgp1V5NfV0r9R.pgp
Description: OpenPGP digital signature

Re: managing transitions (was: python-networkx_1.10-1_amd64.changes ACCEPTED into experimental)

[Thomas Goirand]

On 10/05/2015 11:11 PM, Barry Warsaw wrote:
> On Oct 05, 2015, at 02:51 PM, Thomas Goirand wrote:
> 
>> In other distributions (Red Hat and Ubuntu), everyone is aware of this
>> kind of issue before uploading, and this kinds of things don't happen.
> 
> Ubuntu at least does have a technical solution that helps ameliorate
> archive-wide breakages, and that is -proposed migration.  When you upload
> e.g. to wily, it gets diverted to wily-proposed and to get promoted it has to
> pass a number of tests.  The package and their reverses have to build.  DEP-8
> tests have to pass, etc.  You can get a nice report about which -proposed
> promotions are failing:
> 
> http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html

Oh, nice! We do need something like this too.

> The downside is that you should probably be proactively checking this list
> (poll vs ping) and it can sometimes be difficult to figure out why a promotion
> fails or how to fix it.

It's a super nice tool, though in some cases, I do see that we may want
to ignore it. For example, dozens of packages passing, and just a single
leaf one with some issues.

> But this does mean that the archive itself is very rarely broken, and it can
> be a convenient way to stage package updates that may have effects in parts of
> the archive you might not be aware of.

If we need the compute power to do it, I have a few proposal for that.
I'm all for having a CI / CD also for packages.

This IMO is the same topic as having a Gerrit review system (and not
just Git) which could do tests on each change of a package even before
having them committed to our git.

Thomas

Re: managing transitions (was: python-networkx_1.10-1_amd64.changes ACCEPTED into experimental)

[Brian May]

On Tue, 6 Oct 2015 at 18:46 Thomas Goirand  wrote:

> This IMO is the same topic as having a Gerrit review system (and not
> just Git) which could do tests on each change of a package even before
> having them committed to our git.
>

Sounds like an interesting thing to discuss/test after we move to git...

Re: managing transitions (was: python-networkx_1.10-1_amd64.changes ACCEPTED into experimental)

[Mattia Rizzolo]

On Tue, Oct 06, 2015 at 08:39:48AM +, Brian May wrote:
> On Tue, 6 Oct 2015 at 18:46 Thomas Goirand  wrote:
> 
> > This IMO is the same topic as having a Gerrit review system (and not
> > just Git) which could do tests on each change of a package even before
> > having them committed to our git.
> >
> 
> Sounds like an interesting thing to discuss/test after we move to git...

Moreover, jenkins.debian.org is happening right about now, guess it
would help quite something (and I'd be happy to host such tests there).

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
more about me:  http://mapreri.org  : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: managing transitions

[Thomas Goirand]

On 10/06/2015 01:02 PM, Mattia Rizzolo wrote:
> On Tue, Oct 06, 2015 at 08:39:48AM +, Brian May wrote:
>> On Tue, 6 Oct 2015 at 18:46 Thomas Goirand  wrote:
>>
>>> This IMO is the same topic as having a Gerrit review system (and not
>>> just Git) which could do tests on each change of a package even before
>>> having them committed to our git.
>>>
>>
>> Sounds like an interesting thing to discuss/test after we move to git...
> 
> Moreover, jenkins.debian.org is happening right about now, guess it
> would help quite something (and I'd be happy to host such tests there).

I don't think a single machine is enough for this kind of workload.
Imagine the upload of a new setuptools and rebuilding all packages
build-depending on it. That's a lot of packages to rebuild in
jenkins.d.o. I have offers from some cloud providers which we could use
instead for this kind of job. We "only" need to do the work...

Cheers,

Thomas

Re: managing transitions

[Thomas Goirand]

On 10/06/2015 05:42 PM, Barry Warsaw wrote:
> On Oct 06, 2015, at 08:39 AM, Brian May wrote:
> 
>> On Tue, 6 Oct 2015 at 18:46 Thomas Goirand  wrote:
>>
>>> This IMO is the same topic as having a Gerrit review system (and not
>>> just Git) which could do tests on each change of a package even before
>>> having them committed to our git.
>>>
>>
>> Sounds like an interesting thing to discuss/test after we move to git...
> 
> I don't intend to bikeshed this, nor do I have time to do the work, so in our
> do-it-ocracy any online review system would be a fantastic addition.  I'll
> just point to GitLab community edition as a nice open source option in this
> space.

Interesting. It's the first time I hear about it, I thought it was just
closed source. Does it have test capabilities as well, so that we could
run tests on each new patch, like Gerrit does? Can we plug zuul and
nodepool to it?

FYI, I have Zuul and Nodepool nearly done. I am just waiting for the
support of statsd 3.x to land upstream (as it'd be broken in Sid otherwise).

Cheers,

Thomas Goirand (zigo)