Re: Proposed pocket racing uninstallability and SRU verification around release time

[Steve Langasek]

On Thu, Oct 20, 2016 at 01:09:27PM +0100, Iain Lane wrote:
> > That's backwards.  If you want the default to be safe, and the exception to
> > be the wild west, we should configure it *just* like backports.  
> > NotAutomatic
> > on the server side, no surprises for people without the right apt 
> > preferences
> > bits in place.

> > The buildd chroots already carry a pin for backports to reverse the effect 
> > of
> > NotAutomatic, and could as easily do so for proposed, if there was consensus
> > that this behaviour change is the Way and the Light.

> FYI, there's an ancient bug for turning this on per-series, with a
> maybe-stale implementation.

>   https://bugs.launchpad.net/launchpad/+bug/1016776

> If I remember correctly I didn't push on it more because of the LoC
> policy that was in place at the time.

Ah ;)

Any reason that this needs to be per-series, anyway?

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Proposed pocket racing uninstallability and SRU verification around release time

[Steve Langasek]

On Thu, Oct 20, 2016 at 09:58:45AM +, Adam Conrad wrote:
> On Wed, Oct 19, 2016 at 06:02:35PM -0700, Steve Langasek wrote:

> > The only complication I'm aware of is that we absolutely must have different
> > apt preferences behavior for chroots - such as buildd chroots - than for end
> > user systems; but that should be solvable by using a preferences.d file
> > shipped in the right package.  Probably either software-properties-common or
> > ubuntu-release-upgrader-core?

> That's backwards.  If you want the default to be safe, and the exception
> to be the wild west, we should configure it *just* like backports. 
> NotAutomatic on the server side, no surprises for people without the right
> apt preferences bits in place.

> The buildd chroots already carry a pin for backports to reverse the effect
> of NotAutomatic, and could as easily do so for proposed, if there was
> consensus that this behaviour change is the Way and the Light.

The reason I thought of doing it this way around is that I have existing
chroots where I enable -proposed (for the obvious reasons), and it would be
convenient for me to not have to touch them to add this apt config manually;
I assume there will be other developers in the same boat.

But if there's a preference to change this on the archive side to make the
pocket NotAutomatic and then adjust the buildd chroots, I can live with
that.  I'd just like us to fix it so it's sensible behavior for end users by
default.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

de-crufting of -proposed [Was, Re: Proposed pocket racing uninstallability and SRU verification around release time]

[Steve Langasek]

On Thu, Oct 20, 2016 at 02:32:45PM +0200, Dimitri John Ledkov wrote:

> We literarly have devel-proposed littered with things, and on release
> day people do see those packages even though it has never been
> intented to release them.

If people are "seeing" them on release day, that is still a bug in the
client configuration, not in the contents of -proposed.

> For example:
> https://launchpad.net/ubuntu/+source/jitsi/2.4.4997-1.2ubuntu1/+publishinghistory

> Has been stuck in -proposed for 848 days old and it has not met
> release criteria since utopic.

> Yet, it was published in yakkety-proposed until yesterday, at which
> point it was finally deleted from yakkety-proposed as zesty-proposed
> is initialised.
> However, duing that time it was not an SRU but a broken devel package
> that should be stuck in devel.

> My argument is that we should just remove these packages that have not
> migrated for years. Others in release team say they don't hurt
> anything. Yet, they are visible in a stable-proposed when they should
> not be. Without launchpad engineering, we could be moving those out of
> $stable-proposed into a ppa between release day and next series
> initialisation.

For the record, I don't say that they don't hurt anything.  I consider cruft
hanging out in devel-proposed a problem, because it means people waste time
trying to find the signal in
.
 
I have heard more than once a recommendation that people find things to work
on on this page by scrolling to "somewhere in the middle".  But we shouldn't
have to tell them that; we should drive the cruft down to zero, so that it's
possible to work from the bottom.

However, removing all of that cruft in an appropriate way takes time on a
per-package basis.  In the example of jitsi, first of all, it's not true
that it has been "stuck in -proposed for 848 days"; unfortunately
proposed-migration sorts by upload date, and so if a package is in the
release pocket, and is then demoted to -proposed, it will sort to the
bottom.  You'll see that the publishing history shows that jitsi *was* in
the release pocket, during utopic, and then was demoted because of a libav
transition.  (Ok, that means it's actually been in -proposed for 776 days
instead of 848 days, kind of an extreme case all the same...) I've argued
that wherever sensible, we should remove packages instead of just demoting
them to -proposed.  I believe the archive team has been much more consistent
about this, at least since around March of this year, so there should be
much less cruft flowing in this way.

But the historical cruft, we still haven't gotten rid of.  And I say that we
should remove packages "whenever sensible".  There are two basic classes
where it's not obviously sensible to remove instead of demote:

 - the package has an Ubuntu delta; removing it will cause the package to be
   re-synced from Debian (now or in the future), and the synced version
   might make its way back into the devel pocket but be buggy in a way
   that's not acceptable to the Ubuntu community.

 - the package was removed as uninstallable from the release pocket because
   one of its dependencies had to be removed, but that dependency could
   theoretically become installable again in the future (e.g. we still have
   the source around, but it currently FTBFS) in which case the package
   should be allowed back into the release pocket - but won't be if we
   make it disappear now.

If each of these has to wait for an archive admin to manually review and
assess the impact of removal, we're going to be a bottleneck.  But both
classes of problem can be addressed upstream in Debian by anyone:

 - Get the Ubuntu delta upstreamed, so the package can be in sync.  If it's
   in sync and still uninstallable / unbuildable, the package can be removed
   from -proposed and we can wait for a fixed package from Debian.

 - If the package is broken in Debian and not fixable / being fixed, help
   Debian get it removed from their archive.  The removal will then
   propagate to -proposed.


In the specific instance of jitsi, the delta isn't one we care about - it's
a build fix for -Wl,--as-needed.  Maybe this is still needed, maybe it's
fixed upstream, I don't know; but either way, the correct fix is for us to
take the new version from Debian (as a merge or as a sync) and see what
happens.  If the new version is still unbuildable in Ubuntu, we can then
drop it from -proposed and wait for a fixed version to arrive from
Debian[1].

I've force-synced the new jitsi now, and it's waiting in the unapproved
queue for zesty to open.

Thanks,
-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org

[1] 

Re: Proposed pocket racing uninstallability and SRU verification around release time

[Adam Conrad]

On Thu, Oct 20, 2016 at 02:32:45PM +0200, Dimitri John Ledkov wrote:
> 
> However, since introduction of proposed-migration we have been using
> -proposed for everything in devel series; and stable releases
> -proposed for SRUs.  This does pose a problem on release day, when we
> don't have next release name announced. Meaning that on release day,
> yakkety-proposed was a mixture of 0day SRUs & things that never
> migrated and were to be moved to zesty-proposed. Maybe at final freeze
> we should move all packages in -proposed into a Silo, such that in the
> final week of a series development series-proposed only has 0day SRUs
> and does not have anything that will never migrate. And then on new
> series opening copy things back from that stash silo into
> zesty-proposed with a hope that next series it will migrate.

You're overthinking this (or undereducated on fancy launchpad features)
and I already have a tool half written and a plan in place to smooth
this over for the next release.

There's no need for moving things to silos and such.  That's gross.  We
just mass delete everything that's not a 0-day SRU, record the versions,
and copy them into next-proposed when it opens.  LP will happily copy
deleted packages back into existence.

(The only reason this needs a tool is because we also need to track down
if the source had binaries or not, to pass the right args to copyPackage
later, but that's all done here, just needs a bit of a tidy).

... Adam

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Proposed pocket racing uninstallability and SRU verification around release time

[Dimitri John Ledkov]

On 17 October 2016 at 18:57, Robie Basak  wrote:
> I just filed this bug: https://bugs.launchpad.net/ubuntu/+bug/1634201; I
> Cc'd the bug so as to try and not fragment any discussion.
>
> During development, we have packages in -proposed that fail to migrate,
> as expected, for good reason.
>
> At release time, these packages are still present. For example, Yakkety
> released with libhcrypto4-heimdal:amd64 1.7~git20160703+dfsg-1 in
> proposed, which is not in the release pocket because it is broken (see
> bug 1617963).
>
> I think we should be encouraging users to volunteer to risk testing
> proposed in stable releases. This helps with SRU verification.
>
> However, our current release process breaks these users when they
> upgrade to a new release (which, given that they are testing the cutting
> edge, they are likely to do early, before the proposed pocket has been
> cleaned out).
>

My understanding was that on upgrade -proposed was disabled, and not
re-enabled after the upgrade.

Usually, on release day the release-proposed pocket only has the 0-day SRUs.

However, since introduction of proposed-migration we have been using
-proposed for everything in devel series; and stable releases
-proposed for SRUs.  This does pose a problem on release day, when we
don't have next release name announced. Meaning that on release day,
yakkety-proposed was a mixture of 0day SRUs & things that never
migrated and were to be moved to zesty-proposed. Maybe at final freeze
we should move all packages in -proposed into a Silo, such that in the
final week of a series development series-proposed only has 0day SRUs
and does not have anything that will never migrate. And then on new
series opening copy things back from that stash silo into
zesty-proposed with a hope that next series it will migrate.

We literarly have devel-proposed littered with things, and on release
day people do see those packages even though it has never been
intented to release them.

For example:
https://launchpad.net/ubuntu/+source/jitsi/2.4.4997-1.2ubuntu1/+publishinghistory

Has been stuck in -proposed for 848 days old and it has not met
release criteria since utopic.

Yet, it was published in yakkety-proposed until yesterday, at which
point it was finally deleted from yakkety-proposed as zesty-proposed
is initialised.
However, duing that time it was not an SRU but a broken devel package
that should be stuck in devel.

My argument is that we should just remove these packages that have not
migrated for years. Others in release team say they don't hurt
anything. Yet, they are visible in a stable-proposed when they should
not be. Without launchpad engineering, we could be moving those out of
$stable-proposed into a ppa between release day and next series
initialisation.

> This means that users, instead of being encouraged, are being
> discouraged from testing the SRU proposed pocket since we are breaking
> them with known bugs but delaying removal of those breakages.
>
> Bug 1633653 is an example: a user with xenial-proposed enabled upgraded
> to Yakkety one day after release, and this broke.
>
> How can we adjust our release process to stop this happening?
>
> --
> Ubuntu-release mailing list
> Ubuntu-release@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-release



-- 
Regards,

Dimitri.

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Proposed pocket racing uninstallability and SRU verification around release time

[Iain Lane]

On Thu, Oct 20, 2016 at 09:58:45AM +, Adam Conrad wrote:
> On Wed, Oct 19, 2016 at 06:02:35PM -0700, Steve Langasek wrote:
> > 
> > The only complication I'm aware of is that we absolutely must have different
> > apt preferences behavior for chroots - such as buildd chroots - than for end
> > user systems; but that should be solvable by using a preferences.d file
> > shipped in the right package.  Probably either software-properties-common or
> > ubuntu-release-upgrader-core?
> 
> That's backwards.  If you want the default to be safe, and the exception to
> be the wild west, we should configure it *just* like backports.  NotAutomatic
> on the server side, no surprises for people without the right apt preferences
> bits in place.
> 
> The buildd chroots already carry a pin for backports to reverse the effect of
> NotAutomatic, and could as easily do so for proposed, if there was consensus
> that this behaviour change is the Way and the Light.

FYI, there's an ancient bug for turning this on per-series, with a
maybe-stale implementation.

  https://bugs.launchpad.net/launchpad/+bug/1016776

If I remember correctly I didn't push on it more because of the LoC
policy that was in place at the time.

Cheers,

-- 
Iain Lane  [ i...@orangesquash.org.uk ]
Debian Developer   [ la...@debian.org ]
Ubuntu Developer   [ la...@ubuntu.com ]


signature.asc
Description: PGP signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Proposed pocket racing uninstallability and SRU verification around release time

[Steve Langasek]

On Wed, Oct 19, 2016 at 09:19:50PM +0100, Robie Basak wrote:
> On Wed, Oct 19, 2016 at 12:41:26PM -0700, Steve Langasek wrote:
> > I understand the intent.  Testing -proposed as a whole then still leaves you
> > the problem of root-causing any regressions.  Have you had success in
> > feeding back the results of your testing into the SRU process?

> I'd like to present another use case. In the "devops" world, it is
> considered best practice to have tests for automated deployments. This
> means that these users have CI - they can see at a glance whether
> everything they need in their deployment still works.

> For users who do this, it would be trivial to add -proposed to their
> matrix. Root causing on a red would still be necessary, but they would
> need to do this anyway if they don't tell us about a regression and
> their deployments from -updates goes red.

> I don't currently know of anyone actually doing this for -proposed, so
> perhaps it isn't worth the effort to fix the other issues you point out
> that currently stop testing -proposed as a whole being useful in this
> way. Perhaps we can revisit this (for this particular use case) if users
> actually tell us they want this.

FWIW my view here is that -proposed would always be too noisy to be useful
in this way.  We don't run our own archive-level CI (i.e. autopkgtest) this
way; instead, we cherry-pick individual SRUs from -proposed and regression
test them vs. the existing "trunk" archive, which is release+updates.  I
would not expect a downstream consumer of these to want to try to stay on
top of failures introduced by packages that haven't passed our own upstream
CI gate yet.  You're certainly not going to deploy the lot from -proposed
into production (if you're sane), so why track it with CI?

If you're not committed to keeping the CI green, it's just a waste of
cycles.  And if it's not in the pipeline to be deployed, you're not actually
committed to keeping it green.  So just run the CI for the thing you really
care about the answer to.


> > AIUI Robie's concern was that -proposed is a mess immediately after a
> > stable release.  And it was *my* argument that this is a quantitative,
> > rather than qualitative, difference from the rest of the cycle.  :)

> Point taken, thanks. We should document that enabling proposed wholesale
> is not recommended, as I think the documentation currently is the
> opposite of this. It should probably be removed from the GUI option that
> can enable it then, too, since to my knowledge that doesn't enable any
> kind of apt pinning to stop a wholesale update to proposed.

> IOW, the top part of https://wiki.ubuntu.com/Testing/EnableProposed is
> broken. "Selective upgrading from -proposed" (the second section) should
> be the default, and the GUI shouldn't lead people into doing the first
> but not the second.

> Should we perhaps ship a proposed pin in preferences.d by default?

Yes, this is what I would like to see happen.  But somehow, despite this
having been discussed before, there's been a reluctance to implement it.  I
don't remember there being any problematic side effects to us making this
change; perhaps someone else on the list does.

The only complication I'm aware of is that we absolutely must have different
apt preferences behavior for chroots - such as buildd chroots - than for end
user systems; but that should be solvable by using a preferences.d file
shipped in the right package.  Probably either software-properties-common or
ubuntu-release-upgrader-core?

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Proposed pocket racing uninstallability and SRU verification around release time

[Robie Basak]

On Wed, Oct 19, 2016 at 12:41:26PM -0700, Steve Langasek wrote:
> I understand the intent.  Testing -proposed as a whole then still leaves you
> the problem of root-causing any regressions.  Have you had success in
> feeding back the results of your testing into the SRU process?

I'd like to present another use case. In the "devops" world, it is
considered best practice to have tests for automated deployments. This
means that these users have CI - they can see at a glance whether
everything they need in their deployment still works.

For users who do this, it would be trivial to add -proposed to their
matrix. Root causing on a red would still be necessary, but they would
need to do this anyway if they don't tell us about a regression and
their deployments from -updates goes red.

I don't currently know of anyone actually doing this for -proposed, so
perhaps it isn't worth the effort to fix the other issues you point out
that currently stop testing -proposed as a whole being useful in this
way. Perhaps we can revisit this (for this particular use case) if users
actually tell us they want this.

> AIUI Robie's concern was that -proposed is a mess immediately after a stable
> release.  And it was *my* argument that this is a quantitative, rather than
> qualitative, difference from the rest of the cycle. :)

Point taken, thanks. We should document that enabling proposed wholesale
is not recommended, as I think the documentation currently is the
opposite of this. It should probably be removed from the GUI option that
can enable it then, too, since to my knowledge that doesn't enable any
kind of apt pinning to stop a wholesale update to proposed.

IOW, the top part of https://wiki.ubuntu.com/Testing/EnableProposed is
broken. "Selective upgrading from -proposed" (the second section) should
be the default, and the GUI shouldn't lead people into doing the first
but not the second.

Should we perhaps ship a proposed pin in preferences.d by default?

Robie


signature.asc
Description: PGP signature
-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release

Re: Proposed pocket racing uninstallability and SRU verification around release time

[Steve Langasek]

Hi Jason,

On Wed, Oct 19, 2016 at 10:37:25AM -0600, Jason DeRose wrote:
> >I don't think end users should be installing random packages from -proposed
> >for purposes of fuzz testing of SRUs; it's just not worth the collateral
> >damage nowadays.

> While I agree that *most* users shouldn't be bothering with -proposed, there
> is still an important question: is the intent of -propsed to reflect the
> likely soon-to-be state of a stable release via SRUs?

Well, as long as you filter out packages that:

 - have not yet built on all architectures
 - are out of date on one or more architectures because they have failed to
   build
 - have one or more SRU bugs marked verification-failed
 - have undeclared / unrepresentable requirements for other packages to be
   in sync (e.g. grub2 + grub-signed)
 - have autopkgtest regressions that should block the package being promoted
   to -updates

then yes, -proposed reflects the "likely" soon-to-be state via SRUs.

And since there's no automated way to filter on the above via apt, a
dist-upgrade to -proposed is a dice roll for getting a working set of
packages, or whether reporting any issues you find this way are useful
input to the SRU process.

> I think I have a good use-case for this: System76 would like to be much
> more aggressive about testing -proposed to prevent regressions from
> reaching our customers.  In this case, it's not a matter of testing a
> specific package in -proposed, it's a matter of testing *all* packages in
> -proposed (and their interactions)...  for the purpose of testing the
> expected state of a stable release in the near future.

I understand the intent.  Testing -proposed as a whole then still leaves you
the problem of root-causing any regressions.  Have you had success in
feeding back the results of your testing into the SRU process?

I don't see any reason that you fundamentally /shouldn't/ do this testing, I
just wonder how much it's benefitting you to do so.

> I think the concern that Robie Basak originally brought up (please correct
> me if I'm wrong, Robbie) is that -proposed so often is littered with
> packaging that will most likely *never* make their way into -updates for a
> stable release. Yes, this tends to be worse during the dev cycle for a new
> stable release and immediately after its release, but it can also be quite
> bad during the lifetime of an LTS release.

AIUI Robie's concern was that -proposed is a mess immediately after a stable
release.  And it was *my* argument that this is a quantitative, rather than
qualitative, difference from the rest of the cycle. :)  You agree with the
latter, but argue that this means we should be trying to fix the problem
more broadly; my position is that this is fundamental to the function of
-proposed, and we can't significantly improve it on this axis without
significant tradeoffs elsewhere that we can't justify paying.

> For example, if you look at:

> http://people.canonical.com/~ubuntu-archive/pending-sru.html

> Xenial already has a package in -proposed that has been sitting there for
> 138 days (librarian-puppet-simple). Grated, this package isn't installed by
> default and so understandably is less critical in the grand scheme of
> things. But previous experience watching this SRU page suggests to me that
> this proposed `librarian-puppet-simple` package will *never* make it to
> xenial-updates... so why is this package still there?

Because removing failed-SRU cruft from -proposed is a low-priority task,
compared to all the other things the SRU team works on.

Note that this page does also have a section about '-proposed cleanup',
which does include packages that should be removed due to non-verification. 
It seems a bug that we don't include in this section old packages that have
*failed* verification.

But anyway, removing these packages from -proposed more quickly doesn't help
the user who has already upgraded to them for testing.  There's definitely
no committment that a bad package in -proposed will later be replaced by a
good package in -proposed that you can update to.  

> I guess in summary, I see two important use cases for -proposed:

> 1) For stakeholders to test a single specific (source) package in -proposed,
> especially for SRU verification when said stake holder either filed a
> specific bug or is effected by a specific bug

> 2) For stakeholders to test, in total across all -proposed packages, the
> soon-to-be expected state of the stable release, without awareness of the
> details of any of these -proposed packages (especially without needing to
> guess which specific ones are test-worthy)

> I think making -proposed work like -backports can totally fix use case (1),
> but I don't see how it really helps use case (2).

True; but I think currently we have a situation of serving two masters
poorly, and it would be an overall improvement to handle the first case
better out of the box (because the stakeholders in that case have a more
casual relationship 

Proposed pocket racing uninstallability and SRU verification around release time

[Robie Basak]

I just filed this bug: https://bugs.launchpad.net/ubuntu/+bug/1634201; I
Cc'd the bug so as to try and not fragment any discussion.

During development, we have packages in -proposed that fail to migrate,
as expected, for good reason.

At release time, these packages are still present. For example, Yakkety
released with libhcrypto4-heimdal:amd64 1.7~git20160703+dfsg-1 in
proposed, which is not in the release pocket because it is broken (see
bug 1617963).

I think we should be encouraging users to volunteer to risk testing
proposed in stable releases. This helps with SRU verification.

However, our current release process breaks these users when they
upgrade to a new release (which, given that they are testing the cutting
edge, they are likely to do early, before the proposed pocket has been
cleaned out).

This means that users, instead of being encouraged, are being
discouraged from testing the SRU proposed pocket since we are breaking
them with known bugs but delaying removal of those breakages.

Bug 1633653 is an example: a user with xenial-proposed enabled upgraded
to Yakkety one day after release, and this broke.

How can we adjust our release process to stop this happening?

-- 
Ubuntu-release mailing list
Ubuntu-release@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release