Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Julian Andres Klode]

On Mon, Jun 03, 2024 at 08:49:04PM -0700, Steve Langasek wrote:
> On Fri, May 03, 2024 at 08:43:11AM +0200, Heinrich Schuchardt wrote:
> > On Debian I have seen apt-update downloading diff files. Why don't we use
> > those for Ubuntu
> 
> Disclaimer: I have never benchmarked this, but am going by my observations
> from 20 years ago when this was adopted by Debian.
> 
> In the common case, pdiff files decrease the *data transfer size* for
> downloading indices, but increase the *clock time* it takes to update the
> apt database.  Therefore there has never been evidence that it's a net win
> given modern Internet connections, and no one has ever agitated for this to
> be a priority in Ubuntu/Launchpad.

It's much faster these days. Debian also moved to server-side merged
pdiffs now, so it's just one file to download and apply, whereas before
multiple files were downloaded in the pipeline and then merged locally
before being applied.

And I guess in the very old days it was not merged at any point.

But in any case, this has been brought up a couple of times and
the main issue why we don't have pdiffs for Ubuntu is arguably
that we don't have like 4 dinstall runs a day, but more like 72
launchpad publisher runs, and having 18x as many deltas is quite
expensive both in storage space and calculating them.

Which is why a zsync-style synchronization would be better where
we compress the file block wise and can fetch changed blocks; however
the APT HTTP code is fairly unreliable and that's fairly complex,
and you need to use something like zstd with a custom dictionary
to make this still compress efficiently.

An alternative would be replacing the pdiff format with a tagged
one, where each hunk is associated with a timestamp, such that you
can then ship one larger patch file per day or so, and you only
apply the bits you actually need.

Essentially you just need to keep the previous run's pdiff2 around
to keep pdiffs working on upgrades, so it will be 2 diffs for the
current day, and 1 for each past day or so.

I don't believe there is much value in this for stable releases
though as has been pointed out before. This may change with LTS.

But also we could come up with a much simpler approach where we
just split the -updates/-security pockets into 6 months cycles
(i.e. add 2024H1 or 24.04.1 subdirectories, add some fancy way
to InRelease files to state you need to fetch these subdirectories
too) and then you just fetch the current cycle on update. But
first fetches would be annoying.

But still not convinced that is worth the effort.
-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Steve Langasek]

On Fri, May 03, 2024 at 08:43:11AM +0200, Heinrich Schuchardt wrote:
> On Debian I have seen apt-update downloading diff files. Why don't we use
> those for Ubuntu

Disclaimer: I have never benchmarked this, but am going by my observations
from 20 years ago when this was adopted by Debian.

In the common case, pdiff files decrease the *data transfer size* for
downloading indices, but increase the *clock time* it takes to update the
apt database.  Therefore there has never been evidence that it's a net win
given modern Internet connections, and no one has ever agitated for this to
be a priority in Ubuntu/Launchpad.

> especially for the large files like Contents?

The tradeoffs there are indeed likely to be different (larger files, smaller
incremental deltas), but as pointed out, downloading of Contents files is
not a common use case and so any implementation would certainly be
prioritized accordingly.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Robie Basak]

On Fri, May 03, 2024 at 06:23:05PM +1200, Michael Hudson-Doyle wrote:
> If we want to make apt update quicker / lighter on resources we should
> figure out if we can stop publishing some of the hashes (which entirely
> dominate the size of the compressed package lists). We currently have 4
> hashes in the lists (md5, sha1, sha256, sha512) -- I know Dimitri was
> trying to get us to the point that we could stop publishing MD5 at least
> but there are a few things out there that hardcode a dependence on it.
> Maybe oracular is a good time to turn off some hashes and see what breaks.

I did some further analysis.

Summary of results:

Adding proposed increases download size by 7% at worst. Stripping older
hashes reduces download size by around 35% to 40%. With stripping,
adding proposed would make a difference to download size of 4% at worst,
and overall we'd still have a >30% improvement.

Note however that when I say that there's an increase in download size
of 7% at worst, that's 2.4 MB. IMHO, that's negligible. The most we
could get in savings by stripping hashes is 15 MB, and that's assuming
no previous/ongoing cache.

Analysis

I suggest therefore that we don't need to worry about size from the
perspective of adding proposed. Stripping hashes will provide some
worthwhile benefit but I don't think we need to block adding proposed on
this. I've filed LP: #2067752 to track the removing of the old hashes.

Detailed results:

Noble

Download

| Without proposed | 29.2 MB   |
| With proposed| 29.9 MB   |
| Difference   | 0.7 MB / 102% |

Considering just the Packages files from the download:

|  | Not stripped |Stripped |  Difference |
| Without proposed |   16431k |  11035k | 5396k / 67% |
| With proposed|   16880k |  11199k | 5681k / 66% |
| Difference   |  449k / 103% | 164k / 101% | 5232k / 68% |

Jammy

Download

| Without proposed | 33.9 MB   |
| With proposed| 36.3 MB   |
| Difference   | 2.4 MB / 107% |

Considering just the Packages files from the download:

|  | Not stripped |Stripped |  Difference |
| Without proposed |   23661k |  15229k | 8432k / 64% |
| With proposed|   25135k |  15805k | 9330k / 63% |
| Difference   | 1474k / 106% | 576k / 104% | 7856k / 67% |

Focal

| Without proposed |   33.2 MB |
| With proposed|   34.5 MB |
| Difference   | 1.3 MB / 104% |

Considering just the Packages files from the download:

|  | Not stripped |Stripped |   Difference |
| Without proposed |   23385k |  13256k | 10129k / 57% |
| With proposed|   24214k |  13756k | 10458k / 57% |
| Difference   |  829k / 104% | 500k / 104% |  9629k / 59% |

Notes:

To compare like for like, I used `xz -9` from each corresponding series
both for the stripped estimate, and recompressed using that xz for the
not stripped estimate. In practice, Launchpad would presumably use a
newer-ish xz across all series.

Method:

Using lxd ubuntu: container images
find /var/lib/apt/lists /var/cache/apt -type f -delete
apt-get update  # note how much it says it downloaded, eg. "Fetched 33.9 MB in 
5s (6519 kB/s)"
Add proposed (`add-apt-repository -p proposed` or edit deb822 by hand on Noble 
due to LP: #2061128 and also manually on Focal)
find /var/lib/apt/lists /var/cache/apt -type f -delete
apt-get update  # note how much it says it downloaded, eg. "Fetched 33.9 MB in 
5s (6519 kB/s)"
apt-get install -y dctrl-tools
mkdir {un,}stripped
cp /var/lib/apt/lists/*Packages unstripped
cd unstripped
for i in *; do grep-dctrl -I -s MD5sum,SHA1,SHA256 . < $i > ../stripped/$i;done
xz -9 *
find -type f|xargs du -c  # record "unstripped" "with proposed" sizes
find -type f|grep -v proposed|xargs du -c  # record "unstripped" "without 
proposed" sizes
cd ../stripped
xz -9 *
find -type f|xargs du -c  # record "stripped" "with proposed" sizes
find -type f|grep -v proposed|xargs du -c  # record "stripped" "without 
proposed" sizes



signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Robie Basak]

On Fri, May 03, 2024 at 08:43:11AM +0200, Heinrich Schuchardt wrote:
> On Debian I have seen apt-update downloading diff files. Why don't we use
> those for Ubuntu especially for the large files like Contents?

This would require implementation inside Launchpad I think, which
presumably should come with an assessment that quantifies the potential
benefit first.

But I think we've established that your data isn't representative of
regular users?


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Robie Basak]

On Mon, May 27, 2024 at 03:11:18PM +0200, Marco Trevisan wrote:
> In the past [1] I was suggesting something similar, and IMHO it would be
> quite useful for having better SRU testing tooling too.
> 
> It's true that using `apt -t *-proposed` would work well for testing,
> but I was wondering if that should also come with an
> `ubuntu-sru-verifier` (or better name) tool that would handle for the
> testers installation and potential revert of SRU testing packages.

This would be great. Are you volunteering to write such a tool? :)


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Robie Basak]

On Thu, May 09, 2024 at 07:23:09AM -0400, David A. Desrosiers wrote:
> Let's also not lose sight of the fact that if proposed had been enabled by
> default with the current LTS release, the xz exposure and impact would have
> been a lot broader than it was, and also a lot harder to clean up and
> retract from.

I don't think that's true. With NotAutomatic, users would still have
required to explicitly install the package from proposed (with -t or
equivalent), and what would have caused them to do that?

> As it was, the customer I support mirrored -proposed into their internal
> aptly during the Feb 28-March 30 window when the exploited versions of xz
> packages were resident in noble-proposed, and some of their machines had it
> deployed as part of internal automation. They had to go through a manual
> exercise to delete the pocket from their mirror and specifically the
> xz-utils packages for a daily span of 30 days of mirroring and resilver all
> of their aptly package lists to redact that and remove their own potential
> for exposure.

This sounds like a counterexample to me - it sounds like a user
deliberately chose to opt in to the cutting edge and faced the
consequences. That's always going to be the case for those who opt-in.
If had chosen to already add proposed by default, that's wouldn't have
changed the impact for this particular user.

> Let's err on the side of being a bit more cautious here, so we don't leave
> ourselves open to another possible 'adventure' that could sneak through
> unnoticed, before our users/customers are impacted. -proposed explicitly
> disabled by default has a purpose and requires being manually enabled, and
> once we flip that position, we may lose the value that explicit testing of
> packages in -proposed provides.

From an exposure perspective, I don't see how requiring manual
enablement via sources.list is different from requiring manual opt-in
through apt with -t. In both cases the user has to take an explicit
opt-in step. Further, before we had NotAutomatic for proposed, it was
one step before (add-apt-repository -p proposed) and I'm proposing that
it be one step now that we have NotAutomatic (apt install -t
-proposed). Why you think this is worse?

Robie


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Julian Andres Klode]

I expect apt commands to automatically download missing sources
in 24.10 or 25.04. It stands to reason we could add Disabled
sources too, and if you specify apt install foo/oracular-proposed
it just goes tweaks the bit, runs the update on it, and then
installs the package from it.

Also useful will be --no-strict-pinning such that you can use
apt install foo/oracular-proposed --no-strict-pinning to install
foo and satisfy as many dependencies from the release pocket as
possible*. You can use that with apt in oracular now. The next
version 2.9.5 should have some more improvements.

* This is formally an approximation. The APT solver 3.0 applies a
deterministic SAT solving algorithm with backtracking, which will
first try the candidate, then the installed version, and then ordered
by priority (including negative ones, fwiw) but it does not use MaxSAT
techniques to fully optimize the result. This means that given weird
dependencies, it could pull in more than strictly needed.

On Mon, May 27, 2024 at 03:11:18PM +0200, Marco Trevisan wrote:
> In the past [1] I was suggesting something similar, and IMHO it would be
> quite useful for having better SRU testing tooling too.
> 
> It's true that using `apt -t *-proposed` would work well for testing,
> but I was wondering if that should also come with an
> `ubuntu-sru-verifier` (or better name) tool that would handle for the
> testers installation and potential revert of SRU testing packages.
> 
> Cheers
> 
> [1] 
> https://discourse.ubuntu.com/t/should-we-enable-proposed-by-default-with-lower-pin-priority/28580
> 
> On mag 2 2024, at 3:30 pm, Robie Basak  wrote:
> 
> > On Thu, Jan 21, 2021 at 02:15:59AM -0800, Julian Andres Klode wrote:
> >> I'd like to suggest that we start setting NotAutomatic: yes for the
> >> proposed pocket with hirsute+1, such that things like SRU verification
> >> will be easier, and all those people who enable proposed in sources.list
> >> for I don't know what reasons don't get their systems destroyed as much.
> > 
> > Now that we have an LTS out with NotAutomatic: yes, I wonder if it would
> > be worth looking to add the proposed pocket in apt sources by default
> > everywhere in future releases, like we do for backports[1].
> > 
> > Upside: it would make for even simpler instructions for users to test
> > something from proposed.
> > 
> > Downside: users would have yet more downloading on "apt update",
> > although perhaps we should expect the proposed lists to be small?
> > 
> > To be clear, I'm on the fence, and polling for opinions.
> > 
> > Robie
> > 
> > [1] There are so many ways of deploying Ubuntu now that perhaps it's
> > worth reviewing them for consistent behaviour.
> > -- 
> > ubuntu-devel mailing list
> > ubuntu-devel@lists.ubuntu.com
> > Modify settings or unsubscribe at: 
> > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
> > 
> 
> -- 
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Marco Trevisan]

In the past [1] I was suggesting something similar, and IMHO it would be
quite useful for having better SRU testing tooling too.

It's true that using `apt -t *-proposed` would work well for testing,
but I was wondering if that should also come with an
`ubuntu-sru-verifier` (or better name) tool that would handle for the
testers installation and potential revert of SRU testing packages.

Cheers

[1] 
https://discourse.ubuntu.com/t/should-we-enable-proposed-by-default-with-lower-pin-priority/28580

On mag 2 2024, at 3:30 pm, Robie Basak  wrote:

> On Thu, Jan 21, 2021 at 02:15:59AM -0800, Julian Andres Klode wrote:
>> I'd like to suggest that we start setting NotAutomatic: yes for the
>> proposed pocket with hirsute+1, such that things like SRU verification
>> will be easier, and all those people who enable proposed in sources.list
>> for I don't know what reasons don't get their systems destroyed as much.
> 
> Now that we have an LTS out with NotAutomatic: yes, I wonder if it would
> be worth looking to add the proposed pocket in apt sources by default
> everywhere in future releases, like we do for backports[1].
> 
> Upside: it would make for even simpler instructions for users to test
> something from proposed.
> 
> Downside: users would have yet more downloading on "apt update",
> although perhaps we should expect the proposed lists to be small?
> 
> To be clear, I'm on the fence, and polling for opinions.
> 
> Robie
> 
> [1] There are so many ways of deploying Ubuntu now that perhaps it's
> worth reviewing them for consistent behaviour.
> -- 
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
> 

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[David A. Desrosiers]

On 5/3/24 22:08, Seth Arnold wrote:

But, I also expect very few of our users would use -proposed. What
percentage do you expect? I'm guessing less than 1%.

Instead of configuring proposed by default, I suggest that we should
make this work:

$ sudo add-apt-repository proposed
Unable to handle repository shortcut 'proposed'


Let's also not lose sight of the fact that if proposed had been enabled 
by default with the current LTS release, the xz exposure and impact 
would have been a lot broader than it was, and also a lot harder to 
clean up and retract from.


As it was, the customer I support mirrored -proposed into their internal 
aptly during the Feb 28-March 30 window when the exploited versions of 
xz packages were resident in noble-proposed, and some of their machines 
had it deployed as part of internal automation. They had to go through a 
manual exercise to delete the pocket from their mirror and specifically 
the xz-utils packages for a daily span of 30 days of mirroring and 
resilver all of their aptly package lists to redact that and remove 
their own potential for exposure.


Let's err on the side of being a bit more cautious here, so we don't 
leave ourselves open to another possible 'adventure' that could sneak 
through unnoticed, before our users/customers are impacted. -proposed 
explicitly disabled by default has a purpose and requires being manually 
enabled, and once we flip that position, we may lose the value that 
explicit testing of packages in -proposed provides.


--
David A. Desrosiers
Principal Support Engineer (PSE/DSE), Canonical US

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Robie Basak]

On Sat, May 04, 2024 at 02:08:16AM +, Seth Arnold wrote:
> But, I also expect very few of our users would use -proposed. What
> percentage do you expect? I'm guessing less than 1%.

I agree. But making it easier to test proposed for that tiny percentage
of users would benefit all users[1] in terms of the quality of SRUs that
they receive. Although this assumes that this change would actually be
effective in increasing proposed testing. If this were the case, then
users who do not use -proposed would still benefit from some regression
that was prevented from hitting -updates, at the cost of the increased
"apt update" downloads.

So perhaps the proportion of users who use -proposed isn't quite the
right thing to measure.

Again, to be clear I'm still on the fence!

> Instead of configuring proposed by default, I suggest that we should
> make this work:
> 
> $ sudo add-apt-repository proposed
> Unable to handle repository shortcut 'proposed'

I'd be fine with this, but note that we have "add-apt-repository -p
proposed" (although temporarily broken for Noble due to LP: #2061128) so
we're not too far off.

> The wiki instructions for using proposed is pretty rough. If it were just
> one command I think it'd be a lot easier to encourage our users to provide
> feedback on pre-production packages.

Agreed - that's also mentioned elsewhere in this thread.

Robie

[1] Those that consume from -updates, anyway, which I think is near
enough everyone.


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Seth Arnold]

On Thu, May 02, 2024 at 03:46:18PM +0100, Robie Basak wrote:
> Jammy:
> main 253K
> universe 75K
> Bionic (as an example of a mature release with fewer SRUs in flight):
> main 131K
> universe 9.7K

"omg the size" was my first reaction when I read the proposal, but these
sizes are far more reasonable than I expected.

But, I also expect very few of our users would use -proposed. What
percentage do you expect? I'm guessing less than 1%.

Instead of configuring proposed by default, I suggest that we should
make this work:

$ sudo add-apt-repository proposed
Unable to handle repository shortcut 'proposed'

https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1829588

The wiki instructions for using proposed is pretty rough. If it were just
one command I think it'd be a lot easier to encourage our users to provide
feedback on pre-production packages.

On Fri, May 03, 2024 at 06:23:05PM +1200, Michael Hudson-Doyle wrote:
> Maybe oracular is a good time to turn off some hashes and see what breaks.

Yes, please. Some people have to justify every use of md5 and sha-1 in
their environment. This is silly but it represents a real cost to some
of our users and no value to us.

I did a bit of quick experimenting with Noble:

$ gzip -cd */binary-amd64/Packages.gz > /tmp/Packages
$ grep -v MD5sum:  < /tmp/Packages | grep -v SHA1:  > /tmp/Packages-good-hashes
$ gzip -k9 /tmp/Packages-good-hashes
$ xz -k /tmp/Packages-good-hashes
$ gzip -k9 /tmp/Packages
$ xz -k /tmp/Packages
$ ls -l /tmp/Packages*
-rw-rw-r-- 1 sarnold sarnold 82509409 May  3 18:26 /tmp/Packages
-rw-rw-r-- 1 sarnold sarnold 76129409 May  3 18:31 /tmp/Packages-good-hashes
-rw-rw-r-- 1 sarnold sarnold 17909244 May  3 18:27 /tmp/Packages-good-hashes.gz
-rw-rw-r-- 1 sarnold sarnold 13842660 May  3 18:27 /tmp/Packages-good-hashes.xz
-rw-rw-r-- 1 sarnold sarnold 21542896 May  3 18:26 /tmp/Packages.gz
-rw-rw-r-- 1 sarnold sarnold 16844820 May  3 18:26 /tmp/Packages.xz

Maybe it's not fair to include -release since that is unlikely to change,
but this is rough and easy.

And since I can't help myself:

$ zstd -k -9 /tmp/Packages -o /tmp/Packages.zstd-9
$ zstd -k -16 /tmp/Packages -o /tmp/Packages.zstd-16
$ zstd -k -9 /tmp/Packages-good-hashes -o /tmp/Packages-good-hashes.zstd-9
$ zstd -k -16 /tmp/Packages-good-hashes -o /tmp/Packages-good-hashes.zstd-16
$ ls -l /tmp/Packages*zstd*
-rw-rw-r-- 1 sarnold sarnold 14408261 May  3 18:31 
/tmp/Packages-good-hashes.zstd-16
-rw-rw-r-- 1 sarnold sarnold 16304863 May  3 18:31 
/tmp/Packages-good-hashes.zstd-9
-rw-rw-r-- 1 sarnold sarnold 17456128 May  3 18:26 /tmp/Packages.zstd-16
-rw-rw-r-- 1 sarnold sarnold 19746359 May  3 18:26 /tmp/Packages.zstd-9

On Fri, May 03, 2024 at 08:43:11AM +0200, Heinrich Schuchardt wrote:
> On Debian I have seen apt-update downloading diff files. Why don't we use
> those for Ubuntu especially for the large files like Contents?

My experience with the 'pdiff' files was drastically increased apt update
time -- because the fixed costs of downloading files was a lot higher than
the throughput on larger files. Plus, these diffs also take up additional
space on the mirrors vs only storing the latest version. Maybe modern
http/2 or http/3 means it wouldn't be as bad as it used to be, but I'm
skeptical.

Thanks


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Heinrich Schuchardt]

On 5/3/24 08:23, Michael Hudson-Doyle wrote:



On Fri, 3 May 2024 at 03:05, Heinrich Schuchardt 
> wrote:


On 02.05.24 16:46, Robie Basak wrote:
 > On Thu, May 02, 2024 at 04:05:31PM +0200, Heinrich Schuchardt wrote:
 >> Often I see apt-get update downloads exceeding 100 MiB. That is
without a
 >> single package download.
 >
 > I think it might be worth quantifying this. Right now, for amd64
 > proposed pocket Packages.xz files for the following:

This is what I see:

> Translation-en [118 kB]

Fetched 147 MB in 10s (14,3 MB/s)


Reading package lists... Done


Yes sure but that's not the common experience for at least three reasons:

  1. it's the devel series so the release pocket gets republished all 
the time
  2. you have apt-file installed and are downloading the Contents files. 
those are always big
  3. you have deb-src enabled (this makes much less difference than the 
previous 2 though)


If we want to make apt update quicker / lighter on resources we should 
figure out if we can stop publishing some of the hashes (which entirely 
dominate the size of the compressed package lists). We currently have 4 
hashes in the lists (md5, sha1, sha256, sha512) -- I know Dimitri was 
trying to get us to the point that we could stop publishing MD5 at least 
but there are a few things out there that hardcode a dependence on it. 
Maybe oracular is a good time to turn off some hashes and see what breaks.




With unattended upgrades or cron-apt the meta information is downloaded 
at least daily. Most of the downloaded data doesn't change.


On Debian I have seen apt-update downloading diff files. Why don't we 
use those for Ubuntu especially for the large files like Contents?


Best regards

Heinrich

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Michael Hudson-Doyle]

On Fri, 3 May 2024 at 03:05, Heinrich Schuchardt <
heinrich.schucha...@canonical.com> wrote:

> On 02.05.24 16:46, Robie Basak wrote:
> > On Thu, May 02, 2024 at 04:05:31PM +0200, Heinrich Schuchardt wrote:
> >> Often I see apt-get update downloads exceeding 100 MiB. That is without
> a
> >> single package download.
> >
> > I think it might be worth quantifying this. Right now, for amd64
> > proposed pocket Packages.xz files for the following:
>
> This is what I see:
>
> $ sudo apt-get update
> Get:1 http://archive.ubuntu.com/ubuntu oracular InRelease [64,6 kB]
> Hit:2 http://dl.google.com/linux/chrome/deb stable InRelease
>
>
> Hit:3 https://ppa.launchpadcontent.net/xypron/qemu/ubuntu noble
> InRelease
>
> Hit:4 https://ppa.launchpadcontent.net/xypron/rsyslog2312/ubuntu noble
> InRelease
>
> Get:5 http://security.ubuntu.com/ubuntu oracular-security InRelease
> [64,6 kB]
> Hit:6
>
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/private/ubuntu
> mantic InRelease
> Get:7 http://archive.ubuntu.com/ubuntu oracular-updates InRelease [64,6
> kB]
> Get:8 http://archive.ubuntu.com/ubuntu oracular-backports InRelease
> [64,7 kB]
> Hit:9
>
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/private/ubuntu
> noble InRelease
> Get:10 http://archive.ubuntu.com/ubuntu oracular/multiverse Sources [299
> kB]
> Hit:11
>
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/ventana/ubuntu
> noble InRelease
> Hit:12
>
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/ventana/ubuntu
> mantic InRelease
> Get:13 http://archive.ubuntu.com/ubuntu oracular/restricted Sources
> [18,7 kB]
> Get:14 http://archive.ubuntu.com/ubuntu oracular/universe Sources [19,9
> MB]
> Hit:15
> https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team/private/ubuntu
> mantic InRelease
> Get:16 http://archive.ubuntu.com/ubuntu oracular/main Sources [1.378 kB]
> Get:17 http://archive.ubuntu.com/ubuntu oracular/main amd64 Packages
> [1.401 kB]
> Get:18 http://archive.ubuntu.com/ubuntu oracular/main i386 Packages
> [1.041 kB]
> Get:19 http://archive.ubuntu.com/ubuntu oracular/main Translation-en
> [512 kB]
> Get:20 http://archive.ubuntu.com/ubuntu oracular amd64 Contents (deb)
> [51,3 MB]
> Get:21 http://archive.ubuntu.com/ubuntu oracular i386 Contents (deb)
> [40,3 MB]
> Get:22 http://archive.ubuntu.com/ubuntu oracular/restricted i386
> Packages [14,7 kB]
> Get:23 http://archive.ubuntu.com/ubuntu oracular/restricted amd64
> Packages [93,9 kB]
> Get:24 http://archive.ubuntu.com/ubuntu oracular/restricted
> Translation-en [18,7 kB]
> Get:25 http://archive.ubuntu.com/ubuntu oracular/universe amd64 Packages
> [15,5 MB]
> Get:26 http://archive.ubuntu.com/ubuntu oracular/universe i386 Packages
> [8.166 kB]
> Get:27 http://archive.ubuntu.com/ubuntu oracular/universe Translation-en
> [5.980 kB]
> Get:28 http://archive.ubuntu.com/ubuntu oracular/multiverse amd64
> Packages [269 kB]
> Get:29 http://archive.ubuntu.com/ubuntu oracular/multiverse i386
> Packages [126 kB]
> Get:30 http://archive.ubuntu.com/ubuntu oracular/multiverse
> Translation-en [118 kB]
> Fetched 147 MB in 10s (14,3 MB/s)
>
>
> Reading package lists... Done
>

Yes sure but that's not the common experience for at least three reasons:

 1. it's the devel series so the release pocket gets republished all the
time
 2. you have apt-file installed and are downloading the Contents files.
those are always big
 3. you have deb-src enabled (this makes much less difference than the
previous 2 though)

If we want to make apt update quicker / lighter on resources we should
figure out if we can stop publishing some of the hashes (which entirely
dominate the size of the compressed package lists). We currently have 4
hashes in the lists (md5, sha1, sha256, sha512) -- I know Dimitri was
trying to get us to the point that we could stop publishing MD5 at least
but there are a few things out there that hardcode a dependence on it.
Maybe oracular is a good time to turn off some hashes and see what breaks.

Cheers,
mwh
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Heinrich Schuchardt]

On 02.05.24 16:46, Robie Basak wrote:

On Thu, May 02, 2024 at 04:05:31PM +0200, Heinrich Schuchardt wrote:

Often I see apt-get update downloads exceeding 100 MiB. That is without a
single package download.


I think it might be worth quantifying this. Right now, for amd64
proposed pocket Packages.xz files for the following:


This is what I see:

$ sudo apt-get update
Get:1 http://archive.ubuntu.com/ubuntu oracular InRelease [64,6 kB]
Hit:2 http://dl.google.com/linux/chrome/deb stable InRelease 



Hit:3 https://ppa.launchpadcontent.net/xypron/qemu/ubuntu noble 
InRelease 

Hit:4 https://ppa.launchpadcontent.net/xypron/rsyslog2312/ubuntu noble 
InRelease 

Get:5 http://security.ubuntu.com/ubuntu oracular-security InRelease 
[64,6 kB]
Hit:6 
https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/private/ubuntu 
mantic InRelease
Get:7 http://archive.ubuntu.com/ubuntu oracular-updates InRelease [64,6 
kB]
Get:8 http://archive.ubuntu.com/ubuntu oracular-backports InRelease 
[64,7 kB]
Hit:9 
https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/private/ubuntu 
noble InRelease

Get:10 http://archive.ubuntu.com/ubuntu oracular/multiverse Sources [299 kB]
Hit:11 
https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/ventana/ubuntu 
noble InRelease
Hit:12 
https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team-private/ventana/ubuntu 
mantic InRelease
Get:13 http://archive.ubuntu.com/ubuntu oracular/restricted Sources 
[18,7 kB]
Get:14 http://archive.ubuntu.com/ubuntu oracular/universe Sources [19,9 
MB]
Hit:15 
https://private-ppa.launchpadcontent.net/ubuntu-risc-v-team/private/ubuntu 
mantic InRelease

Get:16 http://archive.ubuntu.com/ubuntu oracular/main Sources [1.378 kB]
Get:17 http://archive.ubuntu.com/ubuntu oracular/main amd64 Packages 
[1.401 kB]
Get:18 http://archive.ubuntu.com/ubuntu oracular/main i386 Packages 
[1.041 kB]
Get:19 http://archive.ubuntu.com/ubuntu oracular/main Translation-en 
[512 kB]
Get:20 http://archive.ubuntu.com/ubuntu oracular amd64 Contents (deb) 
[51,3 MB]
Get:21 http://archive.ubuntu.com/ubuntu oracular i386 Contents (deb) 
[40,3 MB]
Get:22 http://archive.ubuntu.com/ubuntu oracular/restricted i386 
Packages [14,7 kB]
Get:23 http://archive.ubuntu.com/ubuntu oracular/restricted amd64 
Packages [93,9 kB]
Get:24 http://archive.ubuntu.com/ubuntu oracular/restricted 
Translation-en [18,7 kB]
Get:25 http://archive.ubuntu.com/ubuntu oracular/universe amd64 Packages 
[15,5 MB]
Get:26 http://archive.ubuntu.com/ubuntu oracular/universe i386 Packages 
[8.166 kB]
Get:27 http://archive.ubuntu.com/ubuntu oracular/universe Translation-en 
[5.980 kB]
Get:28 http://archive.ubuntu.com/ubuntu oracular/multiverse amd64 
Packages [269 kB]
Get:29 http://archive.ubuntu.com/ubuntu oracular/multiverse i386 
Packages [126 kB]
Get:30 http://archive.ubuntu.com/ubuntu oracular/multiverse 
Translation-en [118 kB]
Fetched 147 MB in 10s (14,3 MB/s) 



Reading package lists... Done

Best regards

Heinrich



Jammy:

main 253K
universe 75K

Bionic (as an example of a mature release with fewer SRUs in flight):

main 131K
universe 9.7K

There are maybe some extra round trip times to consider, although apt
does do HTTP pipelining so I'm not sure.

I'm not counting sources, since presumably those with bandwidth
constraints would not have them enabled.

It's worth noting that I don't expect these to cache (unlike the release
pocket) since they change regularly.

But these results are much smaller than I was expecting!


When working with a mobile connection this is already problematic. We should
strive to get this number down. Adding proposed is going into the wrong
direction and would only help a tiny fraction of all users.


Everything is a trade-off, of course. If we were adding just one byte
then perhaps you wouldn't object. So, given the above sizes, do you
still hold the same opinion?

Robie



--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Robie Basak]

On Thu, May 02, 2024 at 04:05:31PM +0200, Heinrich Schuchardt wrote:
> Often I see apt-get update downloads exceeding 100 MiB. That is without a
> single package download.

I think it might be worth quantifying this. Right now, for amd64
proposed pocket Packages.xz files for the following:

Jammy:

main 253K
universe 75K

Bionic (as an example of a mature release with fewer SRUs in flight):

main 131K
universe 9.7K

There are maybe some extra round trip times to consider, although apt
does do HTTP pipelining so I'm not sure.

I'm not counting sources, since presumably those with bandwidth
constraints would not have them enabled.

It's worth noting that I don't expect these to cache (unlike the release
pocket) since they change regularly.

But these results are much smaller than I was expecting!

> When working with a mobile connection this is already problematic. We should
> strive to get this number down. Adding proposed is going into the wrong
> direction and would only help a tiny fraction of all users.

Everything is a trade-off, of course. If we were adding just one byte
then perhaps you wouldn't object. So, given the above sizes, do you
still hold the same opinion?

Robie


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Heinrich Schuchardt]

On 02.05.24 15:30, Robie Basak wrote:

On Thu, Jan 21, 2021 at 02:15:59AM -0800, Julian Andres Klode wrote:

I'd like to suggest that we start setting NotAutomatic: yes for the
proposed pocket with hirsute+1, such that things like SRU verification
will be easier, and all those people who enable proposed in sources.list
for I don't know what reasons don't get their systems destroyed as much.


Now that we have an LTS out with NotAutomatic: yes, I wonder if it would
be worth looking to add the proposed pocket in apt sources by default
everywhere in future releases, like we do for backports[1].

Upside: it would make for even simpler instructions for users to test
something from proposed.

Downside: users would have yet more downloading on "apt update",
although perhaps we should expect the proposed lists to be small?


Often I see apt-get update downloads exceeding 100 MiB. That is without 
a single package download.


When working with a mobile connection this is already problematic. We 
should strive to get this number down. Adding proposed is going into the 
wrong direction and would only help a tiny fraction of all users.


Best regards

Heinrich



To be clear, I'm on the fence, and polling for opinions.

Robie

[1] There are so many ways of deploying Ubuntu now that perhaps it's
worth reviewing them for consistent behaviour.





--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Make proposed available by default? [was: Setting NotAutomatic for hirsute+1-proposed]

[Robie Basak]

On Thu, Jan 21, 2021 at 02:15:59AM -0800, Julian Andres Klode wrote:
> I'd like to suggest that we start setting NotAutomatic: yes for the
> proposed pocket with hirsute+1, such that things like SRU verification
> will be easier, and all those people who enable proposed in sources.list
> for I don't know what reasons don't get their systems destroyed as much.

Now that we have an LTS out with NotAutomatic: yes, I wonder if it would
be worth looking to add the proposed pocket in apt sources by default
everywhere in future releases, like we do for backports[1].

Upside: it would make for even simpler instructions for users to test
something from proposed.

Downside: users would have yet more downloading on "apt update",
although perhaps we should expect the proposed lists to be small?

To be clear, I'm on the fence, and polling for opinions.

Robie

[1] There are so many ways of deploying Ubuntu now that perhaps it's
worth reviewing them for consistent behaviour.


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Athos Ribeiro]

On Sat, Jan 21, 2023 at 08:30:33PM +0100, Gunnar Hjalmarsson wrote:

On 2023-01-20 21:47, Steve Langasek wrote:

On Thu, Jan 19, 2023 at 01:44:12PM +0100, Gunnar Hjalmarsson wrote:

On 2021-01-21 11:20, Robie Basak wrote:

https://wiki.ubuntu.com/Testing/EnableProposed will need a
rework though. We link to that from every SRU bug.



That page was last updated on 2020-06-19, and is obsolete. Would be
great i someone could follow up the change and edit the
Testing/EnableProposed page so it makes sense again.


Have you run into problems with the instructions on that page?  I
skimmed it and aside from mentioning "Selective upgrading from
-proposed" which is redundant for newer releases, I don't immediately
see anything incorrect there.  If you can point out where it's wrong,
I'll happily edit it to correct it but I don't have time just now to
run through the full instructions there to find out what doesn't
work.


Maybe the change of behavior here is the confusing bit?

Previously, following the first part of the guide (before the "selective
upgrading" docs) would be enough to install the desired package(s) from
proposed without specifying the pocket in the apt command.

First I have to admit that I thought the change had been implemented 
already in jammy. I see now that that's not the case, at least not 
yet.


As regards the contents of the wiki page, you are right and I stand 
corrected — there is not much of directly incorrect information.


I think it's mostly about my personal taste: I have long thought that 
the page is overly complicated. I made use of Ask Ubuntu to illustrate 
what a less wordy page might look like:


https://askubuntu.com/questions/1451256

That Ask Ubuntu answer is not exactly targeted people like the ones 
who have participated in this list thread. I have rather the not so 
tech savvy user in mind, like a bug reporter who wants to help verify 
a proposed SRU fix.


I simply fear that the current wiki page contains so much information, 
so a prospective tester may be discouraged to follow through.


Perhaps, in the first section, after

"It is recommended to enable selective upgrading from -proposed as
described in the next section!"

we could metion the NotAutomatic setting and point out that just
creating the file under /etc/apt/sources.list.d/, as shown in the guide,
will no loger make the user's system to prefer newer packages from
proposed, as it would happen in the past. Instead, they should follow
the next section (selective upgrading from -proposed). If they desire
the former behavior they could change the preferences file priority as
well.



--
Gunnar Hjalmarsson
https://launchpad.net/~gunnarhj

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel


--
Athos Ribeiro

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Gunnar Hjalmarsson]

On 2023-01-20 21:47, Steve Langasek wrote:

On Thu, Jan 19, 2023 at 01:44:12PM +0100, Gunnar Hjalmarsson wrote:

On 2021-01-21 11:20, Robie Basak wrote:

https://wiki.ubuntu.com/Testing/EnableProposed will need a
rework though. We link to that from every SRU bug.



That page was last updated on 2020-06-19, and is obsolete. Would be
great i someone could follow up the change and edit the
Testing/EnableProposed page so it makes sense again.


Have you run into problems with the instructions on that page?  I
skimmed it and aside from mentioning "Selective upgrading from
-proposed" which is redundant for newer releases, I don't immediately
see anything incorrect there.  If you can point out where it's wrong,
I'll happily edit it to correct it but I don't have time just now to
run through the full instructions there to find out what doesn't
work.


First I have to admit that I thought the change had been implemented 
already in jammy. I see now that that's not the case, at least not yet.


As regards the contents of the wiki page, you are right and I stand 
corrected — there is not much of directly incorrect information.


I think it's mostly about my personal taste: I have long thought that 
the page is overly complicated. I made use of Ask Ubuntu to illustrate 
what a less wordy page might look like:


https://askubuntu.com/questions/1451256

That Ask Ubuntu answer is not exactly targeted people like the ones who 
have participated in this list thread. I have rather the not so tech 
savvy user in mind, like a bug reporter who wants to help verify a 
proposed SRU fix.


I simply fear that the current wiki page contains so much information, 
so a prospective tester may be discouraged to follow through.


--
Gunnar Hjalmarsson
https://launchpad.net/~gunnarhj

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Steve Langasek]

On Thu, Jan 19, 2023 at 01:44:12PM +0100, Gunnar Hjalmarsson wrote:
> On 2021-01-21 11:20, Robie Basak wrote:
> > https://wiki.ubuntu.com/Testing/EnableProposed will need a rework
> > though. We link to that from every SRU bug.

> That page was last updated on 2020-06-19, and is obsolete. Would be great i
> someone could follow up the change and edit the Testing/EnableProposed page
> so it makes sense again.

Have you run into problems with the instructions on that page?  I skimmed it
and aside from mentioning "Selective upgrading from -proposed" which is
redundant for newer releases, I don't immediately see anything incorrect
there.  If you can point out where it's wrong, I'll happily edit it to
correct it but I don't have time just now to run through the full
instructions there to find out what doesn't work.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Gunnar Hjalmarsson]

On 2021-01-21 11:20, Robie Basak wrote:

https://wiki.ubuntu.com/Testing/EnableProposed will need a rework
though. We link to that from every SRU bug.


That page was last updated on 2020-06-19, and is obsolete. Would be 
great i someone could follow up the change and edit the 
Testing/EnableProposed page so it makes sense again.


--
Thanks,

Gunnar Hjalmarsson
https://launchpad.net/~gunnarhj

--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Paride Legovini]

Sebastien Bacher wrote on 22/11/2022:
> Le 22/11/2022 à 20:24, Julian Andres Klode a écrit :
>> We have no idea why the test dependencies are failing to install in a 
>> normal setup,
> 
> Which case are you talking about? The dbus issue is clear, the base 
> image used by the autopkgtest infra is build from kinetic with 
> kinety-security versions of packages but the apt source are lunar ones 
> without proposed. The packages are failing to install because 
> kinetic-security has updates which are in lunar-proposed but didn't 
> migrate to lunar.

A follow-up on this. The curl package also had the same issue, as
the version in kinetic was newer than the version in lunar-release. Now
curl migrated, and we are not aware of other problematic packages (i.e.
packages in the lunar autopkgtest images which are not available in
lunar-release). So for the Lunar cycle this issue is behind us.

Starting from the MM cycle we'll generate the early autopkgtest images
using the *release* images of the previous release, instead of using
dailies. In this way we'll avoid having packages in the autopkgtest
images which are not in the release pocket of the new devel release.

Paride

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Julian Andres Klode]

I'm saying the error message here is not entirely relevant as I've said
multiple times before, because it is the result of a broken autopkgtest
pinning setup.

There might be other reasons the test dependencies are not installable that
we do not see because our fallback is broken and we don't have a useful
error message before that when we had working pinning.

If that dbus update is the only issue it can be worked around by retrying
with an addition dbus trigger so that libdbus-1-dev from proposed is used.

Sebastien Bacher  schrieb am Di., 22. Nov. 2022, 20:49:

> Le 22/11/2022 à 20:24, Julian Andres Klode a écrit :
> > We have no idea why the test dependencies are failing to install in a
> > normal setup,
>
>
> Which case are you talking about? The dbus issue is clear, the base
> image used by the autopkgtest infra is build from kinetic with
> kinety-security versions of packages but the apt source are lunar ones
> without proposed. The packages are failing to install because
> kinetic-security has updates which are in lunar-proposed but didn't
> migrate to lunar.
>
> Cheers,
> Sebastien
>
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Sebastien Bacher]

Le 22/11/2022 à 20:24, Julian Andres Klode a écrit :
We have no idea why the test dependencies are failing to install in a 
normal setup,



Which case are you talking about? The dbus issue is clear, the base 
image used by the autopkgtest infra is build from kinetic with 
kinety-security versions of packages but the apt source are lunar ones 
without proposed. The packages are failing to install because 
kinetic-security has updates which are in lunar-proposed but didn't 
migrate to lunar.


Cheers,
Sebastien


--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Julian Andres Klode]

No bumping it doesn't help. The version in the release pocket is still not
installable, and what's broken here is the fallback in autopkgtest to retry
with all-proposed, because it removes the pin (thus restoring NotAutomatic,
aka no-proposed) rather than adding a pin for proposed at priority 500.

We have no idea why the test dependencies are failing to install in a
normal setup, before the all-proposed retry, but people are of course free
to retry with the dpkg in proposed as an additional trigger, thus ensuring
it's not broken before the broken all-proposed fallback.

Dimitri John Ledkov  schrieb am Di., 22. Nov.
2022, 19:59:

> In such cases it is usually best to bump version number, and do a fresh
> upload to lunar-proposed such that it is higher than any of (kinetic,
> lunar).
>
> Might make sense to still upload no change rebuild of dbus into
> lunar-proposed.
>
> On Tue, 22 Nov 2022, 18:45 Sebastien Bacher,  wrote:
>
>> Hey there,
>>
>> Le 22/11/2022 à 18:21, Paride Legovini a écrit :
>> > The dbus package has now need force-migrated from lunar-proposed to
>> > -release (currently pending publication). This should fix the issue.
>>
>> I've stated that via chat but I still disagree that was the right thing
>> to do. Without tests result how are we confident that the update isn't
>> bringing a regression (one other update in lunar could create issue for
>> the dbus in proposed)? Why didn't we fix the autopkgtest setup to use an
>> lunar image or enable proposed to allow the tests to be correctly tried
>> instead?
>>
>> Also are we confident that they aren't other packages in the same
>> situations?
>>
>> Cheers,
>> Sebastien
>>
>>
>> --
>> ubuntu-devel mailing list
>> ubuntu-devel@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Dimitri John Ledkov]

In such cases it is usually best to bump version number, and do a fresh
upload to lunar-proposed such that it is higher than any of (kinetic,
lunar).

Might make sense to still upload no change rebuild of dbus into
lunar-proposed.

On Tue, 22 Nov 2022, 18:45 Sebastien Bacher,  wrote:

> Hey there,
>
> Le 22/11/2022 à 18:21, Paride Legovini a écrit :
> > The dbus package has now need force-migrated from lunar-proposed to
> > -release (currently pending publication). This should fix the issue.
>
> I've stated that via chat but I still disagree that was the right thing
> to do. Without tests result how are we confident that the update isn't
> bringing a regression (one other update in lunar could create issue for
> the dbus in proposed)? Why didn't we fix the autopkgtest setup to use an
> lunar image or enable proposed to allow the tests to be correctly tried
> instead?
>
> Also are we confident that they aren't other packages in the same
> situations?
>
> Cheers,
> Sebastien
>
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Sebastien Bacher]

Hey there,

Le 22/11/2022 à 18:21, Paride Legovini a écrit :

The dbus package has now need force-migrated from lunar-proposed to
-release (currently pending publication). This should fix the issue.


I've stated that via chat but I still disagree that was the right thing 
to do. Without tests result how are we confident that the update isn't 
bringing a regression (one other update in lunar could create issue for 
the dbus in proposed)? Why didn't we fix the autopkgtest setup to use an 
lunar image or enable proposed to allow the tests to be correctly tried 
instead?


Also are we confident that they aren't other packages in the same 
situations?


Cheers,
Sebastien


--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Paride Legovini]

Hi Jeremy,

Jeremy Bicha wrote on 22/11/2022:
> On Tue, Nov 8, 2022 at 6:05 AM Paride Legovini  wrote:
>> Steve Langasek wrote on 01/11/2022:
>>> I have set the flag now for lunar as it came up in discussion with
>>> Foundations.  The question of whether to change this for stable series is
>>> still open (now with some further series that are stable) but can be
>>> discussed separately.
>>
>> I very much welcome this change, but I think we're now missing an easy
>> way to build (sbuild) packages with -proposed fully enabled. schroots
>> created by mk-sbuild and sbuild-launchpad-chroot may have -proposed in
>> sources.list, but that's not going to be used in >= Lunar. On Launchpad
>> some extra pinning is done to fully enable -proposed [1].
>>
>> One way to re-enable easy builds with full -proposed could be modifying
>> sbuild-launchpad-chroot (/etc/schroot/setup.d/90apt-sources) to do the
>> same pinning that Launchpad does [1]. Once we have a way to sbuild with
>> -proposed enabled I think enabling NotAutomatic for the stable releases
>> would be a good idea: other than helping with SRU verification, the
>> change will keep the -proposed behavior uniform across releases.
>>
>> Paride
>>
>> [1]
>> https://git.launchpad.net/launchpad-buildd/commit/?id=c2ebcb6752af496166a5fffd9df3a4d6df6048ef
> 
> Gianfranco pointed out that there are dbus installability problems in
> the lunar autopkgtests. For instance:
> https://autopkgtest.ubuntu.com/packages/c/cinnamon-control-center/lunar/amd64
> 
> libdbus-1-dev : Depends: libdbus-1-3 (= 1.14.0-2ubuntu2) but
> 1.14.0-2ubuntu3 is to be installed
> 
> dbus 1.14.0-2ubuntu3 is built and I think it was published correctly
> but it hasn't migrated out of lunar-proposed yet.
> 
> Is the breakage fallout from the changes to how proposed is handled for lunar?

Not really: this is due to a version skew in the archive caused by
a dbus SRU upload done to Kinetic before Lunar was open. This caused the
dbus version in kinetic-updates to be newer than the version in
lunar-release.

The dbus package has now need force-migrated from lunar-proposed to
-release (currently pending publication). This should fix the issue.


One (unrelated) autopkgtest aspect that is affected by NotAutomatic: yes
is triggering tests with all-proposed=1. This is currently not working
as expected as we're not setting the pin yet.

Paride

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Jeremy Bicha]

On Tue, Nov 8, 2022 at 6:05 AM Paride Legovini  wrote:
> Steve Langasek wrote on 01/11/2022:
> > I have set the flag now for lunar as it came up in discussion with
> > Foundations.  The question of whether to change this for stable series is
> > still open (now with some further series that are stable) but can be
> > discussed separately.
>
> I very much welcome this change, but I think we're now missing an easy
> way to build (sbuild) packages with -proposed fully enabled. schroots
> created by mk-sbuild and sbuild-launchpad-chroot may have -proposed in
> sources.list, but that's not going to be used in >= Lunar. On Launchpad
> some extra pinning is done to fully enable -proposed [1].
>
> One way to re-enable easy builds with full -proposed could be modifying
> sbuild-launchpad-chroot (/etc/schroot/setup.d/90apt-sources) to do the
> same pinning that Launchpad does [1]. Once we have a way to sbuild with
> -proposed enabled I think enabling NotAutomatic for the stable releases
> would be a good idea: other than helping with SRU verification, the
> change will keep the -proposed behavior uniform across releases.
>
> Paride
>
> [1]
> https://git.launchpad.net/launchpad-buildd/commit/?id=c2ebcb6752af496166a5fffd9df3a4d6df6048ef

Gianfranco pointed out that there are dbus installability problems in
the lunar autopkgtests. For instance:
https://autopkgtest.ubuntu.com/packages/c/cinnamon-control-center/lunar/amd64

libdbus-1-dev : Depends: libdbus-1-3 (= 1.14.0-2ubuntu2) but
1.14.0-2ubuntu3 is to be installed

dbus 1.14.0-2ubuntu3 is built and I think it was published correctly
but it hasn't migrated out of lunar-proposed yet.

Is the breakage fallout from the changes to how proposed is handled for lunar?

Thank you,
Jeremy Bicha

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Paride Legovini]

Steve Langasek wrote on 01/11/2022:
> I have set the flag now for lunar as it came up in discussion with
> Foundations.  The question of whether to change this for stable series is
> still open (now with some further series that are stable) but can be
> discussed separately.

I very much welcome this change, but I think we're now missing an easy
way to build (sbuild) packages with -proposed fully enabled. schroots
created by mk-sbuild and sbuild-launchpad-chroot may have -proposed in
sources.list, but that's not going to be used in >= Lunar. On Launchpad
some extra pinning is done to fully enable -proposed [1].

One way to re-enable easy builds with full -proposed could be modifying
sbuild-launchpad-chroot (/etc/schroot/setup.d/90apt-sources) to do the
same pinning that Launchpad does [1]. Once we have a way to sbuild with
-proposed enabled I think enabling NotAutomatic for the stable releases
would be a good idea: other than helping with SRU verification, the
change will keep the -proposed behavior uniform across releases.

Paride

[1]
https://git.launchpad.net/launchpad-buildd/commit/?id=c2ebcb6752af496166a5fffd9df3a4d6df6048ef

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Colin Watson]

On Tue, Nov 01, 2022 at 12:22:00PM +0100, Steve Langasek wrote:
> Unfortunately this discussion foundered on lack of consensus about whether
> to make this change after the fact for stable series; which resulted in both
> jammy and kinetic going out without this being set.
> 
> I have set the flag now for lunar as it came up in discussion with
> Foundations.  The question of whether to change this for stable series is
> still open (now with some further series that are stable) but can be
> discussed separately.

Cool, thanks.

> Colin, will this flag be inherited in the future at series opening?

Yes.

-- 
Colin Watson (he/him)  [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Steve Langasek]

Unfortunately this discussion foundered on lack of consensus about whether
to make this change after the fact for stable series; which resulted in both
jammy and kinetic going out without this being set.

I have set the flag now for lunar as it came up in discussion with
Foundations.  The question of whether to change this for stable series is
still open (now with some further series that are stable) but can be
discussed separately.

Colin, will this flag be inherited in the future at series opening?

On Thu, Dec 09, 2021 at 04:27:31PM +, Colin Watson wrote:
> On Wed, Feb 03, 2021 at 11:12:57AM +, Iain Lane wrote:
> > I think the Launchpad support is still missing, although we started on 
> > this several years ago. That will need to be picked up and finished off:
> > 
> >   https://bugs.launchpad.net/launchpad/+bug/1016776
> > 
> > That bug report talks about doing it pre-release (for devel only) but I 
> > think I'm now in favour of doing it always, and the proposed 
> > implementation in there would allow that. For devel, the main reason is 
> > that I frequently come across users who have misunderstood what proposed 
> > is for and manually enabled it themsleves, resulting in various degrees 
> > of brokenness on their systems and bug reports that take developers' 
> > time to triage and eventually close. These are not (always) people who 
> > have updated from a previous release, where we could have had tools 
> > disable -proposed for them, but also people who have explicitly turned 
> > it on after installing a daily out of an attempt to help test the 
> > upcoming release.
> > 
> > On the client side, as Robie says, we will at least need to update 
> > documentation. I'm also not sure what update-manager will do if there 
> > are NotAutomatic updates present. It might need some tweaking to show 
> > them differently. This could be checked by looking at something in 
> > -backports, which is already present with these flags set.
> > 
> > And finally, there's some implication for package builds; both Launchpad 
> > buildds and other builders would need to ignore this. Launchpad does  
> > this for -backports currently, i.e. -backports builds get Build-Depends 
> > from -backports wholesale; hoepfully that means the buildd side isn't 
> > too hard because we can reuse that.
> 
> This is now ready to use from the Launchpad point of view.  There's a
> "proposed_not_automatic" flag on distro series exported over the API; if
> this is set to True, Launchpad writes "NotAutomatic: yes" and
> "ButAutomaticUpgrades: yes" to the Release file.  We've also arranged
> for *-proposed to be pinned to 500 in launchpad-buildd, so Launchpad
> builds will ignore this; I can't speak for other build environments.
> 
> Thus, from the Launchpad point of view this is ready to use, although
> somebody may want to check the behaviour of things like sbuild and
> pbuilder first.
> 
> Somebody in ~techboard would need to make the actual change, if you
> think it's appropriate.  For example, the following in "lp-shell
> production devel" would do it for all supported Ubuntu series:
> 
>   for name in ("bionic", "focal", "hirsute", "impish", "jammy"):
>   series = lp.distributions["ubuntu"].getSeries(name_or_version=name)
>   series.proposed_not_automatic = True
>   series.lp_save()
> 
> -- 
> Colin Watson (he/him)  [cjwat...@ubuntu.com]
> 
> -- 
> technical-board mailing list
> technical-bo...@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/technical-board

Thanks,
-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Steve Langasek]

On Mon, Dec 13, 2021 at 09:44:11AM -0500, Dan Streetman wrote:
> On Fri, Dec 10, 2021 at 9:10 AM Robie Basak  wrote:

> > On Thu, Dec 09, 2021 at 04:27:31PM +, Colin Watson wrote:
> > > This is now ready to use from the Launchpad point of view.  There's a
> > > "proposed_not_automatic" flag on distro series exported over the API; if
> > > this is set to True, Launchpad writes "NotAutomatic: yes" and
> > > "ButAutomaticUpgrades: yes" to the Release file.  We've also arranged
> > > for *-proposed to be pinned to 500 in launchpad-buildd, so Launchpad
> > > builds will ignore this; I can't speak for other build environments.

> > > Thus, from the Launchpad point of view this is ready to use, although
> > > somebody may want to check the behaviour of things like sbuild and
> > > pbuilder first.

> > Thank you Colin for the work!

> > If sbuild/pbuilder need adjusting, then maybe we need to do that and
> > then give developers some time to update their chroots so that we don't
> > break them (in non-obvious ways) all at once.

> > Another thought is that if there turns out to be an unintended
> > consequence for users enabling jammy-proposed (after Jammy's release),
> > then we'll have done that to them in an LTS instead of hitting an
> > interim release first.

> This is certainly a concern for me...this kind of change seems like
> it's more appropriate for an interim release.

The consequences of the current behavior are sufficiently heinous (users
running apt dist-upgrade after enabling -proposed for testing of an
unrelated SRU may break their systems, up to and including them unbootable)
that I am strongly opposed to deferring this change for after the LTS and
delaying another 2 years before it starts to benefit the vast majority of
affected users.

There may be knock-on consequences in terms of SRU workflows and
documentation that needs updated; but we should eat that cost now, not delay
this change another LTS cycle.

If it were up to me alone, I would want this enabled retroactively for all
supported releases.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developer   https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Dan Streetman]

On Fri, Dec 10, 2021 at 9:10 AM Robie Basak  wrote:
>
> On Thu, Dec 09, 2021 at 04:27:31PM +, Colin Watson wrote:
> > This is now ready to use from the Launchpad point of view.  There's a
> > "proposed_not_automatic" flag on distro series exported over the API; if
> > this is set to True, Launchpad writes "NotAutomatic: yes" and
> > "ButAutomaticUpgrades: yes" to the Release file.  We've also arranged
> > for *-proposed to be pinned to 500 in launchpad-buildd, so Launchpad
> > builds will ignore this; I can't speak for other build environments.
> >
> > Thus, from the Launchpad point of view this is ready to use, although
> > somebody may want to check the behaviour of things like sbuild and
> > pbuilder first.
>
> Thank you Colin for the work!
>
> If sbuild/pbuilder need adjusting, then maybe we need to do that and
> then give developers some time to update their chroots so that we don't
> break them (in non-obvious ways) all at once.
>
> Another thought is that if there turns out to be an unintended
> consequence for users enabling jammy-proposed (after Jammy's release),
> then we'll have done that to them in an LTS instead of hitting an
> interim release first.

This is certainly a concern for me...this kind of change seems like
it's more appropriate for an interim release.

> That might adversely affect SRU verification
> workflow. But given that jammy-proposed is (after release) specifically
> for opt-in and more-risky-than-stable testing, perhaps that doesn't
> warrant delaying until Keen.
>
> --
> technical-board mailing list
> technical-bo...@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/technical-board

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Dan Streetman]

On Mon, Dec 13, 2021 at 9:04 AM Colin Watson  wrote:
>
> On Mon, Dec 13, 2021 at 08:32:53AM -0500, Dan Streetman wrote:
> > Just to clarify, people won't need to manually specify all
> > dependencies, right? For example, if testing the 'systemd' package
> > from -proposed, simply doing 'apt install systemd/jammy-proposed'
> > would install the proposed version of systemd *and also* the proposed
> > version of libsystemd0?
>
> That's how it behaves in my tests, yes - if a dependency imposes a
> version constraint requiring a lower-priority version, then apt tries to
> satisfy it.
>
> > Also, is this really needed? Is it really so hard for people to just do:
> >
> > $ sudo add-apt-repository -p proposed
> >
> > ...install proposed package(s) normally and do tests...
> >
> > $ sudo add-apt-repository -r -p proposed
>
> This has been an issue on and off for at least a decade, so my
> impression is that we have solid empirical evidence that this is indeed
> too hard for many testers in practice.

Ok, but the (non-graphical) method of enabling/disabling the proposed
pocket is quite painful on focal and earlier, so maybe now that users
can simply use add-apt-repository to enable/disable it with a 1-line
command, it might not be as much of an issue?

Updating the 'EnableProposed' wiki page might help, since currently it
seems hugely over-complicated and out of date.

Anyway, if the change is made so apt treats the proposed pocket the
same as the backports pocket, i assume (hope) all new systems will
have the proposed pocket enabled by default in their sources.list?

>
> --
> Colin Watson (he/him)  [cjwat...@ubuntu.com]
>
> --
> technical-board mailing list
> technical-bo...@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/technical-board

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Colin Watson]

On Mon, Dec 13, 2021 at 08:32:53AM -0500, Dan Streetman wrote:
> Just to clarify, people won't need to manually specify all
> dependencies, right? For example, if testing the 'systemd' package
> from -proposed, simply doing 'apt install systemd/jammy-proposed'
> would install the proposed version of systemd *and also* the proposed
> version of libsystemd0?

That's how it behaves in my tests, yes - if a dependency imposes a
version constraint requiring a lower-priority version, then apt tries to
satisfy it.

> Also, is this really needed? Is it really so hard for people to just do:
> 
> $ sudo add-apt-repository -p proposed
> 
> ...install proposed package(s) normally and do tests...
> 
> $ sudo add-apt-repository -r -p proposed

This has been an issue on and off for at least a decade, so my
impression is that we have solid empirical evidence that this is indeed
too hard for many testers in practice.

-- 
Colin Watson (he/him)  [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Dan Streetman]

On Thu, Dec 9, 2021 at 11:27 AM Colin Watson  wrote:
>
> On Wed, Feb 03, 2021 at 11:12:57AM +, Iain Lane wrote:
> > I think the Launchpad support is still missing, although we started on
> > this several years ago. That will need to be picked up and finished off:
> >
> >   https://bugs.launchpad.net/launchpad/+bug/1016776
> >
> > That bug report talks about doing it pre-release (for devel only) but I
> > think I'm now in favour of doing it always, and the proposed
> > implementation in there would allow that. For devel, the main reason is
> > that I frequently come across users who have misunderstood what proposed
> > is for and manually enabled it themsleves, resulting in various degrees
> > of brokenness on their systems and bug reports that take developers'
> > time to triage and eventually close. These are not (always) people who
> > have updated from a previous release, where we could have had tools
> > disable -proposed for them, but also people who have explicitly turned
> > it on after installing a daily out of an attempt to help test the
> > upcoming release.
> >
> > On the client side, as Robie says, we will at least need to update
> > documentation. I'm also not sure what update-manager will do if there
> > are NotAutomatic updates present. It might need some tweaking to show
> > them differently. This could be checked by looking at something in
> > -backports, which is already present with these flags set.
> >
> > And finally, there's some implication for package builds; both Launchpad
> > buildds and other builders would need to ignore this. Launchpad does
> > this for -backports currently, i.e. -backports builds get Build-Depends
> > from -backports wholesale; hoepfully that means the buildd side isn't
> > too hard because we can reuse that.
>
> This is now ready to use from the Launchpad point of view.  There's a
> "proposed_not_automatic" flag on distro series exported over the API; if
> this is set to True, Launchpad writes "NotAutomatic: yes" and
> "ButAutomaticUpgrades: yes" to the Release file.  We've also arranged
> for *-proposed to be pinned to 500 in launchpad-buildd, so Launchpad
> builds will ignore this; I can't speak for other build environments.

Just to clarify, people won't need to manually specify all
dependencies, right? For example, if testing the 'systemd' package
from -proposed, simply doing 'apt install systemd/jammy-proposed'
would install the proposed version of systemd *and also* the proposed
version of libsystemd0?

Also, is this really needed? Is it really so hard for people to just do:

$ sudo add-apt-repository -p proposed

...install proposed package(s) normally and do tests...

$ sudo add-apt-repository -r -p proposed

>
> Thus, from the Launchpad point of view this is ready to use, although
> somebody may want to check the behaviour of things like sbuild and
> pbuilder first.
>
> Somebody in ~techboard would need to make the actual change, if you
> think it's appropriate.  For example, the following in "lp-shell
> production devel" would do it for all supported Ubuntu series:
>
>   for name in ("bionic", "focal", "hirsute", "impish", "jammy"):
>   series = lp.distributions["ubuntu"].getSeries(name_or_version=name)
>   series.proposed_not_automatic = True
>   series.lp_save()
>
> --
> Colin Watson (he/him)  [cjwat...@ubuntu.com]
>
> --
> technical-board mailing list
> technical-bo...@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/technical-board

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Robie Basak]

On Thu, Dec 09, 2021 at 04:27:31PM +, Colin Watson wrote:
> This is now ready to use from the Launchpad point of view.  There's a
> "proposed_not_automatic" flag on distro series exported over the API; if
> this is set to True, Launchpad writes "NotAutomatic: yes" and
> "ButAutomaticUpgrades: yes" to the Release file.  We've also arranged
> for *-proposed to be pinned to 500 in launchpad-buildd, so Launchpad
> builds will ignore this; I can't speak for other build environments.
> 
> Thus, from the Launchpad point of view this is ready to use, although
> somebody may want to check the behaviour of things like sbuild and
> pbuilder first.

Thank you Colin for the work!

If sbuild/pbuilder need adjusting, then maybe we need to do that and
then give developers some time to update their chroots so that we don't
break them (in non-obvious ways) all at once.

Another thought is that if there turns out to be an unintended
consequence for users enabling jammy-proposed (after Jammy's release),
then we'll have done that to them in an LTS instead of hitting an
interim release first. That might adversely affect SRU verification
workflow. But given that jammy-proposed is (after release) specifically
for opt-in and more-risky-than-stable testing, perhaps that doesn't
warrant delaying until Keen.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Iain Lane]

On Thu, Dec 09, 2021 at 05:53:20PM +0100, Julian Andres Klode wrote:
> We only should do it for jammy though, and not risk breaking scripts
> for testing stable releases IMO.

The benefit of this change to people running stable releases is just as 
great as for those running devel. Arguably moreso, as we actually 
instruct people to enable proposed to test updates. There's a strong 
case for making that far less risky.

I can see a case for trying to anticipate any scripts which might break 
though, and/or trying by enabling *one* stable release to see how it 
goes. And there's documentation to update as previously mentioned in 
this thread. This will probably want to be squared off before enabling, 
so it may well make sense to do dev first and stables some time later 
on.

Thanks a lot to Colin for working on this. 

Cheers,

-- 
Iain Lane  [ i...@orangesquash.org.uk ]
Debian Developer   [ la...@debian.org ]
Ubuntu Developer   [ la...@ubuntu.com ]


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Julian Andres Klode]

On Thu, Dec 09, 2021 at 04:27:31PM +, Colin Watson wrote:
> On Wed, Feb 03, 2021 at 11:12:57AM +, Iain Lane wrote:
> > I think the Launchpad support is still missing, although we started on 
> > this several years ago. That will need to be picked up and finished off:
> > 
> >   https://bugs.launchpad.net/launchpad/+bug/1016776
> > 
> > That bug report talks about doing it pre-release (for devel only) but I 
> > think I'm now in favour of doing it always, and the proposed 
> > implementation in there would allow that. For devel, the main reason is 
> > that I frequently come across users who have misunderstood what proposed 
> > is for and manually enabled it themsleves, resulting in various degrees 
> > of brokenness on their systems and bug reports that take developers' 
> > time to triage and eventually close. These are not (always) people who 
> > have updated from a previous release, where we could have had tools 
> > disable -proposed for them, but also people who have explicitly turned 
> > it on after installing a daily out of an attempt to help test the 
> > upcoming release.
> > 
> > On the client side, as Robie says, we will at least need to update 
> > documentation. I'm also not sure what update-manager will do if there 
> > are NotAutomatic updates present. It might need some tweaking to show 
> > them differently. This could be checked by looking at something in 
> > -backports, which is already present with these flags set.
> > 
> > And finally, there's some implication for package builds; both Launchpad 
> > buildds and other builders would need to ignore this. Launchpad does  
> > this for -backports currently, i.e. -backports builds get Build-Depends 
> > from -backports wholesale; hoepfully that means the buildd side isn't 
> > too hard because we can reuse that.
> 
> This is now ready to use from the Launchpad point of view.  There's a
> "proposed_not_automatic" flag on distro series exported over the API; if
> this is set to True, Launchpad writes "NotAutomatic: yes" and
> "ButAutomaticUpgrades: yes" to the Release file.  We've also arranged
> for *-proposed to be pinned to 500 in launchpad-buildd, so Launchpad
> builds will ignore this; I can't speak for other build environments.
> 
> Thus, from the Launchpad point of view this is ready to use, although
> somebody may want to check the behaviour of things like sbuild and
> pbuilder first.
> 
> Somebody in ~techboard would need to make the actual change, if you
> think it's appropriate.  For example, the following in "lp-shell
> production devel" would do it for all supported Ubuntu series:
> 
>   for name in ("bionic", "focal", "hirsute", "impish", "jammy"):
>   series = lp.distributions["ubuntu"].getSeries(name_or_version=name)
>   series.proposed_not_automatic = True
>   series.lp_save()

We only should do it for jammy though, and not risk breaking scripts
for testing stable releases IMO.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Colin Watson]

On Wed, Feb 03, 2021 at 11:12:57AM +, Iain Lane wrote:
> I think the Launchpad support is still missing, although we started on 
> this several years ago. That will need to be picked up and finished off:
> 
>   https://bugs.launchpad.net/launchpad/+bug/1016776
> 
> That bug report talks about doing it pre-release (for devel only) but I 
> think I'm now in favour of doing it always, and the proposed 
> implementation in there would allow that. For devel, the main reason is 
> that I frequently come across users who have misunderstood what proposed 
> is for and manually enabled it themsleves, resulting in various degrees 
> of brokenness on their systems and bug reports that take developers' 
> time to triage and eventually close. These are not (always) people who 
> have updated from a previous release, where we could have had tools 
> disable -proposed for them, but also people who have explicitly turned 
> it on after installing a daily out of an attempt to help test the 
> upcoming release.
> 
> On the client side, as Robie says, we will at least need to update 
> documentation. I'm also not sure what update-manager will do if there 
> are NotAutomatic updates present. It might need some tweaking to show 
> them differently. This could be checked by looking at something in 
> -backports, which is already present with these flags set.
> 
> And finally, there's some implication for package builds; both Launchpad 
> buildds and other builders would need to ignore this. Launchpad does  
> this for -backports currently, i.e. -backports builds get Build-Depends 
> from -backports wholesale; hoepfully that means the buildd side isn't 
> too hard because we can reuse that.

This is now ready to use from the Launchpad point of view.  There's a
"proposed_not_automatic" flag on distro series exported over the API; if
this is set to True, Launchpad writes "NotAutomatic: yes" and
"ButAutomaticUpgrades: yes" to the Release file.  We've also arranged
for *-proposed to be pinned to 500 in launchpad-buildd, so Launchpad
builds will ignore this; I can't speak for other build environments.

Thus, from the Launchpad point of view this is ready to use, although
somebody may want to check the behaviour of things like sbuild and
pbuilder first.

Somebody in ~techboard would need to make the actual change, if you
think it's appropriate.  For example, the following in "lp-shell
production devel" would do it for all supported Ubuntu series:

  for name in ("bionic", "focal", "hirsute", "impish", "jammy"):
  series = lp.distributions["ubuntu"].getSeries(name_or_version=name)
  series.proposed_not_automatic = True
  series.lp_save()

-- 
Colin Watson (he/him)  [cjwat...@ubuntu.com]

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Brian Murray]

On Wed, Feb 03, 2021 at 11:12:57AM +, Iain Lane wrote:
> On Thu, Jan 21, 2021 at 02:15:59AM -0800, Julian Andres Klode wrote:
> > Hi people,
> > 
> > I'd like to suggest that we start setting NotAutomatic: yes for the
> > proposed pocket with hirsute+1, such that things like SRU verification
> > will be easier, and all those people who enable proposed in sources.list
> > for I don't know what reasons don't get their systems destroyed as much.
> > 
> > This would obviously require some changes to pin the repo back up on the
> > builders, but I think it would be useful overall.
> 
> Sounds good to me.
> 
> I think the Launchpad support is still missing, although we started on 
> this several years ago. That will need to be picked up and finished off:
> 
>   https://bugs.launchpad.net/launchpad/+bug/1016776
> 
> That bug report talks about doing it pre-release (for devel only) but I 
> think I'm now in favour of doing it always, and the proposed 
> implementation in there would allow that. For devel, the main reason is 
> that I frequently come across users who have misunderstood what proposed 
> is for and manually enabled it themsleves, resulting in various degrees 
> of brokenness on their systems and bug reports that take developers' 
> time to triage and eventually close. These are not (always) people who 
> have updated from a previous release, where we could have had tools 
> disable -proposed for them, but also people who have explicitly turned 
> it on after installing a daily out of an attempt to help test the 
> upcoming release.

Just as a reminder ubuntu-release-upgrader does disable -proposed during
the upgrade to a development release of Ubuntu.

Cheers,
--
Brian Murray

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Iain Lane]

On Thu, Jan 21, 2021 at 02:15:59AM -0800, Julian Andres Klode wrote:
> Hi people,
> 
> I'd like to suggest that we start setting NotAutomatic: yes for the
> proposed pocket with hirsute+1, such that things like SRU verification
> will be easier, and all those people who enable proposed in sources.list
> for I don't know what reasons don't get their systems destroyed as much.
> 
> This would obviously require some changes to pin the repo back up on the
> builders, but I think it would be useful overall.

Sounds good to me.

I think the Launchpad support is still missing, although we started on 
this several years ago. That will need to be picked up and finished off:

  https://bugs.launchpad.net/launchpad/+bug/1016776

That bug report talks about doing it pre-release (for devel only) but I 
think I'm now in favour of doing it always, and the proposed 
implementation in there would allow that. For devel, the main reason is 
that I frequently come across users who have misunderstood what proposed 
is for and manually enabled it themsleves, resulting in various degrees 
of brokenness on their systems and bug reports that take developers' 
time to triage and eventually close. These are not (always) people who 
have updated from a previous release, where we could have had tools 
disable -proposed for them, but also people who have explicitly turned 
it on after installing a daily out of an attempt to help test the 
upcoming release.

On the client side, as Robie says, we will at least need to update 
documentation. I'm also not sure what update-manager will do if there 
are NotAutomatic updates present. It might need some tweaking to show 
them differently. This could be checked by looking at something in 
-backports, which is already present with these flags set.

And finally, there's some implication for package builds; both Launchpad 
buildds and other builders would need to ignore this. Launchpad does  
this for -backports currently, i.e. -backports builds get Build-Depends 
from -backports wholesale; hoepfully that means the buildd side isn't 
too hard because we can reuse that.

So yes, let's get this work scheduled if we can. :-)

Cheers,

-- 
Iain Lane  [ i...@orangesquash.org.uk ]
Debian Developer   [ la...@debian.org ]
Ubuntu Developer   [ la...@ubuntu.com ]


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: Setting NotAutomatic for hirsute+1-proposed

[Robie Basak]

Hi Julian,

On Thu, Jan 21, 2021 at 02:15:59AM -0800, Julian Andres Klode wrote:
> I'd like to suggest that we start setting NotAutomatic: yes for the
> proposed pocket with hirsute+1, such that things like SRU verification
> will be easier, and all those people who enable proposed in sources.list
> for I don't know what reasons don't get their systems destroyed as much.
> 
> This would obviously require some changes to pin the repo back up on the
> builders, but I think it would be useful overall.

I have no objection, and if it works without UX edge cases then I think
it would be useful to users.

https://wiki.ubuntu.com/Testing/EnableProposed will need a rework
though. We link to that from every SRU bug.


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Setting NotAutomatic for hirsute+1-proposed

[Julian Andres Klode]

Hi people,

I'd like to suggest that we start setting NotAutomatic: yes for the
proposed pocket with hirsute+1, such that things like SRU verification
will be easier, and all those people who enable proposed in sources.list
for I don't know what reasons don't get their systems destroyed as much.

This would obviously require some changes to pin the repo back up on the
builders, but I think it would be useful overall.
-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel