Re: addressing universe/multiverse merges in +1 maintenance
Hi Matthias, On Wed, Feb 3, 2021 at 12:06 PM Matthias Klose wrote: > > With the feature and Debian import freeze approaching in about three weeks, > I'd > like to propose to also address universe/multiverse merges in our +1 > maintenance > tasks within these three weeks. There's a lot untouched for more than one or > two years. > > For merges of the main pocket, owned by the Foundations team, we had some time > addressed after a regular meeting to evaluate those merges, starting with the > oldest (and maybe scariest ones) first. Outcome of these were either removal > requests, or merges. We have up to three +1 maintainers per day, so such an > evaluation could be done within a group as well. > > Please make sure that a merge still builds (maybe keep a browser tab open), > and > bonus points for a successful migration to the release pocket ;) Thanks for raising this! I fully support the proposal, collect all the bonus points! Cheers, Balint > Matthias > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: +1 maintenance report (dogtag-pki vs 389-ds-base)
Hi Brian, On Fri, Jan 22, 2021 at 5:09 PM Brian Murray wrote: > > On Thu, Jan 21, 2021 at 03:25:02PM -0500, Sergio Durigan Junior wrote: > > On Thursday, January 21 2021, Timo Aaltonen wrote: > > > > > On 21.1.2021 18.59, Lukas Märdian wrote: > > >> NO - dogtag-pki vs ['389-ds-base/1.4.4.9-1build2', > > >> 'net-snmp/5.9+dfsg-3ubuntu1'] > > >> > > >> So I had a closer look into the dogtag-pki failure on s390x. I could > > >> easily reproduce the problem inside a s390x LXD container, but > > >> wasn't able to isolate the root cause... After quite some > > >> investigation I was able to produce a debug trace of the problem, > > >> and to me it looks like the issue is actually not inside this > > >> package, but rather inside the LDAP server (i.e. 389-ds-base), as > > >> the debug log shows an "SEVERE: Unable to modify o=pki-tomcat-CA: > > >> netscape.ldap.LDAPException: error result (1); Operations error", > > >> i.e. "Internal Server Error" > > >> (https://docs.oracle.com/cd/E19957-01/816-5618-10/netscape/ldap/LDAPException.html#OPERATION_ERROR > > >> <https://docs.oracle.com/cd/E19957-01/816-5618-10/netscape/ldap/LDAPException.html#OPERATION_ERROR>). > > >> I > > >> do not really understand why the LDAP server would behave > > >> differently on s390x than on all the other architectures, but I > > >> guess this is for another time... > > >> > > >> Debug log: https://paste.ubuntu.com/p/vx9JB6VTjF/ > > >> <https://paste.ubuntu.com/p/vx9JB6VTjF/> > > > > > > This seems to go back to at least 389-ds-base 1.4.4.4, probably > > > longer. Would be useful to know where it regressed. > > > > > > As to why it only happens on s390x my guess would be that it's related > > > to endianness (it's big-endian). Upstream tests only on amd64 (LE). > > > > I'm also very interested in solving this, because it's blocking net-snmp > > and a bunch of other packages from migrating. > > > > Last week I did some investigation and pretty much stopped at the same > > point as Lukas did. I wasn't able to pinpoint exactly what the root > > cause is, but Timo's guess is a good starting point. > > > > I fiddled a bit with autopkgtest.db and confirmed that the failures > > started with 389-ds-base/1.4.4.4-1: > > > > sqlite> SELECT test.package, result.version, result.triggers, > > result.exitcode FROM result INNER JOIN test ON test.id = result.test_id > > where test.package = 'dogtag-pki' and result.triggers LIKE '%389-ds-base%'; > > I find your querying of the SQLite database pretty interesting. Is there > any documentation about the structure of the database and queries that > might be useful? I've created a few queries to analyze the data for identifying the infrastructure improvement opportunities and learn more about interesting packages: https://github.com/rbalint/autopkgtest-db-reports You can even find the generated reports on the GitHub Actions page. Cheers, Balint > Cheers, > -- > Brian Murray > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: dpkg -V does not consider path-exclude config, fails on ubuntu-minimal
Hi Andreas,
On Fri, Nov 20, 2020 at 3:29 PM Andreas Hasenack wrote:
>
> Hi,
>
> I noticed that `dpkg -V` does not take into consideration potential
> exclusions defined in its configuration.
>
> An ubuntu-minimal image, for example, has this:
> ```
> $ cat /etc/dpkg/dpkg.cfg.d/excludes
> # Drop all man pages
> path-exclude=/usr/share/man/*
>
> # Drop all translations
> path-exclude=/usr/share/locale/*/LC_MESSAGES/*.mo
>
> # Drop all documentation ...
> path-exclude=/usr/share/doc/*
>
> # ... except copyright files ...
> path-include=/usr/share/doc/*/copyright
>
> # ... and Debian changelogs
> path-include=/usr/share/doc/*/changelog.Debian.*
> ```
>
> Which results in a lot of noise when `dpkg -V` is run:
> ```
> $ dpkg -V | head
> ??5?? /usr/share/doc/python3-pkg-resources/pkg_resources.txt.gz
> ??5?? /usr/share/doc/fonts-ubuntu-console/README
> ??5?? /usr/share/doc/cryptsetup-bin/NEWS.Debian.gz
> ??5?? /usr/share/man/man8/cryptsetup-reencrypt.8.gz
> ??5?? /usr/share/man/man8/cryptsetup.8.gz
> ??5?? /usr/share/man/man8/integritysetup.8.gz
> ??5?? /usr/share/man/man8/veritysetup.8.gz
> ??5?? /usr/share/doc/python3-distutils/README.Debian
> ??5?? /usr/share/doc/libip4tc2/NEWS.Debian.gz
> ??5?? /usr/share/doc/libksba8/AUTHORS
> ...
> ```
>
> Shouldn't `dpkg -V` take the path-{exclude,include} into
> consideration? Is this just a bug (which I can file), or was it a
> conscious decision?
IMO this is just a dpkg bug, observable on Sid, too.
This is not the only related dpkg bug, see for example:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859646
Cheers,
Balint
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Ubuntu Glibc News
Hi, Groovy Gorilla (20.10) has just been released shipping glibc 2.32 [1] bringing many new and also numerous deprecated features. The deprecations such as the removal of Sun RPC and NIS development files are partially mitigated by adding rpcsvc-proto, libtirpc-dev and libnsl-dev to libc6-dev's dependencies to minimize the number of packages failing to build in 20.10. Eventually those dependencies will be dropped thus if you discover that a package build relies on files not shipped in libc6-dev anymore please add the new build-dependency directly. If you would like to offer a Ubuntu-specific patch for glibc you are welcome to file merge proposals against the new glibc packaging repository [2]. Focal Fossa (20.04 LTS) has recently received a glibc update [3] improving support for the POWER10 [4] architecture and working around a limitation that prevented upgrading to and using 20.04 LTS in many ways on WSL1 [5] (WSL2 did not have this limitation). Bionic Beaver (18.04 LTS) is about to receive a glibc update [6] including a vast amount of stability and performance fixes by merging all of upstream's back-ported changes for 2.27 [7]. The update also adds a new binary package, libc6-lse on arm64 for optimized performance on systems supporting the large-system extensions (LSE) [8]. Cheers, Balint [1] https://sourceware.org/pipermail/libc-announce/2020/29.html [2] https://code.launchpad.net/~ubuntu-core-dev/ubuntu/+source/glibc/+git/glibc [3] https://launchpad.net/ubuntu/+source/glibc/2.31-0ubuntu9.1 [4] https://launchpad.net/bugs/1889190 [5] https://launchpad.net/bugs/1871129 [6] https://launchpad.net/ubuntu/+source/glibc/2.27-3ubuntu1.3 [7] https://launchpad.net/bugs/1851263 [8] https://ubuntu.com/blog/ubuntu-aws-graviton2 -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Ubuntu +1 Maintenance Report (Aug 28, Sept 10-11)
Hi Everyone Aug 28: * Retried double-conversion/i386, cl-usocket/* and cl-ironclad/*. I've filed LP: #1893452 with a hint for double-conversion, cl-ironclad migrated, cl-usocket is still failing in -proposed. * Extended https://code.launchpad.net/~rbalint/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/384609 to trigger tests again on clock skew and pinged Laney to take a new look * One class of failures was caused by tests failing in riscv64 builds, but our team already discussed that and decided to disable tests at build time, thus I've postponed the solution to after fthe +1 maintenance shift. Steve Langasek fixed that since then. Thanks! * I've found a few Golang tests that timed out on armhf LP: #1893640. It was known that ARM test runners are generally slower, but I did not have data to back this and to tell the speed difference, thus I've started creating reports about the test runs at: https://code.launchpad.net/~ubuntu-core-dev/+git/autopkgtest-db-reports Based on the observations I've bumped golang-1.14's and golang-1.15's default test timeout on ARM by 50%, which made tests pass. * I've bumped into systemd-timesycd setting the clock backwards (LP: #1880839) in a few failed tests and I'll propose switching to chrony as the default NTP client on most Ubuntu systems from systemd-timesyncd but I need to put together a formal proposal.b In my September 10-11 shift I've worked on random old stuck packages and glibc in parallel. While the rule is that we should not be working on packages with we are responsible for there were no other big transition in groovy-proposed and glibc gets entangled with other newly arriving packages thus getting it migrated is the most urgent. * Few package rebuilds * Retried riscv64 build of vcmi, now it built and is waiting for glibc 2.32. * Filed LP: #1895277 for dired-du/armhf and submitted a hint for it. * Filed LP: #1895283 to document and track incompatibility between libdevel-declare-perl 0.006022-1 and libperl5i-perl 2.13.2-1. * Merged supervisor bugfix release to fix FTBFS * Asked for removal of libxml-commons-external-java-doc on #ubuntu-release to let xml-commons-external migrate. * Filed hint for 9base/s390x and LP: #1895321 to document the triaging. * Fixed apitrace FTBFS. Cheers, Balint -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Bileto and RiscV64
Hi Lukasz, On Thu, Aug 27, 2020 at 5:56 PM Lukasz Zemczak wrote: > > I think the problem here is the general Bileto architecture, where > autopkgtests can only be triggered when all arch builds are finished > and 'diffed'. But if there are no objections, I could maybe change > Bileto to not care about riscv when performing checks, maybe besides > the final publishing step (in case someone wants to do Bileto publish > to the archive). Yes, please make Bileto consistent with the archive's CI's behaviour. Cheers, Balint PS: Also having an option to automatically start tests when builds are finished would help in not having to stay up a few more hours just to press the button. > > On Thu, 27 Aug 2020 at 16:28, Balint Reczey > wrote: > > > > Hi, > > > > On Thu, Aug 27, 2020 at 1:44 AM Steve Langasek > > wrote: > > > > > > On Wed, Aug 26, 2020 at 06:00:58PM -0300, Andreas Hasenack wrote: > > > > Hi, > > > > > > > The builders for riscv64 are very slow, and since bileto wails for all > > > > builds to be ready, each ticket can take dozens of hours. Even if I > > > > disable that arch in the ppa (via a #webops request), bileto later > > > > enables it again. > > > > > > > Could we do one of the following: > > > > - disable riscv64 by default on bileto, and make it so it can be > > > > enabled (and remain enabled) in the ppa if the user so wants it > > > > - start bileto tests as soon as an arch build is ready, instead of > > > > waiting for them all to be ready as it is today > > > > - something else I haven't thought of :) > > > > Internally we already had discussions about disabling tests in riscv64 > > builds to speed them up. > > https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1891686 > > > > As I understand it got a green light, just no one uploaded the fix yet. > > > > Cheers, > > Balint > > > > > > > > I mean, the other alternative is to not use bileto, which is not part of > > > the > > > normal workflow and results in duplicate tests anyway? > > > > > > > I know we want to have packages working on riscv64, but since it's not > > > > blocking migration in the real archive, it seems unfair that it blocks > > > > bileto so much. > > > > > > It is possible that updating the britney instance used for bileto to match > > > the current code used for -proposed would address this. > > > > > > -- > > > Steve Langasek Give me a lever long enough and a Free OS > > > Debian Developer to set it on, and I can move the world. > > > Ubuntu Developer https://www.debian.org/ > > > slanga...@ubuntu.com vor...@debian.org > > > -- > > > ubuntu-devel mailing list > > > ubuntu-devel@lists.ubuntu.com > > > Modify settings or unsubscribe at: > > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel > > > > > > > > -- > > Balint Reczey > > Ubuntu & Debian Developer > > > > -- > > ubuntu-devel mailing list > > ubuntu-devel@lists.ubuntu.com > > Modify settings or unsubscribe at: > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel > > > > -- > Łukasz 'sil2100' Zemczak > Foundations Team > lukasz.zemc...@canonical.com > www.canonical.com -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Bileto and RiscV64
Hi, On Thu, Aug 27, 2020 at 1:44 AM Steve Langasek wrote: > > On Wed, Aug 26, 2020 at 06:00:58PM -0300, Andreas Hasenack wrote: > > Hi, > > > The builders for riscv64 are very slow, and since bileto wails for all > > builds to be ready, each ticket can take dozens of hours. Even if I > > disable that arch in the ppa (via a #webops request), bileto later > > enables it again. > > > Could we do one of the following: > > - disable riscv64 by default on bileto, and make it so it can be > > enabled (and remain enabled) in the ppa if the user so wants it > > - start bileto tests as soon as an arch build is ready, instead of > > waiting for them all to be ready as it is today > > - something else I haven't thought of :) Internally we already had discussions about disabling tests in riscv64 builds to speed them up. https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1891686 As I understand it got a green light, just no one uploaded the fix yet. Cheers, Balint > > I mean, the other alternative is to not use bileto, which is not part of the > normal workflow and results in duplicate tests anyway? > > > I know we want to have packages working on riscv64, but since it's not > > blocking migration in the real archive, it seems unfair that it blocks > > bileto so much. > > It is possible that updating the britney instance used for bileto to match > the current code used for -proposed would address this. > > -- > Steve Langasek Give me a lever long enough and a Free OS > Debian Developer to set it on, and I can move the world. > Ubuntu Developer https://www.debian.org/ > slanga...@ubuntu.com vor...@debian.org > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Switching iptables to use the nftables backend (again) on Sept 3
Hi Everyone, Switching iptables to use the nftables backend already happened before once, but was reverted later due to LXD and possibly other parts of the Ubuntu software ecosystem were not ready [1]. The 20.04 LTS release cycle was not an ideal time to perform the switch either, but Groovy Gorilla, the 20.10 interim release can use nftables as the default and let us fix any surfacing issue for the next LTS release. Debian already made the switch in Buster thus the packages in the archive should be generally ready for the switch. Going through the packages I found only sshguard that needs to be modified, dropping the Ubuntu delta. The switch is simply swapping the two alternative backends' priority and prefer nftables backend over legacy, without promoting the nftables package to be recommended by the iptables package in this development cycle. No regression showed up while testing the changes in Bileto [2], nor while performing a release-upgrade to the changed packages. LXD have added nftables support [3] and I've tried the microk8s snap and it worked with the switched default but created legacy tables [4]. It will still be possible to change iptables/ip6tables/arptables/ebtables back to use the legacy backend [5] after the switch, but ideally software projects should already have nftables support or have a plan to implement it in the near future [6]. If you have concerns regarding the planned switch please raise them here. The September 3 target date is after Feature Freeze and I'll formally ask for a Feature Freeze Exception. Cheers, Balint [1] https://lists.ubuntu.com/archives/ubuntu-devel/2019-September/040801.html [2] https://bileto.ubuntu.com/#/ticket/4044 [3] https://github.com/lxc/lxd/issues/6223 [4] https://github.com/ubuntu/microk8s/issues/892#issuecomment-681033084 [5] https://wiki.debian.org/nftables#Reverting_to_legacy_xtables [6] https://wiki.nftables.org/wiki-nftables/index.php/Adoption -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Ubuntu +1 Maintenance Report (Aug 14)
Hi Everyone, * Fixed regressions I caused in ubuntu-archive-tools' retry-autopkgtest-regressions. I actually did that in my July 30-31 shift which became very short due to various reasons. * Took a look at the icu transition and it is was blocked only on the building libreoffice package thus I could not move that forward * Retried build of rust-onig-sys package on riscv64 to unblock a few packages, but it failed in tests thus filed LP: #1891686 to disable tests for all riscv64 which we already started discussing and LP: #1891688 for the actual FTBFS tagged with 'update-excuse' * Asked for demotion of rust-tokio-core to -proposed which ended up being a removal because it needs a sourceful upload to ever migrate. This unblocked rust-scoped-tls which migrated. * Filed LP: #1891691 with 'update-excuse' against rust-cookie-store explaining that it needs rust-cookie to be fixed first * Synced rust-gtk up from Eoan to restore it. This needs only upload rights, but then AA powers were needed to accept it from the New queue. This may unblock qwertone. The restoration brought back i386 binaries, too, which should be removed, see LP: #1891754 * Asked for demotion of rust-crossbeam which also became a removal * Asked for removal of predictprotein due to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963927 * retried intermittent autopkgtest failured with ./retry-autopkgtest-regressions --log-regex ... * Reworked the fix for automatically retrying on intermittent autopkgtest failures https://code.launchpad.net/~rbalint/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/38460 * Followed up on migrating Britney hints to git Cheers, Balint -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Ubuntu +1 Maintenance Report (July 16 -17)
Hi Everyone, I've asked for the removal of libselinux 3.1-1 from groovy-proposed because .symbols' symbol version were broken and broke packages built with it. Luckily the only reverse dependency affected was systemd which I rebuilt. libselinux also breaks glibc's autopkgtest which I plan fixing outside of +1 maintenance, next week: LP: #1887919 I've rebased https://code.launchpad.net/~rbalint/ubuntu-archive-tools/retry-intermittent/+merge/384468 hoping that eventually it will be merged. I used the pending changes in my shift again to re-trigger a fair amount of tests. I've NMU-d hyperkitty and postorius via Debian and that will hopefully unblock the migration of glewlwyd. The new proposed-migration page [0] helps a lot in triaging installability issues, thanks Laney for rebasing Britney! Mysql-8.0 regressed in the release pocket, thus I've filed LP: #1887979 and a hint. BTW the hints I've filed in my previous +1 maintenance shift were not merged for 10 days, just now, after explicitly adding reviewing users despite that I've pinged ubuntu-release on 2020-07-06. I'd like to suggest that every active team in Ubuntu should adopt a monitoring process to handle reviews without users set explicitly and ensuring timely reply to merge requests. I started going through the rust-* packages and introduced rust-jobserver to get rust-*git* migrated. The rust-jobserver was removed from the archive but I could not find any trace of why, since there was no removal bug. This is an example of the systematic issue we are having in Ubuntu. There are packages without Ubuntu-specific delta which are stuck to a version lower than in Debian because or completely missing from Ubuntu no new Debian upload triggered the sync since the package was removed. Sometimes the absence of the package or the lower version does match the intention because the package was not ready for release or should not be part of Ubuntu, but in rust-jobserver's case it looks like the package was just lost. Most likely if the package entered testing in Debian it is in good enough shape to be released in Ubuntu so I've came up with the following UDD query: SELECT sources.source, ubuntu_groovy.max_version AS ubuntu_version, sources.version FROM sources LEFT JOIN (SELECT source, MAX(version) as max_version FROM ubuntu_sources WHERE release LIKE 'groovy%' GROUP BY source) as ubuntu_groovy ON sources.source = ubuntu_groovy.source WHERE sources.release='bullseye' AND ( (ubuntu_groovy.max_version < sources.version AND ubuntu_groovy.max_version NOT LIKE '%ubuntu%') OR ubuntu_groovy.max_version is NULL) AND NOT sources.section = 'debian-installer' GROUP by sources.source, sources.version, ubuntu_groovy.max_version ORDER BY sources.source; There are a few rough edges like versions are compared as strings and debian-installer components should be filtered better, but the results [1] is worth checking. This would probably be better as an archive report automatically linking to removal bug reports and I may start working on that after my previous merge request gets merged [2]. Did I mention stale merge requests? ;-) Cheers, Balint [0] https://lists.ubuntu.com/archives/ubuntu-devel/2020-July/041078.html [1] https://paste.ubuntu.com/p/pnwp2v2vVv/ [2] https://code.launchpad.net/~rbalint/ubuntu-archive-scripts/bzr-by-team-report/+merge/359222 -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Ubuntu +1 Maintenance Report (July 2 -3)
Hi All, * retried autopkgtests using "./retry-autopkgtest-regressions --log-regex 'Temporary failure resolving'" and simliar patterns * retriggered fdroidserver/armhf for ruamel.yaml 0.16.10-2 which had new temprorary failure cause 'Could not resolve proxy: squid.internal' * rebuilt hoel glewlwyd to finish orcania transition, but finally the sync of hoel fixed the ftbfs of glewlwyd and ulfius still needs to be fixed * dropped Python 2 PyCryptodome support from kodi to let pycryptodome migrate * NMUd wordpress-shibboleth to finish shibboleth-sp transition * retriggered several tests for node-boom * prepared MP against britney1 to support hints in Git: https://code.launchpad.net/~rbalint/britney/britney1-ubuntu-hints-from-git/+merge/386856 * prepared a few hints for regressions: https://code.launchpad.net/~rbalint/britney/hints-ubuntu/+merge/386878 * updated python-cogent to new upstream release to fix autopkgtest, but there are new failures so it stayed in Salsa (https://salsa.debian.org/med-team/python-cogent) for now and I added a hint for it I had some issues reproducing autopkgtest failures on exotic architectures because autopkgtest-build-lxd fails on bos01 vms due to https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1878225 but it can be worked around by logging into the lxc container being prepared before the timeout and stopping snapd and snapd.seeded services. Cheers, Balint -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Ubuntu +1 Maintenance Report (June 18 -19)
Hi All, I've retried 20+ autopkgtests using "./retry-autopkgtest-regressions --log-regex 'Temporary failure resolving'" and similar patterns. Those will be automatically retried when https://code.launchpad.net/~rbalint/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/384609 gets merged. Transitions: libwebsockets: rebuilt packages to finish transition: driftnet forked-daapd mosquitto python-tornado: filed LP: #1884061 to remove mitmproxy and let it migrate (needs ubuntu-archive action) node-commander: needs to go in with uglify-js, triggered probably flaky arm64 test libvorbis: sponsored Olivier Tilloy's fix R packages: - added multiple hints suggested by ginggs - uploaded r-cran-rsample 0.0.7-0ubuntu1 - uploaded r-cran-sf 0.9-4+dfsg-0ubuntu1 - uploaded r-cran-brms 2.13.0-1ubuntu1 - uploaded r-cran-stanheaders 2.21.0-5+really2.21.0-1-0ubuntu1 - uploaded r-cran-dplyr 1.0.0-1ubuntu1 - opened LP: #1884451 to remove r-cran-openmx with this removal r-base will be ready to migrate Filing hints reminded me that hints are still maintained in Bazaar, thus I offered a Git repository to switch to: https://code.launchpad.net/~rbalint/britney/hints-ubuntu-bzr-to-git/+merge/386139 Cheers, Balint -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Ubuntu +1 Maintenance Report (May 21, )
Hi All, Welcome to the +1 Maintenance Team's [1] reports again. The proposed migration queue is greener, but Perl still did not get through. For more detailed current status please check [3]. * In my shift many autopkgtests were still running which gave the opportunity for improving retry-autopkgtest-regressions from ubuntu-archive-tools to not trigger running/queued tests again and also find intermittent issues and trigger them [4]. I'm looking forward not having to go through logs manually to find out that the test failure was due to a temporal network issue because cron/a systemd timer already did that for me and retriggered the test! * With Ubuntu 20.04 dropping Bazaar git-remote-bzr stopped working without any replacement LP: #1869231. That made working with Bazaar repositories even more painful for people working primarily with git. Ubuntu-archive-tools is still in Bazaar, so I've recreated my Bionic setup for converting repos and offered a migrated repo [5]. * To help others working on the next Ubuntu release I went through the sponsorship queue [6] and sponsored the non-SRU uploads which seemed to have stalled: freecad LP: #1754084 - marked as invalid bluebird-gtk-theme LP: #1836975 - sponsored django-piston3 LP: #1842043 - sponsored kpatch LP: #1837886 - already fixed ovn LP: #1853431 - already fixed friendly-recovery - merged lp:~debian-janitor/friendly-recovery/lintian-fixes qalculate-gtk LP: #1862546 - deferred sync from experimental to wait for the transition to take place in Debian first openscap LP: #1851682 - sponsored synaptic LP: #1870900 - marked as incomplete, sync can't be done because of FTBFS installation-guide - merged lp:~dsmythies/installation-guide/ubuntu-20.04 gammu LP: #1874554 - sponsored via Debian Thanks to everyone joining Ubuntu's development. If you can't upload packages yet to the archive please use the sponsorship process [7] to propose fixes. I hope the infrastructure improvements also help making contribution easier and more fun. Cheers, Balint [1] https://wiki.ubuntu.com/PlusOneMaintenanceTeam [2] https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html [3] https://wiki.ubuntu.com/PlusOneMaintenanceTeam/Status [4] https://code.launchpad.net/~rbalint/ubuntu-archive-tools/retry-intermittent/+merge/384468 [5] https://code.launchpad.net/~rbalint/ubuntu-archive-tools/bzr-to-git/+merge/384436 [6] http://reqorts.qa.ubuntu.com/reports/sponsoring/ [7] https://wiki.ubuntu.com/SponsorshipProcess -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
PSA: New update-excuse bug tag
Dear Ubuntu Developers, The new 'update-excuse' tag can be used to make a bug appear on update excuses [1] page among the excuses why a particular package is not migrating to the release pocket: casper (1.427 to 1.428) * Maintainer: Ubuntu Developers * 6 days old * Also see bug 1850184 last updated on Mon Oct 25 16:59:32 2019 * autopkgtest for casper/1.428: amd: Regression , arm64 Pass, ... The tag can be useful when triaging and fixing an issue takes a longer time or needs coordination with others. For marking package versions in stable releases the update-excuse-$SERIES tags can be used. For more details please see [2] and feel free to share the news on Twitter [3]. Cheers, Balint [1]: https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html [2]: https://balintreczey.hu/blog/new-tags-on-the-block-update-excuse-and-friends/ [3]: https://twitter.com/balintreczey/status/1189526405559848960 -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Supporting LZ4 as initramfs compressor
Hi Dimitri, Thanks for keeping this thread alive. On Wed, Jun 5, 2019 at 2:16 PM Dimitri John Ledkov wrote: > > On Fri, 31 May 2019 at 09:13, Seth Arnold wrote: > > > > On Thu, May 30, 2019 at 11:46:57AM +0100, Dimitri John Ledkov wrote: > > > As if lz4 kernel & xz initrd would yield the fastest boot time? That > > > > I'm lacking some context here, but I think building the initrds is already > > too slow and I'm afraid xz on initrd rebuilds would be significantly > > worse than lz4 or zstd or even low-compression levels of zlib. > > > > Boot times are important but every now and then people do run apt upgrade > > by hand and waiting forever to rebuild an initrd that may or may not be > > used is pretty tedious. xz is great if the really slow compress times will > > be made up for by better transfers or disk savings or similar. > > > > I've tried to dig up past benchmarks in private discussions, public > discussions and kernel mailing list. > > Zstd patches have not made it into the upstream kernel yet. > > As used by mkinitramfs: > - lz4 is faster to compress than gzip > - lz4 is blazingly fast to decompress > - lzma is dog slow to compress and decompress, but is tiny > - lz4 size weight over gzip is marginal (14%) but imho worth the > improved boot time & initrd creation time > - xz is potentially even slower and even smaller than lzma > > In places where size is an absolute premium (tiny embedded iot > devices) and performance is irrelevant, xz or lzma should be used. > > In all other places, our performance profile is in favor of lz4. > > Imho that includes the kernel image itself, thus we should consider switching: > - initramfs tools to default to lz4 > - livecd-rootfs to default to lz4 > - kernels to compress kernel image with lz4 > - grub to include lz4 support Please also note that it is not just the compressor that should be tweaked but the parameters, too. Initramfs-tools uses lz -9 because it provided the best space - speed tradeoff, but comparisons testing the kernel compressors were using the defaults which were not -9. IMO lz4 (-9) would be the best choice for kernels, too, but to see more accurate measurements it may be a good idea to run tests again with setting compression level. Cheers, Balint > > I shall proceed with changing the defaults on the above to improve our > responsiveness experience on installer, cloud, core and classic > devices. > If our firstboot & subsequent boot speed degrades or disk space > becomes a concern, we can look into tweaking these changes further. > > -- > Regards, > > Dimitri. > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Unattended-upgrades starts upgrading packages in Devel 21 days before the release
Dear Disco Lovers, With the Final Release approaching and new intrusive changes hopefully not landing anymore we need to get feedback about how the unattended application of updates work and give it a test flight. After a series of discussions in the Foundations Team we implemented turning unattended-upgrades on 21 days before the planned release date which is hoped to be the sweet spot between running it during the whole development period and turning it on right after the release without wider-scale testing. 21 days before the release we expect the quality of the archive to permit such testing of package upgrades. To add a little context unattended-upgrades is enabled during the whole development period of Debian, both in unstable and testing releases, but the migration from unstable to testing is generally slower than the -proposed to release migration in Ubuntu. Please report issues found in either unattended-upgrades or in packages failing to be upgraded unattended to make Dingo's life full of fun without breakdance! Cheers, Balint -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Proposal: drop rails and its reverse-dependencies from Ubuntu 19.04 [Re: Maintaining language-specific module package stacks]
> we should put this into effect immediately for the Ubuntu 19.04 release, > > despite the nearness of the release date. > > > > Provided that there are no objections here, I will plan to start removing > > this stack from disco and disco-proposed on Friday, April 5. > > I disagree with this approach. There are at least two packages with failing > autopkg tests which were removed in Debian. So why are those not removed in > Ubuntu as well? I think we should fix our ubuntu-archive tasks first, and see > what kind of actions we are missing, and only then decide if those packages > should be removed or not. > > With your AA hat on, why are those removals not handled before proposing the > removal of every dependency? > > > (These removals are reversible right up until the day before release, so I > > don't see much point in a long comment period. If objections are raised > > after Friday, we can still revisit.) > > > > Here's a list of rails packages that look like immediate candidates for > > removal based on this hackish script: > > > [...] > > debci > > Is this a commitment of the ubuntu-release team to maintain the stack needed > for > the autopkg tests outside the archive? > > Please don't get me wrong, I'm not a user of the rails stack, however I think > we > should get our internal processes fixed first. > > Matthias > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Migrating Bazaar repositories to Git on Launchpad
Hi Scott, On Mon, Nov 26, 2018 at 4:05 PM Scott Moser wrote: > > Hi balint, > Thanks for bzr-git-maas-convert. I wonder if you'd be interested in > attempting to add a few features to it that I have used in my bzr2git > I have a gist at > https://gist.github.com/smoser/e4e5388faa6dcc92d8acfb1b7fbabb7c > which i utilize and hand-patch a installed bzr-fastimport. > > The changes maintain some vcs data that would otherwise be lost: > > a.) bug metadata converted: Bzr contains 'fixes' metadata > (bzr commit --fixes) that does not get moved over to git. > This updates the commit message exported to git to > contain LP: #XX for each --fixes=lp:XX in bzr. > > b.) bzr revno info: Often times bug or other references to code may > say 'fixed in revno XXX'. That information gets lost in a conversion to git. > The patch here updates commit messages to contain bzr-revno: XXX > for each bzr revision. Thanks, this is really useful! I have set up a PPA for the package with a slightly modified patch here: https://launchpad.net/~rbalint/+archive/ubuntu/bzr-to-git Now the format conforms a bit more to git conventions: commit b35d0353ee30ef8d72d14b25fff8662264e6c2a6 Author: Foo <...> Date: Mon Nov 26 19:00:07 2018 +0100 Have a nice day! LP: #12345 LP: #44 Co-Authored-By: Bar Bzr-Rev: 864.3 Cheers, Balint > > On Sat, Nov 24, 2018 at 5:31 PM Balint Reczey > wrote: > > > > Dear Ubuntu Contributors! > > > > Launchpad already supports git but there are still many active bzr > > repositories there. > > > > If you would like to migrate some of them to git I'd like to suggest > > taking a look at bzr-git-mass-convert [1] based on bzr fast-export > > (verifying the result with git-remote-bzr). It is a simple tool for > > merging multiple bzr branches to a single git repository, set up for > > pushing it back to Launchpad. > > > > We (at the Foundations Team) use it for migrating our repositories and > > there is also a wiki [2] page for tracking the migration schedule [3] > > of popular repositories. > > > > For example we plan migrating livecd-rootfs on December 12, please get > > your open MP's merged by then to avoid the extra work of resubmitting > > them against the new git repo. > > > > There is also a new tracker to list packages still maintained in bzr, > > grouped by the teams [4]. > > > > Cheers, > > Balint > > > > [1] > > https://code.launchpad.net/~ubuntu-core-dev/+git/bzr-git-mass-convert/+ref/master > > [2] https://wiki.ubuntu.com/UbuntuDevelopment/MigratingFromBzrToGit > > [3] > > https://wiki.ubuntu.com/UbuntuDevelopment/MigratingFromBzrToGit#Migration_Schedule > > [4] http://people.canonical.com/~rbalint/bzr_repos_by_team.html > > > > -- > > Balint Reczey > > Ubuntu & Debian Developer > > > > -- > > ubuntu-devel mailing list > > ubuntu-devel@lists.ubuntu.com > > Modify settings or unsubscribe at: > > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Migrating Bazaar repositories to Git on Launchpad
Dear Ubuntu Contributors! Launchpad already supports git but there are still many active bzr repositories there. If you would like to migrate some of them to git I'd like to suggest taking a look at bzr-git-mass-convert [1] based on bzr fast-export (verifying the result with git-remote-bzr). It is a simple tool for merging multiple bzr branches to a single git repository, set up for pushing it back to Launchpad. We (at the Foundations Team) use it for migrating our repositories and there is also a wiki [2] page for tracking the migration schedule [3] of popular repositories. For example we plan migrating livecd-rootfs on December 12, please get your open MP's merged by then to avoid the extra work of resubmitting them against the new git repo. There is also a new tracker to list packages still maintained in bzr, grouped by the teams [4]. Cheers, Balint [1] https://code.launchpad.net/~ubuntu-core-dev/+git/bzr-git-mass-convert/+ref/master [2] https://wiki.ubuntu.com/UbuntuDevelopment/MigratingFromBzrToGit [3] https://wiki.ubuntu.com/UbuntuDevelopment/MigratingFromBzrToGit#Migration_Schedule [4] http://people.canonical.com/~rbalint/bzr_repos_by_team.html -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: Supporting LZ4 as initramfs compressor
Hi Steve, On Wed, Mar 21, 2018 at 12:40 AM Steve Langasek wrote: > > Hi Balint, > > On Mon, Mar 19, 2018 at 02:59:24PM +0000, Balint Reczey wrote: > > > Initramfs-tools uses gzip compression by default which served us well > > for quite some time but LZ4 offers way faster decompression while > > making a only slightly bigger initramfs files. > > When people have previously discussed lz4 on IRC as a possible choice for > default compression algorithm, I had the impression that this was with the > expectation that the resulting initramfs files would be *smaller* than with > using gzip. > > (I think. It happens that names for compression algorithms are remarkably > unoriginal, so it's possible I've confused lz4 with another.) > > But your data shows that lz4-compressed initramfs is definitely larger than > gzip, which means that there are tradeoffs here that should be fully > examined. > > After all, an initramfs that's not compressed at all would take even less > time to decompress at boot (0s) but would occupy more space. But you aren't > advocating for this, so there must be some reason you believe lz4 is more of > a sweet spot than gzip? > > > The first thing that I see missing from this analysis is the time it takes > for the firmware/bootloader to load the initramfs into memory. I know from > experience that some bootloader filesystem drivers have quite poor > performance; and the time spent loading the initramfs into memory will scale > roughly linearly with the file size. So any analysis of lz4 impact on boot > speed needs to take this into account. We should show that the overall > bootspeed from bootloader to pid 1 has actually improved, and this should be > measured with multiple bootloader driver implementations (across > architectures; UEFI vs BIOS; possibly on multiple virtualization substrates > vs. x86 bare metal). > > > The second thing to consider is that, regardless of any improvements in our > autoremoval of kernels, we have had some historical default sizing for /boot > partitions in the installer which are now on the small side for even a fully > correct kernel upgrade path. > > In trusty, the default (max) size for a /boot partition was 256M. In > xenial, it was 512M. In artful, we have bumped this up to 768M with a > minimum of 512M because of LP: #1716999. > > The actual space consumed by the static files in the 4.13.0-7-generic kernel > in artful - not counting the current .efi.signed duplication which will > hopefully go away soon - is just under 13MiB. My bootloader here is 8MiB, > but 10MiB is not out of the question in some configurations. My initramfs > is 52MiB rather than the 55MiB in that bug, but again 55MiB is plausible - > and your own numbers seem to show 56MiB. > > That means that anyone who installed with trusty, has /boot as a separate > partition, and has plymouth in the initramfs (such as for encrypted root > disk, which would be a common reason to have a separate /boot) has already > run out of space on their /boot while using gzip; so must already reinstall > or switch to a non-default initrd compression algorithm on upgrade. This > can therefore be ignored for the choice between gzip and lz4 as default. > > Anyone who installed with xenial is borderline today with artful; > 56*4+3*13+10 == 273MiB, which is more than xenial's minimum /boot size of > 256MiB but less than the max of 512MiB. So some number of users have a boot > partition that's large enough to accommodate gzipped initrd, but will run > out of space once you switch to lz4 (64*4+3*13+10 == 305MiB). And those > that don't run out of space immediately as a result of the switch to lz4 > would still run out of space sooner as the kernel size grows (since the > kernel definitely seems to only grow over time with new drivers). > > Systems installed with artful or newer seem to still be fine for a while, > with either gzip or lz4. > > So there's a decision to be made about whether we care about upgrades > working with the default compression on systems installed with smaller boot, > and for how long. If we decide this shouldn't block switching the default > compression, we also need to sort out how we will communicate this to > affected users - and in particular, how to avoid problems on upgrade when > the user runs out of disk space at the worst possible time. > > > > Base on the results I plan adding LZ4 compression support to > > initramfs-tools as requested in LP: #1488620 [1] in the next days > > without setting it as default and I propose setting LZ4 as default for > > 18.10. > > Since this is a non-default option and doesn't introduce any new > dependen
Re: Supporting LZ4 as initramfs compressor
On Mon, Mar 19, 2018 at 3:12 PM, Julian Andres Klode wrote: > On Mon, Mar 19, 2018 at 02:59:24PM +0000, Balint Reczey wrote: >> Hi, >> >> Initramfs-tools uses gzip compression by default which served us well >> for quite some time but LZ4 offers way faster decompression while >> making a only slightly bigger initramfs files. >> >> On my old laptop the initramfs extraction time decreased from ~1.2s to >> ~0.24s: >> (with lz4) >> kernel: [0.297726] Unpacking initramfs... >> kernel: [0.535061] Freeing initrd memory: 77940K >> kernel: [0.301637] Unpacking initramfs... >> kernel: [0.539109] Freeing initrd memory: 77940K >> (with gzip) >> kernel: [0.273748] Unpacking initramfs... >> kernel: [1.490066] Freeing initrd memory: 57140K >> kernel: [0.281729] Unpacking initramfs... >> kernel: [1.498493] Freeing initrd memory: 57140K >> >> The increase in the initrd.img size is ~14%: >> (lz4) >> -rw-r--r-- 1 root root 66709065 márc 19 14:24 >> /boot/initrd.img-4.15.0-12-generic >> (gzip) >> -rw-r--r-- 1 root root 58510993 márc 19 12:57 >> /boot/initrd.img-4.15.0-12-generic.bak >> >> Initramfs creation speed also improved a bit from ~24s to ~21s wall clock >> time: >> (lz4) >> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic >> 14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata >> 22368maxresident)k >> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic >> 15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata >> 22308maxresident)k >> (gzip) >> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic >> 18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata >> 22396maxresident)k >> update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic >> 18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata >> 22292maxresident)k > > For me it was 16 -> 10 I think. > >> >> Base on the results I plan adding LZ4 compression support to >> initramfs-tools as requested in LP: #1488620 [1] in the next days >> without setting it as default >> > > +1 > >> and I propose setting LZ4 as default for 18.10. > > We might have zstd support by that time (I hope), it might make sense > to use that then (better space/time tradeoff), but we'll have to see. That would also be an option, but I expect zstd to bring little speed advantage over LZ4 here while LZ4 support could be easily backported to releases with older kernels. The proposed patch enabling lz4 uses lz4 -9. I tried lz4's default compression, -1, but decompression speed difference was barely noticeable - most likely due to copying being the bottleneck. In fact I just realized that i copied the results created with lz4 -1, where the compressed initrd size was 77940K. The correct results for lz4 -9 are the following, taking the same ~0.24s to decompress: kernel: [0.285692] Unpacking initramfs... kernel: [0.518806] Freeing initrd memory: 65148K kernel: [0.289731] Unpacking initramfs... kernel: [0.522823] Freeing initrd memory: 65148K Cheers, Balint -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Supporting LZ4 as initramfs compressor
Hi, Initramfs-tools uses gzip compression by default which served us well for quite some time but LZ4 offers way faster decompression while making a only slightly bigger initramfs files. On my old laptop the initramfs extraction time decreased from ~1.2s to ~0.24s: (with lz4) kernel: [0.297726] Unpacking initramfs... kernel: [0.535061] Freeing initrd memory: 77940K kernel: [0.301637] Unpacking initramfs... kernel: [0.539109] Freeing initrd memory: 77940K (with gzip) kernel: [0.273748] Unpacking initramfs... kernel: [1.490066] Freeing initrd memory: 57140K kernel: [0.281729] Unpacking initramfs... kernel: [1.498493] Freeing initrd memory: 57140K The increase in the initrd.img size is ~14%: (lz4) -rw-r--r-- 1 root root 66709065 márc 19 14:24 /boot/initrd.img-4.15.0-12-generic (gzip) -rw-r--r-- 1 root root 58510993 márc 19 12:57 /boot/initrd.img-4.15.0-12-generic.bak Initramfs creation speed also improved a bit from ~24s to ~21s wall clock time: (lz4) update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic 14.97user 6.31system 0:20.47elapsed 103%CPU (0avgtext+0avgdata 22368maxresident)k update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic 15.18user 6.49system 0:20.48elapsed 105%CPU (0avgtext+0avgdata 22308maxresident)k (gzip) update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic 18.23user 6.77system 0:23.61elapsed 105%CPU (0avgtext+0avgdata 22396maxresident)k update-initramfs: Generating /boot/initrd.img-4.15.0-12-generic 18.38user 6.83system 0:23.82elapsed 105%CPU (0avgtext+0avgdata 22292maxresident)k Base on the results I plan adding LZ4 compression support to initramfs-tools as requested in LP: #1488620 [1] in the next days without setting it as default and I propose setting LZ4 as default for 18.10. I'm aware of the old problem of /boot filling up with kernels and initramfs files but unattended-upgrades already removes old kernels [2] and update-manager is planned to do the same starting with 18.04 [3] thus Ubuntu systems will have enough space in /boot to allow slightly bigger initrd files. Cheers, Balint [1] https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1488620 [2] https://launchpad.net/ubuntu/+source/unattended-upgrades/1.0ubuntu1 [3] https://code.launchpad.net/~rbalint/update-manager/remove-autoremovable-kernels/+merge/341599 -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: zstd compression for packages
Hi All, On Sat, Mar 17, 2018 at 3:09 PM, Dimitri John Ledkov wrote: > On 16 March 2018 at 22:13, Steve Langasek wrote: >> In other words: if we want to make this the default, we should quantify >> Daniel's remark that he would prefer a 6% faster download over a 10% faster >> unpack. >> > > Well, I think it does not make sense to think about this in absolute > terms. Thinking about user stories is better. > > A stable series user will be mostly upgrading packages from -security > and -updates. The download speed and/or size of debs does not matter > much in this case, as these are scheduled to be done in the background > over the course of the day, via unattended upgrades download timer. > Installation speed matters, as that is the window of time when the > system is actually somewhat in a maintenance mode / degraded > performance (apt is locked, there are CPU and disk-io loads). > > New instance initialization - e.g. spinning up a cloud instance, with > cloud-init, and installing a bunch of things; deploying juju charm / > conjure-up spell; configuring things with puppet / ansible / etc => > these are download & install heavy. However, users that do that > heavily, will be in a corporate / bussiness / datacentre environment > and thus it is reasonable to expect them to have either a fat internet > pipe, and/or a local mirror. Meaning download speed & size, are not > critical. > > Then there are devel series users, developers who do sbuild builds, > etc. These users are most likely to be on slower home-user connections > and watch things a lot more closely interactively, who indeed care > about the total download+install time. These users, are most likely > very vocal / visible, but are not ultimately the target audience as to > why we develop Ubuntu in the first place. Thus I would be willing to > trade personal developer/devel-series user experience, in favor of the > stable series user. I'm not sure how much it makes sense to > proxy/cache/local-mirror devel series, if it is only a single machine > in use. > > -- > Regards, > > Dimitri. > > ps. I fight for the user > pss. /me goes to setup a local mirror proxy cache, with dns spoofing > to make sure all my sbuilds / lxd containers / VM / cloud-images use > local mirror out of the box I agree with Dimitri's analysis and I'would also like to add one more thing to consider. During unpacking of packages the system is in a transient state where programs may not work correctly. Minimizing the time spent in that transient state is and important additional benefit of speeding up decompression. The speedup varies a lot across use cases and IMO the 10% speed increase is an understatement for many very important use cases. Cheers, Balint -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: zstd compression for packages
Hi Daniel, On Mon, Mar 12, 2018 at 2:11 PM, Daniel Axtens wrote: > Hi, > > I looked into compression algorithms a bit in a previous role, and to be > honest I'm quite surprised to see zstd proposed for package storage. zstd, > according to its own github repo, is "targeting real-time compression > scenarios". It's not really designed to be run at its maximum compression > level, it's designed to really quickly compress data coming off the wire - > things like compressing log files being streamed to a central server, or I > guess writing random data to btrfs where speed is absolutely an issue. > > Is speed of decompression a big user concern relative to file size? I admit > that I am biased - as an Australian and with the crummy internet that my > location entails, I'd save much more time if the file was 6% smaller and > took 10% longer to decompress than the other way around. Yes, decompression speed is a big issue in some cases. Please consider the case of provisioning cluoud/container instances, where after booting the image plenty of packages need to be installed and saving seconds matter a lot. Zstd format also allows parallel decompression which can make package installation even quicker in wall-clock time. Internet connection speed increases by ~50% (according to this [3] study which matches my experience) on average per year which is more than 6% for every two months. > > Did you consider Google's Brotli? We did consider it but it was less promising. Cheers, Balint [3] http://xahlee.info/comp/bandwidth.html > > Regards, > Daniel > > On Mon, Mar 12, 2018 at 9:58 PM, Julian Andres Klode > wrote: >> >> On Mon, Mar 12, 2018 at 11:06:11AM +0100, Julian Andres Klode wrote: >> > Hey folks, >> > >> > We had a coding day in Foundations last week and Balint and Julian added >> > support for zstd compression to dpkg [1] and apt [2]. >> > >> > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664 >> > [2] https://salsa.debian.org/apt-team/apt/merge_requests/8 >> > >> > Zstd is a compression algorithm developed by Facebook that offers far >> > higher decompression speeds than xz or even gzip (at roughly constant >> > speed and memory usage across all levels), while offering 19 compression >> > levels ranging from roughly comparable to gzip in size (but much faster) >> > to 19, which is roughly comparable to xz -6: >> > >> > In our configuration, we run zstd at level 19. For bionic main amd64, >> > this causes a size increase of about 6%, from roughly 5.6 to 5.9 GB. >> > Installs speed up by about 10%, or, if eatmydata is involved, by up to >> > 40% - user time generally by about 50%. >> > >> > Our implementations for apt and dpkg support multiple frames as used by >> > pzstd, so packages can be compressed and decompressed in parallel >> > eventually. >> >> More links: >> >> PPA: >> https://launchpad.net/~canonical-foundations/+archive/ubuntu/zstd-archive >> APT merge request: https://salsa.debian.org/apt-team/apt/merge_requests/8 >> dpkg patches: https://bugs.debian.org/892664 >> >> I'd also like to talk a bit more about libzstd itself: The package is >> currently in universe, but btrfs recently gained support for zstd, >> so we already have a copy in the kernel and we need to MIR it anyway >> for btrfs-progs. >> >> -- >> debian developer - deb.li/jak | jak-linux.org - free software dev >> ubuntu core developer i speak de, en >> >> -- -- Balint Reczey Ubuntu & Debian Developer -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
