Processed: Re: Bug#982598: incomplete logs for autopkg tests
Processing control commands: > reassign -1 debci Bug #982598 [release.debian.org] incomplete logs for autopkg tests Bug reassigned from package 'release.debian.org' to 'debci'. Ignoring request to alter found versions of bug #982598 to the same values previously set Ignoring request to alter fixed versions of bug #982598 to the same values previously set > retitle -1 trunkate huge logs at the start instead of the end Bug #982598 [debci] incomplete logs for autopkg tests Changed Bug title to 'trunkate huge logs at the start instead of the end' from 'incomplete logs for autopkg tests'. -- 982598: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982598 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#982598: incomplete logs for autopkg tests
Control: reassign -1 debci Control: retitle -1 trunkate huge logs at the start instead of the end Hi, On Fri, 12 Feb 2021 20:35:11 +0100 Paul Gevers wrote: > reassign -1 debci Oops. > Hi Matthias, > > On 12-02-2021 10:54, Matthias Klose wrote: > > Package: release.debian.org > > It's not the release team that runs ci.debian.net. Reassigning > appropriately. > > > As seen with glibc autopkg tests [1], the Debian CI infrastructure doesn't > > store > > complete build logs, cutting these to 20MB (uncompressed), resulting in > > ~450k > > compressed logs. This might not be important for successful tests, but it > > doesn't provide any information for failed tests, as the summary at the end > > is > > always cut. Looking at the Ubuntu CI testers, you see that the glibc log > > for a > > successful test can reach 150MB, compressed 3.5GB [2]. With a glibc patch > > to > > not stop on test failures with the first pass [3], logs for failed tests can > > also reach that size. > > > > The outcome of tests is used by the release team to make decisions about the > > upcoming release [4]. Pointing out to a failed log which doesn't provide > > useful > > information is not helpful, and does cost volunteer time to analyze. In > > this > > case it turned out to be the flaky test infrastructure, and retries > > resulted in > > a successful test. Good luck with that for a real regression ... > > > > As pointed out in another context, "machine time is cheap, volunteer time is > > not" [5], as well as machine storage is cheap compared to volunteer time, so > > please stop cutting the autopkg test logs. > > Unfortunately, we're hitting infrastructure issues if we don't cap the > logs [1]. However, I think we should save the last part (and not the > first part) if we have to cap, because normally the failure happens in > the end. > > Paul > > [1] > https://salsa.debian.org/ci-team/debci/-/commit/9240d93a3e8a017c303d4d1604808fbc1d0f > OpenPGP_signature Description: OpenPGP digital signature
Bug#982598: incomplete logs for autopkg tests
On Fri, Feb 12, 2021 at 08:35:11PM +0100, Paul Gevers wrote: > Unfortunately, we're hitting infrastructure issues if we don't cap the > logs [1]. However, I think we should save the last part (and not the > first part) if we have to cap, because normally the failure happens in > the end. I'd keep a bit from the beginning and a bigger bit from the end. Often the beginning is useful too. -- cheers, Holger ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C ⠈⠳⣄ signature.asc Description: PGP signature
Bug#982598: incomplete logs for autopkg tests
reassign -1 debci Hi Matthias, On 12-02-2021 10:54, Matthias Klose wrote: > Package: release.debian.org It's not the release team that runs ci.debian.net. Reassigning appropriately. > As seen with glibc autopkg tests [1], the Debian CI infrastructure doesn't > store > complete build logs, cutting these to 20MB (uncompressed), resulting in ~450k > compressed logs. This might not be important for successful tests, but it > doesn't provide any information for failed tests, as the summary at the end is > always cut. Looking at the Ubuntu CI testers, you see that the glibc log for > a > successful test can reach 150MB, compressed 3.5GB [2]. With a glibc patch to > not stop on test failures with the first pass [3], logs for failed tests can > also reach that size. > > The outcome of tests is used by the release team to make decisions about the > upcoming release [4]. Pointing out to a failed log which doesn't provide > useful > information is not helpful, and does cost volunteer time to analyze. In this > case it turned out to be the flaky test infrastructure, and retries resulted > in > a successful test. Good luck with that for a real regression ... > > As pointed out in another context, "machine time is cheap, volunteer time is > not" [5], as well as machine storage is cheap compared to volunteer time, so > please stop cutting the autopkg test logs. Unfortunately, we're hitting infrastructure issues if we don't cap the logs [1]. However, I think we should save the last part (and not the first part) if we have to cap, because normally the failure happens in the end. Paul [1] https://salsa.debian.org/ci-team/debci/-/commit/9240d93a3e8a017c303d4d1604808fbc1d0f OpenPGP_signature Description: OpenPGP digital signature
Bug#982598: incomplete logs for autopkg tests
Package: release.debian.org Severity: important Tags: sid bullseye As seen with glibc autopkg tests [1], the Debian CI infrastructure doesn't store complete build logs, cutting these to 20MB (uncompressed), resulting in ~450k compressed logs. This might not be important for successful tests, but it doesn't provide any information for failed tests, as the summary at the end is always cut. Looking at the Ubuntu CI testers, you see that the glibc log for a successful test can reach 150MB, compressed 3.5GB [2]. With a glibc patch to not stop on test failures with the first pass [3], logs for failed tests can also reach that size. The outcome of tests is used by the release team to make decisions about the upcoming release [4]. Pointing out to a failed log which doesn't provide useful information is not helpful, and does cost volunteer time to analyze. In this case it turned out to be the flaky test infrastructure, and retries resulted in a successful test. Good luck with that for a real regression ... As pointed out in another context, "machine time is cheap, volunteer time is not" [5], as well as machine storage is cheap compared to volunteer time, so please stop cutting the autopkg test logs. Matthias [1] https://ci.debian.net/packages/g/glibc/unstable/amd64/ [2] https://autopkgtest.ubuntu.com/packages/g/glibc/hirsute/amd64 [3] https://bugs.debian.org/982360 [4] https://lists.debian.org/debian-release/2021/02/msg6.html [5] https://lists.debian.org/debian-devel/2021/02/msg00175.html
