On Mon, 2024-10-14 at 14:46 +0100, Richard Purdie via lists.openembedded.org wrote: > It was suggested I should write down a list of other issues we have > on > the autobuilder right now. Some of these are related to the > transition > to new infrastructure, some are previously seen issues occurring more > frequently, some are entirely new. I appreciate these should all have > bugs. I'm copying various people including those helping with SWAT so > we can transition those we don't get solved into bugzilla. > > a) insane do_package_qa intermittent failures > --------------------------------------------- > > Two levels of probems. The cachedpath code returns False instead of > exceptions so there was unexpected API changes. The recent insane > changes also stopped handling DEBIAN files correctly leading to > races. > This is perhaps the best backtrace we have: > > https://valkyrie.yoctoproject.org//#/builders/43/builds/246 > > Who/Plan: RP has a partial revert for the first part to fix builds. > Ross is aware and working on the second.
I tested Ross' patch for this and unfortunate it caused a new problem to appear. I accidentally triggered two a-full builds at once, there was also a scarthgap build running at the time. The patch from Ross caused all the do_package_qa tasks to rerun at once in parallel which caused writes to the hash equivalence server. This caused widespread timeout errors and took out all the builds including the scarthgap one in parallel. https://valkyrie.yoctoproject.org/#/builders/4/builds/257 https://valkyrie.yoctoproject.org/#/builders/2/builds/253 https://valkyrie.yoctoproject.org/#/builders/89/builds/236/steps/16/logs/stdio https://valkyrie.yoctoproject.org/#/builders/40/builds/242/steps/11/logs/stdio and many more. Joshua: Is there a performance issue with many writes in parallel to hashequiv? The stats info off the server was: "requests": { "average": 0.042532518429623176, "max_time": 3.0534732329833787, "num": 73325629, "stdev": 0.06959859907598737, "total_time": 3118723.6668062117 } Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#205803): https://lists.openembedded.org/g/openembedded-core/message/205803 Mute This Topic: https://lists.openembedded.org/mt/108984574/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
