Paul, you suggest to use different configurations for doing the "build the kernel" validation and the benchmarking? Why wouldn't we run in the same heat issues while benchmarking?
Can we somehow put the SoC into a very low power state for 2 minutes for cool down and simply wait before starting the build/test? Anyway, I assume that the strict environment requirement where toolchain WG needs to sign off on the setup would mostly apply to benchmarking only and that we could probably choose any stable image for doing the build validation of the toolchain that is rock solid. Matt? If so, it sounds sensible to just pick a recent release with thermal enabled for the build job and use the special configuration for the benchmarking parts - maybe with a cooling step as above. On Mon, Apr 8, 2013 at 7:56 PM, Paul Sokolovsky <[email protected]>wrote: > Hello, > > On Sat, 6 Apr 2013 15:40:15 +0300 > Paul Sokolovsky <[email protected]> wrote: > > [] > > > Ok, so here're these 2 builds: > > > > gcc-4.8~svn196132 > > > > panda-es02 > > https://validation.linaro.org/lava-server/scheduler/job/50993 > > > > panda-es05 > > https://validation.linaro.org/lava-server/scheduler/job/50994 > > > > #50993 went well. > > > > #50994 midway in compilation started to get invalid data (grep for > > "is not valid in preprocessor expressions"), then got kernel fault, > > then got caught in reboot loop, apparently due to: > > > > [ 6.631256] thermal_init_thermal_state: Getting initial temp for > > cpu domain [ 6.638702] thermal_request_temp > > [ 6.642150] omap_fatal_zone:FATAL ZONE (hot spot temp: 128490) > > > > - all these behaviors were seen by me before (actually, previously, I > > didn't see such explicit messages from kernel that it's a thermal > > faults). > > > > So, CBuild/LAVA can do (successful) builds, but some builds fail due > > to thermal issues. Actually, let me load up all boards with the same > > build now, towards assessing thermal failure rate more scientifically. > > Well, let's count: > > > https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/9dea0c78604ce5e65178ec8d71edac8d8a499c4e/ > OK > > > https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/cf346215160e778be13e031378b36328eeb315c3/ > Failed, thermal (see above) > > > https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/bce0a440e5cece819678e6fd7f276a43c2a44ecd/ > "../../../gcc-4.8~svn196132/gcc/gengtype.c:4106:39: error: cannot > convert 'flisT*' to 'flist*' in assignment" flipped bit in file (so > that before, including on local Panda), then other random failures. > > > https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/6c31d8e21dbc716255c7ec13702b5252434f9ced/ > "rsync: getaddrinfo: toolchain64 2000: Temporary failure in name > resolution" - network flip > > > https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/de94c85961e59dbaba41f8dec6c20f4740384241/ > OK > > > https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/c032912d494e80857b931ab8c2dafc4e4f83c762/ > Lot of "malloc: ../bash/jobs.c:743: assertion botched" > > > https://validation.linaro.org/lava-server/dashboard/streams/anonymous/cbuild/bundles/d36badcd7310aa0db4a1efcd1554d11da06ef6bc/ > Reboot during configure, then reboot cycle. > > So, out of 7 builds, only 2 were successful, the rest fail due to > thermal issues (apparently, all but the one with network issues). We > can try to consider why LAVA-based builds have such low yield rate > comparing to native Cbuild builds (one explanation is that LAVA does > pretty heavy lifting to install OS, etc., so when build starts, CPU is > already pretty hot), but it's clear that using kernel with > voltage/frequency scaling disabled simply doesn't work that well for > *builds* (vs benchmarks). > > So, we can consider preparing OS image which uses the same basic OS, > but normal kernel, and use that for builds. Would TCWG be able to > prepare such image? If not, we can add it to the list of tasks for (now > combined) LAVA/Infra team, but with all the other tasks we have, it > may take some time to get to. > > > -- > Best Regards, > Paul > > Linaro.org | Open source software for ARM SoCs > Follow Linaro: http://www.facebook.com/pages/Linaro > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog > -- Alexander Sack Director, Linaro Platform Engineering http://www.linaro.org | Open source software for ARM SoCs http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog
_______________________________________________ linaro-validation mailing list [email protected] http://lists.linaro.org/mailman/listinfo/linaro-validation
