Re: [OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys and wyrding timeouts largest firmwares
On 25.10.2015 13:14, Zoltan HERPAI wrote: Hannu Nyman wrote: wyrding timeouts the imagebuilder/SDK tar file creation step for the largest firmwares. I have noticed that as ar71xx has suffered from that several times in the last few weeks. The 20 minutes given for waiting for any output from the process is sometimes too short for the tar file creation step and the buildbot breaks the build. Apparently "tar" does not produce any intermediate output. (When that timeout happens, the file cleanup step may also take hours, so there may be something wrong with the buildslave's disk or I/O in general.) We're checking these, hopefully both will resolve with the next round of builds. Thanks, -w- Good to hear. One piece of additional information: I noticed today that also buildslave2 may timeout the SDK tar generation phase, so it might be that Openwrt has grown so large that the 20 min timeout is generally too short. http://buildbot.openwrt.org:8010/builders/ath25/builds/107 http://buildbot.openwrt.org:8010/builders/ath25/builds/107/steps/compile_8/logs/stdio * make[2]: Entering directory `/home/openwrt/slave/ath25/build/target/sdk' ... tar -xf - -C /home/openwrt/slave/ath25/build/build_dir/target-mips_mips32_musl-1.1.11/OpenWrt-SDK-ath25_gcc-4.8-linaro_musl-1.1.11.Linux-x86_64 command timed out: 1200 seconds without output, attempting to kill process killed by signal 9 ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys and wyrding timeouts largest firmwares
Hannu Nyman wrote: Buildbot needs some maintenance, as some of the buildslaves are behaving badly :-( 1) gb15 has a wrong host key and silently fails all uploads. It just completed 4 builds successfully, but did not upload them. That has been going on for some time now. It is rather sad that the buildbot script silently fails the binary uploads, because that makes the error hard to notice. The build shows green on the status screen, so unless you wonder about the missing binaries of a successful build and really check the upload step's log file, you won't notice the error. It would be great if the buildbot upload script could be modified to show the upload error more clearly. 2) wyrding timeouts the imagebuilder/SDK tar file creation step for the largest firmwares. I have noticed that as ar71xx has suffered from that several times in the last few weeks. The 20 minutes given for waiting for any output from the process is sometimes too short for the tar file creation step and the buildbot breaks the build. Apparently "tar" does not produce any intermediate output. (When that timeout happens, the file cleanup step may also take hours, so there may be something wrong with the buildslave's disk or I/O in general.) [snip] We're checking these, hopefully both will resolve with the next round of builds. Thanks, -w- ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
[OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys and wyrding timeouts largest firmwares
Buildbot needs some maintenance, as some of the buildslaves are behaving badly :-( 1) gb15 has a wrong host key and silently fails all uploads. It just completed 4 builds successfully, but did not upload them. That has been going on for some time now. Build status green, but upload has failed: http://buildbot.openwrt.org:8010/builders/brcm47xx/builds/123 http://buildbot.openwrt.org:8010/builders/brcm47xx/builds/123/steps/shell_11/logs/stdio http://buildbot.openwrt.org:8010/builders/x86.kvm_guest/builds/113/steps/shell_11/logs/stdio Upload Snapshot to Openwrt Host key verification failed. rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9] Host key verification failed. Somebody should fix the ssh host key (or whatever the rdate uses at the update step). It is rather sad that the buildbot script silently fails the binary uploads, because that makes the error hard to notice. The build shows green on the status screen, so unless you wonder about the missing binaries of a successful build and really check the upload step's log file, you won't notice the error. It would be great if the buildbot upload script could be modified to show the upload error more clearly. 2) wyrding timeouts the imagebuilder/SDK tar file creation step for the largest firmwares. I have noticed that as ar71xx has suffered from that several times in the last few weeks. The 20 minutes given for waiting for any output from the process is sometimes too short for the tar file creation step and the buildbot breaks the build. Apparently "tar" does not produce any intermediate output. (When that timeout happens, the file cleanup step may also take hours, so there may be something wrong with the buildslave's disk or I/O in general.) Example: http://buildbot.openwrt.org:8010/builders/ar71xx/builds/103 http://buildbot.openwrt.org:8010/builders/ar71xx/builds/103/steps/compile_8/logs/stdio http://buildbot.openwrt.org:8010/builders/ar71xx/builds/97/steps/compile_8/logs/stdio The cleanup took 1 h 56 min build build 97: shell_15 './cleanup.sh wyrding ...' ( 1 hrs, 56 mins, 24 secs ) ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
