Re: [OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys

[Zoltan HERPAI]

Hannu A Nyman wrote:
Well, both gb15 and buildslave2 still fail to upload binaries, and as 
they are making 9 of the 22 concurrent builds, almost half of the 
compiled binaries are discarded. Sad.


Logs from today:
http://buildbot.openwrt.org:8010/builders/x86.kvm_guest/builds/119/steps/shell_11/logs/stdio 

http://buildbot.openwrt.org:8010/builders/brcm2708/builds/126/steps/shell_11/logs/stdio 



 Upload Snapshot to Openwrt
 Host key verification failed.
 rsync: connection unexpectedly closed (0 bytes received so far) [sender]
 rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]

gb15 is fixed now.

Regards,
-w-
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys

[Zoltan HERPAI]

On Mon, 23 Nov 2015, Dave Taht wrote:


On Mon, Nov 23, 2015 at 11:07 AM, Zoltan HERPAI  wrote:

Hannu A Nyman wrote:


Well, both gb15 and buildslave2 still fail to upload binaries, and as they
are making 9 of the 22 concurrent builds, almost half of the compiled
binaries are discarded. Sad.

Logs from today:

http://buildbot.openwrt.org:8010/builders/x86.kvm_guest/builds/119/steps/shell_11/logs/stdio

http://buildbot.openwrt.org:8010/builders/brcm2708/builds/126/steps/shell_11/logs/stdio

 Upload Snapshot to Openwrt
 Host key verification failed.
 rsync: connection unexpectedly closed (0 bytes received so far) [sender]
 rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]


gb15 is fixed now.


I note that I have long been paying for the google openwrt build
cluster out of my own pocket (after the google grant expired last year
- which was costing 1700/month for 5? 6? servers), but I had cut it
down to just gb10, which is cost me about $250/month.

I am hoping some more grant money arrives (not a lot!) next year.

I was unaware of gb15 (not coming out of my account, thankfully), and
am glad someone else is covering costs on it...

IF there is a need to turn around openwrt builds even quicker, (how
much quicker can it be done?) we can re-scale up my old google cluster
if there is funding for it from somewhere. I can keep the one server
I'm contributing up and running indefinitely. Or... ?

I have always wished we could do a full build in under 3 hours, and
could circulate package changes around in minutes, rather than in a
full build.


This is not a googlebuild server. I've asked Travis to rename the host to 
make the name less confusing, that's yet to happen.


Thanks,
-w-
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys

[Dave Taht]

On Mon, Nov 23, 2015 at 11:07 AM, Zoltan HERPAI  wrote:
> Hannu A Nyman wrote:
>>
>> Well, both gb15 and buildslave2 still fail to upload binaries, and as they
>> are making 9 of the 22 concurrent builds, almost half of the compiled
>> binaries are discarded. Sad.
>>
>> Logs from today:
>>
>> http://buildbot.openwrt.org:8010/builders/x86.kvm_guest/builds/119/steps/shell_11/logs/stdio
>>
>> http://buildbot.openwrt.org:8010/builders/brcm2708/builds/126/steps/shell_11/logs/stdio
>>
>>  Upload Snapshot to Openwrt
>>  Host key verification failed.
>>  rsync: connection unexpectedly closed (0 bytes received so far) [sender]
>>  rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]
>
> gb15 is fixed now.

I note that I have long been paying for the google openwrt build
cluster out of my own pocket (after the google grant expired last year
- which was costing 1700/month for 5? 6? servers), but I had cut it
down to just gb10, which is cost me about $250/month.

I am hoping some more grant money arrives (not a lot!) next year.

I was unaware of gb15 (not coming out of my account, thankfully), and
am glad someone else is covering costs on it...

IF there is a need to turn around openwrt builds even quicker, (how
much quicker can it be done?) we can re-scale up my old google cluster
if there is funding for it from somewhere. I can keep the one server
I'm contributing up and running indefinitely. Or... ?

I have always wished we could do a full build in under 3 hours, and
could circulate package changes around in minutes, rather than in a
full build.



>
> Regards,
>
> -w-
> ___
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys

[Hannu A Nyman]

Well, both gb15 and buildslave2 still fail to upload binaries, and as they 
are making 9 of the 22 concurrent builds, almost half of the compiled 
binaries are discarded. Sad.


Logs from today:
http://buildbot.openwrt.org:8010/builders/x86.kvm_guest/builds/119/steps/shell_11/logs/stdio
http://buildbot.openwrt.org:8010/builders/brcm2708/builds/126/steps/shell_11/logs/stdio

 Upload Snapshot to Openwrt
 Host key verification failed.
 rsync: connection unexpectedly closed (0 bytes received so far) [sender]
 rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]


On 25.10.2015 13:14, Zoltan HERPAI wrote:

Hannu Nyman wrote:
Buildbot needs some maintenance, as some of the buildslaves are behaving 
badly :-(


1)
gb15 has a wrong host key and silently fails all uploads. It just 
completed 4 builds successfully, but did not upload them. That has been 
going on for some time now.
It is rather sad that the buildbot script silently fails the binary 
uploads, because that makes the error hard to notice. The build shows 
green on the status screen, so unless you wonder about the missing 
binaries of a successful build and really check the upload step's log 
file, you won't notice the error. It would be great if the buildbot upload 
script could be modified to show the upload error more clearly.




We're checking these, hopefully both will resolve with the next round of 
builds.


Thanks,
-w-



___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys and wyrding timeouts largest firmwares

[Zoltan HERPAI]

Hannu Nyman wrote:
Buildbot needs some maintenance, as some of the buildslaves are 
behaving badly :-(


1)
gb15 has a wrong host key and silently fails all uploads. It just 
completed 4 builds successfully, but did not upload them. That has 
been going on for some time now.
It is rather sad that the buildbot script silently fails the binary 
uploads, because that makes the error hard to notice. The build shows 
green on the status screen, so unless you wonder about the missing 
binaries of a successful build and really check the upload step's log 
file, you won't notice the error. It would be great if the buildbot 
upload script could be modified to show the upload error more clearly.


2)
wyrding timeouts the imagebuilder/SDK tar file creation step for the 
largest firmwares. I have noticed that as ar71xx has suffered from 
that several times in the last few weeks. The 20 minutes given for 
waiting for any output from the process is sometimes too short for the 
tar file creation step and the buildbot breaks the build. Apparently 
"tar" does not produce any intermediate output.  (When that timeout 
happens, the file cleanup step may also take hours, so there may be 
something wrong with the buildslave's disk or I/O in general.)

[snip]

We're checking these, hopefully both will resolve with the next round of 
builds.


Thanks,
-w-
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

Re: [OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys and wyrding timeouts largest firmwares

[Hannu Nyman]

On 25.10.2015 13:14, Zoltan HERPAI wrote:

Hannu Nyman wrote:
wyrding timeouts the imagebuilder/SDK tar file creation step for the 
largest firmwares. I have noticed that as ar71xx has suffered from that 
several times in the last few weeks. The 20 minutes given for waiting for 
any output from the process is sometimes too short for the tar file 
creation step and the buildbot breaks the build. Apparently "tar" does not 
produce any intermediate output.  (When that timeout happens, the file 
cleanup step may also take hours, so there may be something wrong with the 
buildslave's disk or I/O in general.)


We're checking these, hopefully both will resolve with the next round of 
builds.


Thanks,
-w-



Good to hear.

One piece of additional information:
I noticed today that also buildslave2 may timeout the SDK tar generation 
phase, so it might be that Openwrt has grown so large that the 20 min timeout 
is generally too short.


http://buildbot.openwrt.org:8010/builders/ath25/builds/107
http://buildbot.openwrt.org:8010/builders/ath25/builds/107/steps/compile_8/logs/stdio

*

make[2]: Entering directory `/home/openwrt/slave/ath25/build/target/sdk'
...
tar -xf - -C 
/home/openwrt/slave/ath25/build/build_dir/target-mips_mips32_musl-1.1.11/OpenWrt-SDK-ath25_gcc-4.8-linaro_musl-1.1.11.Linux-x86_64


command timed out: 1200 seconds without output, attempting to kill
process killed by signal 9
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel

[OpenWrt-Devel] buildbot: gb15 fails to upload due to host keys and wyrding timeouts largest firmwares

[Hannu Nyman]

Buildbot needs some maintenance, as some of the buildslaves are behaving 
badly :-(


1)
gb15 has a wrong host key and silently fails all uploads. It just completed 4 
builds successfully, but did not upload them. That has been going on for some 
time now.


Build status green, but upload has failed:
http://buildbot.openwrt.org:8010/builders/brcm47xx/builds/123
http://buildbot.openwrt.org:8010/builders/brcm47xx/builds/123/steps/shell_11/logs/stdio
http://buildbot.openwrt.org:8010/builders/x86.kvm_guest/builds/113/steps/shell_11/logs/stdio

  Upload Snapshot to Openwrt
  Host key verification failed.
  rsync: connection unexpectedly closed (0 bytes received so far) [sender]
  rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]
  Host key verification failed.

Somebody should fix the ssh host key (or whatever the rdate uses at the 
update step).


It is rather sad that the buildbot script silently fails the binary uploads, 
because that makes the error hard to notice. The build shows green on the 
status screen, so unless you wonder about the missing binaries of a 
successful build and really check the upload step's log file, you won't 
notice the error. It would be great if the buildbot upload script could be 
modified to show the upload error more clearly.


2)
wyrding timeouts the imagebuilder/SDK tar file creation step for the largest 
firmwares. I have noticed that as ar71xx has suffered from that several times 
in the last few weeks. The 20 minutes given for waiting for any output from 
the process is sometimes too short for the tar file creation step and the 
buildbot breaks the build. Apparently "tar" does not produce any intermediate 
output.  (When that timeout happens, the file cleanup step may also take 
hours, so there may be something wrong with the buildslave's disk or I/O in 
general.)


Example:
http://buildbot.openwrt.org:8010/builders/ar71xx/builds/103
http://buildbot.openwrt.org:8010/builders/ar71xx/builds/103/steps/compile_8/logs/stdio
http://buildbot.openwrt.org:8010/builders/ar71xx/builds/97/steps/compile_8/logs/stdio
 The cleanup took 1 h 56 min build build 97:
  shell_15 './cleanup.sh wyrding ...' ( 1 hrs, 56 mins, 24 secs )
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel