Your message dated Tue, 13 Jun 2017 13:51:02 +0000
with message-id <[email protected]>
and subject line Re: [Qa-jenkins-dev] Bug#864686: reproducible: reduce number
of armhf builds per node
has caused the Debian Bug report #864686,
regarding reproducible: reduce number of armhf builds per node
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [email protected]
immediately.)
--
864686: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864686
Debian Bug Tracking System
Contact [email protected] with problems
--- Begin Message ---
Package: jenkins.debian.org
Severity: normal
Please consider merging the following patch, available in the
"armhf-reduced-builds" branch:
https://anonscm.debian.org/cgit/users/vagrant/jenkins.debian.net.git/
This may help reduce the impacts of Out-of-Memory errors on nodes when
building many packages simultaneously, though will likely result in a
slower build network...
live well,
vagrant
commit 314becd4473586e1fe47cb05477276aef9630a87
Author: Vagrant Cascadian <[email protected]>
Date: Mon Jun 12 13:08:44 2017 -0700
reproducible: Reduce number of armhf build jobs to 2 for 1gb of ram, 4
for 2gb of ram, and 6 for 4gb of ram, for a total of 51 jobs. Remove
disabled jobs for odc2a.
diff --git a/README b/README
index 02854897..ded67155 100644
--- a/README
+++ b/README
@@ -135,7 +135,7 @@ Installation tests inside chroot environments.
* The (current) purpose of https://tests.reproducible-builds.org is to show
the potential of reproducible builds for
Debian - and six other projects currently. This is research, showing what could
(and should) be done... check
https://wiki.debian.org/ReproducibleBuilds for the real status of the project
for Debian!
-* For Debian, three suites, 'testing', 'unstable' and 'experimental', are
tested on four architectures: 'amd64',
'i386', 'arm64' and 'armhf'. The tests are done using 'pbuilder' through
several concurrent workers: 40 for 'amd64',
24 for 'i386', 32 for 'arm64' and 76 for 'armhf', which are each constantly
testing packages and saving the results
of these tests. There's a single
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build_service.sh[systemd
service]
starting all of these
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_worker.sh[workers]
which in turn
launch the actual
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build.sh[build
script]. (So the actual builds and tests are happening outside the jenkins
service.)
+* For Debian, three suites, 'testing', 'unstable' and 'experimental', are
tested on four architectures: 'amd64',
'i386', 'arm64' and 'armhf'. The tests are done using 'pbuilder' through
several concurrent workers: 40 for 'amd64',
24 for 'i386', 32 for 'arm64' and 51 for 'armhf', which are each constantly
testing packages and saving the results
of these tests. There's a single
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build_service.sh[systemd
service]
starting all of these
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_worker.sh[workers]
which in turn
launch the actual
link:https://anonscm.debian.org/git/qa/jenkins.debian.net.git/tree/bin/reproducible_build.sh[build
script]. (So the actual builds and tests are happening outside the jenkins
service.)
* These builds on remote nodes run on very different hardware:
** for 'amd64' we are using four virtual machines,
profitbricks-build(1+5+11+15)-amd64, which have 15 or 16 cores
and 48gb ram each. These nodes are sponsored by
link:https://jenkins.debian.net/userContent/thanks.html[Profitbricks].
diff --git a/bin/reproducible_build_service.sh
b/bin/reproducible_build_service.sh
index 79bf24ee..500f15a6 100755
--- a/bin/reproducible_build_service.sh
+++ b/bin/reproducible_build_service.sh
@@ -116,93 +116,68 @@ choose_nodes() {
arm64_31) NODE1=codethink-sled14-arm64
NODE2=codethink-sled11-arm64 ;;
arm64_32) NODE1=codethink-sled13-arm64
NODE2=codethink-sled14-arm64 ;;
# to choose new armhf jobs:
- # for i in cb3a hb0 rpi2b rpi2c wbd0 bpi0 bbx15
cbxi4pro0 ff2a ff2b ff64a jtk1a odc2a odxu4
odxu4b odxu4c odu3a opi2a opi2b opi2c p64b p64c wbq0 cbxi4a cbxi4b ff4a
jtx1a ; do echo "$i: " ; grep NODE1
bin/reproducible_build_service.sh|grep armhf|grep $i-armhf ; done
- # 8 jobs for quad-cores with 4 gb ram
- # 6 jobs for octo-cores with 2 gb ram
- # 6 jobs for hexa-cores with 2 gb ram
- # 6 jobs for quad-cores with 2 gb ram
- # 3 jobs for dual-cores with 1 gb ram
- # 3 jobs for quad-cores with 1 gb ram
+ # for i in cb3a hb0 rpi2b rpi2c wbd0 bpi0 bbx15 cbxi4pro0 ff2a
ff2b ff64a jtk1a odxu4 odxu4b odxu4c
odu3a opi2a opi2b opi2c p64b p64c wbq0 cbxi4a cbxi4b ff4a jtx1a ; do echo "$i:
" ; grep NODE1
bin/reproducible_build_service.sh|grep armhf|grep $i-armhf ; done
+ # 6 jobs for quad-cores with 4 gb ram
+ # 4 jobs for octo-cores with 2 gb ram
+ # 4 jobs for hexa-cores with 2 gb ram
+ # 4 jobs for quad-cores with 2 gb ram
+ # 4 jobs for dual-cores with 2 gb ram
+ # 2 jobs for dual-cores with 1 gb ram
+ # 2 jobs for quad-cores with 1 gb ram
#
# Don't forget to update README with the number of builders…!
#
- armhf_1) NODE1=bbx15-armhf-rb
NODE2=odxu4-armhf-rb ;;
- armhf_2) NODE1=wbq0-armhf-rb
NODE2=p64c-armhf-rb ;;
- armhf_3) NODE1=hb0-armhf-rb
NODE2=p64b-armhf-rb ;;
- armhf_4) NODE1=ff4a-armhf-rb
NODE2=wbq0-armhf-rb ;;
- armhf_5) NODE1=cbxi4pro0-armhf-rb
NODE2=bpi0-armhf-rb ;;
- armhf_6) NODE1=ff4a-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
- armhf_7) NODE1=wbq0-armhf-rb
NODE2=odxu4-armhf-rb ;;
- armhf_8) NODE1=hb0-armhf-rb
NODE2=wbq0-armhf-rb ;;
- armhf_9) NODE1=ff4a-armhf-rb
NODE2=bpi0-armhf-rb ;;
- armhf_10) NODE1=odxu4-armhf-rb
NODE2=rpi2b-armhf-rb ;;
- armhf_11) NODE1=odxu4-armhf-rb
NODE2=wbd0-armhf-rb ;;
- armhf_12) NODE1=wbd0-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
- armhf_13) NODE1=cbxi4pro0-armhf-rb
NODE2=rpi2b-armhf-rb ;;
- armhf_14) NODE1=cbxi4a-armhf-rb
NODE2=odxu4b-armhf-rb ;;
- armhf_15) NODE1=rpi2b-armhf-rb
NODE2=odxu4c-armhf-rb ;;
- armhf_16) NODE1=odxu4b-armhf-rb
NODE2=wbd0-armhf-rb ;;
- armhf_17) NODE1=odxu4c-armhf-rb
NODE2=hb0-armhf-rb ;;
- armhf_18) NODE1=odxu4b-armhf-rb
NODE2=odu3a-armhf-rb ;;
- armhf_19) NODE1=odxu4c-armhf-rb
NODE2=opi2c-armhf-rb ;;
- armhf_20) NODE1=opi2b-armhf-rb
NODE2=odxu4b-armhf-rb ;;
- armhf_21) NODE1=ff2a-armhf-rb
NODE2=odxu4c-armhf-rb ;;
- armhf_22) NODE1=ff2a-armhf-rb
NODE2=rpi2c-armhf-rb ;;
- armhf_23) NODE1=rpi2c-armhf-rb
NODE2=odxu4b-armhf-rb ;;
- armhf_24) NODE1=rpi2c-armhf-rb
NODE2=odxu4c-armhf-rb ;;
- armhf_25) NODE1=odxu4b-armhf-rb
NODE2=ff2b-armhf-rb ;;
- armhf_26) NODE1=jtk1a-armhf-rb
NODE2=ff2a-armhf-rb ;;
- armhf_27) NODE1=odxu4c-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
- armhf_28) NODE1=jtk1a-armhf-rb
NODE2=ff2b-armhf-rb ;;
- armhf_29) NODE1=ff2b-armhf-rb
NODE2=jtk1a-armhf-rb ;;
- armhf_30) NODE1=ff2b-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
- armhf_31) NODE1=ff2b-armhf-rb
NODE2=opi2b-armhf-rb ;;
- armhf_32) NODE1=jtk1a-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
- armhf_33) NODE1=ff2a-armhf-rb
NODE2=opi2b-armhf-rb ;;
- armhf_34) NODE1=cbxi4a-armhf-rb
NODE2=opi2b-armhf-rb ;;
- armhf_35) NODE1=opi2a-armhf-rb
NODE2=ff2b-armhf-rb ;;
- armhf_36) NODE1=opi2a-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
- armhf_37) NODE1=opi2a-armhf-rb
NODE2=wbq0-armhf-rb ;;
- armhf_38) NODE1=cbxi4b-armhf-rb
NODE2=jtk1a-armhf-rb ;;
- armhf_39) NODE1=cbxi4b-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
- armhf_40) NODE1=opi2b-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
- armhf_41) NODE1=opi2b-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
- armhf_42) NODE1=cbxi4b-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
- armhf_43) NODE1=cbxi4a-armhf-rb
NODE2=opi2c-armhf-rb ;;
- armhf_44) NODE1=bbx15-armhf-rb
NODE2=ff4a-armhf-rb ;;
- armhf_45) NODE1=ff4a-armhf-rb
NODE2=p64b-armhf-rb ;;
- armhf_46) NODE1=wbq0-armhf-rb
NODE2=bbx15-armhf-rb ;;
- armhf_47) NODE1=cbxi4pro0-armhf-rb
NODE2=bbx15-armhf-rb ;;
- armhf_48) NODE1=bbx15-armhf-rb
NODE2=p64c-armhf-rb ;;
- armhf_49) NODE1=bpi0-armhf-rb
NODE2=ff4a-armhf-rb ;;
- armhf_50) NODE1=odxu4-armhf-rb
NODE2=odu3a-armhf-rb ;;
- armhf_51) NODE1=odu3a-armhf-rb
NODE2=cb3a-armhf-rb ;;
- armhf_52) NODE1=opi2c-armhf-rb
NODE2=cb3a-armhf-rb ;;
- armhf_53) NODE1=cb3a-armhf-rb
NODE2=ff4a-armhf-rb ;;
- armhf_54) NODE1=odu3a-armhf-rb
NODE2=opi2c-armhf-rb ;;
- armhf_55) NODE1=opi2c-armhf-rb
NODE2=odu3a-armhf-rb ;;
- armhf_56) NODE1=odu3a-armhf-rb
NODE2=ff2a-armhf-rb ;;
- armhf_57) NODE1=opi2c-armhf-rb
NODE2=ff64a-armhf-rb ;;
- armhf_58) NODE1=cbxi4a-armhf-rb
NODE2=ff64a-armhf-rb ;;
- armhf_59) NODE1=jtk1a-armhf-rb
NODE2=ff64a-armhf-rb ;;
- armhf_60) NODE1=cbxi4b-armhf-rb
NODE2=jtx1a-armhf-rb ;;
- armhf_61) NODE1=p64c-armhf-rb
NODE2=jtx1a-armhf-rb ;;
- armhf_62) NODE1=p64b-armhf-rb
NODE2=jtx1a-armhf-rb ;;
- armhf_63) NODE1=p64b-armhf-rb
NODE2=jtx1a-armhf-rb ;;
-# armhf_64) NODE1=p64c-armhf-rb
NODE2=odc2a-armhf-rb ;;
-# armhf_65) NODE1=p64b-armhf-rb
NODE2=odc2a-armhf-rb ;;
-# armhf_66) NODE1=p64c-armhf-rb
NODE2=odc2a-armhf-rb ;;
- armhf_67) NODE1=ff64a-armhf-rb
NODE2=ff2a-armhf-rb ;;
- armhf_68) NODE1=ff64a-armhf-rb
NODE2=p64b-armhf-rb ;;
- armhf_69) NODE1=ff64a-armhf-rb
NODE2=p64c-armhf-rb ;;
- armhf_70) NODE1=jtx1a-armhf-rb
NODE2=opi2a-armhf-rb ;;
- armhf_71) NODE1=jtx1a-armhf-rb
NODE2=opi2a-armhf-rb ;;
- armhf_72) NODE1=jtx1a-armhf-rb
NODE2=opi2a-armhf-rb ;;
- armhf_73) NODE1=jtx1a-armhf-rb
NODE2=ff4a-armhf-rb ;;
-# armhf_74) NODE1=odc2a-armhf-rb
NODE2=bbx15-armhf-rb ;;
-# armhf_75) NODE1=odc2a-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
-# armhf_76) NODE1=odc2a-armhf-rb
NODE2=odxu4-armhf-rb ;;
-
+ armhf_1) NODE1=bbx15-armhf-rb
NODE2=jtx1a-armhf-rb ;;
+ armhf_2) NODE1=bbx15-armhf-rb
NODE2=rpi2b-armhf-rb ;;
+ armhf_3) NODE1=cb3a-armhf-rb
NODE2=bbx15-armhf-rb ;;
+ armhf_4) NODE1=cb3a-armhf-rb
NODE2=bbx15-armhf-rb ;;
+ armhf_5) NODE1=hb0-armhf-rb
NODE2=cb3a-armhf-rb ;;
+ armhf_6) NODE1=rpi2b-armhf-rb
NODE2=cb3a-armhf-rb ;;
+ armhf_7) NODE1=rpi2c-armhf-rb
NODE2=wbd0-armhf-rb ;;
+ armhf_8) NODE1=wbd0-armhf-rb
NODE2=rpi2c-armhf-rb ;;
+ armhf_9) NODE1=bpi0-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
+ armhf_10) NODE1=cbxi4pro0-armhf-rb
NODE2=bpi0-armhf-rb ;;
+ armhf_11) NODE1=cbxi4pro0-armhf-rb
NODE2=ff2a-armhf-rb ;;
+ armhf_12) NODE1=ff2a-armhf-rb
NODE2=cbxi4pro0-armhf-rb ;;
+ armhf_13) NODE1=ff2a-armhf-rb
NODE2=ff2b-armhf-rb ;;
+ armhf_14) NODE1=ff2b-armhf-rb
NODE2=ff2a-armhf-rb ;;
+ armhf_15) NODE1=ff2b-armhf-rb
NODE2=ff64a-armhf-rb ;;
+ armhf_16) NODE1=ff64a-armhf-rb
NODE2=ff2b-armhf-rb ;;
+ armhf_17) NODE1=ff64a-armhf-rb
NODE2=jtk1a-armhf-rb ;;
+ armhf_18) NODE1=jtk1a-armhf-rb
NODE2=ff64a-armhf-rb ;;
+ armhf_19) NODE1=jtk1a-armhf-rb
NODE2=odxu4-armhf-rb ;;
+ armhf_20) NODE1=odxu4-armhf-rb
NODE2=jtk1a-armhf-rb ;;
+ armhf_21) NODE1=odxu4-armhf-rb
NODE2=odxu4b-armhf-rb ;;
+ armhf_22) NODE1=odxu4b-armhf-rb
NODE2=odxu4-armhf-rb ;;
+ armhf_23) NODE1=odxu4b-armhf-rb
NODE2=odxu4c-armhf-rb ;;
+ armhf_24) NODE1=odxu4c-armhf-rb
NODE2=odxu4b-armhf-rb ;;
+ armhf_25) NODE1=odxu4c-armhf-rb
NODE2=odu3a-armhf-rb ;;
+ armhf_26) NODE1=odu3a-armhf-rb
NODE2=odxu4c-armhf-rb ;;
+ armhf_27) NODE1=odu3a-armhf-rb
NODE2=opi2a-armhf-rb ;;
+ armhf_28) NODE1=opi2a-armhf-rb
NODE2=odu3a-armhf-rb ;;
+ armhf_29) NODE1=opi2a-armhf-rb
NODE2=opi2b-armhf-rb ;;
+ armhf_30) NODE1=opi2b-armhf-rb
NODE2=opi2a-armhf-rb ;;
+ armhf_31) NODE1=opi2b-armhf-rb
NODE2=opi2c-armhf-rb ;;
+ armhf_32) NODE1=opi2c-armhf-rb
NODE2=opi2b-armhf-rb ;;
+ armhf_33) NODE1=opi2c-armhf-rb
NODE2=p64b-armhf-rb ;;
+ armhf_34) NODE1=p64b-armhf-rb
NODE2=opi2c-armhf-rb ;;
+ armhf_35) NODE1=p64b-armhf-rb
NODE2=p64c-armhf-rb ;;
+ armhf_36) NODE1=p64c-armhf-rb
NODE2=p64b-armhf-rb ;;
+ armhf_37) NODE1=p64c-armhf-rb
NODE2=wbq0-armhf-rb ;;
+ armhf_38) NODE1=wbq0-armhf-rb
NODE2=p64c-armhf-rb ;;
+ armhf_39) NODE1=wbq0-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
+ armhf_40) NODE1=cbxi4a-armhf-rb
NODE2=wbq0-armhf-rb ;;
+ armhf_41) NODE1=cbxi4a-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
+ armhf_42) NODE1=cbxi4a-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
+ armhf_43) NODE1=cbxi4b-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
+ armhf_44) NODE1=cbxi4b-armhf-rb
NODE2=cbxi4a-armhf-rb ;;
+ armhf_45) NODE1=cbxi4b-armhf-rb
NODE2=ff4a-armhf-rb ;;
+ armhf_46) NODE1=ff4a-armhf-rb
NODE2=cbxi4b-armhf-rb ;;
+ armhf_47) NODE1=ff4a-armhf-rb
NODE2=jtx1a-armhf-rb ;;
+ armhf_48) NODE1=ff4a-armhf-rb
NODE2=jtx1a-armhf-rb ;;
+ armhf_49) NODE1=jtx1a-armhf-rb
NODE2=ff4a-armhf-rb ;;
+ armhf_50) NODE1=jtx1a-armhf-rb
NODE2=ff4a-armhf-rb ;;
+ armhf_51) NODE1=jtx1a-armhf-rb
NODE2=hb0-armhf-rb ;;
*) NODE1=undefined
;;
esac
signature.asc
Description: PGP signature
--- End Message ---
--- Begin Message ---
On Mon, Jun 12, 2017 at 01:33:00PM -0700, Vagrant Cascadian wrote:
> Please consider merging the following patch, available in the
> "armhf-reduced-builds" branch:
>
> https://anonscm.debian.org/cgit/users/vagrant/jenkins.debian.net.git/
>
> This may help reduce the impacts of Out-of-Memory errors on nodes when
> building many packages simultaneously, though will likely result in a
> slower build network...
I still don't believe this will be useful, but seeing is believing, so I
merged (*) and deployed it. We'll see how this goes.
We're already detecting build problems due to out of filesystem space,
so that they "dont ftbfs" and automatically reschedule those builds,
I suppose we could do the same for out of memory errors…
(*) please check git commit 43d7cc544… - a better solution for that would
also be welcome but the impact is rather minor. we could also set MAX=100
for all archs…
--
cheers,
Holger
signature.asc
Description: Digital signature
--- End Message ---
