Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-12 Thread Lukas Zapletal
Mass scratch rebuild was a success, we have all packages green except few: foreman-plugins-nightly-fedora24, rubygem-smart_proxy_abrt-0.0.8-3.git.0.5b8894c.git.0.5b8894c.fc24.src.rpm foreman-plugins-nightly-fedora24, rubygem-foreman_docker-3.0.0-1.fm1_16.git.700.5b8894c.fc24.src.rpm foreman-plugin

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-12 Thread Lukas Zapletal
Assuming the DNS still points to the restored koji, you need to do this: 34.226.218.207 koji.katello.org kojihub.katello.org I kicked off scratch builds of all our packages into nightly, looks good so far. It is not yet finished but processing at very good rate of about several tasks per minute (

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-11 Thread Bryan Kearney
I am gonna take the optimistic route, and hope that this approach works. 1.15 should be back up: foreman-1.15-fedora24 foreman-plugins-1.15-fedora24 foreman-1.15-nonscl-rhel7 foreman-plugins-1.15-nonscl-rhel7 foreman-1.15-rhel7 foreman-plugins-1.15-rhel7 Please kick off some builds against thes

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-11 Thread Eric D Helms
The foreman-release package defines macro that sets the fm1_ addition to some RPMs that define foremandist in the spec. The foreman-release package needs to exist in the rpm and srpm build groups for the proper version. On Jun 10, 2017 11:25 AM, "Lukas Zapletal" wrote: I was trying to rebuild 1.

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-10 Thread Bryan Kearney
Update. Per Lzap's suggestion, I wrote down the rough approach at [1]. I believe that foreman-nightly should be ready to test on this machine. Specifically, building into these tags: foreman-nightly-fedora24 foreman-plugins-nightly-fedora24 foreman-nightly-nonscl-rhel7 foreman-plugins-nightly

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-10 Thread Lukas Zapletal
I was trying to rebuild 1.14 in old koji but it did not work well. I am running into several issues. Some packages fails to complete build with: GenericError: Unable to complete build: release mismatch (build: 1.fm1_14.fc24, rpm: 1.fm1_12.fc24) like http://ec2-54-163-236-166.compute-1.amazonaws.co

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-10 Thread Bryan Kearney
ok.. I have builds running http://34.226.218.207/koji/taskinfo?taskID=749 but what i dont get is the build requires. The foreman spec file for this requires npm-babel-core but the package I have http://34.226.218.207/koji/buildinfo?buildID=15280 provides bundled-npm-babel-core so something

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-10 Thread Bryan Kearney
On http://34.226.218.207/koji (cleankoji) I think I have recreated the fedora24 tags for foreman (not plugins) (see http://34.226.218.207/koji/taginfo?tagID=2) . What I did was go back through and add the packaegs in, and then tag in the builds from the repos which are loctaed in http://34.226

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-10 Thread Lukas Zapletal
Then find missing packages in tags: kojikat watch-task $(seq 521218 521565) | tee mass-rebuild-1.13-a.log grep -o "package .* not in list for tag .*" mass-rebuild-1.13-a.log | awk '{print "kojikat add-pkg --owner=lzapleta", $8, $2}' I was missing these: kojikat add-pkg --owner=lzapleta foreman-

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-10 Thread Lukas Zapletal
So I am mass rebuilding 1.13 now, I was able to figure out a GNU parallel command that will upload all SRPMs in few minutes. Run this in foreman-packaging branch rpm/1.13 (or the version you want): First all non-foreman packages (dependencies): mkdir /tmp/tito; find . -type f -name \*.spec | egre

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
Also one thought - if your SSD VM is fast enough in building, just send all packages in a loop and mark first and last task ID, and then regen repos and resubmit all tasks once again. Those which were built will not build again (will be red tho but skipped) and those missing dependencies will build

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
Bryan I think I found out, turn off kojira service and then you should not get regens on each successful build! LZ On Fri, Jun 9, 2017 at 11:46 PM, Lukas Zapletal wrote: > Since your clean koji has SSD I can set dmcache for you for the EBS > volume to speed up workload, but I am wrapping up for

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
Since your clean koji has SSD I can set dmcache for you for the EBS volume to speed up workload, but I am wrapping up for today, so tomorrow. The key is to turn off repo regen after each build, not sure how to do this. I know it can be done somewhere in configs... LZ On Fri, Jun 9, 2017 at 11:43

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
Basically you go to foreman-packaging rpm/develop and then: git annex get (few won't download so delete them for now and skip them) for each package directory (you can do in loop) tito release koji-foreman tito release koji-foreman-plugins First build all packages without "foreman" in it, then r

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Bryan Kearney
also.. which rhel7 build is used? scl or nonscl? -- bk On 06/09/2017 05:26 PM, Bryan Kearney wrote: Is there a script to rebuild for nightlies? The tags and external repos are up and running for nightlies at http://34.226.218.207/koji/ (cleankoji) I am planning to reverse-engineer from the

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Bryan Kearney
Is there a script to rebuild for nightlies? The tags and external repos are up and running for nightlies at http://34.226.218.207/koji/ (cleankoji) I am planning to reverse-engineer from the composes since I have all the builds loaded already. But, if there is an easy rebuild that would be gr

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
Quick update. Bryan is working on clean koji installation creating tags from scratch. I have koji running from backup, last version built is 1.12. My plan is: - use our release engineering script to create new 1.13 tags - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild - mak

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
Hmmm if I'd have to create new koji I'd rather start from scratch reimporting our repos back in. Current problems we had: - small vm with not enough cores/RAM to run builds - outdated system (RHEL6) and koji version - no use of LVM - data on ephemeral volumes - lots of symlinks which were added a

Re: [foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Bryan Kearney
On 06/09/2017 08:27 AM, Lukas Zapletal wrote: I have some bad news to share. I started on moving from our 2nd koji to 3rd koji created as m1.xxlarge instance so we would avoid NFS and could run on a single VM. I prepared everything and when I was shutting down all the builders and test VMs I acci

[foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
I have some bad news to share. I started on moving from our 2nd koji to 3rd koji created as m1.xxlarge instance so we would avoid NFS and could run on a single VM. I prepared everything and when I was shutting down all the builders and test VMs I accidentaly stopped the 2nd koji. Unfortunately the

[foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
After I increased capacity it started to pick tasks, but all were stuck, I tracked it down to NFS hard connection which was stuck, I rebooted the builder and it does not come up. If I won't fix this in a reasonable time, I will shutdown htttp on our new koji to prevent more tasks to queue up. NFS

[foreman-dev] Re: Koji outage on Sunday

2017-06-09 Thread Lukas Zapletal
Hey, we are suffering from capacity issues - builder3 (that's the only extra builder we currently have) was spawned on hypervisor which is under stress, therefore kernel reports load of 3 and kojid does not schedule any builds there since this exceeds the capacity. The instance is simply idling, I

[foreman-dev] Re: Koji outage on Sunday

2017-06-05 Thread Lukas Zapletal
Hello, during migration I noticed we are running out of space on the data volume, I am going to remove Fedora 22 and 23 external repositories tomorrow. If you still need them, speak up! LZ On Sun, Jun 4, 2017 at 6:11 PM, Lukas Zapletal wrote: > I was wondering, if someone can put the /etc/hosts

[foreman-dev] Re: Koji outage on Sunday

2017-06-04 Thread Lukas Zapletal
I was wondering, if someone can put the /etc/hosts entry to jenkins, I assume it's a change somewhere in puppet or foreman-infra? Thanks On Sun, Jun 4, 2017 at 5:59 PM, Lukas Zapletal wrote: > The koji was re-spinned as a new instance with new IP 54.163.225.45, > please change your /etc/hosts un

[foreman-dev] Re: Koji outage on Sunday

2017-06-04 Thread Lukas Zapletal
The koji was re-spinned as a new instance with new IP 54.163.225.45, please change your /etc/hosts until DNS gets populated: 54.163.225.45 koji.katello.org The very last job successful that was processed on the old koji was: http://koji.katello.org/koji/taskinfo?taskID=644141 Just for the recor