So I am mass rebuilding 1.13 now, I was able to figure out a GNU
parallel command that will upload all SRPMs in few minutes. Run this
in foreman-packaging branch rpm/1.13 (or the version you want):
First all non-foreman packages (dependencies):
mkdir /tmp/tito; find . -type f -name \*.spec | egrep -v
'(foreman|proxy)' | parallel --eta 'pushd $(dirname {}); /usr/bin/tito
release koji-foreman; /usr/bin/tito release koji-foreman-plugins;
popd'
Then regen repos:
koji list-targets --quiet | awk '{print $2}' | sort -u | grep 1.13 |
xargs -n 1 koji regen-repo
Then foreman packages:
mkdir /tmp/tito; find . -type f -name \*.spec | egrep
'(foreman|proxy)' | parallel --eta 'pushd $(dirname {}); /usr/bin/tito
release koji-foreman; /usr/bin/tito release koji-foreman-plugins;
popd'
You will get one package stuck, that is rubygem-locale which fails to
download locale-2.0.9.gem and timeouts for hours. Just skip that for
now.
This is a rough mass rebuild just to test how much feasible is this
path, I will compare meshed repository with what we have.
On Fri, Jun 9, 2017 at 11:54 PM, Lukas Zapletal <l...@redhat.com> wrote:
> Also one thought - if your SSD VM is fast enough in building, just
> send all packages in a loop and mark first and last task ID, and then
> regen repos and resubmit all tasks once again. Those which were built
> will not build again (will be red tho but skipped) and those missing
> dependencies will build. Repeat several times, N^2 in the worst case
> :)
>
> LZ
>
> On Fri, Jun 9, 2017 at 11:50 PM, Lukas Zapletal <l...@redhat.com> wrote:
>> Bryan I think I found out, turn off kojira service and then you should
>> not get regens on each successful build!
>>
>> LZ
>>
>> On Fri, Jun 9, 2017 at 11:46 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>> Since your clean koji has SSD I can set dmcache for you for the EBS
>>> volume to speed up workload, but I am wrapping up for today, so
>>> tomorrow.
>>>
>>> The key is to turn off repo regen after each build, not sure how to do
>>> this. I know it can be done somewhere in configs...
>>>
>>> LZ
>>>
>>> On Fri, Jun 9, 2017 at 11:43 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>>> Basically you go to foreman-packaging rpm/develop and then:
>>>>
>>>> git annex get (few won't download so delete them for now and skip them)
>>>>
>>>> for each package directory (you can do in loop)
>>>> tito release koji-foreman
>>>> tito release koji-foreman-plugins
>>>>
>>>> First build all packages without "foreman" in it, then regen
>>>> buildroots, then all with "foreman" in it and then solve all failed
>>>> builds. The order matters.
>>>>
>>>> But I suggest to turn off repo regeneration after each build, I don't
>>>> remember how this can be done (anyone?) Then you regenerate at the end
>>>> of the build with:
>>>>
>>>> koji list-targets --quiet | awk '{print $2}' | sort -u | egrep
>>>> '(foreman|katello|pulp)' | grep nightly | xargs -n 1 koji regen-repo
>>>>
>>>> Also tune your kojibuilder1, I set maxjobs to 7 and capacity to 999 so
>>>> it is under nice load of 5-7 concurrent jobs.
>>>>
>>>> On Fri, Jun 9, 2017 at 11:32 PM, Bryan Kearney <bryan.kear...@gmail.com>
>>>> wrote:
>>>>> also.. which rhel7 build is used? scl or nonscl?
>>>>>
>>>>> -- bk
>>>>>
>>>>>
>>>>> On 06/09/2017 05:26 PM, Bryan Kearney wrote:
>>>>>>
>>>>>> Is there a script to rebuild for nightlies? The tags and external repos
>>>>>> are up and running for nightlies at
>>>>>>
>>>>>> http://34.226.218.207/koji/
>>>>>>
>>>>>> (cleankoji)
>>>>>>
>>>>>> I am planning to reverse-engineer from the composes since I have all the
>>>>>> builds loaded already. But, if there is an easy rebuild that would be
>>>>>> great.
>>>>>>
>>>>>> -- bk
>>>>>>
>>>>>> On 06/09/2017 05:15 PM, Lukas Zapletal wrote:
>>>>>>>
>>>>>>> Quick update. Bryan is working on clean koji installation creating
>>>>>>> tags from scratch.
>>>>>>>
>>>>>>> I have koji running from backup, last version built is 1.12. My plan is:
>>>>>>>
>>>>>>> - use our release engineering script to create new 1.13 tags
>>>>>>> - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild
>>>>>>> - make a compose and compare that against what we have on
>>>>>>> yum.theforeman.org
>>>>>>>
>>>>>>> If this is success, I will do the same for 1.15.
>>>>>>>
>>>>>>> New koji recovered from backup is at
>>>>>>> http://ec2-54-163-236-166.compute-1.amazonaws.com/koji/ and we
>>>>>>> requested DNS change already, hopefully it's ready on Monday at least
>>>>>>> for nightly builds.
>>>>>>>
>>>>>>> On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>>>>>>>
>>>>>>>> Hmmm if I'd have to create new koji I'd rather start from scratch
>>>>>>>> reimporting our repos back in.
>>>>>>>>
>>>>>>>> Current problems we had:
>>>>>>>>
>>>>>>>> - small vm with not enough cores/RAM to run builds
>>>>>>>> - outdated system (RHEL6) and koji version
>>>>>>>> - no use of LVM
>>>>>>>> - data on ephemeral volumes
>>>>>>>> - lots of symlinks which were added as we grew
>>>>>>>>
>>>>>>>> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
>>>>>>>> Local volumes should be only used as cache, swap and temp, external
>>>>>>>> repos should be on external NFS, everything else on dedicated block
>>>>>>>> device with regular backups.
>>>>>>>>
>>>>>>>> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney <bryan.kear...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have some bad news to share. I started on moving from our 2nd koji
>>>>>>>>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>>>>>>>>> could run on a single VM. I prepared everything and when I was
>>>>>>>>>> shutting down all the builders and test VMs I accidentaly stopped the
>>>>>>>>>> 2nd koji. Unfortunately the original koji instance was stopped
>>>>>>>>>> yesterday as well. We lost all data from epheremal storage volumes,
>>>>>>>>>> these were /mnt/tmp holding most importantly Postgres database for
>>>>>>>>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>>>>>>>>> across two local volumes and EBS.
>>>>>>>>>>
>>>>>>>>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>>>>>>>>> stopped being dumped summer 2016
>>>>>>>>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one).
>>>>>>>>>> Here
>>>>>>>>>> are the options we have:
>>>>>>>>>>
>>>>>>>>>> 1) Restore the DB and attempt to start services back - but this will
>>>>>>>>>> give us instance back with huge gap of almost one year. I am not sure
>>>>>>>>>> how much useful that would be - we will not be able to bu ild into
>>>>>>>>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>>>>>>>>
>>>>>>>>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>>>>>>>>> and start building there
>>>>>>>>>>
>>>>>>>>>> 3) Speed up Copr transition
>>>>>>>>>>
>>>>>>>>>> 4) Build whole new Koji from scratch
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> A variation here is to use a new DB on the instance we have. So, same
>>>>>>>>> certs
>>>>>>>>> just a blank DB.
>>>>>>>>>
>>>>>>>>> -- bk
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Later,
>>>>>>>> Lukas @lzap Zapletal
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "foreman-dev" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>> --
>>>> Later,
>>>> Lukas @lzap Zapletal
>>>
>>>
>>>
>>> --
>>> Later,
>>> Lukas @lzap Zapletal
>>
>>
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
--
Later,
Lukas @lzap Zapletal
--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
