Also one thought - if your SSD VM is fast enough in building, just
send all packages in a loop and mark first and last task ID, and then
regen repos and resubmit all tasks once again. Those which were built
will not build again (will be red tho but skipped) and those missing
dependencies will build. Repeat several times, N^2 in the worst case
:)
LZ
On Fri, Jun 9, 2017 at 11:50 PM, Lukas Zapletal <l...@redhat.com> wrote:
> Bryan I think I found out, turn off kojira service and then you should
> not get regens on each successful build!
>
> LZ
>
> On Fri, Jun 9, 2017 at 11:46 PM, Lukas Zapletal <l...@redhat.com> wrote:
>> Since your clean koji has SSD I can set dmcache for you for the EBS
>> volume to speed up workload, but I am wrapping up for today, so
>> tomorrow.
>>
>> The key is to turn off repo regen after each build, not sure how to do
>> this. I know it can be done somewhere in configs...
>>
>> LZ
>>
>> On Fri, Jun 9, 2017 at 11:43 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>> Basically you go to foreman-packaging rpm/develop and then:
>>>
>>> git annex get (few won't download so delete them for now and skip them)
>>>
>>> for each package directory (you can do in loop)
>>> tito release koji-foreman
>>> tito release koji-foreman-plugins
>>>
>>> First build all packages without "foreman" in it, then regen
>>> buildroots, then all with "foreman" in it and then solve all failed
>>> builds. The order matters.
>>>
>>> But I suggest to turn off repo regeneration after each build, I don't
>>> remember how this can be done (anyone?) Then you regenerate at the end
>>> of the build with:
>>>
>>> koji list-targets --quiet | awk '{print $2}' | sort -u | egrep
>>> '(foreman|katello|pulp)' | grep nightly | xargs -n 1 koji regen-repo
>>>
>>> Also tune your kojibuilder1, I set maxjobs to 7 and capacity to 999 so
>>> it is under nice load of 5-7 concurrent jobs.
>>>
>>> On Fri, Jun 9, 2017 at 11:32 PM, Bryan Kearney <bryan.kear...@gmail.com>
>>> wrote:
>>>> also.. which rhel7 build is used? scl or nonscl?
>>>>
>>>> -- bk
>>>>
>>>>
>>>> On 06/09/2017 05:26 PM, Bryan Kearney wrote:
>>>>>
>>>>> Is there a script to rebuild for nightlies? The tags and external repos
>>>>> are up and running for nightlies at
>>>>>
>>>>> http://34.226.218.207/koji/
>>>>>
>>>>> (cleankoji)
>>>>>
>>>>> I am planning to reverse-engineer from the composes since I have all the
>>>>> builds loaded already. But, if there is an easy rebuild that would be
>>>>> great.
>>>>>
>>>>> -- bk
>>>>>
>>>>> On 06/09/2017 05:15 PM, Lukas Zapletal wrote:
>>>>>>
>>>>>> Quick update. Bryan is working on clean koji installation creating
>>>>>> tags from scratch.
>>>>>>
>>>>>> I have koji running from backup, last version built is 1.12. My plan is:
>>>>>>
>>>>>> - use our release engineering script to create new 1.13 tags
>>>>>> - regen repos for 1.13 and rebuild all 1.13 packages using tito rebuild
>>>>>> - make a compose and compare that against what we have on
>>>>>> yum.theforeman.org
>>>>>>
>>>>>> If this is success, I will do the same for 1.15.
>>>>>>
>>>>>> New koji recovered from backup is at
>>>>>> http://ec2-54-163-236-166.compute-1.amazonaws.com/koji/ and we
>>>>>> requested DNS change already, hopefully it's ready on Monday at least
>>>>>> for nightly builds.
>>>>>>
>>>>>> On Fri, Jun 9, 2017 at 2:46 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>>>>>>
>>>>>>> Hmmm if I'd have to create new koji I'd rather start from scratch
>>>>>>> reimporting our repos back in.
>>>>>>>
>>>>>>> Current problems we had:
>>>>>>>
>>>>>>> - small vm with not enough cores/RAM to run builds
>>>>>>> - outdated system (RHEL6) and koji version
>>>>>>> - no use of LVM
>>>>>>> - data on ephemeral volumes
>>>>>>> - lots of symlinks which were added as we grew
>>>>>>>
>>>>>>> If I'd need recreate Koji from scratch, I'd try to avoid all of these.
>>>>>>> Local volumes should be only used as cache, swap and temp, external
>>>>>>> repos should be on external NFS, everything else on dedicated block
>>>>>>> device with regular backups.
>>>>>>>
>>>>>>> On Fri, Jun 9, 2017 at 2:33 PM, Bryan Kearney <bryan.kear...@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On 06/09/2017 08:27 AM, Lukas Zapletal wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have some bad news to share. I started on moving from our 2nd koji
>>>>>>>>> to 3rd koji created as m1.xxlarge instance so we would avoid NFS and
>>>>>>>>> could run on a single VM. I prepared everything and when I was
>>>>>>>>> shutting down all the builders and test VMs I accidentaly stopped the
>>>>>>>>> 2nd koji. Unfortunately the original koji instance was stopped
>>>>>>>>> yesterday as well. We lost all data from epheremal storage volumes,
>>>>>>>>> these were /mnt/tmp holding most importantly Postgres database for
>>>>>>>>> kojihub and not sure what else. Our deployment was dozen of symlinks
>>>>>>>>> across two local volumes and EBS.
>>>>>>>>>
>>>>>>>>> I found some backups of pgsql on EBS volume, but unfortunately these
>>>>>>>>> stopped being dumped summer 2016
>>>>>>>>> (koji.katello.org-kojidb-20160830-230505.dump is the newest one). Here
>>>>>>>>> are the options we have:
>>>>>>>>>
>>>>>>>>> 1) Restore the DB and attempt to start services back - but this will
>>>>>>>>> give us instance back with huge gap of almost one year. I am not sure
>>>>>>>>> how much useful that would be - we will not be able to bu ild into
>>>>>>>>> 1.15/1.14 repos, missing tags, builds, logs.
>>>>>>>>>
>>>>>>>>> 2) Move to https://cbs.centos.org CentOS CBS, they run their own Koji
>>>>>>>>> and start building there
>>>>>>>>>
>>>>>>>>> 3) Speed up Copr transition
>>>>>>>>>
>>>>>>>>> 4) Build whole new Koji from scratch
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> A variation here is to use a new DB on the instance we have. So, same
>>>>>>>> certs
>>>>>>>> just a blank DB.
>>>>>>>>
>>>>>>>> -- bk
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Later,
>>>>>>> Lukas @lzap Zapletal
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "foreman-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> Later,
>>> Lukas @lzap Zapletal
>>
>>
>>
>> --
>> Later,
>> Lukas @lzap Zapletal
>
>
>
> --
> Later,
> Lukas @lzap Zapletal
--
Later,
Lukas @lzap Zapletal
--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
