Send hpx-devel mailing list submissions to
[email protected]
To subscribe or unsubscribe via the World Wide Web, visit
https://mail.cct.lsu.edu/mailman/listinfo/hpx-devel
or, via email, send a message with subject or body 'help' to
[email protected]
You can reach the person managing the list at
[email protected]
When replying, please edit your Subject line so it is more specific
than "Re: Contents of hpx-devel digest..."
Today's Topics:
1. Re: Switch to gitlab? (Thomas Heller)
2. pycicle (Biddiscombe, John A.)
----------------------------------------------------------------------
Message: 1
Date: Sat, 2 Dec 2017 19:55:18 +0100
From: Thomas Heller <[email protected]>
Subject: Re: [hpx-devel] Switch to gitlab?
To: [email protected]
Message-ID:
<cajcxaeyf098znmdxr-azynlbtzsw2rtyudvfo2s0emjo+mt...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
The mirroring seems to only affect the repository, that is branches and
tags. Not issues or PRs.
This seems like a dead end then. Setting up a bridge to test PRs with a
gitlab runners is an effort that's probably not worth it. The search
continues.
Am 02.12.2017 4:48 nachm. schrieb "Marcin Copik" <[email protected]>:
> In terms of cooperations with open-source community and visibility, I
> think that GitHub is far more superior to both BitBucket and GitLab.
> Perhaps someone has a different experience but I can count on the fingers
> of one hand the number of important projects which are hosted on GitLab.
>
> We can still use GitHub for development and interaction with contributors
> and run private GitLab repo as a backend for CI. There is a builtin support
> for that in GitLab.
> https://docs.gitlab.com/ee/workflow/repository_mirroring.html
>
> sob., 2 gru 2017 o 16:09 u?ytkownik Hartmut Kaiser <
> [email protected]> napisa?:
>
>>
>> > I think it depends what we intend to do. When we want to replace github
>> > it is important for our contributors or? When we keep github, you are
>> > right it does not matter.
>>
>> I'm strongly opposed to switching away from github as our main repository
>> for mostly political reasons (as said before).
>>
>> Regards Hartmut
>> ---------------
>> http://boost-spirit.com
>> http://stellar.cct.lsu.edu
>>
>>
>> >
>> > On 12/02/2017 09:42 AM, Hartmut Kaiser wrote:
>> > >
>> > >> Obe questions is where would the gitlab be hosted? Do new
>> contributors
>> > >> easily access the code, report issues and contribute?
>> > >
>> > > That's not important as long as the CI can access it and we can see
>> the
>> > CI results. I'd even encourage a system where people _don't_ see the
>> > gitlab repo at all.
>> > >
>> > > Regards Hartmut
>> > > ---------------
>> > > http://boost-spirit.com
>> > > http://stellar.cct.lsu.edu
>> > >
>> > >
>> > >>
>> > >> On 12/02/2017 08:32 AM, Hartmut Kaiser wrote:
>> > >>> I?d like to leave the github repo as our user-facing frontend,
>> mostly
>> > >>> for political reasons. If gitlab can synchronize from and to github
>> > >>> seamlessly (as you say) we still could have such a repo coordinating
>> > the
>> > >>> CI business in the background.
>> > >>>
>> > >>>
>> > >>>
>> > >>> Regards Hartmut
>> > >>>
>> > >>> ---------------
>> > >>>
>> > >>> http://boost-spirit.com
>> > >>>
>> > >>> http://stellar.cct.lsu.edu
>> > >>>
>> > >>>
>> > >>>
>> > >>> *From:*[email protected]
>> > >>> [mailto:[email protected]] *On Behalf Of
>> *Thomas
>> > >> Heller
>> > >>> *Sent:* Saturday, December 2, 2017 4:34 AM
>> > >>> *To:* [email protected]
>> > >>> *Subject:* [hpx-devel] Switch to gitlab?
>> > >>>
>> > >>>
>> > >>>
>> > >>> Hi all,
>> > >>>
>> > >>>
>> > >>>
>> > >>> In the past months, there has been increased interest to improve our
>> > CI
>> > >>> system. One of the biggest problems we have currently is that PRs
>> > aren't
>> > >>> tested properly due to insufficient resources on circleci. On the
>> > other
>> > >>> hand, we have a buildbot instance running at cct which is testing
>> > >>> multiple configurations for the master branch. The problem with
>> > buildbot
>> > >>> is that it's not easily scalable to the various other resources we
>> > have
>> > >>> access to. Integrating all resources would allow us to test PRs more
>> > >>> thoroughly.
>> > >>>
>> > >>> The biggest problem here is that we don't have a ready made solution
>> > >>> which would allow us to implement a richer CI solution.
>> > >>>
>> > >>>
>> > >>>
>> > >>> What I'd like to suggest here is a rather radical change for our
>> > >>> infrastructure:
>> > >>>
>> > >>> A switch to gitlab.
>> > >>>
>> > >>> Gitlab provides the same features and more that GitHub provides.
>> > >>>
>> > >>> Among the biggest feature over GitHub is an extensible solution for
>> CI
>> > >>> runners:
>> > >>>
>> > >>> https://about.gitlab.com/features/gitlab-ci-cd/
>> > >>>
>> > >>>
>> > >>>
>> > >>> An overview of all gitlab features can be seen here:
>> > >>>
>> > >>> https://about.gitlab.com/features
>> > >>>
>> > >>>
>> > >>>
>> > >>> I think this is something worth investigating and looks like a
>> > promising
>> > >>> solution (we could still mirror to GitHub).
>> > >>>
>> > >>>
>> > >>>
>> > >>> What do you guys think?
>> > >>>
>> > >>>
>> > >>>
>> > >>> Cheers,
>> > >>>
>> > >>> Thomas
>> > >>>
>> > >>>
>> > >>>
>> > >>> _______________________________________________
>> > >>> hpx-devel mailing list
>> > >>> [email protected]
>> > >>> https://mail.cct.lsu.edu/mailman/listinfo/hpx-devel
>> > >>>
>> > >>
>> > >> --
>> > >> Patrick Diehl
>> > >> diehlpk.github.io
>> > >
>> > >
>> > > _______________________________________________
>> > > hpx-devel mailing list
>> > > [email protected]
>> > > https://mail.cct.lsu.edu/mailman/listinfo/hpx-devel
>> > >
>> >
>> > --
>> > Patrick Diehl
>> > diehlpk.github.io
>>
>>
>> _______________________________________________
>> hpx-devel mailing list
>> [email protected]
>> https://mail.cct.lsu.edu/mailman/listinfo/hpx-devel
>>
>
> _______________________________________________
> hpx-devel mailing list
> [email protected]
> https://mail.cct.lsu.edu/mailman/listinfo/hpx-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mail.cct.lsu.edu/pipermail/hpx-devel/attachments/20171202/0a89fdbe/attachment-0001.html
------------------------------
Message: 2
Date: Sun, 3 Dec 2017 17:56:29 +0000
From: "Biddiscombe, John A." <[email protected]>
Subject: [hpx-devel] pycicle
To: "[email protected]" <[email protected]>
Message-ID:
<[email protected]>
Content-Type: text/plain; charset="us-ascii"
As seen on IRC, I have created a smiple tool to poll github and spawn build for
PR's
Currently it is limited to
Poll github
Get list of PRs
Compare the current SHA of the PR with the last time we checked
if it is different - start a build
To start a build, it ssh's to a machine, generates a slurm script and uses
sbatch to launch it
the build/test is controlled by a cmake script using ctest.
I was driven by two main objectives
1) one and only one config per machine-build
2) the minimal amount of other scripts etc
Delievery of results is via ctest. It does
Configure/Build/Test/Submit to the cscs dashboard where results are displayed.
We can see build errors/warnings, test fails and simple interface to see which
tests pass/fail etc.
Tweaks - Getting the correct update step so that the PR shows the files that
have changed between master and PR was tricky, so I checkout master, create a
branch, merge PR, then go back to master branch - the ctest_update goes back to
the merged branch and shows the diffed files as desired.
Things to do :
daint (cray at cscs) cannot submit results to cdas because the compute nodes
cannot see outside the machine :
solution - have the pycicle script colect the xml configure/build/test files
from the pycicle root path and submit them by using ctest on the machine on
which pycicle is running. Not hard, just need to collect the xml files, scp
them back, and submit them, then delete them so we don't do it twice etc.
Monitor build - at the moment, if you modify a PR whilst a build is still
going, it will submit a new job to slurm for the same build dir etc and two
jobs might run in the same place.
solution - scancel the existing job if it is the same pr (name) on the same
machine, just need to query slurm before submitting a new job
Multiple machines: I've hardcoded greina into the script, but we need a list of
machines and we should pick N machines for each PR and launch jobs on them
Multiple configurations : I'm only building release and one version of boost.
We need to add lists of config options and then generate builds for N of them
Solution : To do the above steps we need to track how many jobs have been
launched, and keeps lists of who is doing what, etc etc.
Make PRs go green if builds pass on github: righ now, cdash displays the
results - but we have no idea if they passed.
solution : like the problem with daint, we should scrape the xml test files
from the machines we submit to, parse for fails and then enable/disable the PR
merge button based on what we've got. (Not sure yet ig pygithub has a function
to do that, but I'm sure it does).
Lots of other small things.
This is an experiment and it may go nowhere - but I'm going to leave it running
for the forseeable future and improve the python stuff to add the features I've
mentioned. I would be happy if others contributed stuff to the pycicle script.
It's simple stuff, but it seems to be working so far. Mypython skills are very
basic, so I need help :)
yours
JB
--
John Biddiscombe, email:biddisco @.at.@ cscs.ch
http://www.cscs.ch/
CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07
Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mail.cct.lsu.edu/pipermail/hpx-devel/attachments/20171203/dc2050b2/attachment.html
------------------------------
_______________________________________________
hpx-devel mailing list
[email protected]
https://mail.cct.lsu.edu/mailman/listinfo/hpx-devel
End of hpx-devel Digest, Vol 35, Issue 4
****************************************
