On Tue, Jun 13, 2017 at 3:24 PM, Brian Kelley <fustiga...@gmail.com> wrote:
> I'm unsure if that staging will help, aren't the builds and tests already
> done in parallel effectively?
>
Kind of. There are currently two jobs that run in parallel:
- job 1: build the core library, the python 2.7 wrappers, and the java
wrappers and test everything
- job 2: build the core library, the python 3.5 wrappers, and the java
wrappers and test everything
Those have no dependencies on each other. I would hope that by pulling the
core library and test builds out into a separate job and then using the
artifacts (=libraries) from that for the python and java builds we could
shorten the individual jobs.
-greg
> On Tue, Jun 13, 2017 at 9:20 AM, Greg Landrum <greg.land...@gmail.com>
> wrote:
>
>> Given that the current variability is almost a factor of two I don't
>> think smaller changes are going to help.
>>
>> What I'm likely going to try is doing staged builds:
>> - stage 1: build and test core library
>> - stage 2a: build and test python 2.7 wrappers
>> - stage 2b: build and test python 3.5 wrappers
>> - stage 2c: build and test java wrappers
>>
>> the three "stage 2" things can run in parallel.
>>
>> From the bit of reading I did on the train this morning, it looks like
>> this is not that horrible to do in jenkins; just need to invest a bit of
>> time.
>>
>> -greg
>>
>>
>> On Tue, Jun 13, 2017 at 3:12 PM, Brian Kelley <fustiga...@gmail.com>
>> wrote:
>>
>>> I tried using ninja for the builds in stead of make, after an evening of
>>> not building I successfully had two builds:
>>>
>>> https://travis-ci.org/bp-kelley/rdkit/builds/242399233
>>>
>>> Commit:
>>>
>>> https://github.com/bp-kelley/rdkit/blob/baac00bc1eb7f6641301
>>> 1aed239413fe74da3388/.travis.yml
>>>
>>> Note that the default ninja build is #cores + 2 which blew out the
>>> memory on the VMs so I used -j3 we could try this in the makefile as well.
>>>
>>> I also think we could shave off a few minutes by
>>>
>>> 1. building in the root conda env
>>> 2. having only one conda install for *everything*
>>>
>>> If you like I can try these fixes.
>>>
>>> Cheers,
>>> Brian
>>>
>>> On Tue, Jun 13, 2017 at 3:44 AM, Greg Landrum <greg.land...@gmail.com>
>>> wrote:
>>>
>>>> Dear all,
>>>>
>>>> We're getting more and more timeouts with RDKit builds on travis. I'm
>>>> reasonably sure that this is due to some variability in the performance of
>>>> the VMs that the builds end up running on. A good demonstration of that is
>>>> this particular build (on a branch where I'm playing with simplifying the
>>>> travis builds), which works fine for py35 but times out for py27:
>>>> https://travis-ci.org/rdkit/rdkit/builds/242301061?utm_sourc
>>>> e=email&utm_medium=notification
>>>>
>>>> If you look into the details you'll see that the main compile "only"
>>>> takes 1475 seconds for the python 3.5 build but it is killed after 2541
>>>> seconds in the python2.7 build. Given that it's the same code being built,
>>>> it seems like this more or less has to be infrastructure variability.
>>>>
>>>> I'll continue to explore, particularly breaking the build into stages,
>>>> but if anyone has suggestions on how to improve this please let me know.
>>>>
>>>> -greg
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> ------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Rdkit-devel mailing list
>>>> Rdkit-devel@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/rdkit-devel
>>>>
>>>>
>>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Rdkit-devel mailing list
Rdkit-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rdkit-devel
