Re: [Rdkit-discuss] Hankering after faster builds

[Tim Dudgeon]

Hi Greg,

On 03/12/2016 13:16, Greg Landrum wrote:
Builds do take a while, but there is *no way* they should be taking 2 
hours unless they are running on extremely overloaded hardware. The 
travis builds, which include running all the tests, typically take 
less than 40 minutes.
yes on my normal machine a build takes about 45 mins. But on Docker hub 
it seems your only get pretty limited hardware (1 core for instance) and 
can't do anything about this.


If, for some reason, you do still need to deal with this, I would 
guess that it would help to start by building an image without the 
java (or python) wrappers and then build an image on top of that which 
adds just the java (or python) wrappers
Yes, I can try that. In fact I'm sort of doing this for the Java 
wrappers, but not quite in that way.


Still, I would hope that you don't need to run too many builds on 
docker. It seems like one of those "once per release" activities. 
Certainly not something you'd want to do after every commit.
Yes, that's the case. Typically I build when there is a new release 
(e.g. every 6 months) and from time to time check that the build on 
master is still OK.


Tim


-greg





On Fri, Dec 2, 2016 at 6:29 PM, Tim Dudgeon > wrote:


Of course builds from source are never fast enough, and the RDKit
one is
pretty big.
So far I've lived with this and made cups of coffee.
But since I've been working with the Release_2016_09_2 release my
Docker
image builds on Docker Hub [1] are timing out as they sometimes exceed
the 2 hour limit. If I try at a quiet time I can sometimes get them to
complete, but I suppose the situation is only going to get worse.

I've tried breaking the build into 2 steps, the first to prepare the
base image [2] and the second to build RDKit [3], and that's helped a
bit, but I don't think there's more mileage here as nearly all the
time
is spend in the 'make' command.

Anticipating things getting worse, does anyone have suggestions for
speeding this up. I hanker after a --fast mode, but I suppose if there
was one it would already be the default.

Any ideas?

In case anyone wonders I'm building from source as I need to also
create
a Java enabled version and one for the Postgres cartridge and want to
use the same approach to generating them all.

Tim

[1] https://hub.docker.com/r/informaticsmatters/rdkit/

[2] https://github.com/InformaticsMatters/rdkit_debian_base

[3] https://github.com/InformaticsMatters/rdkit







--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Rdkit-discuss mailing list
Rdkit-discuss@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/rdkit-discuss





--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Rdkit-discuss mailing list
Rdkit-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rdkit-discuss

Re: [Rdkit-discuss] Hankering after faster builds

[Greg Landrum]

Lest my previous reply be mis-interpreted: I agree that it would be great
if the builds were quicker - I end up needing to do a large number of them
before each release - but I don't see much that can really be done other
than removing a bunch of functionality.

On Sat, Dec 3, 2016 at 8:42 AM, Gianluca Sforna  wrote:

> On Fri, Dec 2, 2016 at 6:29 PM, Tim Dudgeon  wrote:
> > But since I've been working with the Release_2016_09_2 release my Docker
> > image builds on Docker Hub [1] are timing out as they sometimes exceed
> > the 2 hour limit. If I try at a quiet time I can sometimes get them to
> > complete, but I suppose the situation is only going to get worse.
> >
>
> I feel your pain. The RPM build has to compile everything TWICE, one
> against python2 and one against python3...
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Rdkit-discuss mailing list
> Rdkit-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/rdkit-discuss
>
--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Rdkit-discuss mailing list
Rdkit-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rdkit-discuss

Re: [Rdkit-discuss] Hankering after faster builds

[Greg Landrum]

Builds do take a while, but there is *no way* they should be taking 2 hours
unless they are running on extremely overloaded hardware. The travis
builds, which include running all the tests, typically take less than 40
minutes.

If, for some reason, you do still need to deal with this, I would guess
that it would help to start by building an image without the java (or
python) wrappers and then build an image on top of that which adds just the
java (or python) wrappers

Still, I would hope that you don't need to run too many builds on docker.
It seems like one of those "once per release" activities. Certainly not
something you'd want to do after every commit.

-greg





On Fri, Dec 2, 2016 at 6:29 PM, Tim Dudgeon  wrote:

> Of course builds from source are never fast enough, and the RDKit one is
> pretty big.
> So far I've lived with this and made cups of coffee.
> But since I've been working with the Release_2016_09_2 release my Docker
> image builds on Docker Hub [1] are timing out as they sometimes exceed
> the 2 hour limit. If I try at a quiet time I can sometimes get them to
> complete, but I suppose the situation is only going to get worse.
>
> I've tried breaking the build into 2 steps, the first to prepare the
> base image [2] and the second to build RDKit [3], and that's helped a
> bit, but I don't think there's more mileage here as nearly all the time
> is spend in the 'make' command.
>
> Anticipating things getting worse, does anyone have suggestions for
> speeding this up. I hanker after a --fast mode, but I suppose if there
> was one it would already be the default.
>
> Any ideas?
>
> In case anyone wonders I'm building from source as I need to also create
> a Java enabled version and one for the Postgres cartridge and want to
> use the same approach to generating them all.
>
> Tim
>
> [1] https://hub.docker.com/r/informaticsmatters/rdkit/
> [2] https://github.com/InformaticsMatters/rdkit_debian_base
> [3] https://github.com/InformaticsMatters/rdkit
>
>
>
>
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> ___
> Rdkit-discuss mailing list
> Rdkit-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/rdkit-discuss
>
--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Rdkit-discuss mailing list
Rdkit-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rdkit-discuss

Re: [Rdkit-discuss] Hankering after faster builds

[Gianluca Sforna]

On Fri, Dec 2, 2016 at 6:29 PM, Tim Dudgeon  wrote:
> But since I've been working with the Release_2016_09_2 release my Docker
> image builds on Docker Hub [1] are timing out as they sometimes exceed
> the 2 hour limit. If I try at a quiet time I can sometimes get them to
> complete, but I suppose the situation is only going to get worse.
>

I feel your pain. The RPM build has to compile everything TWICE, one
against python2 and one against python3...

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
___
Rdkit-discuss mailing list
Rdkit-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rdkit-discuss