Re: Fixing Trunk tests and getting stable nightly build on b.a.o
Hi, On Fri, May 30, 2014 at 1:05 PM, dev-digest-h...@hive.apache.org wrote: Subject: Re: Fixing Trunk tests and getting stable nightly build on b.a.o Thanks for asking. Wanted to add, due to high hardware requirements of Hive build (20 hours and the memory like you see) and the limited Apache hardware resources at the time, we had moved the Hive builds to our dedicated Amazon EC2 cluster (currently sponsored by Cloudera, open to other sponsorship), currently of 6 EC2 spot instances of size c1.xlarge. I noticed that this is linked to from the Hive front page bigtop01.cloudera.org:8080/view/Hive/ which is dead... or which times out. For this, we have written a parallel test framework that divides the test suite among the nodes as described here: https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure . It has given pretty good throughput. Brock did the original move, and he or others might have more context of old hive builds running on Apache. I think it is always a good idea to set this up on Apache hardware, but only issue is whether it can meet the requirements, as we have a really huge volume of incoming pre-commit builds that need to be run. Yeah this is an issue for sure. Honestly, I don't have a solution for these types of resources being run on Apache builds infrastructure. We have a similar situation with a goraci [0] test suite for Apache Gora... the only option we have is to run this (when we can) on outside infrastructure. Unfortunately we don't have someone kind enough like Cloudera to run this on though. Lewis [0] https://github.com/keith-turner/goraci
Re: Fixing Trunk tests and getting stable nightly build on b.a.o
Hi Lewis, Are there any specific tests that you are seeing trouble with? If so, please feel free to log appropriate JIRAs to get them fixed (and submit patches ;) ) There is a developer guide[1] that explains in quite detail how to run tests. Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when invoking a build? I think it depends. If you are running the full suite, most probably yes. You might need to do something like export MAVEN_OPTS =-Xmx2g -XX:MaxPermSize=256M. Hope that helps. [1] https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ On Thu, May 29, 2014 at 6:28 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Folks, Is there any interest in getting a stable build on builds.apache.org? I just checked out trunk code and it appears broken out of the box... It also eats up memory like mad. Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when invoking a build? Thanks Lewis -- *Lewis* -- Swarnim
Re: Fixing Trunk tests and getting stable nightly build on b.a.o
Thanks for asking. Wanted to add, due to high hardware requirements of Hive build (20 hours and the memory like you see) and the limited Apache hardware resources at the time, we had moved the Hive builds to our dedicated Amazon EC2 cluster (currently sponsored by Cloudera, open to other sponsorship), currently of 6 EC2 spot instances of size c1.xlarge. For this, we have written a parallel test framework that divides the test suite among the nodes as described here: https://cwiki.apache.org/confluence/display/Hive/Hive+PTest2+Infrastructure. It has given pretty good throughput. Brock did the original move, and he or others might have more context of old hive builds running on Apache. I think it is always a good idea to set this up on Apache hardware, but only issue is whether it can meet the requirements, as we have a really huge volume of incoming pre-commit builds that need to be run. Thanks, Szehon On Fri, May 30, 2014 at 7:19 AM, kulkarni.swar...@gmail.com kulkarni.swar...@gmail.com wrote: Hi Lewis, Are there any specific tests that you are seeing trouble with? If so, please feel free to log appropriate JIRAs to get them fixed (and submit patches ;) ) There is a developer guide[1] that explains in quite detail how to run tests. Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when invoking a build? I think it depends. If you are running the full suite, most probably yes. You might need to do something like export MAVEN_OPTS =-Xmx2g -XX:MaxPermSize=256M. Hope that helps. [1] https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ On Thu, May 29, 2014 at 6:28 PM, Lewis John Mcgibbney lewis.mcgibb...@gmail.com wrote: Hi Folks, Is there any interest in getting a stable build on builds.apache.org? I just checked out trunk code and it appears broken out of the box... It also eats up memory like mad. Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when invoking a build? Thanks Lewis -- *Lewis* -- Swarnim
Fixing Trunk tests and getting stable nightly build on b.a.o
Hi Folks, Is there any interest in getting a stable build on builds.apache.org? I just checked out trunk code and it appears broken out of the box... It also eats up memory like mad. Are there any suggested EXTRA_PARAMETERS to be passing in to the JVM when invoking a build? Thanks Lewis -- *Lewis*
