Yeah Sean so the setup I had didn't really care about parallelizing in Maven. It just stashed the built artifacts and moved them onto the slaves running tests and tests for each submodule ran in a separate docker container. After each subbuild was done the build logs were transferred back and aggregated. I actually don't know much about the permissions and capabilities of the ASF Jenkins and how projects get on there but as long as we can use amplab jenkins the same way with docker containers then I think it is totally fine to stay there.
Xin On Wed, Nov 8, 2017 at 1:57 AM, Sean Owen <so...@cloudera.com> wrote: > On a related note, I believe Hyukjin has been requesting access to trigger > tests on amplab Jenkins for some time. Would be great to add him. > > > On Wed, Nov 8, 2017, 4:25 AM Felix Cheung <felixcheun...@hotmail.com> > wrote: > >> We actually have some immediate needs for custom config for some upcoming >> integration tests. >> >> I don’t know if such changes are possible in ASF Jenkins but the work is >> in progress in RISELab Jenkins :) >> >> >> ------------------------------ >> *From:* holden.ka...@gmail.com <holden.ka...@gmail.com> on behalf of >> Holden Karau <hol...@pigscanfly.ca> >> *Sent:* Tuesday, November 7, 2017 2:14:18 PM >> *To:* Sean Owen >> *Cc:* Xin Lu; dev@spark.apache.org >> *Subject:* Re: Jenkins upgrade/Test Parallelization & Containerization >> >> True, I think we've seen that the Amp Lab Jenkins needs to be more >> focused on running AMP Lab projects, and while I don't know how difficult >> the ASF Jenkins is I assume it might be an easier place to make changes >> going forward? (Of course this could be the grass is greener on the other >> side and I don't mean to say it's been hard to make changes on the AMP lab >> hardware, folks have been amazingly helpful - its just the projects on each >> have different needs). >> >> On Tue, Nov 7, 2017 at 12:52 PM, Sean Owen <so...@cloudera.com> wrote: >> >>> Faster tests would be great. I recall that the straightforward ways to >>> parallelize via Maven haven't worked because many tests collide with one >>> another. Is this about running each module's tests in a container? that >>> should work. >>> >>> I can see how this is becoming essential for repeatable and reliable >>> Python/R builds, which depend on the environment to a much greater extent >>> than the JVM does. >>> >>> I don't have a strong preference for AMPLab vs ASF builds. I suppose >>> using the ASF machinery is a little tidier. If it's got a later Jenkins >>> that's required, also a plus, but I assume updating AMPLab isn't so hard >>> here either. I think the key issue is which environment is easier to >>> control and customize over time. >>> >>> >>> On Wed, Nov 1, 2017 at 6:05 AM Xin Lu <x...@salesforce.com> wrote: >>> >>>> Hi everyone, >>>> >>>> I tried sending emails to this list and I'm not sure if it went through >>>> so I'm trying again. Anyway, a couple months ago before I left Databricks >>>> I was working on a proof of concept that parallelized Spark tests on >>>> jenkins. The way it worked was basically it build the spark jars and then >>>> ran all the tests in a docker container on a bunch of slaves in parallel. >>>> This cut the testing time down from 4 hours to approximately 1.5 hours. >>>> This required a newer version of jenkins and the Jenkins Pipeline plugin. >>>> I am wondering if it is possible to do this on amplab jenkins. It looks >>>> like https://builds.apache.org/ has upgraded so Amplabs jenkins is a >>>> year or so behind. I am happy to help with this project if it is something >>>> that people think is worthwhile. >>>> >>>> Thanks >>>> >>>> Xin >>>> >>> >> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> >
