Jonathan, Hm... I seeing the "Using zinc server for incremental compilation" messages in compile output. For instance https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/357/BUILD_ENVIRONMENTS=ubuntu-16.04,COMPONENTS=spark,label=docker-slave/consoleFull
According to https://github.com/davidB/scala-maven-plugin/blob/master/src/main/java/sbt_inc/SbtIncrementalCompiler.java Line 48 it is using the zinc server through nailgun. I am not sure that maven-scala-plugin does start zinc, but it looks like something started a zinc server and is doing weird things in our CI Infra, since I get non predicatable failures. Compare for instance https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/357/ (before disabling zinc and after disabling zinc https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/359/) Yep, the compile time increased from 45 Min to 60 Min for spark 2 in our infra. I am not too happy with that too, but the situation has improved greatly by this patch. I do not have any clue about scala: If you have any other idea how to improve (or debug) the situation please tell us. I disabled incrementalCompile because I thought that it is essential for having Zinc disabled. Now reading through the plugin Source I see that it is possible to have incremental compilation without Zinc. Does it have an influence to the compile time ? Olaf > Am 28.11.2016 um 18:38 schrieb Jonathan Kelly <[email protected]>: > > Olaf, are you absolutely sure this is what is happening though? According > to http://davidb.github.io/scala-maven-plugin/example_incremental.html, the > useZincServer property doesn't actually seem to *start* Zinc; it just > causes the build to use it if it is available. This is how the Spark build > has always worked previously. > > If the Spark 2.x build is in fact starting Zinc by itself now, I didn't > realize that was the case. I've still been starting Zinc prior to the > Bigtop build manually in my own builds. > > BTW, even though we are not doing incremental builds for Spark, this isn't > the only reason to use Zinc; with Zinc running, the Spark build is cut in > *half* (from ~40 mins to ~20 mins last I checked). However, if build time > isn't an issue, of course it's fine to disable it, especially if it's > causing problems in the Bigtop CI. > > ~ Jonathan > > On Thu, Nov 24, 2016 at 12:29 PM Konstantin Boudnik <[email protected]> wrote: > >> +1 - let's get rid of this crap. >> >> On Thu, Nov 24, 2016 at 06:03PM, Olaf Flebbe wrote: >>> Hi, >>> >>> I think I got a clue why spark compile fails in our CI, but this may not >> explain why compile fails for Ganesh. >>> >>> The compile job in CI is started with option >>> >>> docker run --rm -v `pwd`:/ws --workdir /ws -e COMPONENTS=$COMPONENTS >> --net=container:nexus bigtop/slaves:trunk-$BUILD_ENVIRONMENTS \ >>> bash -c '. /etc/profile.d/bigtop.sh; ./gradlew allclean configure-nexus >> $COMPONENTS-pkg' >>> >>> The "--net=container:nexus" is necessary to get access to the private >> subnet of the nexus container to be used as a maven repository securely. >>> >>> However, if a process within the compile job happens to open a server >> port, this port may be accessible from any other container running on the >> same machine as well, since they are shareing their network! >>> >>> I just read about nailgun and I saw that the zinc server is started at >> spark compile job unconditionally in the pom.xml. So the problem may be a >> "cross docker compile" race condition: When the docker container servicing >> the zinc server stops, all other compile host will fail at once, without >> any error message. Or it will confuse libraries in the same namespace. >> There are sign of this errors in the CI logs, too. >>> >>> Given that our build process is not an incremental compile, I vote for >> disabling the zinc madness. >>> (i.e. remove useZincServer in pom.xml of spark) >>> >>> Other possible attempts is to give up caching of artifacts. >>> >>> >>> Thoughts? >>> Olaf >>> >>
