After some experiments, I figured out the following: 1. Druid uses above 8gb of memory for testing. (building-druid.png) 2. With 8gb(physical)+4gb(swap) of memory, the test succeeds regardless of maven version (3.3.9, 3.5.2, 3.5.4) or MAVEN_OPTS. However, with 8gb(physical)+2gb(swap) of memory[^1], some tests failed. The list of failing tests differs between maven 3.3.9 and 3.5.2.
In short, retaining sufficient memory solved the problem - *It seems like 12gb of memory is a recommended setting for building druid.* (I guess lots of you are working with the MacBook Pro with 16gb RAM, right? In that case, you must not have encountered this problem.) If you are okay, may I update the building documentation for the newbies like me? Thanks, Dongjin +1. While building Druid, I found another problem. But this issue should be discussed in another thread. [^1]: You know, the other processes also occupy the memory. On Tue, Jul 24, 2018 at 3:07 AM Jihoon Son <jihoon...@apache.org> wrote: > I'm also using Maven 3.5.2 and not using any special configurations for > Maven, but I have never seen that error too. > Most of our Travis jobs have been working with only 512 MB of direct > memory. Only the 'strict compilation' Travis job requires 3 GB of memory. > > I think it's worthwhile to look into this more. Maybe we somehow use more > memory when we run all tests by 'mvn install'. Maybe this relates to the > frequent transient failures of 'processing module test', one of our Travis > jobs. > > Jihoon > > On Mon, Jul 23, 2018 at 9:32 AM Gian Merlino <g...@apache.org> wrote: > > > Interesting. Fwiw, I am using Maven 3.5.2 for building Druid and it has > > been working for for me. I don't think I"m using any special Maven > > overrides (at least, I don't see anything interesting in my ~/.m2 > directory > > or in my environment variables). It might have to do with how much memory > > our machines have? I do most of my builds on a Mac with 16GB RAM. Maybe > try > > checking .travis.yml in the druid repo. It sets -Xmx3000m for mvn install > > commands, which might be needed for more low memory environments. > > > > $ mvn --version > > Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; > > 2017-10-18T00:58:13-07:00) > > Maven home: /usr/local/Cellar/maven/3.5.2/libexec > > Java version: 1.8.0_161, vendor: Oracle Corporation > > Java home: > > /Library/Java/JavaVirtualMachines/jdk1.8.0_161.jdk/Contents/Home/jre > > Default locale: en_US, platform encoding: UTF-8 > > OS name: "mac os x", version: "10.13.5", arch: "x86_64", family: "mac" > > > > On Mon, Jul 23, 2018 at 6:40 AM Dongjin Lee <dong...@apache.org> wrote: > > > > > Finally, it seems like I found the reason. It was a composition of > > several > > > problems: > > > > > > - Druid should not be built with maven 3.5.x. With 3.5.2, Test suites > > like > > > `GroupByQueryRunnerFailureTest` fails. After I switched into 3.3.9 > which > > is > > > built in the latest version of IntelliJ, those errors disappeared. It > > seems > > > like maven 3.5.x is not stable yet - it applied a drastic change, and > it > > is > > > also why they skipped 3.4.x. > > > - It seems like Druid requires some MaxDirectMemorySize configuration > for > > > some test suites. With some JVM parameter like > > `-XX:MaxDirectMemorySize=4g` > > > some test suites were passed, but not all. I am now trying the other > > > options with enlarged swap space. > > > > > > Question: How much MaxDirectMemorySize configuration are you using? > > > > > > Best, > > > Dongjin > > > > > > On Sat, Jul 21, 2018 at 3:01 AM Jihoon Son <ghoon...@gmail.com> wrote: > > > > > > > Hi Dongjin, > > > > > > > > that is weird. It looks like the vm crashed because of out of memory > > > while > > > > testing. > > > > It might be a real issue or not. > > > > Have you set any memory configuration for your maven? > > > > > > > > Jihoon > > > > > > > > On Thu, Jul 19, 2018 at 7:09 PM Dongjin Lee <dong...@apache.org> > > wrote: > > > > > > > > > Hi Jihoon, > > > > > > > > > > I ran `mvn clean package` following development/build > > > > > < > > > > > > > > > > > > > > > https://github.com/apache/incubator-druid/blob/master/docs/content/development/build.md > > > > > > > > > > > . > > > > > > > > > > Dongjin > > > > > > > > > > On Fri, Jul 20, 2018 at 12:30 AM Jihoon Son <jihoon...@apache.org> > > > > wrote: > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > what maven command did you run? > > > > > > > > > > > > Jihoon > > > > > > > > > > > > On Wed, Jul 18, 2018 at 10:38 PM Dongjin Lee <dong...@apache.org > > > > > > wrote: > > > > > > > > > > > > > Hello. I am trying to build druid, but it fails. My environment > > is > > > > like > > > > > > the > > > > > > > following: > > > > > > > > > > > > > > - CPU: Intel(R) Core(TM) i7-7560U CPU @ 2.40GHz > > > > > > > - RAM: 7704 MB > > > > > > > - OS: ubuntu 18.04 > > > > > > > - JDK: openjdk version "1.8.0_171" (default configuration, with > > > > > > MaxHeapSize > > > > > > > = 1928 MB) > > > > > > > - Branch: master (commit: cd8ea3d) > > > > > > > > > > > > > > The error message I got is: > > > > > > > > > > > > > > [INFO] > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > [INFO] Reactor Summary: > > > > > > > > [INFO] > > > > > > > > [INFO] io.druid:druid ..................................... > > > > SUCCESS [ > > > > > > > > 50.258 s] > > > > > > > > [INFO] java-util .......................................... > > > SUCCESS > > > > > > > [03:57 > > > > > > > > min] > > > > > > > > [INFO] druid-api .......................................... > > > > SUCCESS [ > > > > > > > > 22.694 s] > > > > > > > > [INFO] druid-common ....................................... > > > > SUCCESS [ > > > > > > > > 14.083 s] > > > > > > > > [INFO] druid-hll .......................................... > > > > SUCCESS [ > > > > > > > > 17.126 s] > > > > > > > > [INFO] extendedset ........................................ > > > > SUCCESS [ > > > > > > > > 10.856 s] > > > > > > > > > > > > > > > > *[INFO] druid-processing ................................... > > > > FAILURE > > > > > > > > [04:36 min]*[INFO] druid-aws-common > > > > > ................................... > > > > > > > > SKIPPED > > > > > > > > [INFO] druid-server ....................................... > > > SKIPPED > > > > > > > > [INFO] druid-examples ..................................... > > > SKIPPED > > > > > > > > ... > > > > > > > > [INFO] > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > [INFO] BUILD FAILURE > > > > > > > > [INFO] > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > [INFO] Total time: 10:29 min > > > > > > > > [INFO] Finished at: 2018-07-19T13:23:31+09:00 > > > > > > > > [INFO] Final Memory: 88M/777M > > > > > > > > [INFO] > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > > > > > *[ERROR] Failed to execute goal > > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test > > > > > > (default-test) > > > > > > > > on project druid-processing: Execution default-test of goal > > > > > > > > org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test > > > failed: > > > > > The > > > > > > > > forked VM terminated without properly saying goodbye. VM > crash > > or > > > > > > > > System.exit called?*[ERROR] Command was /bin/sh -c cd > > > > > > > > /home/djlee/workspace/java/druid/processing && > > > > > > > > /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx3000m > > > > > > > -Duser.language=en > > > > > > > > -Duser.country=US -Dfile.encoding=UTF-8 -Duser.timezone=UTC > > > > > > > > > > > -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager > > > > > > > > -Ddruid.indexing.doubleStorage=double -jar > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/djlee/workspace/java/druid/processing/target/surefire/surefirebooter1075382243904099051.jar > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/djlee/workspace/java/druid/processing/target/surefire/surefire5444459351134757209tmp > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > /home/djlee/workspace/java/druid/processing/target/surefire/surefire_5173894389718744688tmp > > > > > > > > > > > > > > > > > > > > > It seems like it fails when it runs tests on `druid-processing` > > > > module > > > > > > but > > > > > > > I can't certain. Is there anyone who can give me some hints? > > Thanks > > > > in > > > > > > > advance. > > > > > > > > > > > > > > Best, > > > > > > > Dongjin > > > > > > > > > > > > > > -- > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > > > > <http://github.com/dongjinleekr>linkedin: > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > > > > > www.slideshare.net/dongjinleekr > > > > > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > > > > > > > > > > > -- > > > > > > *Dongjin Lee* > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > > > <http://github.com/dongjinleekr>linkedin: > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > > > www.slideshare.net/dongjinleekr > > > > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > > > > > > > > > > > -- > > > > *Dongjin Lee* > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > <http://github.com/dongjinleekr>linkedin: > > > kr.linkedin.com/in/dongjinleekr > > > > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > > > www.slideshare.net/dongjinleekr > > > > <http://www.slideshare.net/dongjinleekr>* > > > > > > > > > > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > *github: <http://goog_969573159/>github.com/dongjinleekr > <http://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <http://kr.linkedin.com/in/dongjinleekr>slideshare: > www.slideshare.net/dongjinleekr > <http://www.slideshare.net/dongjinleekr>* >
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
