[
https://issues.apache.org/jira/browse/GRIFFIN-295?focusedWorklogId=337209&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337209
]
ASF GitHub Bot logged work on GRIFFIN-295:
------------------------------------------
Author: ASF GitHub Bot
Created on: 01/Nov/19 06:39
Start Date: 01/Nov/19 06:39
Worklog Time Spent: 10m
Work Description: joohnnie commented on issue #546: [GRIFFIN-295]Limit
the memory used by test case
URL: https://github.com/apache/griffin/pull/546#issuecomment-548677470
HI @wankunde ,
First, you are right, there is a memory limit for the Travis container.
As i mentioned in this PR https://github.com/apache/griffin/pull/543.
I copied my testing result below
```
I tried to build with the following Maven Options and they are running
successfully.
export MAVEN_OPTS="-Xms500m -Xmx1g -XX:MaxPermSize=512m
https://travis-ci.org/joohnnie/griffin/builds/601486973
and
export MAVEN_OPTS="-Xms256m -Xmx1g -XX:MaxPermSize=256m
https://travis-ci.org/joohnnie/griffin/builds/601561953
Looks like there is memory limit for each travis CI JVM.
```
This kind of constraint is related to the Travis CI container.
IMO, I think it is a good practice to not put this kind of limitation to our
code and POM configs if it can be solved by the travis config .travis.yml.
To put it another way, if in the future we use other CI tools to replace
Travis and there is no this kind
of memory limitation, we need to change the config one more time to make it
work on the new CI tools.
Based on my experience, the Travis issues mainly related to the 2
constraints,
- one is memory
- the other one is No Log in 10 minutes/ or too much log to exceed the log
size.
That's why we need to carefully define the log level and memory.
If the issue cannot be solved by changing travis config, we need to think
about changing the code or config.
Thanks
Johnnie
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 337209)
Time Spent: 1h (was: 50m)
> Limit the memory used by test case
> ----------------------------------
>
> Key: GRIFFIN-295
> URL: https://issues.apache.org/jira/browse/GRIFFIN-295
> Project: Griffin
> Issue Type: Improvement
> Reporter: Wan Kun
> Priority: Major
> Attachments: 1.png, 2.png, 3.png, 4.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The container memory size is 3G in travis, but out test cases always uses
> more than 3G memory, so `Cannot allocate memory` will be thrown.
>
> {code:java}
> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
> os::commit_memory(0x00000000fe980000, 23592960, 0) failed; error='Cannot
> allocate memory' (errno=12)
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 23592960 bytes for committing
> reserved memory.
> # An error report file with more information is saved as:
> # /home/travis/build/apache/griffin/measure/hs_err_pid11948.log
> # [ timer expired, abort... ]
> {code}
>
> There are two kind of programs in our tests, the maven main program and the
> tests run by maven-surefire-plugin and scalatest-maven-plugin.
> If the memory is not limited, test cases will occupy as much memory as
> possible especially spark jobs.
> Spark jobs will not free the memory until a full GC occurs , even if we have
> stopped the spark context .so we need to limit the momery used by test cases.
> We can limit the maven memory used by set export MAVEN_OPTS=" -Xmx1024m
> -XX:ReservedCodeCacheSize=128m" , and we can limit the memory used by spark
> job tests by configuring the maven-surefire-plugin and scalatest-maven-plugin.
>
> For example:
> Before we limit the memory used, maven program occupy 1.5G memory and spark
> job occupy 1.8G memory.
> !1.png!
> !2.png!
>
> After we limit the memory used, maven program occupy 1G memory and spark job
> occupy 1G memory.
>
> !3.png!
>
> !4.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
