[
https://issues.apache.org/jira/browse/GRIFFIN-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wan Kun updated GRIFFIN-295:
----------------------------
Description:
The container memory size is 3G in travis, but out test cases always uses more
than 3G memory, so `Cannot allocate memory` will be thrown.
{code:java}
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00000000fe980000, 23592960, 0) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 23592960 bytes for committing
reserved memory.
# An error report file with more information is saved as:
# /home/travis/build/apache/griffin/measure/hs_err_pid11948.log
# [ timer expired, abort... ]
{code}
There are two kind of programs in our tests, the maven main program and the
tests run by maven-surefire-plugin and scalatest-maven-plugin.
If the memory is not limited, test cases will occupy as much memory as possible
especially spark jobs.
Spark jobs will not free the memory until a full GC occurs , even if we have
stopped the spark context .so we need to limit the momery used by test cases.
We can limit the maven memory used by set export MAVEN_OPTS=" -Xmx1024m
-XX:ReservedCodeCacheSize=128m" , and we can limit the memory used by spark job
tests by configuring the maven-surefire-plugin and scalatest-maven-plugin.
For example:
Before we limit the memory used, maven program occupy 1.5G memory and spark
job occupy 1.8G memory.
!1.png!
!2.png!
After we limit the memory used, maven program occupy 1G memory and spark job
occupy 1G memory.
!3.png!
!4.png!
was:
The container memory size is 3G in travis, but out test cases always uses more
than 3G memory, so `Cannot allocate memory` will be thrown.
{code:java}
Java HotSpot(TM) 64-Bit Server VM warning: INFO:
os::commit_memory(0x00000000fe980000, 23592960, 0) failed; error='Cannot
allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 23592960 bytes for committing
reserved memory.
# An error report file with more information is saved as:
# /home/travis/build/apache/griffin/measure/hs_err_pid11948.log
# [ timer expired, abort... ]
{code}
There are two kind of programs in our tests, the maven main program and the
tests run by maven-surefire-plugin and scalatest-maven-plugin.
If the memory is not limited, test cases will occupy more and more memory
especially spark jobs.
Each spark job has a HeapMemoryAllocator object for managing memory,which
stores many internal objects in memory to speed up calculation.
But even if we stop the spark context, it will not release memory, which is
strange. so we need to limit the momery used by test cases.
We can limit the maven memory used by set export MAVEN_OPTS=" -Xmx1024m
-XX:ReservedCodeCacheSize=128m" , and we can limit the memory used by spark job
tests by configuring the maven-surefire-plugin and scalatest-maven-plugin.
For example:
Before we limit the memory used, maven program occupy 1.5G memory and spark job
occupy 1.8G memory.
!1.png!
!2.png!
After we limit the memory used, maven program occupy 1G memory and spark job
occupy 1G memory.
!3.png!
!4.png!
> Limit the memory used by test case
> ----------------------------------
>
> Key: GRIFFIN-295
> URL: https://issues.apache.org/jira/browse/GRIFFIN-295
> Project: Griffin
> Issue Type: Improvement
> Reporter: Wan Kun
> Priority: Major
> Attachments: 1.png, 2.png, 3.png, 4.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The container memory size is 3G in travis, but out test cases always uses
> more than 3G memory, so `Cannot allocate memory` will be thrown.
>
> {code:java}
> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
> os::commit_memory(0x00000000fe980000, 23592960, 0) failed; error='Cannot
> allocate memory' (errno=12)
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 23592960 bytes for committing
> reserved memory.
> # An error report file with more information is saved as:
> # /home/travis/build/apache/griffin/measure/hs_err_pid11948.log
> # [ timer expired, abort... ]
> {code}
>
> There are two kind of programs in our tests, the maven main program and the
> tests run by maven-surefire-plugin and scalatest-maven-plugin.
> If the memory is not limited, test cases will occupy as much memory as
> possible especially spark jobs.
> Spark jobs will not free the memory until a full GC occurs , even if we have
> stopped the spark context .so we need to limit the momery used by test cases.
> We can limit the maven memory used by set export MAVEN_OPTS=" -Xmx1024m
> -XX:ReservedCodeCacheSize=128m" , and we can limit the memory used by spark
> job tests by configuring the maven-surefire-plugin and scalatest-maven-plugin.
>
> For example:
> Before we limit the memory used, maven program occupy 1.5G memory and spark
> job occupy 1.8G memory.
> !1.png!
> !2.png!
>
> After we limit the memory used, maven program occupy 1G memory and spark job
> occupy 1G memory.
>
> !3.png!
>
> !4.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
