[
https://issues.apache.org/jira/browse/SAMZA-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cameron Lee updated SAMZA-2481:
-------------------------------
Description:
Gradle 5 reduced the default memory allocation for test executors. Before
Gradle 5, our unit tests were given plenty of memory, so they ran fine. When
upgrading to Gradle 5, we noticed some issues with running tests. Setting the
memory (to 3GB heap) explicitly seems to have resolved the issues, but it seems
odd that unit tests need so much. It would be good to investigate this possible
inefficiency. It might also help build times, since heavy memory usage could
mean a unnecessarily heavy unit test.
One potential example is TestZkMetadataStore. It tests large zookeeper values
(10MB), and that seems to cause some issues. If heap usage is set to 1GB or
1.5GB, then that test takes a long time to execute (seems like it might be
running out of memory). This can be reproduced by changing the heap usage for
"test" for the samza-core module in build.gradle, and then running "./gradlew
:samza-core:test". By setting the zookeeper value to 1MB (and using 1GB of
heap), the tests will execute fine. It is odd that the test works on the scale
of MB but seems to have memory requirements on the scale of GB.
was:
Gradle 5 reduced the default memory allocation for test executors. Before
Gradle 5, our unit tests were given plenty of memory, so they ran fine. When
upgrading to Gradle 5, we noticed some issues with running tests. Setting the
memory (to 3GB heap) explicitly seems to have resolved the issues, but it seems
odd that unit tests need so much. It would be good to investigate this possible
inefficiency. It might also help build times, since heavy memory usage could
mean a unnecessarily heavy unit test.
One potential example is TestZkMetadataStore. It tests large zookeeper values
(10MB), and that seems to cause some issues. If heap usage is set to 1GB or
1.5GB, then that test takes a long time to execute (seems like it might be
running out of memory). This can be reproduced by changing the heap usage for
"test" for the samza-core module in build.gradle, and then running "./gradlew
:samza-core:test". By setting the zookeeper value to 1MB, the tests will
execute fine. It is odd that the test works on the scale of MB but seems to
have memory requirements on the scale of GB.
> Improve memory usage in samza-core unit tests
> ---------------------------------------------
>
> Key: SAMZA-2481
> URL: https://issues.apache.org/jira/browse/SAMZA-2481
> Project: Samza
> Issue Type: Improvement
> Reporter: Cameron Lee
> Priority: Major
>
> Gradle 5 reduced the default memory allocation for test executors. Before
> Gradle 5, our unit tests were given plenty of memory, so they ran fine. When
> upgrading to Gradle 5, we noticed some issues with running tests. Setting the
> memory (to 3GB heap) explicitly seems to have resolved the issues, but it
> seems odd that unit tests need so much. It would be good to investigate this
> possible inefficiency. It might also help build times, since heavy memory
> usage could mean a unnecessarily heavy unit test.
> One potential example is TestZkMetadataStore. It tests large zookeeper values
> (10MB), and that seems to cause some issues. If heap usage is set to 1GB or
> 1.5GB, then that test takes a long time to execute (seems like it might be
> running out of memory). This can be reproduced by changing the heap usage for
> "test" for the samza-core module in build.gradle, and then running "./gradlew
> :samza-core:test". By setting the zookeeper value to 1MB (and using 1GB of
> heap), the tests will execute fine. It is odd that the test works on the
> scale of MB but seems to have memory requirements on the scale of GB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
