Cameron Lee created SAMZA-2481:
----------------------------------
Summary: Improve memory usage in samza-core unit tests
Key: SAMZA-2481
URL: https://issues.apache.org/jira/browse/SAMZA-2481
Project: Samza
Issue Type: Improvement
Reporter: Cameron Lee
Gradle 5 reduced the default memory allocation for test executors. Before
Gradle 5, our unit tests were given plenty of memory, so they ran fine. When
upgrading to Gradle 5, we noticed some issues with running tests. Setting the
memory (to 3GB heap) explicitly seems to have resolved the issues, but it seems
odd that unit tests need so much. It would be good to investigate this possible
inefficiency. It might also help build times, since heavy memory usage could
mean a unnecessarily heavy unit test.
One potential example is TestZkMetadataStore. It tests large zookeeper values
(10MB), and that seems to cause some issues. If heap usage is set to 1GB or
1.5GB, then that test takes a long time to execute (seems like it might be
running out of memory). This can be reproduced by changing the heap usage for
"test" for the samza-core module in build.gradle, and then running "./gradlew
:samza-core:test". By setting the zookeeper value to 1MB, the tests will
execute fine. It is odd that the test works on the scale of MB but seems to
have memory requirements on the scale of GB.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
