[
https://issues.apache.org/jira/browse/HIVE-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609224#comment-15609224
]
Siddharth Seth commented on HIVE-14887:
---------------------------------------
[~spena] - the latest patch essentially sets up the cluster properly. Earlier,
the intent was to use a 128MB task, but YARN was setup with 1GB containers.
That's changed to move to 128MB containers, and the AM size updated to 128MB.
It also cuts the maven heap size to 256MB (this is for the launcher, and not
the VM running the tests). The mvn opts in source-prep remains unchanged - and
more than 1G is required to compile Hive.
The initial patch was also changing the heapsize used for regular tests (unit
tests, cliDriver, MiniLlapLocal, MiniSparkLocal etc). I'm going to split that
into another patch though, given the extreme flakiness of the tests.
In terms of what is being used - it's really about trying a low enough value
which does not result in failures. Not measuring what is actually being used.
Since 128MB is what was configured - that should be sufficient.
For the regular unit tests - they're setup to use 2GB - that seems a little
excessive. Will file a follow up jira to reduce this.
Need a review on this. cc [~prasanth_j], [~spena]
> Reduce the memory requirements for tests
> ----------------------------------------
>
> Key: HIVE-14887
> URL: https://issues.apache.org/jira/browse/HIVE-14887
> Project: Hive
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: HIVE-14887.01.patch, HIVE-14887.02.patch,
> HIVE-14887.03.patch, HIVE-14887.04.patch, HIVE-14887.05.patch,
> HIVE-14887.06.patch
>
>
> The clusters that we spin up end up requiring 16GB at times. Also the maven
> arguments seem a little heavy weight.
> Reducing this will allow for additional ptest drones per box, which should
> bring down the runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
