[
https://issues.apache.org/jira/browse/HIVE-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613093#comment-15613093
]
Siddharth Seth edited comment on HIVE-14887 at 10/27/16 8:27 PM:
-----------------------------------------------------------------
Thanks for the review. Committing.
Failing tests fixed by HIVE-14964.
In terms of #of HDFS nodes. Each node is launched as a thread within the same
process (the process launched by maven to execute tests). That's currently 2GB,
and I'll file a separate jira to reduce this after this one is committed. When
moving this to 1GB - I didn't really see any failures related to HDFS itself.
A 4 node MiniYARNCluster runs all 4 nodes as threads as well. However, the
difference is that the containers that run on the NodeManagers launch as
separate processes. Without this patch - we would run 4 NMs, with 4 1GB
containers each - so each test could end up running with 16GB. Post the changes
there's 2 NMs, with 4x128MB. So this gets way lower.
was (Author: sseth):
Thanks for the review. Committing.
In terms of #of HDFS nodes. Each node is launched as a thread within the same
process (the process launched by maven to execute tests). That's currently 2GB,
and I'll file a separate jira to reduce this after this one is committed. When
moving this to 1GB - I didn't really see any failures related to HDFS itself.
A 4 node MiniYARNCluster runs all 4 nodes as threads as well. However, the
difference is that the containers that run on the NodeManagers launch as
separate processes. Without this patch - we would run 4 NMs, with 4 1GB
containers each - so each test could end up running with 16GB. Post the changes
there's 2 NMs, with 4x128MB. So this gets way lower.
> Reduce the memory used by MiniMr, MiniTez, MiniLlap tests
> ---------------------------------------------------------
>
> Key: HIVE-14887
> URL: https://issues.apache.org/jira/browse/HIVE-14887
> Project: Hive
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Fix For: 2.2.0
>
> Attachments: HIVE-14887.01.patch, HIVE-14887.02.patch,
> HIVE-14887.03.patch, HIVE-14887.04.patch, HIVE-14887.05.patch,
> HIVE-14887.06.patch
>
>
> The clusters that we spin up end up requiring 16GB at times. Also the maven
> arguments seem a little heavy weight.
> Reducing this will allow for additional ptest drones per box, which should
> bring down the runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
