[
https://issues.apache.org/jira/browse/TEZ-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308556#comment-15308556
]
Siddharth Seth commented on TEZ-3282:
-------------------------------------
I didn't really see it like that. Was contemplating whether to open the jira as
a Bug or an Improvement. Was always under the impression that we set Xms and
Xmx values when computing this.
The motivation here was that Hive precommit runs were running far too many
containers on a node - this would cause all kind of random failures with the
mvn test runner being killed, allocate failures etc. I'm sure containers were
seeing the same errors. I would say it's better to get an error up front saying
there isn't enough memory, rather than getting random failures when a container
may end up hitting a limit.
For now - should we put this behind a config ? Until pluggable java_opt
providers are in place.
> Set the Xms value when Tez computes the heap size for a container
> -----------------------------------------------------------------
>
> Key: TEZ-3282
> URL: https://issues.apache.org/jira/browse/TEZ-3282
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-3282.01.txt
>
>
> Currently, we only set the Xmx value.
> Another useful change would be to change the calculation to not use the
> fraction blindly. For large containers - .e.g 10GB - this ends up leaving 2G
> of the container unused.
> Instead we can add another parameter which indicates the max amount of memory
> to leave outside the heap. Maybe a separate jira for this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
