[
https://issues.apache.org/jira/browse/HIVE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278679#comment-17278679
]
Zoltan Matyus commented on HIVE-24734:
--------------------------------------
My proposal would be this:
{code:java}
if (getContext() != null) {
totalResource = getContext().getTotalAvailableResource().getMemory();
taskResource = getContext().getVertexTaskResource().getMemory();
availableSlots = totalResource / taskResource;
LOG.info(
"Using availableSlots: {} calculated from (total memory: {} / task
memory: {})",
availableSlots, totalResource, taskResource);
if(availableSlots < 1) {
LOG.warn("Correcting availableSlots to 1 (from {}).", availableSlots);
availableSlots = 1;
}
}
{code}
I'm not opening a PR yet, I'm not a Tez expert and I'd like some opinions
before starting the precommit machinery...
> Sanity check in HiveSplitGenerator available slot calculation
> -------------------------------------------------------------
>
> Key: HIVE-24734
> URL: https://issues.apache.org/jira/browse/HIVE-24734
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Affects Versions: 4.0.0
> Reporter: Zoltan Matyus
> Priority: Minor
>
> HiveSplitGenerator calculates the number of available slots from available
> memory like this:
> {code:java}
> if (getContext() != null) {
> totalResource = getContext().getTotalAvailableResource().getMemory();
> taskResource = getContext().getVertexTaskResource().getMemory();
> availableSlots = totalResource / taskResource;
> }
> {code}
> I had a scenario where the total memory was calculated correctly, but the
> task memory returned -1. This led to error like these:
> {noformat}
> tez.HiveSplitGenerator: Number of input splits: 1. -3641 available slots, 1.7
> waves. Input format is: org.apache.hadoop.hive.ql.io.HiveInputFormat
> Estimated number of tasks: -6189 for bucket 1
> java.lang.IllegalArgumentException: Illegal Capacity: -6189
> {noformat}
> Admittedly, this happened during development, and hopefully will not occur on
> a properly configured cluster. (Although I'm not sure what the issue was on
> my setup, possibly XMX set higher than physical memory.)
> In any case, it feels like setting availableSlots < 1 will never lead to
> desired behavior, so in such cases we could emit a warning and correct the
> value to 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
