[
https://issues.apache.org/jira/browse/HIVE-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282660#comment-17282660
]
Zoltan Matyus commented on HIVE-24734:
--------------------------------------
[~abstractdog]: could you comment on this? This is probably an edge case, but I
see no harm in making the change...
> Sanity check in HiveSplitGenerator available slot calculation
> -------------------------------------------------------------
>
> Key: HIVE-24734
> URL: https://issues.apache.org/jira/browse/HIVE-24734
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Affects Versions: 4.0.0
> Reporter: Zoltan Matyus
> Priority: Minor
>
> HiveSplitGenerator calculates the number of available slots from available
> memory like this:
> {code:java}
> if (getContext() != null) {
> totalResource = getContext().getTotalAvailableResource().getMemory();
> taskResource = getContext().getVertexTaskResource().getMemory();
> availableSlots = totalResource / taskResource;
> }
> {code}
> I had a scenario where the total memory was calculated correctly, but the
> task memory returned -1. This led to error like these:
> {noformat}
> tez.HiveSplitGenerator: Number of input splits: 1. -3641 available slots, 1.7
> waves. Input format is: org.apache.hadoop.hive.ql.io.HiveInputFormat
> Estimated number of tasks: -6189 for bucket 1
> java.lang.IllegalArgumentException: Illegal Capacity: -6189
> {noformat}
> Admittedly, this happened during development, and hopefully will not occur on
> a properly configured cluster. (Although I'm not sure what the issue was on
> my setup, possibly XMX set higher than physical memory.)
> In any case, it feels like setting availableSlots < 1 will never lead to
> desired behavior, so in such cases we could emit a warning and correct the
> value to 1.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
