[
https://issues.apache.org/jira/browse/TEZ-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941308#comment-13941308
]
Bikas Saha commented on TEZ-962:
--------------------------------
It means that the initial desired number of splits is too large.
The code calculates the length per split by dividing the total length by the
desired-num-splits. If length per split < min-length-per-split then the code
thinks that the desired number of splits is too large (because the length per
splits is too small). Does that clarify the log? Patch to improve the language
is welcome.
In the case of extremely small total length (say 0), because of the small
length, the above logic kicks in even though the desired num splits is actually
small. But the new value of desired num splits still a sane value.
> Logs need to be improved
> ------------------------
>
> Key: TEZ-962
> URL: https://issues.apache.org/jira/browse/TEZ-962
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.4.0
> Reporter: Vikram Dixit K
>
> 2014-03-19 17:55:29,044 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapred.split.TezMapredSplitsGrouper: Desired splits: 1 too
> large. Desired splitLength: 224 Min splitLength: 16777216 New desired
> splits: 1 Total length: 224 Original splits: 4
> 2014-03-19 17:55:29,045 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapred.split.TezMapredSplitsGrouper: Number of splits
> desired: 1 created: 3 splitsProcessed: 4
> Desired splits too large? But had 4 splits and generated 4.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
