Peter Bacsko created MAPREDUCE-7056:
---------------------------------------
Summary: Ensure that mapreduce.job.reduces is not negative
Key: MAPREDUCE-7056
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7056
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Peter Bacsko
Assignee: Peter Bacsko
Recently we've seen a strange problem that was related to
{{mapreduce.job.reduces}} being set to -1. If this value is negative, two
things can happen:
1. If we use the old API, then the mappers will pass, but the number of
reducers will be recorded as "-1" if we open it later from JHS. This can
confuse Hadoop users.
2. If use the new API, then we'll see a not-so-obvious stack trace:
{noformat}
2018-02-20 06:37:35,493 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2018-02-20 06:37:35,507 INFO [main] org.apache.hadoop.mapred.MapTask: Starting
flush of map output
2018-02-20 06:37:35,507 INFO [main] org.apache.hadoop.mapred.MapTask: kvbuffer
is null. Skipping flush.
2018-02-20 06:37:35,508 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalArgumentException
at java.nio.ByteBuffer.allocate(ByteBuffer.java:334)
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:51)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1891)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1527)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:735)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:805)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
{noformat}
and the job fails.
We should either fail if this property is negative or set to "0" to avoid this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
