[jira] [Created] (MAPREDUCE-6009) Map-only job with new-api runs wrong OutputCommitter when cleanup scheduled in a reduce slot

[Gera Shegalov (JIRA)]

Gera Shegalov created MAPREDUCE-6009:


 Summary: Map-only job with new-api runs wrong OutputCommitter when 
cleanup scheduled in a reduce slot
 Key: MAPREDUCE-6009
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6009
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, job submission
Affects Versions: 1.2.1
Reporter: Gera Shegalov
Assignee: Gera Shegalov
Priority: Blocker


In branch 1 job commit is executed in a JOB_CLEANUP task that may run in either 
map or reduce slot

in org.apache.hadoop.mapreduce.Job#setUseNewAPI there is a logic setting 
new-api flag only for reduce-ful jobs.
{code}
if (numReduces != 0) {
  conf.setBooleanIfUnset(mapred.reducer.new-api,
 conf.get(oldReduceClass) == null);
  ...
{code}
Therefore, when cleanup runs in a reduce slot, ReduceTask inits using the old 
API and runs incorrect default OutputCommitter, instead of consulting 
OutputFormat.




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Hadoop-Mapreduce-trunk - Build # 1845 - Failure

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1845/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 34853 lines...]
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at 
org.apache.hadoop.mapred.TestLocalModeWithNewApis.testNewApis(TestLocalModeWithNewApis.java:93)


Results :

Tests in error: 
  TestJobClientGetJob.testGetRunningJobFromJobClient:56 ? UnknownHost 
asf901.ygr...
  TestMRWithDistributedCache.testLocalJobRunner:198-testWithConf:186 ? 
UnknownHost
  TestLocalModeWithNewApis.testNewApis:93 ? UnknownHost asf901.ygridcore.net: 
as...

Tests run: 55, Failures: 0, Errors: 3, Skipped: 0

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] hadoop-mapreduce-client ... SUCCESS [  3.034 s]
[INFO] hadoop-mapreduce-client-core .. SUCCESS [ 53.963 s]
[INFO] hadoop-mapreduce-client-common  FAILURE [ 20.967 s]
[INFO] hadoop-mapreduce-client-shuffle ... SKIPPED
[INFO] hadoop-mapreduce-client-app ... SKIPPED
[INFO] hadoop-mapreduce-client-hs  SKIPPED
[INFO] hadoop-mapreduce-client-jobclient . SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] hadoop-mapreduce .. SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 01:19 min
[INFO] Finished at: 2014-07-28T13:21:34+00:00
[INFO] Final Memory: 27M/670M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
project hadoop-mapreduce-client-common: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/target/surefire-reports
 for the individual test results.
[ERROR] - [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn goals -rf :hadoop-mapreduce-client-common
Build step 'Execute shell' marked build as failure
[FINDBUGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Sending artifact delta relative to Hadoop-Mapreduce-trunk #1844
Archived 2 artifacts
Archive block size is 32768
Received 0 blocks and 16975524 bytes
Compression is 0.0%
Took 12 sec
Updating YARN-2247
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
No tests ran.

[jira] [Created] (MAPREDUCE-6010) HistoryServerFileSystemStateStore fails to update tokens

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6010:
-

 Summary: HistoryServerFileSystemStateStore fails to update tokens
 Key: MAPREDUCE-6010
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6010
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver
Affects Versions: 2.3.0
Reporter: Jason Lowe


When token recovery is enabled and the file system state store is being used 
then tokens fail to be updated due to a rename destination conflict.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (MAPREDUCE-6011) Improve history server behavior during a recovery error

[Jason Lowe (JIRA)]

Jason Lowe created MAPREDUCE-6011:
-

 Summary: Improve history server behavior during a recovery error
 Key: MAPREDUCE-6011
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6011
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver
Affects Versions: 2.3.0
Reporter: Jason Lowe


Currently when the history server encounters an error during recovery it is 
fatal without specific details on the error (e.g. which token was involved 
during the recovery error).  We should either allow the history server to 
proceed past recovery errors or provide more specifics on the offending token 
involved in the fatal error to aid in manual recovery.



--
This message was sent by Atlassian JIRA
(v6.2#6252)