[
https://issues.apache.org/jira/browse/MAPREDUCE-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-5749:
----------------------------------
Attachment: MAPREDUCE-5749v2.patch
The problem is caused by TestRMContainerAllocator.MyContainerAllocator#register
setting the login user to a bogus app attempt user. This means anything after
that ends up getting the wrong username if they're expecting the user that
submitted the job.
The original patch for this "fixes" the problem by making the user for the job
be the bogus app attempt user which means we match up filenames but it isn't
really the right thing to do.
Posting a patch where we avoid creating a bogus app attempt user and instead
attach the necessary tokens to the current user. Also added a
UserGroupInformation.setLoginUser(null) to the Before clause so we don't
accidentally leak tokens across the test cases.
> TestRMContainerAllocator#testReportedAppProgress Failed
> -------------------------------------------------------
>
> Key: MAPREDUCE-5749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5749
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: shenhong
> Attachments: MAPREDUCE-5749.patch, MAPREDUCE-5749v2.patch,
> TestRMContainerAllocator_failure.txt
>
>
> When execute "mvn test
> -Dtest=TestRMContainerAllocator#testReportedAppProgress", It failed with
> message:
> {code}
> Caused by: java.io.FileNotFoundException: File
> /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001/job.xml
> does not exist
> {code}
> But in fact, the job.xml exits:
> {code}
> -rw-rw-r-- 1 yuling.sh yuling.sh 65791 2月 10 13:13
> /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/yuling.sh/.staging/job_1392009213299_0001/job.xml
> {code}
> See the following code:
> {code}
> public Job submit(Configuration conf, boolean mapSpeculative,
> boolean reduceSpeculative) throws Exception {
> String user = conf.get(MRJobConfig.USER_NAME, UserGroupInformation
> .getCurrentUser().getShortUserName());
> conf.set(MRJobConfig.USER_NAME, user);
> conf.set(MRJobConfig.MR_AM_STAGING_DIR, testAbsPath.toString());
> conf.setBoolean(MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR, true);
> // TODO: fix the bug where the speculator gets events with
> // not-fully-constructed objects. For now, disable speculative exec
> conf.setBoolean(MRJobConfig.MAP_SPECULATIVE, mapSpeculative);
> conf.setBoolean(MRJobConfig.REDUCE_SPECULATIVE, reduceSpeculative);
> init(conf);
> start();
> DefaultMetricsSystem.shutdown();
> Job job = getContext().getAllJobs().values().iterator().next();
> if (assignedQueue != null) {
> job.setQueueName(assignedQueue);
> }
> // Write job.xml
> String jobFile = MRApps.getJobFile(conf, user,
> TypeConverter.fromYarn(job.getID()));
> LOG.info("Writing job conf to " + jobFile);
> new File(jobFile).getParentFile().mkdirs();
> conf.writeXml(new FileOutputStream(jobFile));
> return job;
> }
> {code}
> At first, user is "yuling.sh", but the UGI is setted to attemptId at
> "start();", after that, job.xml write to
> yuling.sh/.staging/job_1392009213299_0001/job.xml. But when the job is
> running, MRAppMaster can't find the job.xml at
> appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
