[
https://issues.apache.org/jira/browse/MAPREDUCE-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Changming Sun updated MAPREDUCE-4000:
-------------------------------------
Description:
Create jobTokenPassword file fail when run hadoop pipes locally.
I have put such settings in my job conf.
{code:xml}
<property>
<name>mapreduce.jobtracker.address</name>
<value>local</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>local</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>file:///</value>
</property>
{code}
This job has one map task ,and one reduce task. It will fail when running
PipesReducer, because "jobTokenPassword" file is created at current working
directory,with permission 0400.
{code:title=org.apache.hadoop.mapred.pipes.Application.java}
private void writePasswordToLocalFile(String localPasswordFile,
byte[] password, JobConf conf) throws IOException {
FileSystem localFs = FileSystem.getLocal(conf);
Path localPath = new Path(localPasswordFile);
FSDataOutputStream out = FileSystem.create(localFs, localPath,
new FsPermission("400"));
out.write(password);
out.close();
}
{code}
So, it will fail at the second time.
In such situation,the application cannot be initialized, but the
PipesReducer.close() method will be called after that ,so there will be
NullPointer Exception raise in close() method.
was:
Create jobTokenPassword file fail when run hadoop pipes locally.
I have put such settings in my job conf.
<property>
<name>mapreduce.jobtracker.address</name>
<value>local</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>local</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>file:///</value>
</property>
This job has one map task ,and one reduce task. It will fail when running
PipesReducer, because "jobTokenPassword" file is created at current working
directory,with permission 0400.
org.apache.hadoop.mapred.pipes.Application:
private void writePasswordToLocalFile(String localPasswordFile,
byte[] password, JobConf conf) throws IOException {
FileSystem localFs = FileSystem.getLocal(conf);
Path localPath = new Path(localPasswordFile);
FSDataOutputStream out = FileSystem.create(localFs, localPath,
new FsPermission("400"));
out.write(password);
out.close();
}
So, it will fail at the second time.
In such situation,the application cannot be initialized, but the
PipesReducer.close() method will be called after that ,so there will be
NullPointer Exception raise in close() method.
> Create jobTokenPassword file fail when run hadoop pipes locally
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-4000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
> Affects Versions: 0.23.3
> Environment: Fedora release 16
> Linux localhost.localdomain 3.2.7-1.fc16.x86_64 #1 SMP Tue Feb 21 01:40:47
> UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
> Maven home: /usr/local/apache-maven-3.0.4
> Java version: 1.7.0_03, vendor: Oracle Corporation
> Java home: /usr/java/jdk1.7.0_03/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.2.7-1.fc16.x86_64", arch: "amd64", family:
> "unix"
> Reporter: Changming Sun
> Priority: Minor
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Create jobTokenPassword file fail when run hadoop pipes locally.
> I have put such settings in my job conf.
> {code:xml}
> <property>
> <name>mapreduce.jobtracker.address</name>
> <value>local</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>local</value>
> </property>
> <property>
> <name>fs.defaultFS</name>
> <value>file:///</value>
> </property>
> {code}
> This job has one map task ,and one reduce task. It will fail when running
> PipesReducer, because "jobTokenPassword" file is created at current working
> directory,with permission 0400.
> {code:title=org.apache.hadoop.mapred.pipes.Application.java}
> private void writePasswordToLocalFile(String localPasswordFile,
> byte[] password, JobConf conf) throws IOException {
> FileSystem localFs = FileSystem.getLocal(conf);
> Path localPath = new Path(localPasswordFile);
> FSDataOutputStream out = FileSystem.create(localFs, localPath,
> new FsPermission("400"));
> out.write(password);
> out.close();
> }
> {code}
> So, it will fail at the second time.
> In such situation,the application cannot be initialized, but the
> PipesReducer.close() method will be called after that ,so there will be
> NullPointer Exception raise in close() method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
