FileNotFoundException during Mapper.setup for file embedded in job jar
----------------------------------------------------------------------
Key: MAPREDUCE-3981
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3981
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: David Savage
I have a job that is packaged as a jar and contains a configuration file in the
root of the jar "RankingRules.xml".
During the mapping I'm loading this file at Mapper.setup using (effectively)
ClassLoader.getResource("RankingRules.xml").openStream()
This works for a while then fails repeatedly 20-40 mins into the job causing
the maximum retry failures to trigger and then the job aborts.
The failure is caused by:
java.io.FileNotFoundException:
/tmp/hadoop-hadoop/hadoop-unjar695679302321435361/RankingRules.xml (No such
file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:97)
at
sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)
at
sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)
at java.net.URL.openStream(URL.java:1035)
... 12 more
I'm thinking a good approach is to change the code so it loads the RankingRules
from an HDFS url instead of embedding them in the jar.
However I'm a little baffled as to why this should change between tasks. This
job is creating about 600 map tasks and on the last run it failed after about
300 successful maps.
At the moment this is running in psuedo distributed mode so it's just one
machine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
