[
https://issues.apache.org/jira/browse/CHUKWA-488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved CHUKWA-488.
------------------------------
Fix Version/s: 0.5.0
Resolution: Fixed
I just committed this. Thanks Kirk
> Hadoop cannot find custom Demux class
> -------------------------------------
>
> Key: CHUKWA-488
> URL: https://issues.apache.org/jira/browse/CHUKWA-488
> Project: Hadoop Chukwa
> Issue Type: Bug
> Components: MR Data Processors
> Affects Versions: 0.4.0
> Environment: Linux x86-64
> Java 1.6.0_20
> Reporter: Kirk True
> Fix For: 0.5.0
>
> Attachments: Demux.diff
>
>
> I'm getting ClassNotFoundException errors when running inside Hadoop's map
> phase, unable to find my class
> org.apache.hadoop.chukwa.extraction.demux.processor.mapper.XmlBasedDemux
> which I've packaged in a JAR named data-collection-demux-0.1.jar.
> The problem seems to be in the values of these two properties in the Hadoop
> job configuration:
> {code}
> <property>
> <name>mapred.job.classpath.files</name>
>
> <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> <property>
> <name>mapred.cache.files</name>
>
> <value>hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar</value>
> </property>
> {code}
> The problem seems to stem from the fact that the call to
> DistributedCache.addFileToClassPath is passing in a Path that is in URI form,
> i.e. hdfs://localhost:9000/chukwa/demux/data-collection-demux-0.1.jar whereas
> the DistributedCache API expects it to be a filesystem-based path (i.e.
> /chukwa/demux/data-collection-demux-0.1.jar). I'm not sure why, but the
> FileStatus object returned by FileSystem.listStatus is returning a URL-based
> path instead of a filesystem-based path.
> I kludged the Demux class' addParsers to strip the "hdfs://localhost:9000"
> portion of the string and now my class is found. I will attempt to provide a
> patch today that determines the value of Hadoop's fs.default.name and strips
> that from the value returned in Demux.java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
