[
https://issues.apache.org/jira/browse/HADOOP-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630140#action_12630140
]
sharadag edited comment on HADOOP-3570 at 9/11/08 2:31 AM:
-----------------------------------------------------------------
Amereshwari pointed out that as Streaming is not implementing the Tool, the
libjars are not getting available for streaming.
I fixed it by setting the classpath in RunJar which launches the Streaming
application.
Now I need to set the classloader in GenericOptionParser itself. The issue in
setting it in JobClient is that it makes libjars available only after submitJob
is called. However applications may requires to load a class (inputformat)
before calling submitJob.
was (Author: sharadag):
Amereshwari pointed out that as Streaming is not implementing the Tool, the
libjars are not getting available for streaming.
I fixed it by setting the classpath in RunJar which launches the Streaming
application.
Now I need to set the classloader in GenericOptionParser itself. The issue in
setting it in JobClient is that it makes libjars available only after submitJob
is called. However streaming may requires to load a class (inputformat) before
calling submitJob.
> Including user specified jar files in the client side classpath path in
> Hadoop 0.17 streaming
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3570
> URL: https://issues.apache.org/jira/browse/HADOOP-3570
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.17.0
> Reporter: Suhas Gogate
> Assignee: Sharad Agarwal
> Fix For: 0.19.0
>
> Attachments: 3570_v1.patch, 3570_v2.patch, 3570_v3.patch
>
>
> Hadoop 0.17 streaming allows specifying multiple user specific jar files on
> the command line along with the hadoop-streaming.jar. These user specific jar
> files are not automatically added on the client side class path when hadoop
> runs the streaming job
> For example, in the following streaming job <user-app.jar> is not
> automatically included in client side path before executing the streaming job
>
> hadoop jar -libjars <user-app.jar> $HADOOP_HOME/hadoop-streaming.jar \
> -input </input-data-path> \
> -output </output data path> \
> -inputformat <Input-format> \
> -outputformat <OutputFormat> \
> -mapper /bin/cat -reducer /bin/cat \
> -additionalconfspec <xxxx> \
> -jobconf mapred.reduce.tasks=3\
> Workaround:
> Current work around is to set these user specific jar files in
> HADOOP_CLASSPATH environment variable on the client side before running above
> example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
