Thanks very much , todd. I get it.
On Wed, Aug 17, 2011 at 6:23 AM, Todd Lipcon <t...@cloudera.com> wrote:
> Putting files on the classpath doesn't make them accessible to JVM's
> resource loader. If you have dir/foo.properties, then "dir" needs to
> be on the classpath, not "dir/foo.properties". Since the working dir
> of the task is on the classpath, then -files works since it gets the
> properties file into a directory on the classpath.
>
> -Todd
>
> On Mon, Aug 15, 2011 at 8:09 PM, Anty <anty....@gmail.com> wrote:
>> thanks very much for you reply, todd.
>> I am at a complete loss. I want to ship a configuration file to the
>> cluster to run my mapreduce job.
>>
>> if I use -libjars option to ship the configuration file, the launched
>> child JVM created by task tracker
>> can't find the configuration file,curiously, the configuration file
>> is already on the classpath of the child JVM.
>>
>> if I use -files option to ship the configuration file, the child JVM
>> can find the file.
>> IMO, what's the difference between -libjars and -files is that -files
>> will create a symbol sink to the configuration file
>> in current workding directory of child JVM.
>>
>> I dig into the source code,but it's so complicated, i can't figure out
>> the root cause of this.
>> So my question is :
>> with -libjars option ,the configuration file is already on the
>> classpath, why classload can't the configuration file ,
>> but why JVM classload CAN find the shipped jar with -libjars option?
>>
>> any help will be appreciated.
>>
>> On Tue, Aug 16, 2011 at 1:06 AM, Todd Lipcon <t...@cloudera.com> wrote:
>>> Your "driver" is the program that submits the job. The task is the
>>> thing that runs on the cluster. They have separate classpaths.
>>>
>>> Better to ask on the public lists if you want a more indepth explanation
>>>
>>> -Todd
>>>
>>> On Mon, Aug 15, 2011 at 9:02 AM, Anty <anty....@gmail.com> wrote:
>>>> Hi:Todd
>>>> Would you please explain a litter more?
>>>>
>>>> On Sat, Dec 11, 2010 at 2:08 AM, Todd Lipcon <t...@cloudera.com> wrote:
>>>>>
>>>>> You need to put the library jar on your classpath (eg using
>>>>> HADOOP_CLASSPATH) as well. The -libjars will ship it to the cluster
>>>>> and put it on the classpath of your task, but not the classpath of
>>>>> your "driver" code.
>>>>>
>>>> I still can't understand you mean by " but not the classpath of
>>>> your "driver" code."
>>>>
>>>> THX advance.
>>>>
>>>>
>>>>> -Todd
>>>>>
>>>>> On Thu, Dec 9, 2010 at 10:29 PM, Vipul Pandey <vipan...@gmail.com> wrote:
>>>>> > disclaimer : a newbie!!!
>>>>> > Howdy?
>>>>> > Got a quick question. -libjars option doesn't seem to work for me in -
>>>>> > prettymuch - my first (or mayby second) mapreduce job.
>>>>> > Here's what i'm doing :
>>>>> > $bin/hadoop jar sherlock.jar somepkg.FindSchoolsJob -libjars
>>>>> > HStats-1A18.jar input output
>>>>> >
>>>>> > sherlock.jar has my main class (ofcourse) FindSchoolsJob, which runs
>>>>> > just
>>>>> > fine by itself till I add a dependency on a class in HStats-1A18.jar.
>>>>> > When I run the above command with -libjars specified - it fails to find
>>>>> > my
>>>>> > classes that 'are' inside HStats jar file.
>>>>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> > com/*****/HAgent
>>>>> > at com.*****.FindSchoolsJob.run(FindSchoolsJob.java:46)
>>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>> > at com.******.FindSchoolsJob.main(FindSchoolsJob.java:101)
>>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> > at
>>>>> >
>>>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>> > at
>>>>> >
>>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>> > Caused by: java.lang.ClassNotFoundException:com/*****/HAgent
>>>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>> > at java.security.AccessController.doPrivileged(Native Method)
>>>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>>>> > ... 8 more
>>>>> >
>>>>> > My main class is defined as below :
>>>>> > public class FindSchoolsJob extends Configured implements Tool {
>>>>> > :
>>>>> > public int run(String[] args) throws Exception {
>>>>> > :
>>>>> > :
>>>>> > }
>>>>> > :
>>>>> > public static void main(String[] args) throws Exception {
>>>>> > int res = ToolRunner.run(new Configuration(), new FindSchoolsJob(),
>>>>> > args);
>>>>> > System.exit(res);
>>>>> > }
>>>>> > }
>>>>> > Any hint would be highly appreciated.
>>>>> > Thank You!
>>>>> > ~V
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>> Anty Rao
>>>>
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> Best Regards
>> Anty Rao
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Best Regards
Anty Rao
