In the "standalone" hadoop application, try setting `export
HADOOP_OPTS="-Djava.library.path=..."`
w/o explicitly setting LD_LIBRARY_PATH.
On Mon, May 2, 2011 at 8:07 AM, Alex Kozlov <ale...@cloudera.com> wrote:
> Try adding -Djava.library.path=... to the *mapred.child.java.opts*string.
> Otherwise, upgrade to 0.21.
>
> On Mon, May 2, 2011 at 2:24 AM, Donatella Firmani <
> donatellafirm...@yahoo.com> wrote:
>
>> Hi Alex,
>>
>> thanks for your reply. The value of *mapred.child.java.opts* in the
>> job.xml is just -Xmx200m.
>>
>> Yes, for "standalone" I mean exactly a single JVM. To test my program in
>> this environment I followed the mepreduce tutorial at the section "Standalone
>> Operation".
>>
>> With a single JVM my program, if I - instead exporting the LD_LIBRARY_PATH
>> - use the "-Djava.library.path=..."
>> option of the bin/hadoop command, does not work. Same situation with a
>> similar experiment, i.e. using the option "-Dmapred.child.env="
>> LD_LIBRARY_PATH=....".
>> It works just if I explicitly set the system environment variable
>> LD_LIBRARY_PATH.
>>
>> Do you think that using the 0.21 could be a good idea?
>>
>> Thanks,
>> DF
>>
>>
>> ------------------------------
>> *From:* Alex Kozlov <ale...@cloudera.com>
>> *To:* mapreduce-user@hadoop.apache.org
>> *Sent:* Mon, May 2, 2011 5:11:08 AM
>>
>> *Subject:* Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>
>> And what is the value of *mapred.child.java.opts* in the job.xml? Is it
>> something like `-Djava.library.path=...`? The problem might be that *
>> mapred.child.env* was introduced only in 0.21.
>>
>> I assume by the "standalone" installation you mean a single JVM. Does
>> your program works with -Dava.library.path=... in this environment?
>>
>> On Sun, May 1, 2011 at 9:01 AM, Donatella Firmani <
>> donatellafirm...@yahoo.com> wrote:
>>
>>>
>>> Maybe can help that in a standalone installation of hadoop map reduce it
>>> works.
>>> I know that it is trivial because it is sufficient to type "export
>>> LD_LIBRARY_PATH=..." in the user shell...
>>> It is just to be sure that I did not forget anything that may be useful.
>>>
>>> Cheers,
>>> DF
>>>
>>> ------------------------------
>>> *From:* Donatella Firmani <donatellafirm...@yahoo.com>
>>> *To:* mapreduce-user@hadoop.apache.org
>>> *Sent:* Sun, May 1, 2011 2:32:03 PM
>>>
>>> *Subject:* Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>
>>> Dear Alex,
>>>
>>> thanks for you kind assistance.
>>>
>>> I ran the job giving the option with a -Dmapred.child.env="
>>> LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3"
>>> flag.
>>>
>>> Checking the job.xml file via JT UI, I can verify that the parameters
>>> have the correct values for the job.
>>> There appears a line with :
>>>
>>> mapred.child.env
>>>
>>> LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/
>>> mylibpath3/lib3
>>>
>>> Unfortunately, the value of the call System.getenv("LD_LIBRARY_PATH") is
>>> different (without the libs) and so the job does not still work.
>>>
>>> What do you think about it?
>>> Cheers,
>>> DF
>>>
>>> ------------------------------
>>> *From:* Alex Kozlov <ale...@cloudera.com>
>>> *To:* mapreduce-user@hadoop.apache.org
>>> *Sent:* Fri, April 29, 2011 8:01:30 PM
>>> *Subject:* Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>
>>> The next step is to find the job.xml file and check (either in the
>>> mapred.local.dir in local FS or in the JT web UI)...
>>>
>>> On Fri, Apr 29, 2011 at 10:59 AM, Donatella Firmani <
>>> donatellafirm...@yahoo.com> wrote:
>>>
>>>> Dear Alex,
>>>>
>>>> that's exactly the point. I made my mapper process dump on log files the
>>>> result of
>>>>
>>>> System.getenv("LD_LIBRARY_PATH")
>>>> System.getProperty("java.library.path")
>>>>
>>>> and none of the values seem to be affected neither by the setting of
>>>> mapred.child.java.opts
>>>> or of mapred.child.env. in the mapred-site.xml file.
>>>>
>>>> Maybe is there something else that I have to do to make LD_LIBRARY_PATH
>>>> in the JVM environment be correctly set? There are some restrictions on
>>>> the values that it can assume (i.e. under HDFS an non in the FS of the
>>>> node)?
>>>>
>>>> Cheers,
>>>> DF
>>>>
>>>> ------------------------------
>>>> *From:* Alex Kozlov <ale...@cloudera.com>
>>>> *To:* mapreduce-user@hadoop.apache.org
>>>> *Sent:* Fri, April 29, 2011 7:52:19 PM
>>>>
>>>> *Subject:* Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>>
>>>> The option should be passed to the child JVM environment when it is
>>>> started. You can set most of the environment variables to garbage with no
>>>> side-effect. A more important question what is the LD_LIBRARY_PATH in
>>>> your JVM environment.
>>>>
>>>> Once again, check the job.xml file in the mapred.local.dir (should be
>>>> /tmp/cache/${user.name}/... or something like this in the pseudo-config
>>>> environment) or try to print out the environment variables directly in your
>>>> map/reduce task.
>>>>
>>>> Alex K
>>>>
>>>> On Fri, Apr 29, 2011 at 10:37 AM, Donatella Firmani <
>>>> donatellafirm...@yahoo.com> wrote:
>>>>
>>>>> I just tried giving the option -Dmapred.child.env="LD_LIBRARY_PATH=/
>>>>> home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3"
>>>>> writing no-sense environment variables like -Dmapred.child.env="
>>>>> blahblablah".
>>>>>
>>>>> It continues working... so I think that the option is completely
>>>>> ignored by the bin/hadoop script.
>>>>>
>>>>> Do you think it is an expected behavior?
>>>>>
>>>>> Cheers,
>>>>> DF
>>>>>
>>>>> ------------------------------
>>>>> *From:* Alex Kozlov <ale...@cloudera.com>
>>>>>
>>>>> *To:* mapreduce-user@hadoop.apache.org
>>>>> *Sent:* Fri, April 29, 2011 7:03:50 PM
>>>>>
>>>>> *Subject:* Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>>>
>>>>> You need only to edit the config files on the client or give the option
>>>>> with a -Dmapred.child.env="LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/
>>>>> mylibpath2/lib2;home/mylibpath3/lib3" flag (if you implement Tool).
>>>>> You can check the job.xml file via JT UI to verify that the parameters
>>>>> have
>>>>> the correct values for the job.
>>>>>
>>>>> On Fri, Apr 29, 2011 at 9:05 AM, Donatella Firmani <
>>>>> donatellafirm...@yahoo.com> wrote:
>>>>>
>>>>>> Dear Yin,
>>>>>>
>>>>>> Good point: I can try to install 0.19 and reproduce the problem. I'll
>>>>>> let you know ASAP.
>>>>>>
>>>>>> Thanks,
>>>>>> DF
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* Yin Lou <yin.lou...@gmail.com>
>>>>>>
>>>>>> *To:* mapreduce-user@hadoop.apache.org
>>>>>> *Sent:* Fri, April 29, 2011 5:59:14 PM
>>>>>> *Subject:* Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>>>>
>>>>>> Just curious, can we do this in 0.19?
>>>>>>
>>>>>> Thanks,
>>>>>> Yin
>>>>>>
>>>>>> On Fri, Apr 29, 2011 at 10:29 AM, Robert Evans
>>>>>> <ev...@yahoo-inc.com>wrote:
>>>>>>
>>>>>>> DF,
>>>>>>>
>>>>>>> You can set mapred.child.java.opts to set java options, but you can
>>>>>>> also set mapred.child.env to set environment variables, be careful
>>>>>>> because
>>>>>>> they are space separated with an = in between them.
>>>>>>>
>>>>>>> <property>
>>>>>>>
>>>>>>> <name>mapred.child.env</name>
>>>>>>>
>>>>>>>
>>>>>>> <value>LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3</value>
>>>>>>>
>>>>>>> </property>
>>>>>>>
>>>>>>> --Bobby
>>>>>>>
>>>>>>>
>>>>>>> On 4/29/11 5:58 AM, "Donatella Firmani" <donatellafirm...@yahoo.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> To solve the issue addressed in my previous message, i tried setting
>>>>>>> property
>>>>>>> mapred.child.java.opts in mapred-site.xml. But - even if it seems the
>>>>>>> right
>>>>>>> approach in relation to what said in blogs & forums - there is a big
>>>>>>> problem
>>>>>>> with it.
>>>>>>>
>>>>>>> Following the tutorial (hadoop website) as section Task Execution &
>>>>>>> Environment,
>>>>>>>
>>>>>>> my xml looks like:
>>>>>>>
>>>>>>> <configuration>
>>>>>>> <property>
>>>>>>> <name>mapred.job.tracker</name>
>>>>>>> <value>localhost:9001</value>
>>>>>>> </property>
>>>>>>> <property>
>>>>>>> <name>mapred.child.java.opts</name>
>>>>>>> <value>
>>>>>>>
>>>>>>> -Djava.library.path=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3
>>>>>>>
>>>>>>>
>>>>>>> </value>
>>>>>>> </property>
>>>>>>> </configuration>
>>>>>>>
>>>>>>> The problem arises when executing the job, because it is thrown an
>>>>>>> exception:
>>>>>>>
>>>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>>>
>>>>>>> -Djava/library/path=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any help would be appreciated.
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>> DF
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ----- Original Message ----
>>>>>>> From: Donatella Firmani <donatellafirm...@yahoo.com>
>>>>>>> To: mapreduce-user@hadoop.apache.org
>>>>>>> Sent: Fri, April 29, 2011 12:57:52 PM
>>>>>>> Subject: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi to all,
>>>>>>>
>>>>>>> I just subscribed to this mailing list and I'd like to ask you if
>>>>>>> anyone knows
>>>>>>> how to deal with LD_LIBRARY_PATH.
>>>>>>> I have a Java application that needs a proper setting of this
>>>>>>> environment
>>>>>>> variable to work under Linux-Ubuntu.
>>>>>>> I want to use tis application from a mapreduce job, unfortunately I
>>>>>>> could not
>>>>>>> find a way to make things work against the LD_LIBRARY_PATH
>>>>>>> environment variable.
>>>>>>>
>>>>>>> I tried so many different strategies and I am stuck. Maybe someone of
>>>>>>> you can
>>>>>>> help.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>> Cheers.
>>>>>>>
>>>>>>> DF
>>>>>>>
>>>>>>> PS: I use hadoop-0-20-2
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
