Hi Alex,
thanks for your reply. The value of mapred.child.java.optsin the job.xml is
just
-Xmx200m.
Yes, for "standalone" I mean exactly a single JVM. To test my program in this
environment I followed the mepreduce tutorial at the section "Standalone
Operation".
With a single JVM my program, if I - instead exporting the LD_LIBRARY_PATH -
use
the "-Djava.library.path=..."
option of the bin/hadoop command, does not work. Same situation with a similar
experiment, i.e. using the option "-Dmapred.child.env="LD_LIBRARY_PATH=....".
It works just if I explicitly set the system environment variable
LD_LIBRARY_PATH.
Do you think that using the 0.21 could be a good idea?
Thanks,
DF
________________________________
From: Alex Kozlov <ale...@cloudera.com>
To: mapreduce-user@hadoop.apache.org
Sent: Mon, May 2, 2011 5:11:08 AM
Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
And what is the value of mapred.child.java.opts in the job.xml? Is it
something
like `-Djava.library.path=...`? The problemmight be that mapred.child.env was
introduced only in 0.21.
I assume by the "standalone" installation you mean a single JVM. Does your
program works with -Dava.library.path=... in this environment?
On Sun, May 1, 2011 at 9:01 AM, Donatella Firmani <donatellafirm...@yahoo.com>
wrote:
>
>Maybe can help that in a standalone installation of hadoop map reduce it works.
>I know that it is trivial because it is sufficient to type "export
>LD_LIBRARY_PATH=..." in the user shell...
>It is just to be sure that I did not forget anything that may be useful.
>
>
>Cheers,
>DF
>
>
>
________________________________
>From: Donatella Firmani <donatellafirm...@yahoo.com>
>To: mapreduce-user@hadoop.apache.org
>Sent: Sun, May 1, 2011 2:32:03 PM
>
>Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>
>
>
>Dear Alex,
>
>thanks for you kind assistance.
>
>I ran the job giving the option with a
>-Dmapred.child.env="LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3"
> flag.
>
>Checking the job.xml file via JT UI, I can verify that the parameters have the
>correct values for the job.
>
>There appears a line with :
>
>mapred.child.env
>
>LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3
>
>Unfortunately, the value of the call System.getenv("LD_LIBRARY_PATH") is
>different (without the libs) and so the job does not still work.
>
>
>What do you think about it?
>Cheers,
>DF
>
>
>
________________________________
From: Alex Kozlov <ale...@cloudera.com>
>To: mapreduce-user@hadoop.apache.org
>Sent: Fri, April 29, 2011 8:01:30 PM
>Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>
>The next step is to find the job.xml file and check (either in the
>mapred.local.dir in local FS or in the JT web UI)...
>
>
>On Fri, Apr 29, 2011 at 10:59 AM, Donatella Firmani
><donatellafirm...@yahoo.com>
>wrote:
>
>Dear Alex,
>>
>>that's exactly the point. I made my mapper process dump on log files the
>>result
>>of
>>
>>
>>
>>System.getenv("LD_LIBRARY_PATH")
>>System.getProperty("java.library.path")
>>
>>and none of the values seem to be affected neither by the setting of
>>mapred.child.java.opts or of mapred.child.env. in the mapred-site.xml file.
>>
>>
>>Maybe is there something else that I have to do to make LD_LIBRARY_PATH in
>>the
>>JVM environment be correctly set? There are some restrictions on the values
>>that
>>it can assume (i.e. under HDFS an non in the FS of the node)?
>>
>>Cheers,
>>DF
>>
>>
>>
>>
________________________________
From: Alex Kozlov <ale...@cloudera.com>
>>To: mapreduce-user@hadoop.apache.org
>>Sent: Fri, April 29, 2011 7:52:19 PM
>>
>>Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>
>>
>>The option should be passed to the child JVM environment when it is started.
>>You can set most of the environment variables to garbage with no side-effect.
>> A
>>more important question what is the LD_LIBRARY_PATH in your JVM environment.
>>
>>Once again, check the job.xml file in the mapred.local.dir (should be
>>/tmp/cache/${user.name}/... or something like this in the pseudo-config
>>environment) or try to print out the environment variables directly in your
>>map/reduce task.
>>
>>Alex K
>>
>>
>>On Fri, Apr 29, 2011 at 10:37 AM, Donatella Firmani
>><donatellafirm...@yahoo.com>
>>wrote:
>>
>>I just tried giving the option
>>-Dmapred.child.env="LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3"
>> writing no-sense environment variables like
>> -Dmapred.child.env="blahblablah".
>>>
>>>It continues working... so I think that the option is completely ignored by
>>>the
>>>bin/hadoop script.
>>>
>>>Do you think it is an expected behavior?
>>>
>>>Cheers,
>>>DF
>>>
>>>
>>>
>>>
________________________________
From: Alex Kozlov <ale...@cloudera.com>
>>>
>>>To: mapreduce-user@hadoop.apache.org
>>>Sent: Fri, April 29, 2011 7:03:50 PM
>>>
>>>Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>
>>>
>>>You need only to edit the config files on the client or give the option with
>>>a
>>>-Dmapred.child.env="LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3"
>>> flag (if you implement Tool). You can check the job.xml file via JT UI to
>>>verify that the parameters have the correct values for the job.
>>>
>>>
>>>On Fri, Apr 29, 2011 at 9:05 AM, Donatella Firmani
>>><donatellafirm...@yahoo.com>
>>>wrote:
>>>
>>>Dear Yin,
>>>>
>>>>Good point: I can try to install 0.19 and reproduce the problem. I'll let
>>>>you
>>>>know ASAP.
>>>>
>>>>Thanks,
>>>>DF
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
________________________________
From: Yin Lou <yin.lou...@gmail.com>
>>>>
>>>>To: mapreduce-user@hadoop.apache.org
>>>>Sent: Fri, April 29, 2011 5:59:14 PM
>>>>Subject: Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>>
>>>>
>>>>Just curious, can we do this in 0.19?
>>>>
>>>>Thanks,
>>>>Yin
>>>>
>>>>
>>>>On Fri, Apr 29, 2011 at 10:29 AM, Robert Evans <ev...@yahoo-inc.com> wrote:
>>>>
>>>>DF,
>>>>>
>>>>>You can set mapred.child.java.opts to set java options, but you can also
>>>>>set
>>>>>mapred.child.env to set environment variables, be careful because they are
>>>>>space
>>>>>separated with an = in between them.
>>>>>
>>>>> <property>
>>>>>
>>>>> <name>mapred.child.env</name>
>>>>>>
>>>>>> <value>LD_LIBRARY_PATH=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3</value>
>>>>>>
>>>>>> </property>
>>>>
>>>>--Bobby
>>>>
>>>>
>>>>On 4/29/11 5:58 AM, "Donatella Firmani" <donatellafirm...@yahoo.com> wrote:
>>>>
>>>>
>>>>To solve the issue addressed in my previous message, i tried setting
property
>>>>>mapred.child.java.opts in mapred-site.xml. But - even if it seems the right
>>>>>approach in relation to what said in blogs & forums - there is a big
problem
>>>>>with it.
>>>>>
>>>>>Following the tutorial (hadoop website) as section Task Execution &
>>>>Environment,
>>>>>
>>>>>my xml looks like:
>>>>>
>>>>><configuration>
>>>>> <property>
>>>>> <name>mapred.job.tracker</name>
>>>>> <value>localhost:9001</value>
>>>>> </property>
>>>>> <property>
>>>>> <name>mapred.child.java.opts</name>
>>>>> <value>
>>>>>-Djava.library.path=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3
>>>>>
>>>>>
>>>>>
>>>>> </value>
>>>>> </property>
>>>>></configuration>
>>>>>
>>>>>The problem arises when executing the job, because it is thrown an
>exception:
>>>>>
>>>>>Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>>-Djava/library/path=/home/mylibpath1/lib1;home/mylibpath2/lib2;home/mylibpath3/lib3
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>Any help would be appreciated.
>>>>>Thanks in advance,
>>>>>
>>>>>DF
>>>>>
>>>>>
>>>>>
>>>>>----- Original Message ----
>>>>>From: Donatella Firmani <donatellafirm...@yahoo.com>
>>>>>To: mapreduce-user@hadoop.apache.org
>>>>>Sent: Fri, April 29, 2011 12:57:52 PM
>>>>>Subject: Hadoop Mapreduce jobs and LD_LIBRARY_PATH
>>>>>
>>>>>
>>>>>
>>>>>Hi to all,
>>>>>
>>>>>I just subscribed to this mailing list and I'd like to ask you if anyone
>>knows
>>>>>how to deal with LD_LIBRARY_PATH.
>>>>>I have a Java application that needs a proper setting of this environment
>>>>>variable to work under Linux-Ubuntu.
>>>>>I want to use tis application from a mapreduce job, unfortunately I could
>not
>>>>>find a way to make things work against the LD_LIBRARY_PATH environment
>>>>variable.
>>>>>
>>>>>I tried so many different strategies and I am stuck. Maybe someone of you
>can
>>>>>help.
>>>>>
>>>>>Thanks in advance,
>>>>>Cheers.
>>>>>
>>>>>DF
>>>>>
>>>>>PS: I use hadoop-0-20-2
>>>>>
>>>>>
>>>>
>>>
>>
>
