How to set the LD_LIBRARY_PATH for the child , configuring mapred-site.xml doesn't work. Also setting -Djava.library.path is not good enough since it only gets the reference to the lib I am a trying to load(let's say lib.so) , but that lib has dependencies on other libs like lib1.so resulting in UnsatisfiedLinkError . Thus, LD_LIBRARY_PATH has to be set.
On Thu, Feb 18, 2010 at 10:03 PM, Jason Venner <[email protected]>wrote: > We used do this all the time at attributor. Now if I can remember how we > did > it. > > If the libraries are constant you can just install them on your nodes to > save pushing them through the distributed cache, and then setup the > LD_LIBRARY_PATH correctly. > > The key issue if you push them through the distributed cache is ensuring > that the directory that the library gets dropped in, is actually in the > runtime java.library.path > You can also give explicit paths to System.load > > The -Djava.library.path in the child.options mapred.child.java.opts (if I > have the param correct) should work also. > > On Thu, Feb 18, 2010 at 6:49 PM, Utkarsh Agarwal <[email protected] > >wrote: > > > My .so file has other .so dependencies , so would I have to add them all > in > > the DistributedCache . Also I tried setting LD_LIBRARY_PATH in > > mapred-site.xml as > > > > <property> > > <name>mapred.child.env</name> > > <value>LD_LIBRARY_PATH=/opt/libs/</value> > > </property> > > > > > > doesnt work. the java.library.path is not sufficient to set , have to get > > LD_LIB set. > > > > -Utkarsh > > > > On Thu, Feb 18, 2010 at 3:14 PM, Allen Wittenauer > > <[email protected]>wrote: > > > > > > > > > > > Like this: > > > > > > > > > > > > http://hadoop.apache.org/common/docs/current/native_libraries.html#Loading+n > > > ative+libraries+through+DistributedCache > > > > > > > > > > > > On 2/16/10 5:29 PM, "Jason Rutherglen" <[email protected]> > > wrote: > > > > > > > How would this work? > > > > > > > > On Fri, Feb 12, 2010 at 10:45 AM, Allen Wittenauer > > > > <[email protected]> wrote: > > > >> > > > >> ... or just use distributed cache. > > > >> > > > >> > > > >> On 2/12/10 10:02 AM, "Alex Kozlov" <[email protected]> wrote: > > > >> > > > >>> All native libraries should be on each of the cluster nodes. You > > need > > > to > > > >>> set "java.library.path" property to point to your libraries (or > just > > > put > > > >>> them in the default system dirs). > > > >>> > > > >>> On Fri, Feb 12, 2010 at 9:12 AM, Utkarsh Agarwal > > > >>> <[email protected]>wrote: > > > >>> > > > >>>> Can anybody point me how to use JNI calls in a map reduce program. > > My > > > .so > > > >>>> files have other dependencies also , is there a way to load the > > > >>>> LD_LIBRARY_PATH for child processes . Should all the native stuff > be > > > in > > > >>>> HDFS? > > > >>>> > > > >>>> Thanks, > > > >>>> Utkarsh. > > > >>>> > > > >> > > > >> > > > > > > > > > > > > -- > Pro Hadoop, a book to guide you from beginner to hadoop mastery, > http://www.amazon.com/dp/1430219424?tag=jewlerymall > www.prohadoopbook.com a community for Hadoop Professionals >
