Hi Manuel, You could use the following to add a path to the library search path, --conf spark.driver.extraLibraryPath=PathToLibFolder --conf spark.executor.extraLibraryPath=PathToLibFolder
Thanks, Keith. Regards, Keith. http://keith-chapman.com On Wed, Jan 17, 2018 at 5:39 PM, Manuel Sopena Ballesteros < manuel...@garvan.org.au> wrote: > Dear Spark community, > > > > I have a spark running in a yarn cluster and I am getting some error when > trying to run my python application. > > > > /home/mansop/virtenv/bin/python2.7: error while loading shared libraries: > libpython2.7.so.1.0: cannot open shared object file: No such file or > directory > > > > Is there a way to specify the LD_LIBRARY_PATH in the spark-submit command > or in the config file? > > > > > > *Manuel Sopena Ballesteros *| Big data Engineer > *Garvan Institute of Medical Research * > The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010 > <https://maps.google.com/?q=370+Victoria+Street,+Darlinghurst,+NSW+2010&entry=gmail&source=g> > *T:* + 61 (0)2 9355 5760 <+61%202%209355%205760> | *F:* +61 (0)2 9295 8507 > <+61%202%209295%208507> | *E:* manuel...@garvan.org.au > > > NOTICE > Please consider the environment before printing this email. This message > and any attachments are intended for the addressee named and may contain > legally privileged/confidential/copyright information. If you are not the > intended recipient, you should not read, use, disclose, copy or distribute > this communication. If you have received this message in error please > notify us at once by return email and then delete both messages. We accept > no liability for the distribution of viruses or similar in electronic > communications. This notice should not be removed. >
