The “problem” seems to be related to Zeppelin: the loading of the libraries is
successful when using the spark-shell command. Is it possible that Zeppelin
erases the default configuration of the Spark cluster? And how is it possible
to keep the library path untouched?
Thanks in advance for your answers
Rémy
_____________________________________________
From: PHELIPOT, REMY
Sent: Tuesday, July 21, 2015 9:58 AM
To: 'users@zeppelin.incubator.apache.org'
Subject: Native libraries with Zeppelin and Spark
Hello,
I’am a new user of Zeppelin and I’m encountering a problem when I want to use
native libraries with Spark in cluster mode. The following error message is
displayed in the logs:
15/07/21 07:46:12 ERROR SparkUncaughtExceptionHandler: Uncaught exception in
thread Thread[Executor task launch worker-0,5,main]
java.lang.UnsatisfiedLinkError: no xxxjni in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1865)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1119)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at
org.apache.spark.util.collection.WritablePartitionedIterator$$anon$3.<init>(WritablePartitionedPairCollection.scala:101)
at
org.apache.spark.util.collection.WritablePartitionedIterator$.fromIterator(WritablePartitionedPairCollection.scala:100)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:209)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
The environment variables seem to be correctly set in spark-defaults.conf
(located in the spark configuration directory):
spark.executor.extraLibraryPath=/usr/lib/:/usr/lib/jni/
spark.driver.extraLibraryPath=/usr/lib/:/usr/lib/jni/
spark.driver.extraClassPath=/usr/share/java
spark.executor.extraClassPath=/usr/share/java
If I copy the libraries in /usr/lib, the error message disappears and the code
is executed without errors, but this is an ugly fix.
Do you have any solution to this problem? Is it specific to Zeppelin or Spark?
Kind regards,
Rémy
Ce message et toutes les pièces jointes (ci-après le "message") sont établis à
l’intention exclusive des destinataires désignés. Il contient des informations
confidentielles et pouvant être protégé par le secret professionnel. Si vous
recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et
de détruire le message. Toute utilisation de ce message non conforme à sa
destination, toute diffusion ou toute publication, totale ou partielle, est
interdite, sauf autorisation expresse de l’émetteur. L'internet ne garantissant
pas l'intégrité de ce message lors de son acheminement, Atos (et ses filiales)
décline(nt) toute responsabilité au titre de son contenu. Bien que ce message
ait fait l’objet d’un traitement anti-virus lors de son envoi, l’émetteur ne
peut garantir l’absence totale de logiciels malveillants dans son contenu et ne
pourrait être tenu pour responsable des dommages engendrés par la transmission
de l’un d’eux.
This message and any attachments (the "message") are intended solely for the
addressee(s). It contains confidential information, that may be privileged. If
you receive this message in error, please notify the sender immediately and
delete the message. Any use of the message in violation of its purpose, any
dissemination or disclosure, either wholly or partially is strictly prohibited,
unless it has been explicitly authorized by the sender. As its integrity cannot
be secured on the internet, Atos and its subsidiaries decline any liability for
the content of this message. Although the sender endeavors to maintain a
computer virus-free network, the sender does not warrant that this transmission
is virus-free and will not be liable for any damages resulting from any virus
transmitted.
