bcc dev@ and add user@
This is more a user@ list question rather than a dev@ list question. You
can do something like this:
object MySimpleApp {
def loadResources(): Unit = // define some idempotent way to load
resources, e.g. with a flag or lazy val
def main() = {
...
sc.parallelize(1 to 10).mapPartitions { iter =>
MySimpleApp.loadResources()
// do whatever you want with the iterator
}
}
}
On Fri, Nov 25, 2016 at 2:33 PM, vineet chadha <[email protected]>
wrote:
> Hi,
>
> I am trying to invoke C library from the Spark Stack using JNI interface
> (here is sample application code)
>
>
> class SimpleApp {
> // ---Native methods
> @native def foo (Top: String): String
> }
>
> object SimpleApp {
> def main(args: Array[String]) {
>
> val conf = new
> SparkConf().setAppName("SimpleApplication").set("SPARK_LIBRARY_PATH",
> "lib")
> val sc = new SparkContext(conf)
> System.loadLibrary("foolib")
> //instantiate the class
> val SimpleAppInstance = new SimpleApp
> //String passing - Working
> val ret = SimpleAppInstance.foo("fooString")
> }
>
> Above code work fines.
>
> I have setup LD_LIBRARY_PATH and spark.executor.extraClassPath,
> spark.executor.extraLibraryPath at worker node
>
> How can i invoke JNI library from worker node ? Where should i load it in
> executor ?
> Calling System.loadLibrary("foolib") inside the work node gives me
> following error :
>
> Exception in thread "main" java.lang.UnsatisfiedLinkError:
>
> Any help would be really appreciated.
>
>
>
>
>
>
>
>
>
>
>
>
>
