[
https://issues.apache.org/jira/browse/SPARK-6301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360301#comment-14360301
]
raju patel edited comment on SPARK-6301 at 3/13/15 1:05 PM:
------------------------------------------------------------
I am attaching a sample program to demostrate what exactly I am trying to
achieve :
Java Program HelloWorld.java:
package com.psl.si;
public class HelloWorld {
public void printHello() {
System.out.println("Hello World");
}
public int addNum(int n1,int n2){
return n1+n2;
}
}
Python Program helloworld.py :
#imported SparkConf and SparkContext
from pyspark import SparkConf,SparkContext
from jnius import autoclass
conf = SparkConf().setMaster("local").setAppName("hello world")
sc = SparkContext(conf = conf)
def mapFunction():
def addOneToList(irdd):
#Java Class loading using jnius autoclass method
helloWorld = autoclass('com.psl.si.HelloWorld')
hw = helloWorld()
#Calling printHello() method from HelloWorld Java class
hw.printHello()
#Calling addNum() method from HelloWorld Java class
sum=hw.addNum(4,6)
print "sum=",sum
#Adding 1 to existing list
olist=[]
olist.append(irdd+1)
return olist
data = [1,2,3,4,5]
#rdd from list
distData = sc.parallelize(data)
x=distData.map(addOneToList)
#Calling mapFunction
mapFunction()
Created Jar of HelloWorld program which is inside com.psl.si package
jar -cvf HelloWorld.jar com (as HelloWorld.class is inside
com.psl.si.HelloWorld)
Finally submitting Spark Job:
spark-submit --jars HelloWorld.jar helloworld.py
Error that I am getting:
c = find_javaclass(clsname)
File "jnius_export_func.pxi", line 23, in jnius.find_javaclass
(jnius/jnius.c:12356)
JavaException: Class not found 'com/psl/si/HelloWorld'
It is not able to load the class from the jar. Please give your suggestions if
I am doing anything wrong.
I am using pyjnius-1.3.0 for calling Java functions from Python.
was (Author: rajupats91):
I am attaching a sample program to demostrate what exactly I am trying to
achieve :
Java Program HelloWorld.java:
package com.psl.si;
public class HelloWorld {
public void printHello() {
System.out.println("Hello World");
}
public int addNum(int n1,int n2){
return n1+n2;
}
}
Python Program helloworld.py :
#imported SparkConf and SparkContext
from pyspark import SparkConf,SparkContext
from jnius import autoclass
conf = SparkConf().setMaster("local").setAppName("hello world")
sc = SparkContext(conf = conf)
def mapFunction():
def addOneToList(irdd):
#Java Class loading using jnius autoclass method
helloWorld = autoclass('com.psl.si.HelloWorld')
hw = helloWorld()
#Calling printHello() method from HelloWorld Java class
hw.printHello()
#Calling addNum() method from HelloWorld Java class
sum=hw.addNum(4,6)
print "sum=",sum
#Adding 1 to existing list
olist=[]
olist.append(irdd+1)
return olist
data = [1,2,3,4,5]
#rdd from list
distData = sc.parallelize(data)
x=distData.map(addOneToList)
#Calling mapFunction
mapFunction()
Created Jar of HelloWorld program which is inside com.psl.si package
jar -cvf HelloWorld.jar com (as HelloWorld.class is inside
com.psl.si.HelloWorld)
Finally submitting Spark Job:
spark-submit --jars HelloWorld.jar helloworld.py
Error that I am getting:
c = find_javaclass(clsname)
File "jnius_export_func.pxi", line 23, in jnius.find_javaclass
(jnius/jnius.c:12356)
JavaException: Class not found 'com/psl/si/HelloWorld'
It is not able to load the class from the jar. Please give your suggestions if
I am doing anything wrong.
I am using pyjnius-1.3.0 for calling Java functions from Python
> Unable to load external jars while submitting Spark Job
> -------------------------------------------------------
>
> Key: SPARK-6301
> URL: https://issues.apache.org/jira/browse/SPARK-6301
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Submit
> Affects Versions: 1.2.0
> Reporter: raju patel
>
> We are using Jnius to call Java functions from Python. But when we are trying
> to submit the job using Spark,it is not able to load the java classes that
> are provided in the --jars option, although it is successfully able to load
> python class.
> The Error is like this :
> c = find_javaclass(clsname)
> File "jnius_export_func.pxi", line 23, in jnius.find_javaclass
> (jnius/jnius.c:12815)
> JavaException: Class not found
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
