Error: application failed with exceptionjava.lang.NoSuchMethodError:
org.apache.spark.sql.SQLContext.<init>(Lorg/apache/spark/api/java/JavaSparkContext;)V
at
examples.PersonRecordReader.getPersonRecords(PersonRecordReader.java:35)
at examples.PersonRecordReader.main(PersonRecordReader.java:17) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606) at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367) at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77) at
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Hi All
I am getting the above exception when I am using SQLContext in spark jobs.The
error occurs only with the insertion of these statements. The rdd is fine and
it prints all correctly.The error occurs when creating dataframes. I am using
maven dependencies version 1.3.1
public static void getPersonRecords(String...args) { SparkConf
sparkConf = new SparkConf().setAppName("SQLContext"); JavaSparkContext
javaSparkContext = new JavaSparkContext(sparkConf); JavaRDD<String>
lines = javaSparkContext.textFile(args[0], 1); JavaRDD<Person>
personRecords = lines.map(new Function<String, Person>() {
public Person call(String line) throws Exception {
System.out.println(line);
String[] rec = line.split(","); return new
Person(Integer.parseInt(rec[1].trim()), rec[0]); }
}); for(Person p : personRecords.collect()) {
System.out.println(p.getName()); } SQLContext
sqlContext = new SQLContext(javaSparkContext); DataFrame
dataFrame = sqlContext.createDataFrame(personRecords, Person.class); }
Please help me stuck with this since morning
thanksrakesh
