[
https://issues.apache.org/jira/browse/SPARK-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229161#comment-15229161
]
Sudhakar Thota commented on SPARK-14442:
----------------------------------------
These are the details:
1. Error I am facing: Cannot load main class from JAR
file:/usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar
[spark@appdev1 ~]$ /usr/iop/current/spark-client/bin/spark-submit --verbose
--master yarn-cluster --jars
/usr/iop/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,
/usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar,
/usr/iop/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar --files
/usr/iop/current/spark-client/conf/hive-site.xml --class "SimpleApp"
/tmp/simple-project_2.10-1.0.jar
Using properties file: /usr/iop/4.1.0.0/spark/conf/spark-defaults.conf
Adding default property: spark.history.kerberos.keytab=none
Adding default property: spark.eventLog.enabled=true
Adding default property: spark.yarn.queue=default
Adding default property:
spark.yarn.historyServer.address=appdev1.softlayer.com:18080
Adding default property: spark.yarn.applicationMaster.waitTries=10
Adding default property: spark.history.kerberos.principal=none
Adding default property: spark.yarn.submit.file.replication=3
Adding default property:
spark.yarn.am.extraJavaOptions=-Diop.version={{iop_full_version}}
Adding default property: spark.yarn.containerLauncherMaxThreads=25
Adding default property: spark.yarn.max.executor.failures=3
Adding default property:
spark.driver.extraJavaOptions=-Diop.version={{iop_full_version}}
Adding default property: spark.yarn.preserve.staging.files=false
Adding default property:
spark.eventLog.dir=hdfs://appdev1.softlayer.com:8020/iop/apps/4.1.0.0/spark/logs/history-server
Adding default property: spark.history.ui.port=18080
Adding default property: spark.yarn.executor.memoryOverhead=384
Adding default property: spark.yarn.driver.memoryOverhead=384
Adding default property: spark.yarn.scheduler.heartbeat.interval-ms=5000
Adding default property:
spark.yarn.jar=hdfs://appdev1.softlayer.com:8020/iop/apps/4.1.0.0/spark/jars/spark-assembly.jar
Error: Cannot load main class from JAR
file:/usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar,
Run with --help for usage help or --verbose for debug output
[spark@appdev1 ~]$
2. The source code trying to execute.
/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql
import org.apache.spark.sql.hive
object SimpleApp {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
// println("************************************************************")
// println("We are in SPARK CONTEXT")
// println("************************************************************")
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
// println("************************************************************")
// println("We are in SQL CONTEXT which is actually HiveContext ")
// println("************************************************************")
val hc = new org.apache.spark.sql.hive.HiveContext(sc)
// println("************************************************************")
// println("We are in Hive CONTEXT")
// println("************************************************************")
// //val df =
hc.read.format("parquet").load("/Users/sudhakarthota/spark/examples/src/main/resources/users.parquet")
// // ("namescolors_08172015_4")
//
// //df.write.saveAsTable("namescolors_12162015_1")
// //df.show(5)
hc.sql("drop table mydbs")
hc.sql("show databases").write.saveAsTable("mydbs")
}
}
3. Spark versions that I ran this code on 1.4.1. and 1.5.1
4. Similar JIRA, that the problem is resolved in:
https://issues.apache.org/jira/browse/SPARK-4298
> Error: Cannot load main class from JAR
> file:/usr/iop/current/spark-client/lib/datanucleus-core-3.2.10.jar
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-14442
> URL: https://issues.apache.org/jira/browse/SPARK-14442
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.4.1, 1.5.1
> Reporter: Sudhakar Thota
> Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
