Hi Dean I guess you are using Spark 1.3.
- The JDBC driver class must be visible to the primordial class loader on the client session and on all executors. This is because Java’s DriverManager class does a security check that results in it ignoring all drivers not visible to the primordial class loader when one goes to open a connection. One convenient way to do this is to modify compute_classpath.sh on all worker nodes to include your driver JARs. Take a look at this https://issues.apache.org/jira/browse/SPARK-6913 and see http://stackoverflow.com/questions/30221677/spark-sql-postgresql-jdbc-classpath-issues . Regards Deenar *Think Reactive Ltd* deenar.toras...@thinkreactive.co.uk 07714140812 On 29 October 2015 at 10:34, dean.wood <dean.w...@sparkol.com> wrote: > I'm having a problem building a spark jar with scala. It's a really simple > thing, I want to programatically access a mysql server via JDBC and load it > On 29 October 2015 at 10:34, dean.wood <dean.w...@sparkol.com> wrote: > I'm having a problem building a spark jar with scala. It's a really simple > thing, I want to programatically access a mysql server via JDBC and load it > in to a spark data frame. I can get this to work in the spark shell but I > cannot package a jar that works with spark submit. It will package but when > running, fails with > > Exception in thread "main" java.sql.SQLException: No suitable driver found > for jdbc:mysql://localhost:3310/100million > My spark-submit command is > > ./bin/spark-submit ~/path/to/scala/project/target/scala-2.10/complete.jar > --driver-class-path ~/path/to/mysql-connector-java-5.1.37-bin.jar > > My build.sbt looks like > > name := "sql_querier" > > version := "1.0" > > scalaVersion := "2.10.4" > > sbtVersion := "0.13.7" > > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" % > "provided" > > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" % > "provided" > > libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.37" > > assemblyJarName in assembly := "complete.jar" > > mainClass in assembly := Some("sql_querier") > > offline := true > and my very simple code is > > import org.apache.spark.SparkContext > import org.apache.spark.SparkConf > import org.apache.spark.sql.SQLContext > > object sql_querier{ > > def main(args: Array[String]) { > > val sc = new org.apache.spark.SparkContext() > val sqlContext = new org.apache.spark.sql.SQLContext(sc) > val url="jdbc:mysql://databaseurl:portno/database" > > val prop = new java.util.Properties > prop.setProperty("user","myuser") > prop.setProperty("password","mydatabase") > val cats=sqlContext.read.jdbc(url, "categories", prop) > cats.show > } > } > Where I've hidden the real values for user password and database url. I've > also got a file in projects that adds the sbt assembly plugin and there is > nothing wrong with this. I've successfully used sbt assembly before with > this configuration. When starting a spark shell with the > --driver-class-path > option pointing to the mysql jar, I can run the commands and extract data > from the mysql database. > > I've tried version 5.1.34 and 5.0.8 and neither have worked. I've also > tried > changing --driver-class-path for --jar in the spark submit command and > adding the lines > > > > sc.addJar("/Users/dean.wood/data_science/scala/sqlconn/mysql-connector-java-5.0.8-bin.jar") > Class.forName("com.mysql.jdbc.Driver") > > to the scala code. > > Any clue what I am doing wrong with the build would be greatly appreciated. > > Dean > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Packaging-a-jar-for-a-jdbc-connection-using-sbt-assembly-and-scala-tp25225.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
