I tried with a different version of driver but same error ./bin/spark-submit -v --master yarn-cluster --driver-class-path /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar --jars /home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar, */home/dvasthimal/spark1.3/mysql-connector-java-5.0.8-bin.jar* --files $SPARK_HOME/conf/hive-site.xml --num-executors 1 --driver-memory 4g --driver-java-options "-XX:MaxPermSize=2G" --executor-memory 2g --executor-cores 1 --queue hdmi-express --class com.ebay.ep.poc.spark.reporting.SparkApp spark_reporting-1.0-SNAPSHOT.jar startDate=2015-02-16 endDate=2015-02-16 input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro subcommand=successevents2 output=/user/dvasthimal/epdatasets/successdetail2
On Sat, Mar 28, 2015 at 12:47 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]> wrote: > This is what am seeing > > > > ./bin/spark-submit -v --master yarn-cluster --driver-class-path > /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar > --jars > /home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar > --files $SPARK_HOME/conf/hive-site.xml --num-executors 1 --driver-memory > 4g --driver-java-options "-XX:MaxPermSize=2G" --executor-memory 2g > --executor-cores 1 --queue hdmi-express --class > com.ebay.ep.poc.spark.reporting.SparkApp spark_reporting-1.0-SNAPSHOT.jar > startDate=2015-02-16 endDate=2015-02-16 > input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro > subcommand=successevents2 output=/user/dvasthimal/epdatasets/successdetail2 > > > Caused by: > org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: > The specified datastore driver ("com.mysql.jdbc.Driver") was not found in > the CLASSPATH. Please check your CLASSPATH specification, and the name of > the driver. > > > > > ./bin/spark-submit -v --master yarn-cluster --driver-class-path > /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar > --jars > /home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar,*/home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar > *--files $SPARK_HOME/conf/hive-site.xml --num-executors 1 > --driver-memory 4g --driver-java-options "-XX:MaxPermSize=2G" > --executor-memory 2g --executor-cores 1 --queue hdmi-express --class > com.ebay.ep.poc.spark.reporting.SparkApp spark_reporting-1.0-SNAPSHOT.jar > startDate=2015-02-16 endDate=2015-02-16 > input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro > subcommand=successevents2 output=/user/dvasthimal/epdatasets/successdetail2 > > > Caused by: java.sql.SQLException: No suitable driver found for > jdbc:mysql://db_host_name.vip.ebay.com:3306/HDB > at java.sql.DriverManager.getConnection(DriverManager.java:596) > > > Looks like the driver jar that i got in is not correct, > > On Sat, Mar 28, 2015 at 12:34 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]> > wrote: > >> Could someone please share the spark-submit command that shows their >> mysql jar containing driver class used to connect to Hive MySQL meta store. >> >> Even after including it through >> >> --driver-class-path >> /home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar >> OR (AND) >> --jars /home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar >> >> I keep getting "Suitable driver not found for >> >> >> Command >> ======== >> >> ./bin/spark-submit -v --master yarn-cluster --driver-class-path >> */home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar*:/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar >> --jars >> /home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar, >> */home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.ja*r --files >> $SPARK_HOME/conf/hive-site.xml --num-executors 1 --driver-memory 4g >> --driver-java-options "-XX:MaxPermSize=2G" --executor-memory 2g >> --executor-cores 1 --queue hdmi-express --class >> com.ebay.ep.poc.spark.reporting.SparkApp spark_reporting-1.0-SNAPSHOT.jar >> startDate=2015-02-16 endDate=2015-02-16 >> input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro >> subcommand=successevents2 output=/user/dvasthimal/epdatasets/successdetail2 >> Logs >> ==== >> >> Caused by: java.sql.SQLException: No suitable driver found for >> jdbc:mysql://hostname:3306/HDB >> at java.sql.DriverManager.getConnection(DriverManager.java:596) >> at java.sql.DriverManager.getConnection(DriverManager.java:187) >> at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361) >> at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416) >> ... 68 more >> ... >> ... >> >> 15/03/27 23:56:08 INFO yarn.Client: Uploading resource >> file:/home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar -> hdfs:// >> apollo-phx-nn.vip.ebay.com:8020/user/dvasthimal/.sparkStaging/application_1426715280024_119815/mysql-connector-java-5.1.34.jar >> >> ... >> >> ... >> >> >> >> -sh-4.1$ jar -tvf ../mysql-connector-java-5.1.34.jar | grep Driver >> 61 Fri Oct 17 08:05:36 GMT-07:00 2014 >> META-INF/services/java.sql.Driver >> 3396 Fri Oct 17 08:05:22 GMT-07:00 2014 >> com/mysql/fabric/jdbc/FabricMySQLDriver.class >> * 692 Fri Oct 17 08:05:22 GMT-07:00 2014 com/mysql/jdbc/Driver.class* >> 1562 Fri Oct 17 08:05:20 GMT-07:00 2014 >> com/mysql/jdbc/NonRegisteringDriver$ConnectionPhantomReference.class >> 17817 Fri Oct 17 08:05:20 GMT-07:00 2014 >> com/mysql/jdbc/NonRegisteringDriver.class >> 690 Fri Oct 17 08:05:24 GMT-07:00 2014 >> com/mysql/jdbc/NonRegisteringReplicationDriver.class >> 731 Fri Oct 17 08:05:24 GMT-07:00 2014 >> com/mysql/jdbc/ReplicationDriver.class >> 336 Fri Oct 17 08:05:24 GMT-07:00 2014 org/gjt/mm/mysql/Driver.class >> You have new mail in /var/spool/mail/dvasthimal >> -sh-4.1$ cat conf/hive-site.xml | grep Driver >> <name>javax.jdo.option.ConnectionDriverName</name> >> * <value>com.mysql.jdbc.Driver</value>* >> <description>Driver class name for a JDBC metastore</description> >> -sh-4.1$ >> >> -- >> Deepak >> >> > > > -- > Deepak > > -- Deepak
