This is from my Hive installation -sh-4.1$ ls /apache/hive/lib | grep derby
derby-10.10.1.1.jar derbyclient-10.10.1.1.jar derbynet-10.10.1.1.jar -sh-4.1$ ls /apache/hive/lib | grep datanucleus datanucleus-api-jdo-3.2.6.jar datanucleus-core-3.2.10.jar datanucleus-rdbms-3.2.9.jar -sh-4.1$ ls /apache/hive/lib | grep mysql mysql-connector-java-5.0.8-bin.jar -sh-4.1$ $ hive --version Hive 0.13.0.2.1.3.6-2 Subversion git://ip-10-0-0-90.ec2.internal/grid/0/jenkins/workspace/BIGTOP-HDP_RPM_REPO-HDP-2.1.3.6-centos6/bigtop/build/hive/rpm/BUILD/hive-0.13.0.2.1.3.6 -r 87da9430050fb9cc429d79d95626d26ea382b96c $ On Sat, Mar 28, 2015 at 1:05 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]> wrote: > I tried with a different version of driver but same error > > ./bin/spark-submit -v --master yarn-cluster --driver-class-path > /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar > --jars > /home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar, > */home/dvasthimal/spark1.3/mysql-connector-java-5.0.8-bin.jar* --files > $SPARK_HOME/conf/hive-site.xml --num-executors 1 --driver-memory 4g > --driver-java-options "-XX:MaxPermSize=2G" --executor-memory 2g > --executor-cores 1 --queue hdmi-express --class > com.ebay.ep.poc.spark.reporting.SparkApp spark_reporting-1.0-SNAPSHOT.jar > startDate=2015-02-16 endDate=2015-02-16 > input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro > subcommand=successevents2 output=/user/dvasthimal/epdatasets/successdetail2 > > On Sat, Mar 28, 2015 at 12:47 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]> > wrote: > >> This is what am seeing >> >> >> >> ./bin/spark-submit -v --master yarn-cluster --driver-class-path >> /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar >> --jars >> /home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar >> --files $SPARK_HOME/conf/hive-site.xml --num-executors 1 --driver-memory >> 4g --driver-java-options "-XX:MaxPermSize=2G" --executor-memory 2g >> --executor-cores 1 --queue hdmi-express --class >> com.ebay.ep.poc.spark.reporting.SparkApp spark_reporting-1.0-SNAPSHOT.jar >> startDate=2015-02-16 endDate=2015-02-16 >> input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro >> subcommand=successevents2 output=/user/dvasthimal/epdatasets/successdetail2 >> >> >> Caused by: >> org.datanucleus.store.rdbms.connectionpool.DatastoreDriverNotFoundException: >> The specified datastore driver ("com.mysql.jdbc.Driver") was not found in >> the CLASSPATH. Please check your CLASSPATH specification, and the name of >> the driver. >> >> >> >> >> ./bin/spark-submit -v --master yarn-cluster --driver-class-path >> /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar >> --jars >> /home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar,*/home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar >> *--files $SPARK_HOME/conf/hive-site.xml --num-executors 1 >> --driver-memory 4g --driver-java-options "-XX:MaxPermSize=2G" >> --executor-memory 2g --executor-cores 1 --queue hdmi-express --class >> com.ebay.ep.poc.spark.reporting.SparkApp spark_reporting-1.0-SNAPSHOT.jar >> startDate=2015-02-16 endDate=2015-02-16 >> input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro >> subcommand=successevents2 output=/user/dvasthimal/epdatasets/successdetail2 >> >> >> Caused by: java.sql.SQLException: No suitable driver found for >> jdbc:mysql://db_host_name.vip.ebay.com:3306/HDB >> at java.sql.DriverManager.getConnection(DriverManager.java:596) >> >> >> Looks like the driver jar that i got in is not correct, >> >> On Sat, Mar 28, 2015 at 12:34 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]> >> wrote: >> >>> Could someone please share the spark-submit command that shows their >>> mysql jar containing driver class used to connect to Hive MySQL meta store. >>> >>> Even after including it through >>> >>> --driver-class-path >>> /home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar >>> OR (AND) >>> --jars /home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar >>> >>> I keep getting "Suitable driver not found for >>> >>> >>> Command >>> ======== >>> >>> ./bin/spark-submit -v --master yarn-cluster --driver-class-path >>> */home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar*:/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar >>> --jars >>> /home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar, >>> */home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.ja*r --files >>> $SPARK_HOME/conf/hive-site.xml --num-executors 1 --driver-memory 4g >>> --driver-java-options "-XX:MaxPermSize=2G" --executor-memory 2g >>> --executor-cores 1 --queue hdmi-express --class >>> com.ebay.ep.poc.spark.reporting.SparkApp spark_reporting-1.0-SNAPSHOT.jar >>> startDate=2015-02-16 endDate=2015-02-16 >>> input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro >>> subcommand=successevents2 output=/user/dvasthimal/epdatasets/successdetail2 >>> Logs >>> ==== >>> >>> Caused by: java.sql.SQLException: No suitable driver found for >>> jdbc:mysql://hostname:3306/HDB >>> at java.sql.DriverManager.getConnection(DriverManager.java:596) >>> at java.sql.DriverManager.getConnection(DriverManager.java:187) >>> at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361) >>> at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416) >>> ... 68 more >>> ... >>> ... >>> >>> 15/03/27 23:56:08 INFO yarn.Client: Uploading resource >>> file:/home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar -> hdfs:// >>> apollo-phx-nn.vip.ebay.com:8020/user/dvasthimal/.sparkStaging/application_1426715280024_119815/mysql-connector-java-5.1.34.jar >>> >>> ... >>> >>> ... >>> >>> >>> >>> -sh-4.1$ jar -tvf ../mysql-connector-java-5.1.34.jar | grep Driver >>> 61 Fri Oct 17 08:05:36 GMT-07:00 2014 >>> META-INF/services/java.sql.Driver >>> 3396 Fri Oct 17 08:05:22 GMT-07:00 2014 >>> com/mysql/fabric/jdbc/FabricMySQLDriver.class >>> * 692 Fri Oct 17 08:05:22 GMT-07:00 2014 com/mysql/jdbc/Driver.class* >>> 1562 Fri Oct 17 08:05:20 GMT-07:00 2014 >>> com/mysql/jdbc/NonRegisteringDriver$ConnectionPhantomReference.class >>> 17817 Fri Oct 17 08:05:20 GMT-07:00 2014 >>> com/mysql/jdbc/NonRegisteringDriver.class >>> 690 Fri Oct 17 08:05:24 GMT-07:00 2014 >>> com/mysql/jdbc/NonRegisteringReplicationDriver.class >>> 731 Fri Oct 17 08:05:24 GMT-07:00 2014 >>> com/mysql/jdbc/ReplicationDriver.class >>> 336 Fri Oct 17 08:05:24 GMT-07:00 2014 org/gjt/mm/mysql/Driver.class >>> You have new mail in /var/spool/mail/dvasthimal >>> -sh-4.1$ cat conf/hive-site.xml | grep Driver >>> <name>javax.jdo.option.ConnectionDriverName</name> >>> * <value>com.mysql.jdbc.Driver</value>* >>> <description>Driver class name for a JDBC metastore</description> >>> -sh-4.1$ >>> >>> -- >>> Deepak >>> >>> >> >> >> -- >> Deepak >> >> > > > -- > Deepak > > -- Deepak
