I am able to connect to MySQL Hive metastore from the client
cluster machine.
-sh-4.1$ mysql --user=hiveuser --password=pass
--host=hostname.vip.company.com <http://hostname.vip.company.com>
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9417286
Server version: 5.5.12-eb-5.5.12-log MySQL-eb 5.5.12, Revision 3492
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All
rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current
input statement.
mysql> use eBayHDB;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+---------------------------+
| Tables_in_HDB |
+---------------------------+
Regards,
Deepak
On Sat, Mar 28, 2015 at 12:35 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)
<deepuj...@gmail.com <mailto:deepuj...@gmail.com>> wrote:
Yes am using yarn-cluster and i did add it via --files. I get
"Suitable error not found error"
Please share the spark-submit command that shows mysql jar
containing driver class used to connect to Hive MySQL meta
store.
Even after including it through
--driver-class-path
/home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar
OR (AND)
--jars /home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar
I keep getting "Suitable driver not found for"
Command
========
./bin/spark-submit -v --master yarn-cluster
--driver-class-path
*/home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar*:/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar
--jars
/home/dvasthimal/spark1.3/spark-avro_2.10-1.0.0.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar,/home/dvasthimal/spark1.3/spark-1.3.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar,*/home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.ja*r
--files $SPARK_HOME/conf/hive-site.xml --num-executors 1
--driver-memory 4g --driver-java-options "-XX:MaxPermSize=2G"
--executor-memory 2g --executor-cores 1 --queue hdmi-express
--class com.ebay.ep.poc.spark.reporting.SparkApp
spark_reporting-1.0-SNAPSHOT.jar startDate=2015-02-16
endDate=2015-02-16
input=/user/dvasthimal/epdatasets/successdetail1/part-r-00000.avro
subcommand=successevents2
output=/user/dvasthimal/epdatasets/successdetail2
Logs
====
Caused by: java.sql.SQLException: No suitable driver found
for jdbc:mysql://hostname:3306/HDB
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at
com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
... 68 more
...
...
15/03/27 23:56:08 INFO yarn.Client: Uploading resource
file:/home/dvasthimal/spark1.3/mysql-connector-java-5.1.34.jar ->
hdfs://apollo-NN:8020/user/dvasthimal/.sparkStaging/application_1426715280024_119815/mysql-connector-java-5.1.34.jar
...
...
-sh-4.1$ jar -tvf ../mysql-connector-java-5.1.34.jar | grep
Driver
61 Fri Oct 17 08:05:36 GMT-07:00 2014
META-INF/services/java.sql.Driver
3396 Fri Oct 17 08:05:22 GMT-07:00 2014
com/mysql/fabric/jdbc/FabricMySQLDriver.class
* 692 Fri Oct 17 08:05:22 GMT-07:00 2014
com/mysql/jdbc/Driver.class*
1562 Fri Oct 17 08:05:20 GMT-07:00 2014
com/mysql/jdbc/NonRegisteringDriver$ConnectionPhantomReference.class
17817 Fri Oct 17 08:05:20 GMT-07:00 2014
com/mysql/jdbc/NonRegisteringDriver.class
690 Fri Oct 17 08:05:24 GMT-07:00 2014
com/mysql/jdbc/NonRegisteringReplicationDriver.class
731 Fri Oct 17 08:05:24 GMT-07:00 2014
com/mysql/jdbc/ReplicationDriver.class
336 Fri Oct 17 08:05:24 GMT-07:00 2014
org/gjt/mm/mysql/Driver.class
You have new mail in /var/spool/mail/dvasthimal
-sh-4.1$ cat conf/hive-site.xml | grep Driver
<name>javax.jdo.option.ConnectionDriverName</name>
*<value>com.mysql.jdbc.Driver</value>*
<description>Driver class name for a JDBC
metastore</description>
-sh-4.1$
--
Deepak
On Sat, Mar 28, 2015 at 1:06 AM, Michael Armbrust
<mich...@databricks.com <mailto:mich...@databricks.com>> wrote:
Are you running on yarn?
- If you are running in yarn-client mode, set
HADOOP_CONF_DIR to /etc/hive/conf/ (or the directory
where your hive-site.xml is located).
- If you are running in yarn-cluster mode, the easiest
thing to do is to add--files=/etc/hive/conf/hive-site.xml
(or the path for your hive-site.xml) to your spark-submit
script.
On Fri, Mar 27, 2015 at 5:42 AM, ÐΞ€ρ@Ҝ (๏̯͡๏)
<deepuj...@gmail.com <mailto:deepuj...@gmail.com>> wrote:
I can recreate tables but what about data. It looks
like this is a obvious feature that Spark SQL must be
having. People will want to transform tons of data
stored in HDFS through Hive from Spark SQL.
Spark programming guide suggests its possible.
Spark SQL also supports reading and writing data
stored in Apache Hive <http://hive.apache.org/>. ....
Configuration of Hive is done by placing your
|hive-site.xml| file in |conf/|.
https://spark.apache.org/docs/1.3.0/sql-programming-guide.html#hive-tables
For some reason its not working.
On Fri, Mar 27, 2015 at 3:35 PM, Arush Kharbanda
<ar...@sigmoidanalytics.com
<mailto:ar...@sigmoidanalytics.com>> wrote:
Seems Spark SQL accesses some more columns apart
from those created by hive.
You can always recreate the tables, you would
need to execute the table creation scripts but it
would be good to avoid recreation.
On Fri, Mar 27, 2015 at 3:20 PM, ÐΞ€ρ@Ҝ (๏̯͡๏)
<deepuj...@gmail.com
<mailto:deepuj...@gmail.com>> wrote:
I did copy hive-conf.xml form Hive
installation into spark-home/conf. IT does
have all the meta store connection details,
host, username, passwd, driver and others.
Snippet
======
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://host.vip.company.com:3306/HDB
<http://host.vip.company.com:3306/HDB></value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC
metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hiveuser</value>
<description>username to use against
metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>some-password</value>
<description>password to use against
metastore database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to
remove metastore server or open a new
metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>location of default database for
the warehouse</description>
</property>
......
When i attempt to read hive table, it does
not work. dw_bid does not exists.
I am sure there is a way to read tables
stored in HDFS (Hive) from Spark SQL.
Otherwise how would anyone do analytics since
the source tables are always either persisted
directly on HDFS or through Hive.
On Fri, Mar 27, 2015 at 1:15 PM, Arush
Kharbanda <ar...@sigmoidanalytics.com
<mailto:ar...@sigmoidanalytics.com>> wrote:
Since hive and spark SQL internally use
HDFS and Hive metastore. The only thing
you want to change is the processing
engine. You can try to bring your
hive-site.xml to
%SPARK_HOME%/conf/hive-site.xml.(Ensure
that the hive site xml captures the
metastore connection details).
Its a hack, i havnt tried it. I have
played around with the metastore and it
should work.
On Fri, Mar 27, 2015 at 12:04 PM, ÐΞ€ρ@Ҝ
(๏̯͡๏) <deepuj...@gmail.com
<mailto:deepuj...@gmail.com>> wrote:
I have few tables that are created in
Hive. I wan to transform data stored
in these Hive tables using Spark SQL.
Is this even possible ?
So far i have seen that i can create
new tables using Spark SQL dialect.
However when i run show tables or do
desc hive_table it says table not found.
I am now wondering is this support
present or not in Spark SQL ?
--
Deepak
--
Sigmoid Analytics
<http://htmlsig.com/www.sigmoidanalytics.com>
*Arush Kharbanda* || Technical Teamlead
ar...@sigmoidanalytics.com
<mailto:ar...@sigmoidanalytics.com> ||
www.sigmoidanalytics.com
<http://www.sigmoidanalytics.com/>
--
Deepak
--
Sigmoid Analytics
<http://htmlsig.com/www.sigmoidanalytics.com>
*Arush Kharbanda* || Technical Teamlead
ar...@sigmoidanalytics.com
<mailto:ar...@sigmoidanalytics.com> ||
www.sigmoidanalytics.com
<http://www.sigmoidanalytics.com/>
--
Deepak
--
Deepak
--
Deepak
