Re: Spark SQL 1.5.2 missing JDBC driver for PostgreSQL?

Chris, I have a question about your setup. Does it allow the same usage of Cassandra/HBase data sources? Can I create a table that links to and be used by Spark SQL? The reason for asking is that I see the Cassandra connector package included in your script. Thanks, Ben > On Dec 25, 2015, at

Re: Spark SQL 1.5.2 missing JDBC driver for PostgreSQL?

Hi Chris, I did what you did. It works for me now! Thanks for your help. Have a Merry Christmas! Cheers, Ben > On Dec 25, 2015, at 6:41 AM, Chris Fregly wrote: > > Configuring JDBC drivers with Spark is a bit tricky as the JDBC driver needs > to be on the Java System Classpath per this > <

Re: Spark SQL 1.5.2 missing JDBC driver for PostgreSQL?

Configuring JDBC drivers with Spark is a bit tricky as the JDBC driver needs to be on the Java System Classpath per this troubleshooting section in the Spark SQL programming guide. Here

Re: Spark SQL 1.5.2 missing JDBC driver for PostgreSQL?

Stephen, Let me confirm. I just need to propagate these settings I put in spark-defaults.conf to all the worker nodes? Do I need to do the same with the PostgreSQL driver jar file too? If so, is there a way to have it read from HDFS rather than copying out to the cluster manually. Thanks for your

Re: Spark SQL 1.5.2 missing JDBC driver for PostgreSQL?

HI Benjamin, yes by adding to the thrift server then the create table would work. But querying is performed by the workers: so you need to add to the classpath of all nodes for reads to work. 2015-12-22 18:35 GMT-08:00 Benjamin Kim : > Hi Stephen, > > I forgot to mention that I added these line

Re: Spark SQL 1.5.2 missing JDBC driver for PostgreSQL?

Hi Stephen, I forgot to mention that I added these lines below to the spark-default.conf on the node with Spark SQL Thrift JDBC/ODBC Server running on it. Then, I restarted it. spark.driver.extraClassPath=/usr/share/java/postgresql-9.3-1104.jdbc41.jar spark.executor.extraClassPath=/usr/share/ja

Re: Spark SQL 1.5.2 missing JDBC driver for PostgreSQL?

The postgres jdbc driver needs to be added to the classpath of your spark workers. You can do a search for how to do that (multiple ways). 2015-12-22 17:22 GMT-08:00 b2k70 : > I see in the Spark SQL documentation that a temporary table can be created > directly onto a remote PostgreSQL table. >

Spark SQL 1.5.2 missing JDBC driver for PostgreSQL?

I see in the Spark SQL documentation that a temporary table can be created directly onto a remote PostgreSQL table. CREATE TEMPORARY TABLE USING org.apache.spark.sql.jdbc OPTIONS ( url "jdbc:postgresql:///", dbtable "impressions" ); When I run this against our PostgreSQL server, I get the followi