Re: Advice using Spark SQL and Thrift JDBC Server

, completed Apr 8, 2015 12:33:45 PM Mohammed *From:* Michael Armbrust [mailto:mich...@databricks.com] *Sent:* Wednesday, April 8, 2015 11:54 AM *To:* Mohammed Guller *Cc:* Todd Nist; James Aley; user; Patrick Wendell *Subject:* Re: Advice using Spark SQL and Thrift JDBC Server Sorry guys

RE: Advice using Spark SQL and Thrift JDBC Server

, 2015 5:49 AM To: James Aley Cc: Michael Armbrust; user Subject: Re: Advice using Spark SQL and Thrift JDBC Server To use the HiveThriftServer2.startWithContext, I thought one would use the following artifact in the build: org.apache.spark%% spark-hive-thriftserver % 1.3.0 But I am unable

Re: Advice using Spark SQL and Thrift JDBC Server

To use the HiveThriftServer2.startWithContext, I thought one would use the following artifact in the build: org.apache.spark%% spark-hive-thriftserver % 1.3.0 But I am unable to resolve the artifact. I do not see it in maven central or any other repo. Do I need to build Spark and

Re: Advice using Spark SQL and Thrift JDBC Server

:* Michael Armbrust; user *Subject:* Re: Advice using Spark SQL and Thrift JDBC Server To use the HiveThriftServer2.startWithContext, I thought one would use the following artifact in the build: org.apache.spark%% spark-hive-thriftserver % 1.3.0 But I am unable to resolve

RE: Advice using Spark SQL and Thrift JDBC Server

...@databricks.com] Sent: Wednesday, April 8, 2015 11:54 AM To: Mohammed Guller Cc: Todd Nist; James Aley; user; Patrick Wendell Subject: Re: Advice using Spark SQL and Thrift JDBC Server Sorry guys. I didn't realize that https://issues.apache.org/jira/browse/SPARK-4925 was not fixed yet. You can publish

RE: Advice using Spark SQL and Thrift JDBC Server

: Wednesday, April 8, 2015 6:16 PM To: Todd Nist Cc: Mohammed Guller; Michael Armbrust; James Aley; user Subject: Re: Advice using Spark SQL and Thrift JDBC Server Hey Guys, Someone submitted a patch for this just now. It's a very simple fix and we can merge it soon. However, it's just missed our

Re: Advice using Spark SQL and Thrift JDBC Server

Wendell *Subject:* Re: Advice using Spark SQL and Thrift JDBC Server Sorry guys. I didn't realize that https://issues.apache.org/jira/browse/SPARK-4925 was not fixed yet. You can publish locally in the mean time (sbt/sbt publishLocal). On Wed, Apr 8, 2015 at 8:29 AM, Mohammed Guller

Re: Advice using Spark SQL and Thrift JDBC Server

1) What exactly is the relationship between the thrift server and Hive? I'm guessing Spark is just making use of the Hive metastore to access table definitions, and maybe some other things, is that the case? Underneath the covers, the Spark SQL thrift server is executing queries using a

Re: Advice using Spark SQL and Thrift JDBC Server

That should totally work. The other option would be to run a persistent metastore that multiple contexts can talk to and periodically run a job that creates missing tables. The trade-off here would be more complexity, but less downtime due to the server restarting. On Tue, Apr 7, 2015 at 12:34

Re: Advice using Spark SQL and Thrift JDBC Server

Hi Michael, Thanks so much for the reply - that really cleared a lot of things up for me! Let me just check that I've interpreted one of your suggestions for (4) correctly... Would it make sense for me to write a small wrapper app that pulls in hive-thriftserver as a dependency, iterates my