That should totally work. The other option would be to run a persistent metastore that multiple contexts can talk to and periodically run a job that creates missing tables. The trade-off here would be more complexity, but less downtime due to the server restarting.
On Tue, Apr 7, 2015 at 12:34 PM, James Aley <james.a...@swiftkey.com> wrote: > Hi Michael, > > Thanks so much for the reply - that really cleared a lot of things up for > me! > > Let me just check that I've interpreted one of your suggestions for (4) > correctly... Would it make sense for me to write a small wrapper app that > pulls in hive-thriftserver as a dependency, iterates my Parquet directory > structure to discover "tables" and registers each as a temp table in some > context, before calling HiveThriftServer2.createWithContext as you > suggest? > > This would mean that to add new content, all I need to is restart that > app, which presumably could also be avoided fairly trivially by > periodically restarting the server with a new context internally. That > certainly beats manual curation of Hive table definitions, if it will work? > > > Thanks again, > > James. > > On 7 April 2015 at 19:30, Michael Armbrust <mich...@databricks.com> wrote: > >> 1) What exactly is the relationship between the thrift server and Hive? >>> I'm guessing Spark is just making use of the Hive metastore to access table >>> definitions, and maybe some other things, is that the case? >>> >> >> Underneath the covers, the Spark SQL thrift server is executing queries >> using a HiveContext. In this mode, nearly all computation is done with >> Spark SQL but we try to maintain compatibility with Hive wherever >> possible. This means that you can write your queries in HiveQL, read >> tables from the Hive metastore, and use Hive UDFs UDTs UDAFs, etc. >> >> The one exception here is Hive DDL operations (CREATE TABLE, etc). These >> are passed directly to Hive code and executed there. The Spark SQL DDL is >> sufficiently different that we always try to parse that first, and fall >> back to Hive when it does not parse. >> >> One possibly confusing point here, is that you can persist Spark SQL >> tables into the Hive metastore, but this is not the same as a Hive table. >> We are only use the metastore as a repo for metadata, but are not using >> their format for the information in this case (as we have datasources that >> hive does not understand, including things like schema auto discovery). >> >> HiveQL DDL, run by Hive but can be read by Spark SQL: CREATE TABLE t (x >> INT) SORTED AS PARQUET >> Spark SQL DDL, run by Spark SQL, stored in metastore, cannot be read by >> hive: CREATE TABLE t USING parquet (path '/path/to/data') >> >> >>> 2) Am I therefore right in thinking that SQL queries sent to the thrift >>> server are still executed on the Spark cluster, using Spark SQL, and Hive >>> plays no active part in computation of results? >>> >> >> Correct. >> >> 3) What SQL flavour is actually supported by the Thrift Server? Is it >>> Spark SQL, Hive, or both? I've confused, because I've seen it accepting >>> Hive CREATE TABLE syntax, but Spark SQL seems to work too? >>> >> >> HiveQL++ (with Spark SQL DDL). You can make it use our simple SQL parser >> by `SET spark.sql.dialect=sql`, but honestly you probably don't want to do >> this. The included SQL parser is mostly there for people who have >> dependency conflicts with Hive. >> >> >>> 4) When I run SQL queries using the Scala or Python shells, Spark seems >>> to figure out the schema by itself from my Parquet files very well, if I >>> use createTempTable on the DataFrame. It seems when running the thrift >>> server, I need to create a Hive table definition first? Is that the case, >>> or did I miss something? If it is, is there some sensible way to automate >>> this? >>> >> >> Temporary tables are only visible to the SQLContext that creates them. >> If you want it to be visible to the server, you need to either start the >> thrift server with the same context your program is using >> (see HiveThriftServer2.createWithContext) or make a metastore table. This >> can be done using Spark SQL DDL: >> >> CREATE TABLE t USING parquet (path '/path/to/data') >> >> Michael >> > >
