RE: Spark SQL -Hive transactions support
Do you have any plans for supporting hive transactions in Spark? From: Michael Armbrust [mailto:mich...@databricks.com] Sent: Tuesday, January 19, 2016 3:18 PM To: hnagar Cc: user Subject: Re: Spark SQL -Hive transactions support We don't support Hive style transaction. On Tue, Jan 19, 2016 at 11:32 AM, hnagar mailto:hemang.na...@mobiusws.com> > wrote: Hive has transactions support since version 0.14. I am using Spark 1.6, and Hive 1.2.1, are transactions supported in Spark SQL now. I tried in the Spark-Shell and it gives the following error org.apache.spark.sql.AnalysisException: Unsupported language features in query: insert into test values(1, 'john', 'dong') I am wondering if its a configuration issue, or this is still not supported in Spark. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Hive-transactions-support-tp26012.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org <mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org <mailto:user-h...@spark.apache.org>
Re: Spark SQL -Hive transactions support
Hive's ACID feature (which introduces transactions) is not required for inserts, only updates and deletes. Inserts should be supported on a vanilla Hive shell. I'm not sure how Spark interacts with Hive in that regard but perhaps the HiveSQLContext implementation is lacking support. On a separate note, ACID is rarely configured out of the box and also requires that your tables are declared with some specific attributes. This might be worth checking but does not seem like the issue in your case: https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration For streaming type processing it would be nice to integrate Spark with Hive's streaming API which uses the ACID feature: https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest Additionally, updates and deletes could be supported by Spark by integrating with the not yet released mutation API: http://htmlpreview.github.io/?https://github.com/apache/hive/blob/master/hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/mutate/package.html Elliot. On Tuesday, 19 January 2016, hnagar wrote: > Hive has transactions support since version 0.14. > > I am using Spark 1.6, and Hive 1.2.1, are transactions supported in Spark > SQL now. I tried in the Spark-Shell and it gives the following error > > org.apache.spark.sql.AnalysisException: > Unsupported language features in query: insert into test values(1, 'john', > 'dong') > > I am wondering if its a configuration issue, or this is still not supported > in Spark. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Hive-transactions-support-tp26012.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Spark SQL -Hive transactions support
We don't support Hive style transaction. On Tue, Jan 19, 2016 at 11:32 AM, hnagar wrote: > Hive has transactions support since version 0.14. > > I am using Spark 1.6, and Hive 1.2.1, are transactions supported in Spark > SQL now. I tried in the Spark-Shell and it gives the following error > > org.apache.spark.sql.AnalysisException: > Unsupported language features in query: insert into test values(1, 'john', > 'dong') > > I am wondering if its a configuration issue, or this is still not supported > in Spark. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-Hive-transactions-support-tp26012.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >