Thanks, Michael. From: mich...@databricks.com Date: Fri, 25 Jul 2014 14:49:00 -0700 Subject: Re: Spark SQL and Hive tables To: user@spark.apache.org
>From the programming guide: When working with Hive one must construct a HiveContext, which inherits from SQLContext, and adds support for finding tables in in the MetaStore and writing queries using HiveQL. conf/ is a top level directory in the spark distribution that you downloaded. On Fri, Jul 25, 2014 at 2:35 PM, Sameer Tilak <ssti...@live.com> wrote: Hi Jerry,Thanks for your reply. I was following the steps in this programming guide. It does not mention anything about creating HiveContext or HQL explicitly. http://databricks.com/blog/2014/03/26/spark-sql-manipulating-structured-data-using-spark-2.html Users(userId INT, name String, email STRING, age INT, latitude: DOUBLE, longitude: DOUBLE, subscribed: BOOLEAN)Events(userId INT, action INT) Given the data stored in in these tables, one might want to build a model that will predict which users are good targets for a new campaign, based on users that are similar. // Data can easily be extracted from existing sources, // such as Apache Hive. val trainingDataTable = sql(""" SELECT e.action u.age, u.latitude, u.logitude FROM Users u JOIN Events e ON u.userId = e.userId""") Date: Fri, 25 Jul 2014 17:27:17 -0400 Subject: Re: Spark SQL and Hive tables From: chiling...@gmail.com To: user@spark.apache.org Hi Sameer, Maybe this page will help you: https://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables Best Regards, Jerry On Fri, Jul 25, 2014 at 5:25 PM, Sameer Tilak <ssti...@live.com> wrote: Hi All,I am trying to load data from Hive tables using Spark SQL. I am using spark-shell. Here is what I see: val trainingDataTable = sql("""SELECT prod.prod_num, demographics.gender, demographics.birth_year, demographics.income_group FROM prod p JOIN demographics d ON d.user_id = p.user_id""") 14/07/25 14:18:46 INFO Analyzer: Max iterations (2) reached for batch MultiInstanceRelations14/07/25 14:18:46 INFO Analyzer: Max iterations (2) reached for batch CaseInsensitiveAttributeReferences java.lang.RuntimeException: Table Not Found: prod. I have these tables in hive. I used show tables command to confirm this. Can someone please let me know how do I make them accessible here?
