Re: Nested Query using SparkSQL 1.1.0

2014-10-13 Thread Yin Huai
Hi Shahab, Can you try to use HiveContext? Its should work in 1.1. For SQLContext, this issues was not fixed in 1.1 and you need to use master branch at the moment. Thanks, Yin On Sun, Oct 12, 2014 at 5:20 PM, shahab shahab.mok...@gmail.com wrote: Hi, Apparently is it is possible to query

Re: Nested Query using SparkSQL 1.1.0

2014-10-13 Thread shahab
Thanks Yin. I trued HiveQL and and it solved that problem. But now I have second query requirement : But since you are main developer behind JSON-Spark integration (I saw your presentation on youtube Easy JSON Data Manipulation in Spark), is it possible to perform aggregation kind queries, for

Re: Nested Query using SparkSQL 1.1.0

2014-10-13 Thread Yin Huai
Hi Shahab, Do you mean queries with group by and aggregation functions? Once you register the json dataset as a table, you can write queries like querying a regular table. You can join it with other tables and do aggregations. Is it what you were asking for? If not, can you give me a more

Nested Query using SparkSQL 1.1.0

2014-10-12 Thread shahab
Hi, Apparently is it is possible to query nested json using spark SQL, but , mainly due to lack of proper documentation/examples, I did not manage to make it working. I do appreciate if you could point me to any example or help with this issue, Here is my code: val anotherPeopleRDD =