Re: Question about SparkSQL and Hive-on-Spark
Hi Reynold! Will sparkSQL strictly obey the HQL syntax ? For example, the cube function. In other words, the hiveContext of sparkSQL should only implement the subset of HQL features? Best Regards, Yi Tian tianyi.asiai...@gmail.com On Sep 23, 2014, at 15:49, Reynold Xin r...@databricks.com wrote: On Tue, Sep 23, 2014 at 12:47 AM, Yi Tian tianyi.asiai...@gmail.com wrote: Hi all, I have some questions about the SparkSQL and Hive-on-Spark Will SparkSQL support all the hive feature in the future? or just making hive as a datasource of Spark? Most likely not *ALL* Hive features, but almost all common features. From Spark 1.1.0 , we have thrift-server support running hql on spark. Will this feature be replaced by Hive on Spark? No. The reason for asking these questions is that we found some hive functions are not running well on SparkSQL ( like window function, cube and rollup function) Is it worth for making effort on implement these functions with SparkSQL? Could you guys give some advices ? Yes absolutely. thank you. Best Regards, Yi Tian tianyi.asiai...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Question about SparkSQL and Hive-on-Spark
I don’t think so. For example, we’ve already added extended syntax like CACHE TABLE. On Wed, Sep 24, 2014 at 3:27 PM, Yi Tian tianyi.asiai...@gmail.com wrote: Hi Reynold! Will sparkSQL strictly obey the HQL syntax ? For example, the cube function. In other words, the hiveContext of sparkSQL should only implement the subset of HQL features? Best Regards, Yi Tian tianyi.asiai...@gmail.com On Sep 23, 2014, at 15:49, Reynold Xin r...@databricks.com wrote: On Tue, Sep 23, 2014 at 12:47 AM, Yi Tian tianyi.asiai...@gmail.com wrote: Hi all, I have some questions about the SparkSQL and Hive-on-Spark Will SparkSQL support all the hive feature in the future? or just making hive as a datasource of Spark? Most likely not *ALL* Hive features, but almost all common features. From Spark 1.1.0 , we have thrift-server support running hql on spark. Will this feature be replaced by Hive on Spark? No. The reason for asking these questions is that we found some hive functions are not running well on SparkSQL ( like window function, cube and rollup function) Is it worth for making effort on implement these functions with SparkSQL? Could you guys give some advices ? Yes absolutely. thank you. Best Regards, Yi Tian tianyi.asiai...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Question about SparkSQL and Hive-on-Spark
Hi Yi, So I've been thinking about implementing windowing for some time and started working on it in earnest yesterday. There is already a PR for ROLLUP and CUBE; you may want to look at it and see if you can help the author out or provide some test cases: https://github.com/apache/spark/pull/1567 As far as windowing, I'll be developing my own test cases but would appreciate it if you could also share some kinds of queries you're interested in so that I can incorporate them as well. best, wb - Original Message - From: Yi Tian tianyi.asiai...@gmail.com To: Will Benton wi...@redhat.com Cc: dev@spark.apache.org Sent: Tuesday, September 23, 2014 11:01:57 PM Subject: Re: Question about SparkSQL and Hive-on-Spark Hi, Will We are planning to start implementing these functions. We hope that we could make a general design in following week. Best Regards, Yi Tian tianyi.asiai...@gmail.com On Sep 23, 2014, at 23:39, Will Benton wi...@redhat.com wrote: Hi Yi, I've had some interest in implementing windowing and rollup in particular for some of my applications but haven't had them on the front of my plate yet. If you need them as well, I'm happy to start taking a look this week. best, wb - Original Message - From: Yi Tian tianyi.asiai...@gmail.com To: dev@spark.apache.org Sent: Tuesday, September 23, 2014 2:47:17 AM Subject: Question about SparkSQL and Hive-on-Spark Hi all, I have some questions about the SparkSQL and Hive-on-Spark Will SparkSQL support all the hive feature in the future? or just making hive as a datasource of Spark? From Spark 1.1.0 , we have thrift-server support running hql on spark. Will this feature be replaced by Hive on Spark? The reason for asking these questions is that we found some hive functions are not running well on SparkSQL ( like window function, cube and rollup function) Is it worth for making effort on implement these functions with SparkSQL? Could you guys give some advices ? thank you. Best Regards, Yi Tian tianyi.asiai...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Question about SparkSQL and Hive-on-Spark
Hi all, I have some questions about the SparkSQL and Hive-on-Spark Will SparkSQL support all the hive feature in the future? or just making hive as a datasource of Spark? From Spark 1.1.0 , we have thrift-server support running hql on spark. Will this feature be replaced by Hive on Spark? The reason for asking these questions is that we found some hive functions are not running well on SparkSQL ( like window function, cube and rollup function) Is it worth for making effort on implement these functions with SparkSQL? Could you guys give some advices ? thank you. Best Regards, Yi Tian tianyi.asiai...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Question about SparkSQL and Hive-on-Spark
On Tue, Sep 23, 2014 at 12:47 AM, Yi Tian tianyi.asiai...@gmail.com wrote: Hi all, I have some questions about the SparkSQL and Hive-on-Spark Will SparkSQL support all the hive feature in the future? or just making hive as a datasource of Spark? Most likely not *ALL* Hive features, but almost all common features. From Spark 1.1.0 , we have thrift-server support running hql on spark. Will this feature be replaced by Hive on Spark? No. The reason for asking these questions is that we found some hive functions are not running well on SparkSQL ( like window function, cube and rollup function) Is it worth for making effort on implement these functions with SparkSQL? Could you guys give some advices ? Yes absolutely. thank you. Best Regards, Yi Tian tianyi.asiai...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Question about SparkSQL and Hive-on-Spark
Hi Yi, I've had some interest in implementing windowing and rollup in particular for some of my applications but haven't had them on the front of my plate yet. If you need them as well, I'm happy to start taking a look this week. best, wb - Original Message - From: Yi Tian tianyi.asiai...@gmail.com To: dev@spark.apache.org Sent: Tuesday, September 23, 2014 2:47:17 AM Subject: Question about SparkSQL and Hive-on-Spark Hi all, I have some questions about the SparkSQL and Hive-on-Spark Will SparkSQL support all the hive feature in the future? or just making hive as a datasource of Spark? From Spark 1.1.0 , we have thrift-server support running hql on spark. Will this feature be replaced by Hive on Spark? The reason for asking these questions is that we found some hive functions are not running well on SparkSQL ( like window function, cube and rollup function) Is it worth for making effort on implement these functions with SparkSQL? Could you guys give some advices ? thank you. Best Regards, Yi Tian tianyi.asiai...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Question about SparkSQL and Hive-on-Spark
Hi, Will We are planning to start implementing these functions. We hope that we could make a general design in following week. Best Regards, Yi Tian tianyi.asiai...@gmail.com On Sep 23, 2014, at 23:39, Will Benton wi...@redhat.com wrote: Hi Yi, I've had some interest in implementing windowing and rollup in particular for some of my applications but haven't had them on the front of my plate yet. If you need them as well, I'm happy to start taking a look this week. best, wb - Original Message - From: Yi Tian tianyi.asiai...@gmail.com To: dev@spark.apache.org Sent: Tuesday, September 23, 2014 2:47:17 AM Subject: Question about SparkSQL and Hive-on-Spark Hi all, I have some questions about the SparkSQL and Hive-on-Spark Will SparkSQL support all the hive feature in the future? or just making hive as a datasource of Spark? From Spark 1.1.0 , we have thrift-server support running hql on spark. Will this feature be replaced by Hive on Spark? The reason for asking these questions is that we found some hive functions are not running well on SparkSQL ( like window function, cube and rollup function) Is it worth for making effort on implement these functions with SparkSQL? Could you guys give some advices ? thank you. Best Regards, Yi Tian tianyi.asiai...@gmail.com - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
