Re: Question about SparkSQL and Hive-on-Spark

2014-09-24 Thread Yi Tian
Hi Reynold! Will sparkSQL strictly obey the HQL syntax ? For example, the cube function. In other words, the hiveContext of sparkSQL should only implement the subset of HQL features? Best Regards, Yi Tian tianyi.asiai...@gmail.com On Sep 23, 2014, at 15:49, Reynold Xin

Re: Question about SparkSQL and Hive-on-Spark

2014-09-24 Thread Cheng Lian
I don’t think so. For example, we’ve already added extended syntax like CACHE TABLE. ​ On Wed, Sep 24, 2014 at 3:27 PM, Yi Tian tianyi.asiai...@gmail.com wrote: Hi Reynold! Will sparkSQL strictly obey the HQL syntax ? For example, the cube function. In other words, the hiveContext of

Re: Question about SparkSQL and Hive-on-Spark

2014-09-24 Thread Will Benton
Hi Yi, So I've been thinking about implementing windowing for some time and started working on it in earnest yesterday. There is already a PR for ROLLUP and CUBE; you may want to look at it and see if you can help the author out or provide some test cases:

Re: A couple questions about shared variables

2014-09-24 Thread Nan Zhu
I proposed a fix https://github.com/apache/spark/pull/2524 Glad to receive feedbacks -- Nan Zhu On Tuesday, September 23, 2014 at 9:06 PM, Sandy Ryza wrote: Filed https://issues.apache.org/jira/browse/SPARK-3642 for documenting these nuances. -Sandy On Mon, Sep 22, 2014 at

Spark SQL use of alias in where clause

2014-09-24 Thread Du Li
Hi, The following query does not work in Shark nor in the new Spark SQLContext or HiveContext. SELECT key, value, concat(key, value) as combined from src where combined like ’11%’; The following tweak of syntax works fine although a bit ugly. SELECT key, value, concat(key, value) as combined

do MIMA checking before all test cases start?

2014-09-24 Thread Nan Zhu
Hi, all It seems that, currently, Jenkins makes MIMA checking after all test cases have finished, IIRC, during the first months we introduced MIMA, we do the MIMA checking before running test cases What’s the motivation to adjust this behaviour? In my opinion, if you have some binary

Re: Spark SQL use of alias in where clause

2014-09-24 Thread Yanbo Liang
Maybe it's the way SQL works. The select part is executed after the where filter is applied, so you cannot use alias declared in select part in where clause. Hive and Oracle behavior the same as Spark SQL. 2014-09-25 8:58 GMT+08:00 Du Li l...@yahoo-inc.com.invalid: Hi, The following query

Re: do MIMA checking before all test cases start?

2014-09-24 Thread Patrick Wendell
Have you considered running the mima checks locally? We prefer people not use Jenkins for very frequent checks since it takes resources away from other people trying to run tests. On Wed, Sep 24, 2014 at 6:44 PM, Nan Zhu zhunanmcg...@gmail.com wrote: Hi, all It seems that, currently, Jenkins

Re: do MIMA checking before all test cases start?

2014-09-24 Thread Nan Zhu
yeah, I tried that, but there is always an issue when I ran dev/mima, it always gives me some binary compatibility error on Java API part…. so I have to wait for Jenkins’ result when fixing MIMA issues -- Nan Zhu On Thursday, September 25, 2014 at 12:04 AM, Patrick Wendell wrote: Have