Re: using the Hive SQL parser in Spark

2016-05-19 Thread Reynold Xin
I want to give an update since there have been some new development since my last email. We did import Hive's parser into Spark in Feb, but then in April that was replaced by another ANTLR4 based parser. So the net effect is that this didn't happen (no release was made with the Hive parser). Than

Re: using the Hive SQL parser in Spark

2015-12-18 Thread Reynold Xin
(Please use reply-all so I see the replies) Responses inline. On Fri, Dec 18, 2015 at 1:17 PM, Yin Huai wrote: > Let me add Reynold to the thread. > > On Fri, Dec 18, 2015 at 12:36 PM, Gopal Vijayaraghavan > wrote: > >> >> >We have looked into various options, and it looks like the best optio

Re: using the Hive SQL parser in Spark

2015-12-18 Thread Yin Huai
Let me add Reynold to the thread. On Fri, Dec 18, 2015 at 12:36 PM, Gopal Vijayaraghavan wrote: > > >We have looked into various options, and it looks like the best option is > >to copy the ANTLR grammar file from Hive into Spark. Because the grammar > >file is tightly coupled with Hive's semant

Re: using the Hive SQL parser in Spark

2015-12-18 Thread Gopal Vijayaraghavan
>We have looked into various options, and it looks like the best option is >to copy the ANTLR grammar file from Hive into Spark. Because the grammar >file is tightly coupled with Hive's semantic analysis, we need to refactor >some code to use them so it will end up becoming the .g file plus some >

using the Hive SQL parser in Spark

2015-12-18 Thread Reynold Xin
Hi Hive devs, I would like to share with you Spark's plan w.r.t. SQL parser going forward. As you may (or may not) know, Spark SQL has had two parsers so far: - a very simple one based on Scala's parser combinator; and - one that depends on Hive's The Scala parser combinator one was written quic