RE: [SparkSQL] pre-check syntex before running spark job?
Hi, you can use spark sql Antlr grammer for pre check you syntax. https://github.com/apache/spark/blob/acf71c63cdde8dced8d108260cdd35e1cc992248/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 From: Linyuxin [mailto:linyu...@huawei.com] Sent: Wednesday, February 22, 2017 7:34 AM To: Irving Duran ; Yong Zhang Cc: Jacek Laskowski ; user Subject: 答复: [SparkSQL] pre-check syntex before running spark job? Actually，I want a standalone jar as I can check the syntax without spark execution environment 发件人: Irving Duran [mailto:irving.du...@gmail.com] 发送时间: 2017年2月21日 23:29 收件人: Yong Zhang mailto:java8...@hotmail.com>> 抄送: Jacek Laskowski mailto:ja...@japila.pl>>; Linyuxin mailto:linyu...@huawei.com>>; user mailto:user@spark.apache.org>> 主题: Re: [SparkSQL] pre-check syntex before running spark job? You can also run it on REPL and test to see if you are getting the expected result. Thank You, Irving Duran On Tue, Feb 21, 2017 at 8:01 AM, Yong Zhang mailto:java8...@hotmail.com>> wrote: You can always use explain method to validate your DF or SQL, before any action. Yong From: Jacek Laskowski mailto:ja...@japila.pl>> Sent: Tuesday, February 21, 2017 4:34 AM To: Linyuxin Cc: user Subject: Re: [SparkSQL] pre-check syntex before running spark job? Hi, Never heard about such a tool before. You could use Antlr to parse SQLs (just as Spark SQL does while parsing queries). I think it's a one-hour project. Jacek On 21 Feb 2017 4:44 a.m., "Linyuxin" mailto:linyu...@huawei.com>> wrote: Hi All, Is there any tool/api to check the sql syntax without running spark job actually? Like the siddhiQL on storm here: SiddhiManagerService. validateExecutionPlan https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java it can validate the syntax before running the sql on storm this is very useful for exposing sql string as a DSL of the platform. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
Re: [SparkSQL] pre-check syntex before running spark job?
You can also run it on REPL and test to see if you are getting the expected result. Thank You, Irving Duran On Tue, Feb 21, 2017 at 8:01 AM, Yong Zhang wrote: > You can always use explain method to validate your DF or SQL, before any > action. > > > Yong > > > -- > *From:* Jacek Laskowski > *Sent:* Tuesday, February 21, 2017 4:34 AM > *To:* Linyuxin > *Cc:* user > *Subject:* Re: [SparkSQL] pre-check syntex before running spark job? > > Hi, > > Never heard about such a tool before. You could use Antlr to parse SQLs > (just as Spark SQL does while parsing queries). I think it's a one-hour > project. > > Jacek > > On 21 Feb 2017 4:44 a.m., "Linyuxin" wrote: > > Hi All, > Is there any tool/api to check the sql syntax without running spark job > actually? > > Like the siddhiQL on storm here: > SiddhiManagerService. validateExecutionPlan > https://github.com/wso2/siddhi/blob/master/modules/siddhi- > core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java > it can validate the syntax before running the sql on storm > > this is very useful for exposing sql string as a DSL of the platform. > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > >
Re: [SparkSQL] pre-check syntex before running spark job?
You can always use explain method to validate your DF or SQL, before any action. Yong From: Jacek Laskowski Sent: Tuesday, February 21, 2017 4:34 AM To: Linyuxin Cc: user Subject: Re: [SparkSQL] pre-check syntex before running spark job? Hi, Never heard about such a tool before. You could use Antlr to parse SQLs (just as Spark SQL does while parsing queries). I think it's a one-hour project. Jacek On 21 Feb 2017 4:44 a.m., "Linyuxin" mailto:linyu...@huawei.com>> wrote: Hi All, Is there any tool/api to check the sql syntax without running spark job actually? Like the siddhiQL on storm here: SiddhiManagerService. validateExecutionPlan https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java it can validate the syntax before running the sql on storm this is very useful for exposing sql string as a DSL of the platform. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>
Re: [SparkSQL] pre-check syntex before running spark job?
Hi, Never heard about such a tool before. You could use Antlr to parse SQLs (just as Spark SQL does while parsing queries). I think it's a one-hour project. Jacek On 21 Feb 2017 4:44 a.m., "Linyuxin" wrote: Hi All, Is there any tool/api to check the sql syntax without running spark job actually? Like the siddhiQL on storm here: SiddhiManagerService. validateExecutionPlan https://github.com/wso2/siddhi/blob/master/modules/ siddhi-core/src/main/java/org/wso2/siddhi/core/SiddhiManagerService.java it can validate the syntax before running the sql on storm this is very useful for exposing sql string as a DSL of the platform. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
