Hi Kant,
You can try "explaining" the sql query.
spark.sql(sqlText).explain(true); //the parameter true is to get more
verbose query plan and it is optional.
This is the safest way to validate sql without actually executing/creating
a df/view in spark. It validates syntax as well as schema of tables/views
used.
If there is an issue with your SQL syntax then the method throws below
exception that you can catch
org.apache.spark.sql.catalyst.parser.ParseException
Hope this helps!
Akshay Bhardwaj
+91-97111-33849
On Fri, Mar 1, 2019 at 10:23 PM kant kodali <kanth...@gmail.com> wrote:
> Hi All,
>
> Is there a way to validate the syntax of raw spark SQL query?
>
> for example, I would like to know if there is any isValid API call spark
> provides?
>
> val query = "select * from table"if(isValid(query)) {
> sparkSession.sql(query) } else {
> log.error("Invalid Syntax")}
>
> I tried the following
>
> val query = "select * morf table" // Invalid queryval parser =
> spark.sessionState.sqlParsertry{
> parser.parseExpression(query)} catch (ParseException ex) {
> throw new Exception(ex); //Exception not getting thrown}Dataset<>Row df =
> sparkSession.sql(query) // Exception gets thrown here
> df.writeStream.format("console").start()
>
> Question: parser.parseExpression is not catching the invalid syntax
> before I hit the sparkSession.sql. Other words it is not being helpful in
> the above code. any reason? My whole goal is to catch syntax errors before
> I pass it on to sparkSession.sql
>
>
>
