[GitHub] [spark] dtenedor commented on a diff in pull request #41191: [SPARK-43529][SQL] Support general constant expressions as OPTIONS values in the parser

2023-05-19 Thread via GitHub


dtenedor commented on code in PR #41191:
URL: https://github.com/apache/spark/pull/41191#discussion_r1199511465


##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##
@@ -3187,7 +3189,42 @@ class AstBuilder extends 
SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
   ctx: PropertyListContext): Map[String, String] = withOrigin(ctx) {
 val properties = ctx.property.asScala.map { property =>
   val key = visitPropertyKey(property.key)
-  val value = visitPropertyValue(property.value)
+  // If the expression provided for the option value is a literal, use the 
string
+  // representation of the contents of the literal value provided here. 
Otherwise, if this
+  // expression is constant but non-literal, evaluate the expression and 
use its value instead.
+  val value = if (property.value == null) {
+null
+  } else {
+val parsed = expression(property.value)
+def error: Throwable = {
+  new ParseException(
+errorClass = "INVALID_SQL_SYNTAX",
+messageParameters = Map(
+  "inputString" ->
+s"option or property key $key is invalid; only constant 
expressions are supported"),
+ctx)
+}
+val plan = try {
+  val analyzer: Analyzer = ResolveDefaultColumns.DefaultColumnAnalyzer

Review Comment:
   Good point. I updated this to avoid using the analyzer, and just check if 
the expression is foldable and `eval` it instead. This supports all types of 
literals and some basic operations now. Function calls are not supported yet; I 
can support them later, but that change is more invasive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dtenedor commented on a diff in pull request #41191: [SPARK-43529][SQL] Support general constant expressions as OPTIONS values in the parser

2023-05-17 Thread via GitHub


dtenedor commented on code in PR #41191:
URL: https://github.com/apache/spark/pull/41191#discussion_r1197029651


##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##
@@ -3187,7 +3189,37 @@ class AstBuilder extends 
SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
   ctx: PropertyListContext): Map[String, String] = withOrigin(ctx) {
 val properties = ctx.property.asScala.map { property =>
   val key = visitPropertyKey(property.key)
-  val value = visitPropertyValue(property.value)
+  // If the expression provided for the option value is a literal, use the 
string
+  // representation of the contents of the literal value provided here. 
Otherwise, if this
+  // expression is constant but non-literal, evaluate the expression and 
use its value instead.
+  val value = if (property.value == null) {
+null
+  } else {
+val parsed = expression(property.value)
+def error: Throwable = {
+  new ParseException(
+errorClass = "INVALID_SQL_SYNTAX",
+messageParameters = Map(
+  "inputString" ->
+s"option or property key $key is invalid; only constant 
expressions are supported"),
+ctx)
+}
+val plan = try {
+  val analyzer: Analyzer = ResolveDefaultColumns.DefaultColumnAnalyzer
+  val analyzed = analyzer.execute(Project(Seq(Alias(parsed, "col")()), 
OneRowRelation()))
+  analyzer.checkAnalysis(analyzed)
+  ConstantFolding(analyzed)
+} catch {
+  case _: AnalysisException => throw error
+}
+val result: Expression = plan.collectFirst {
+  case Project(Seq(a: Alias), OneRowRelation()) => a.child
+}.get
+result match {
+  case expr if expr.foldable => expr.eval().toString

Review Comment:
   Update: I found that using `Cast` does not work in all cases, such as 
timestamps since we do not currently support casting timestamps to strings. I 
switched this back to use `.toString` and added a comment explaining this. It 
seems to work well for different types of expressions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org