[GitHub] [spark] dtenedor commented on a diff in pull request #41191: [SPARK-43529][SQL] Support general constant expressions as OPTIONS values in the parser
dtenedor commented on code in PR #41191:
URL: https://github.com/apache/spark/pull/41191#discussion_r1199511465
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##
@@ -3187,7 +3189,42 @@ class AstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
ctx: PropertyListContext): Map[String, String] = withOrigin(ctx) {
val properties = ctx.property.asScala.map { property =>
val key = visitPropertyKey(property.key)
- val value = visitPropertyValue(property.value)
+ // If the expression provided for the option value is a literal, use the
string
+ // representation of the contents of the literal value provided here.
Otherwise, if this
+ // expression is constant but non-literal, evaluate the expression and
use its value instead.
+ val value = if (property.value == null) {
+null
+ } else {
+val parsed = expression(property.value)
+def error: Throwable = {
+ new ParseException(
+errorClass = "INVALID_SQL_SYNTAX",
+messageParameters = Map(
+ "inputString" ->
+s"option or property key $key is invalid; only constant
expressions are supported"),
+ctx)
+}
+val plan = try {
+ val analyzer: Analyzer = ResolveDefaultColumns.DefaultColumnAnalyzer
Review Comment:
Good point. I updated this to avoid using the analyzer, and just check if
the expression is foldable and `eval` it instead. This supports all types of
literals and some basic operations now. Function calls are not supported yet; I
can support them later, but that change is more invasive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dtenedor commented on a diff in pull request #41191: [SPARK-43529][SQL] Support general constant expressions as OPTIONS values in the parser
dtenedor commented on code in PR #41191:
URL: https://github.com/apache/spark/pull/41191#discussion_r1197029651
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##
@@ -3187,7 +3189,37 @@ class AstBuilder extends
SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
ctx: PropertyListContext): Map[String, String] = withOrigin(ctx) {
val properties = ctx.property.asScala.map { property =>
val key = visitPropertyKey(property.key)
- val value = visitPropertyValue(property.value)
+ // If the expression provided for the option value is a literal, use the
string
+ // representation of the contents of the literal value provided here.
Otherwise, if this
+ // expression is constant but non-literal, evaluate the expression and
use its value instead.
+ val value = if (property.value == null) {
+null
+ } else {
+val parsed = expression(property.value)
+def error: Throwable = {
+ new ParseException(
+errorClass = "INVALID_SQL_SYNTAX",
+messageParameters = Map(
+ "inputString" ->
+s"option or property key $key is invalid; only constant
expressions are supported"),
+ctx)
+}
+val plan = try {
+ val analyzer: Analyzer = ResolveDefaultColumns.DefaultColumnAnalyzer
+ val analyzed = analyzer.execute(Project(Seq(Alias(parsed, "col")()),
OneRowRelation()))
+ analyzer.checkAnalysis(analyzed)
+ ConstantFolding(analyzed)
+} catch {
+ case _: AnalysisException => throw error
+}
+val result: Expression = plan.collectFirst {
+ case Project(Seq(a: Alias), OneRowRelation()) => a.child
+}.get
+result match {
+ case expr if expr.foldable => expr.eval().toString
Review Comment:
Update: I found that using `Cast` does not work in all cases, such as
timestamps since we do not currently support casting timestamps to strings. I
switched this back to use `.toString` and added a comment explaining this. It
seems to work well for different types of expressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
