bersprockets opened a new pull request, #38727:
URL: https://github.com/apache/spark/pull/38727
### What changes were proposed in this pull request?
Change `TryToBinary`'s constructor to test whether the format expression is
foldable before creating the replacement expression.
### Why are the changes needed?
`try_to_binary`, when called with non-foldable format, throws an unhelpful
error message:
```
spark-sql> SELECT try_to_binary(col1, col2) from values ('abc', 'utf-8') as
data(col1, col2);
[INTERNAL_ERROR] The Spark SQL phase analysis failed with an internal error.
You hit a bug in Spark or the Spark plugins you use. Please, report this bug to
the corresponding communities or vendors, and provide the full stack trace.
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase
analysis failed with an internal error. You hit a bug in Spark or the Spark
plugins you use. Please, report this bug to the corresponding communities or
vendors, and provide the full stack trace.
at
org.apache.spark.SparkException$.internalError(SparkException.scala:88)
...
Caused by: java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:208)
at
org.apache.spark.sql.catalyst.expressions.ToBinary.$anonfun$replacement$1(stringExpressions.scala:2597)
at scala.Option.map(Option.scala:230)
at
org.apache.spark.sql.catalyst.expressions.ToBinary.replacement$lzycompute(stringExpressions.scala:2596)
at
org.apache.spark.sql.catalyst.expressions.ToBinary.replacement(stringExpressions.scala:2596)
```
`TryToBinary` creates an instance of `ToBinary` as a replacement expression.
However, `ToBinary`'s default constructor does not check whether the format
expression is foldable, so the code that creates `ToBinary`'s own replacement
expression fails with an assertion failure.
`ToBinary` is not typically instantiated using the default constructor. The
function registry uses the second auxiliary constructor, which does check
whether format is foldable. The code that creates `ToBinary`'s replacement
expression assumes this second auxiliary constructor is always used.
`TryToBinary` cannot use `ToBinary`'s second auxiliary constructor because
it needs to set `nullOnInvalidFormat` to `true`.
After this PR, the above example will throw a more useful error message:
```
spark-sql> SELECT try_to_binary(col1, col2) from values ('abc', 'utf-8') as
data(col1, col2);
The 'format' parameter of function 'try_to_binary' needs to be a string
literal.; line 1 pos 7
spark-sql>
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
