himadripal commented on code in PR #1385:
URL: https://github.com/apache/datafusion-comet/pull/1385#discussion_r1968652251
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1126,27 +1129,33 @@ class CometCastSuite extends CometTestBase with
AdaptiveSparkPlanHelper {
val cometMessage =
if (cometException.getCause != null)
cometException.getCause.getMessage
else cometException.getMessage
- if (CometSparkSessionExtensions.isSpark40Plus) {
- // for Spark 4 we expect to sparkException carries the message
- assert(
- sparkException.getMessage
- .replace(".WITH_SUGGESTION] ", "]")
- .startsWith(cometMessage))
- } else if (CometSparkSessionExtensions.isSpark34Plus) {
- // for Spark 3.4 we expect to reproduce the error message
exactly
- assert(cometMessage == sparkMessage)
+ // for comet decimal conversion throws ArrowError(string) from
arrow - across spark versions the message dont match.
+ if (sparkMessage.contains("cannot be represented as")) {
+ cometMessage.contains("cannot be represented as") ||
cometMessage.contains(
+ "too large to store")
} else {
Review Comment:
> Another way to remove the if block is to convert the error message to make
it similar to Spark
This is the place that we are defining Spark error messages
https://github.com/apache/datafusion-comet/blob/main/native/spark-expr/src/error.rs#L36
You can check how these are used
This one I checked - problem here is from native execution, we get back
`Arrow(ArrowError)` which only has a string, precision and scale information is
not present. Also to construct an error message from a string - we need to
check specific string in the message.
We can try to change the ArrowError to have parameter but that will be a big
change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
