kazuyukitanimura commented on code in PR #1385:
URL: https://github.com/apache/datafusion-comet/pull/1385#discussion_r1956612664
##########
docs/source/user-guide/compatibility.md:
##########
@@ -73,76 +73,77 @@ Spark.
The following cast operations are generally compatible with Spark except for
the differences noted here.
| From Type | To Type | Notes |
-|-|-|-|
-| boolean | byte | |
-| boolean | short | |
+|-|---------|-|
Review Comment:
nit: Just checking whether this change is due to the changes in the
producing method
Should be automatically created by `make release`
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1126,27 +1129,36 @@ class CometCastSuite extends CometTestBase with
AdaptiveSparkPlanHelper {
val cometMessage =
if (cometException.getCause != null)
cometException.getCause.getMessage
else cometException.getMessage
- if (CometSparkSessionExtensions.isSpark40Plus) {
- // for Spark 4 we expect to sparkException carries the message
+ // for comet decimal conversion throws ArrowError(string) from
arrow - across spark versions the message dont match.
+ if (sparkMessage.contains("cannot be represented as")) {
assert(
- sparkException.getMessage
- .replace(".WITH_SUGGESTION] ", "]")
- .startsWith(cometMessage))
- } else if (CometSparkSessionExtensions.isSpark34Plus) {
- // for Spark 3.4 we expect to reproduce the error message
exactly
- assert(cometMessage == sparkMessage)
+ cometMessage.contains("cannot be represented as") ||
cometMessage.contains(
+ "too large to store"))
} else {
Review Comment:
There are message modifications below per spark version
Would you mind update them instead of creating another if branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
