jackylee-ch commented on code in PR #7476:
URL: https://github.com/apache/incubator-gluten/pull/7476#discussion_r1810555859
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala:
##########
@@ -706,50 +706,6 @@ class VeloxSparkPlanExecApi extends SparkPlanExecApi {
VeloxGetStructFieldTransformer(substraitExprName, childTransformer,
ordinal, original)
}
- /**
- * To align with spark in casting string type input to other types, add trim
node for trimming
- * space or whitespace. See spark's Cast.scala.
- */
- override def genCastWithNewChild(c: Cast): Cast = {
- // scalastyle:off nonascii
- // Common whitespace to be trimmed, including: ' ', '\n', '\r', '\f', etc.
- val trimWhitespaceStr = " \t\n\u000B\u000C\u000D\u001C\u001D\u001E\u001F"
- // Space separator.
- val trimSpaceSepStr = "\u1680\u2008\u2009\u200A\u205F\u3000" +
- ('\u2000' to '\u2006').toList.mkString
- // Line separator.
- val trimLineSepStr = "\u2028"
Review Comment:
After looking at the velox code, the conversion from string to int/long is
basically similar between [vanilla
Spark](https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java#L200)
and
[velox](https://github.com/facebookincubator/velox/blob/main/velox/functions/lib/string/StringImpl.h#L354).
But for the conversion from string to float/double, [vanilla
Spark](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L390)
uses `String.trim` to remove characters smaller than spaces, which is not same
as velox, so we still need this extra trim.
A new config, which is default false, is good to me. Thanks @Henry2SS for
working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
