[
https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634335#comment-16634335
]
Xiangrui Meng commented on SPARK-25378:
---------------------------------------
I don't think I'm the right person to decide here because I know little about
how UTF8String is being used in Spark SQL. As a user, I do want to use
spark-tensorflow-connector w/ the upcoming Spark 2.4 release.
I already made the change in TF connector to use ObjectType:
https://github.com/tensorflow/ecosystem/pull/100. But they need to wait for TF
1.12 release, which might come out in the second half of Oct. If we won't make
the final 2.4 release by then, maybe we don't have to fix 2.4 branch. The risk
is other data sources might have similar usage that will break, which we don't
really know.
> ArrayData.toArray(StringType) assume UTF8String in 2.4
> ------------------------------------------------------
>
> Key: SPARK-25378
> URL: https://issues.apache.org/jira/browse/SPARK-25378
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Xiangrui Meng
> Priority: Critical
>
> The following code works in 2.3.1 but failed in 2.4.0-SNAPSHOT:
> {code}
> import org.apache.spark.sql.catalyst.util._
> import org.apache.spark.sql.types.StringType
> ArrayData.toArrayData(Array("a", "b")).toArray[String](StringType)
> res0: Array[String] = Array(a, b)
> {code}
> In 2.4.0-SNAPSHOT, the error is
> {code}java.lang.ClassCastException: java.lang.String cannot be cast to
> org.apache.spark.unsafe.types.UTF8String
> at
> org.apache.spark.sql.catalyst.util.GenericArrayData.getUTF8String(GenericArrayData.scala:75)
> at
> org.apache.spark.sql.catalyst.InternalRow$$anonfun$getAccessor$8.apply(InternalRow.scala:136)
> at
> org.apache.spark.sql.catalyst.InternalRow$$anonfun$getAccessor$8.apply(InternalRow.scala:136)
> at org.apache.spark.sql.catalyst.util.ArrayData.toArray(ArrayData.scala:178)
> ... 51 elided
> {code}
> cc: [~cloud_fan] [~yogeshg]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
