[
https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16613157#comment-16613157
]
Liang-Chi Hsieh edited comment on SPARK-25378 at 9/13/18 8:33 AM:
------------------------------------------------------------------
I think a quick fix is to use general `get` method for just `StringType` in
`InternalRow.getAccessor`. This can allow the backward-compatible behavior for
`StringType` when calling `toArray`.
And we may consider to correct it back to `getUTF8String` by 3.0.
WDYT? [~mengxr], [~cloud_fan], [~hyukjin.kwon].
was (Author: viirya):
I think a quick fix is to use general `get` method for just `StringType` in
`InternalRow.getAccessor`. This can allow the backward-compatible behavior for
`StringType` when calling `toArray`.
And we may consider to correct to `getUTF8String` by 3.0.
WDYT? [~mengxr], [~cloud_fan], [~hyukjin.kwon].
> ArrayData.toArray(StringType) assume UTF8String in 2.4
> ------------------------------------------------------
>
> Key: SPARK-25378
> URL: https://issues.apache.org/jira/browse/SPARK-25378
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Xiangrui Meng
> Priority: Critical
>
> The following code works in 2.3.1 but failed in 2.4.0-SNAPSHOT:
> {code}
> import org.apache.spark.sql.catalyst.util._
> import org.apache.spark.sql.types.StringType
> ArrayData.toArrayData(Array("a", "b")).toArray[String](StringType)
> res0: Array[String] = Array(a, b)
> {code}
> In 2.4.0-SNAPSHOT, the error is
> {code}java.lang.ClassCastException: java.lang.String cannot be cast to
> org.apache.spark.unsafe.types.UTF8String
> at
> org.apache.spark.sql.catalyst.util.GenericArrayData.getUTF8String(GenericArrayData.scala:75)
> at
> org.apache.spark.sql.catalyst.InternalRow$$anonfun$getAccessor$8.apply(InternalRow.scala:136)
> at
> org.apache.spark.sql.catalyst.InternalRow$$anonfun$getAccessor$8.apply(InternalRow.scala:136)
> at org.apache.spark.sql.catalyst.util.ArrayData.toArray(ArrayData.scala:178)
> ... 51 elided
> {code}
> cc: [~cloud_fan] [~yogeshg]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
