[ https://issues.apache.org/jira/browse/SPARK-25378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16613157#comment-16613157 ]
Liang-Chi Hsieh edited comment on SPARK-25378 at 9/13/18 7:59 AM: ------------------------------------------------------------------ I think a quick fix is to use general `get` method for just `StringType` in `InternalRow.getAccessor`. This can allow the backward-compatible behavior for `StringType` when calling `toArray`. And we may consider to correct to `getUTF8String` by 3.0. WDYT? [~mengxr], [~cloud_fan], [~hyukjin.kwon]. was (Author: viirya): I think a quick fix is to use general `get` method for just `StringType` in `InternalRow.getAccessor`. This can allow the backward-compatible behavior for `StringType` when calling `toArray`. And we may consider to correct to `getUTF8String` by 3.0. WDYT? [~mengxr][~cloud_fan][~hyukjin.kwon] > ArrayData.toArray(StringType) assume UTF8String in 2.4 > ------------------------------------------------------ > > Key: SPARK-25378 > URL: https://issues.apache.org/jira/browse/SPARK-25378 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.0 > Reporter: Xiangrui Meng > Priority: Critical > > The following code works in 2.3.1 but failed in 2.4.0-SNAPSHOT: > {code} > import org.apache.spark.sql.catalyst.util._ > import org.apache.spark.sql.types.StringType > ArrayData.toArrayData(Array("a", "b")).toArray[String](StringType) > res0: Array[String] = Array(a, b) > {code} > In 2.4.0-SNAPSHOT, the error is > {code}java.lang.ClassCastException: java.lang.String cannot be cast to > org.apache.spark.unsafe.types.UTF8String > at > org.apache.spark.sql.catalyst.util.GenericArrayData.getUTF8String(GenericArrayData.scala:75) > at > org.apache.spark.sql.catalyst.InternalRow$$anonfun$getAccessor$8.apply(InternalRow.scala:136) > at > org.apache.spark.sql.catalyst.InternalRow$$anonfun$getAccessor$8.apply(InternalRow.scala:136) > at org.apache.spark.sql.catalyst.util.ArrayData.toArray(ArrayData.scala:178) > ... 51 elided > {code} > cc: [~cloud_fan] [~yogeshg] -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org