[
https://issues.apache.org/jira/browse/SPARK-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin closed SPARK-9332.
------------------------------
Resolution: Fixed
> CatalystTypeConverters.toScala does not work on UnsafeRows
> ----------------------------------------------------------
>
> Key: SPARK-9332
> URL: https://issues.apache.org/jira/browse/SPARK-9332
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Josh Rosen
> Assignee: Reynold Xin
> Priority: Critical
> Fix For: 1.5.0
>
>
> When CatalystTypeConverters.toScala is applied to an UnsafeRow it may lead to
> an UnsupportedOperationException when it tries to use a generic getter to
> retrieve primitive columns.
> For example:
> {code}
> == Parsed Logical Plan ==
> Sort [c183#208 ASC,c183#208 ASC], true
> Project [c183#208,c184#209]
> LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at
> DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
> == Analyzed Logical Plan ==
> c183: binary, c184: int
> Sort [c183#208 ASC,c183#208 ASC], true
> Project [c183#208,c184#209]
> LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at
> DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
> == Optimized Logical Plan ==
> Sort [c183#208 ASC,c183#208 ASC], true
> LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at
> DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
> == Physical Plan ==
> UnsafeExternalSort [c183#208 ASC,c183#208 ASC], true, 0
> Exchange rangepartitioning(c183#208 ASC,c183#208 ASC)
> PhysicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at
> DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
> {code}
> {code}
> java.lang.UnsupportedOperationException
> at
> org.apache.spark.sql.catalyst.expressions.UnsafeRow.get(UnsafeRow.java:223)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$IdentityConverter$.toScalaImpl(CatalystTypeConverters.scala:143)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:118)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:266)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:233)
> at
> org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToScalaConverter$2.apply(CatalystTypeConverters.scala:386)
> at
> org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480)
> at
> org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
