[ https://issues.apache.org/jira/browse/SPARK-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14643343#comment-14643343 ]
Reynold Xin commented on SPARK-9332: ------------------------------------ Fixed by SPARK-9368. > CatalystTypeConverters.toScala does not work on UnsafeRows > ---------------------------------------------------------- > > Key: SPARK-9332 > URL: https://issues.apache.org/jira/browse/SPARK-9332 > Project: Spark > Issue Type: Sub-task > Components: SQL > Reporter: Josh Rosen > Assignee: Reynold Xin > Priority: Critical > Fix For: 1.5.0 > > > When CatalystTypeConverters.toScala is applied to an UnsafeRow it may lead to > an UnsupportedOperationException when it tries to use a generic getter to > retrieve primitive columns. > For example: > {code} > == Parsed Logical Plan == > Sort [c183#208 ASC,c183#208 ASC], true > Project [c183#208,c184#209] > LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at > DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2 > == Analyzed Logical Plan == > c183: binary, c184: int > Sort [c183#208 ASC,c183#208 ASC], true > Project [c183#208,c184#209] > LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at > DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2 > == Optimized Logical Plan == > Sort [c183#208 ASC,c183#208 ASC], true > LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at > DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2 > == Physical Plan == > UnsafeExternalSort [c183#208 ASC,c183#208 ASC], true, 0 > Exchange rangepartitioning(c183#208 ASC,c183#208 ASC) > PhysicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at > DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2 > {code} > {code} > java.lang.UnsupportedOperationException > at > org.apache.spark.sql.catalyst.expressions.UnsafeRow.get(UnsafeRow.java:223) > at > org.apache.spark.sql.catalyst.CatalystTypeConverters$IdentityConverter$.toScalaImpl(CatalystTypeConverters.scala:143) > at > org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:118) > at > org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:266) > at > org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:233) > at > org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToScalaConverter$2.apply(CatalystTypeConverters.scala:386) > at > org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480) > at > org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org