Josh Rosen created SPARK-9332:
---------------------------------
Summary: CatalystTypeConverters.toScala does not work on UnsafeRows
Key: SPARK-9332
URL: https://issues.apache.org/jira/browse/SPARK-9332
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: Josh Rosen
Priority: Critical
When CatalystTypeConverters.toScala is applied to an UnsafeRow it may lead to
an UnsupportedOperationException when it tries to use a generic getter to
retrieve primitive columns.
For example:
{code}
== Parsed Logical Plan ==
Sort [c183#208 ASC,c183#208 ASC], true
Project [c183#208,c184#209]
LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at
DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
== Analyzed Logical Plan ==
c183: binary, c184: int
Sort [c183#208 ASC,c183#208 ASC], true
Project [c183#208,c184#209]
LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at
DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
== Optimized Logical Plan ==
Sort [c183#208 ASC,c183#208 ASC], true
LogicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at DataFrameFuzzingSuite
at NativeConstructorAccessorImpl.java:-2
== Physical Plan ==
UnsafeExternalSort [c183#208 ASC,c183#208 ASC], true, 0
Exchange rangepartitioning(c183#208 ASC,c183#208 ASC)
PhysicalRDD [c183#208,c184#209], MapPartitionsRDD[367] at
DataFrameFuzzingSuite at NativeConstructorAccessorImpl.java:-2
{code}
{code}
java.lang.UnsupportedOperationException
at
org.apache.spark.sql.catalyst.expressions.UnsafeRow.get(UnsafeRow.java:223)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$IdentityConverter$.toScalaImpl(CatalystTypeConverters.scala:143)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:118)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:266)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:233)
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToScalaConverter$2.apply(CatalystTypeConverters.scala:386)
at
org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480)
at
org.apache.spark.sql.DataFrame$$anonfun$rdd$1$$anonfun$apply$9.apply(DataFrame.scala:1480)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
