[
https://issues.apache.org/jira/browse/SPARK-20299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacek Laskowski updated SPARK-20299:
------------------------------------
Description:
When creating a Dataset from a tuple with {{null}} and a string, NPE is
reported. When either is removed, it works fine.
{code}
scala> Seq((1, null.asInstanceOf[Int]), (2, 1)).toDS
res43: org.apache.spark.sql.Dataset[(Int, Int)] = [_1: int, _2: int]
scala> Seq(("1", null.asInstanceOf[Int]), ("2", 1)).toDS
java.lang.RuntimeException: Error while encoding: java.lang.NullPointerException
staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType,
fromString, assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level
Product input object), - root class: "scala.Tuple2")._1, true) AS _1#474
assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level Product
input object), - root class: "scala.Tuple2")._2 AS _2#475
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290)
at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:454)
at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:377)
at
org.apache.spark.sql.SQLImplicits.localSeqToDatasetHolder(SQLImplicits.scala:246)
... 48 elided
Caused by: java.lang.NullPointerException
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_1$(Unknown
Source)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
Source)
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:287)
... 58 more
{code}
was:
When creating a Dataset from a tuple with {{null}} and a string, NPE is
reported. When either is removed, it works fine.
{code}
scala> Seq((1, null.asInstanceOf[Int]), (2, 1)).toDS
res43: org.apache.spark.sql.Dataset[(Int, Int)] = [_1: int, _2: int]
scala> Seq(("1", null.asInstanceOf[Int]), ("2", 1)).toDS
scala> Seq(("1", null.asInstanceOf[Int]), ("2", 1)).toDS
java.lang.RuntimeException: Error while encoding: java.lang.NullPointerException
staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType,
fromString, assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level
Product input object), - root class: "scala.Tuple2")._1, true) AS _1#474
assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level Product
input object), - root class: "scala.Tuple2")._2 AS _2#475
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290)
at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
at org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:454)
at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:377)
at
org.apache.spark.sql.SQLImplicits.localSeqToDatasetHolder(SQLImplicits.scala:246)
... 48 elided
Caused by: java.lang.NullPointerException
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_1$(Unknown
Source)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
Source)
at
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:287)
... 58 more
{code}
> NullPointerException when null and string are in a tuple while encoding
> Dataset
> -------------------------------------------------------------------------------
>
> Key: SPARK-20299
> URL: https://issues.apache.org/jira/browse/SPARK-20299
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Jacek Laskowski
> Priority: Minor
>
> When creating a Dataset from a tuple with {{null}} and a string, NPE is
> reported. When either is removed, it works fine.
> {code}
> scala> Seq((1, null.asInstanceOf[Int]), (2, 1)).toDS
> res43: org.apache.spark.sql.Dataset[(Int, Int)] = [_1: int, _2: int]
> scala> Seq(("1", null.asInstanceOf[Int]), ("2", 1)).toDS
> java.lang.RuntimeException: Error while encoding:
> java.lang.NullPointerException
> staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType,
> fromString, assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top
> level Product input object), - root class: "scala.Tuple2")._1, true) AS _1#474
> assertnotnull(assertnotnull(input[0, scala.Tuple2, true], top level Product
> input object), - root class: "scala.Tuple2")._2 AS _2#475
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:290)
> at
> org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
> at
> org.apache.spark.sql.SparkSession$$anonfun$2.apply(SparkSession.scala:454)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.List.foreach(List.scala:381)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.immutable.List.map(List.scala:285)
> at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:454)
> at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:377)
> at
> org.apache.spark.sql.SQLImplicits.localSeqToDatasetHolder(SQLImplicits.scala:246)
> ... 48 elided
> Caused by: java.lang.NullPointerException
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_1$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:287)
> ... 58 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
