[jira] [Commented] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133091#comment-16133091 ] Hyukjin Kwon commented on SPARK-21677: -- Issue resolved by pull request 18930 https://github.com/apache/spark/pull/18930 > json_tuple throws NullPointException when column is null as string type. > > > Key: SPARK-21677 > URL: https://issues.apache.org/jira/browse/SPARK-21677 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Hyukjin Kwon >Assignee: Jen-Ming Chung >Priority: Minor > Labels: Starter > Fix For: 2.3.0 > > > I was testing {{json_tuple}} before using this to extract values from JSONs > in my testing cluster but I found it could actually throw > {{NullPointException}} as below sometimes: > {code} > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Hyukjin'))").show() > +---+ > | c0| > +---+ > |224| > +---+ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Jackson'))").show() > ++ > | c0| > ++ > |null| > ++ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(null))").show() > ... > java.lang.NullPointerException > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:367) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:366) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames$lzycompute(jsonExpressions.scala:366) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames(jsonExpressions.scala:365) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields$lzycompute(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.org$apache$spark$sql$catalyst$expressions$JsonTuple$$parseRow(jsonExpressions.scala:417) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:401) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:400) > at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2559) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.eval(jsonExpressions.scala:400) > {code} > It sounds we should show explicit error messages or return {{NULL}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121740#comment-16121740 ] Liang-Chi Hsieh commented on SPARK-21677: - As a given field name {{null}} can't be matched with any field names in json, we just output {{null}} as its column value. I think it's reasonable. > json_tuple throws NullPointException when column is null as string type. > > > Key: SPARK-21677 > URL: https://issues.apache.org/jira/browse/SPARK-21677 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Hyukjin Kwon >Priority: Minor > Labels: Starter > > I was testing {{json_tuple}} before using this to extract values from JSONs > in my testing cluster but I found it could actually throw > {{NullPointException}} as below sometimes: > {code} > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Hyukjin'))").show() > +---+ > | c0| > +---+ > |224| > +---+ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Jackson'))").show() > ++ > | c0| > ++ > |null| > ++ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(null))").show() > ... > java.lang.NullPointerException > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:367) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:366) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames$lzycompute(jsonExpressions.scala:366) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames(jsonExpressions.scala:365) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields$lzycompute(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.org$apache$spark$sql$catalyst$expressions$JsonTuple$$parseRow(jsonExpressions.scala:417) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:401) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:400) > at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2559) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.eval(jsonExpressions.scala:400) > {code} > It sounds we should show explicit error messages or return {{NULL}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121658#comment-16121658 ] Hyukjin Kwon commented on SPARK-21677: -- [~cjm], I was thinking like {code} spark.sql("""SELECT json_tuple('{"a":1, "b":2}', cast(NULL AS STRING), 'b', cast(NULL AS STRING), 'a');""").show() ++---++---+ | c0| c1| c2| c3| ++---++---+ |null| 2|null| 1| ++---++---+ {code} I think this could be at least consistent with Hive's implementation: {code} hive> SELECT json_tuple('{"a":1, "b":2}', cast(NULL AS STRING), 'b', cast(NULL AS STRING), 'a'); ... NULL2 NULL1 {code} > json_tuple throws NullPointException when column is null as string type. > > > Key: SPARK-21677 > URL: https://issues.apache.org/jira/browse/SPARK-21677 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Hyukjin Kwon >Priority: Minor > Labels: Starter > > I was testing {{json_tuple}} before using this to extract values from JSONs > in my testing cluster but I found it could actually throw > {{NullPointException}} as below sometimes: > {code} > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Hyukjin'))").show() > +---+ > | c0| > +---+ > |224| > +---+ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Jackson'))").show() > ++ > | c0| > ++ > |null| > ++ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(null))").show() > ... > java.lang.NullPointerException > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:367) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:366) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames$lzycompute(jsonExpressions.scala:366) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames(jsonExpressions.scala:365) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields$lzycompute(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.org$apache$spark$sql$catalyst$expressions$JsonTuple$$parseRow(jsonExpressions.scala:417) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:401) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:400) > at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2559) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.eval(jsonExpressions.scala:400) > {code} > It sounds we should show explicit error messages or return {{NULL}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121641#comment-16121641 ] Jen-Ming Chung commented on SPARK-21677: to [~hyukjin.kwon], the return {{NULL}} you mentioned does it means all fields should be null in json_tuple, or just the non-existence field as shown in the following. Thanks! {code:language=scala|borderStyle=solid} e.g., spark.sql("""SELECT json_tuple('{"a":1, "b":2}', 'a', 'b', 'not_exising_fields')""").show() +---+---++ | c0| c1| c2| +---+---++ | 1| 2|null| +---+---++ {code} > json_tuple throws NullPointException when column is null as string type. > > > Key: SPARK-21677 > URL: https://issues.apache.org/jira/browse/SPARK-21677 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Hyukjin Kwon >Priority: Minor > Labels: Starter > > I was testing {{json_tuple}} before using this to extract values from JSONs > in my testing cluster but I found it could actually throw > {{NullPointException}} as below sometimes: > {code} > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Hyukjin'))").show() > +---+ > | c0| > +---+ > |224| > +---+ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Jackson'))").show() > ++ > | c0| > ++ > |null| > ++ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(null))").show() > ... > java.lang.NullPointerException > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:367) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:366) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames$lzycompute(jsonExpressions.scala:366) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames(jsonExpressions.scala:365) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields$lzycompute(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.org$apache$spark$sql$catalyst$expressions$JsonTuple$$parseRow(jsonExpressions.scala:417) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:401) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:400) > at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2559) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.eval(jsonExpressions.scala:400) > {code} > It sounds we should show explicit error messages or return {{NULL}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119928#comment-16119928 ] Liang-Chi Hsieh commented on SPARK-21677: - [~hyukjin.kwon] Thanks! Definitely we are interested in this. We will work on this. > json_tuple throws NullPointException when column is null as string type. > > > Key: SPARK-21677 > URL: https://issues.apache.org/jira/browse/SPARK-21677 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Hyukjin Kwon >Priority: Minor > Labels: Starter > > I was testing {{json_tuple}} before using this to extract values from JSONs > in my testing cluster but I found it could actually throw > {{NullPointException}} as below sometimes: > {code} > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Hyukjin'))").show() > +---+ > | c0| > +---+ > |224| > +---+ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Jackson'))").show() > ++ > | c0| > ++ > |null| > ++ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(null))").show() > ... > java.lang.NullPointerException > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:367) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:366) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames$lzycompute(jsonExpressions.scala:366) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames(jsonExpressions.scala:365) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields$lzycompute(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.org$apache$spark$sql$catalyst$expressions$JsonTuple$$parseRow(jsonExpressions.scala:417) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:401) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:400) > at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2559) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.eval(jsonExpressions.scala:400) > {code} > It sounds we should show explicit error messages or return {{NULL}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.
[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119891#comment-16119891 ] Hyukjin Kwon commented on SPARK-21677: -- cc [~viirya], I remember your mentee was checking through JSON related code paths. Does this make sense to you and would you be interested in this? > json_tuple throws NullPointException when column is null as string type. > > > Key: SPARK-21677 > URL: https://issues.apache.org/jira/browse/SPARK-21677 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Hyukjin Kwon >Priority: Minor > Labels: Starter > > I was testing {{json_tuple}} before using this to extract values from JSONs > in my testing cluster but I found it could actually throw > {{NullPointException}} as below sometimes: > {code} > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Hyukjin'))").show() > +---+ > | c0| > +---+ > |224| > +---+ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(' Jackson'))").show() > ++ > | c0| > ++ > |null| > ++ > scala> Seq(("""{"Hyukjin": 224, "John": > 1225}""")).toDS.selectExpr("json_tuple(value, trim(null))").show() > ... > java.lang.NullPointerException > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:367) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$foldableFieldNames$1.apply(jsonExpressions.scala:366) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames$lzycompute(jsonExpressions.scala:366) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.foldableFieldNames(jsonExpressions.scala:365) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields$lzycompute(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.constantFields(jsonExpressions.scala:373) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.org$apache$spark$sql$catalyst$expressions$JsonTuple$$parseRow(jsonExpressions.scala:417) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:401) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple$$anonfun$eval$4.apply(jsonExpressions.scala:400) > at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2559) > at > org.apache.spark.sql.catalyst.expressions.JsonTuple.eval(jsonExpressions.scala:400) > {code} > It sounds we should show explicit error messages or return {{NULL}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org