[ https://issues.apache.org/jira/browse/SPARK-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012031#comment-14012031 ]
Cheng Lian commented on SPARK-1959: ----------------------------------- The problematic line should be [this one|https://github.com/apache/spark/blob/master/sql%2Fhive%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhive%2FhiveOperators.scala#L154]. I wonder under what circumstances, would Hive return a Java string {{"NULL"}} to represent a null value? Is it safe to simply remove this line? [~marmbrus] > String "NULL" is interpreted as null value > ------------------------------------------ > > Key: SPARK-1959 > URL: https://issues.apache.org/jira/browse/SPARK-1959 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.0.0 > Reporter: Cheng Lian > > The {{HiveTableScan}} operator unwraps string "NULL" (case insensitive) into > null values even if the column type is {{STRING}}. > To reproduce the bug, we use > {{sql/hive/src/test/resources/groupby_groupingid.txt}} as test input, copied > to {{/tmp/groupby_groupingid.txt}}. > Hive session: > {code} > hive> CREATE TABLE test_null(key INT, value STRING); > hive> LOAD DATA LOCAL INPATH '/tmp/groupby_groupingid.txt' INTO table > test_null; > hive> SELECT * FROM test_null WHERE value IS NOT NULL; > ... > OK > 1 NULL > 1 1 > 2 2 > 3 3 > 3 NULL > 4 5 > {code} > We can see that the {{NULL}} cells in the original input file are interpreted > as string {{"NULL"}} in Hive. > Spark SQL session ({{sbt/sbt hive/console}}): > {code} > scala> hql("CREATE TABLE test_null(key INT, value STRING)") > scala> hql("LOAD DATA LOCAL INPATH '/tmp/groupby_groupingid.txt' INTO table > test_null") > scala> hql("SELECT * FROM test_null WHERE value IS NOT NULL").foreach(println) > ... > [1,1] > [2,2] > [3,3] > [4,5] > {code} > As we can see, string {{"NULL"}} is interpreted as null values in Spark SQL. -- This message was sent by Atlassian JIRA (v6.2#6252)