[jira] (SPARK-19412) HiveContext hive queries differ from CLI hive queries

[Enrique Sainz Baixauli (JIRA)]

Title: Message Title

Enrique Sainz Baixauli created an issue   Spark / SPARK-19412 HiveContext hive queries differ from CLI hive queries Issue Type: Bug Affects Versions: 1.5.1 Assignee: Unassigned Components: SQL Created: 31/Jan/17 14:29 Priority: Major Reporter: Enrique Sainz Baixauli Hi, I've been wandering around trying to find a report related to this, but haven't found such so far. So here it is. I have found two differences between Hive queries made from SparkSQL and from Hive CLI (or beeswax or Hue): First: a query against a single partitioned table (although other, non-partitioned tables seem to work well) returns all column values in the first column, and null in the rest of them (field separator is '\t'): scala > sqlContext.sql("use [database_name]) scala> val t1 = sqlContext.sql("select * from [table_name]") scala> t1.first.get(0) res9: Any = "0001 1 [...]" scala> t1.first.get(1) res10: Any = null And the same for the following column indexes from 1 on. However, the same query in Hive CLI/Beeswax/Hue shows all columns filled and no null values at the end. Second: a join between two partitioned tables returns an empty DataFrame when the same query in Hive returns rows filled with values: scala> val join = sqlContext.sql("select * from [table_1_name] t1 join [table_2_name] t2 on t1.id = t2.id where t2.data_timestamp_part = '1485428556075'") scala> join.first [...] java.util.NoSuchElementException: next on empty iterator These two issues happen in Spark 1.5 (CDH 5.5.1) but seem to be solved in 1.6 (CDH 5.7.0). Is there a chance of backporting this fix? And is there a bug report that I can't find where this was reported before? Thank you all! Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)