Sushanth Sowmyan created HIVE-6016:
--------------------------------------
Summary: Hadoop23Shims has a bug in listLocatedStatus impl.
Key: HIVE-6016
URL: https://issues.apache.org/jira/browse/HIVE-6016
Project: Hive
Issue Type: Bug
Components: Shims
Affects Versions: 0.13.0
Reporter: Sushanth Sowmyan
Assignee: Prasanth J
Prashant and I discovered that the implementation of the wrapping Iterator in
listLocatedStatus at
https://github.com/apache/hive/blob/2d2f89c21618341987c1257a88691981f1f606c7/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java#L350-L393
is broken.
Basically, if you had files (a,b,_s) , with a filter that is supposed to filter
out _s, we expect an output result of (a,b). Instead, we get (a,b,null), with
hasNext looking at the next value to see if it's null, and using that to decide
if it has any more entries, and thus, (a,b,_s) becomes (a,b).
The problem with this approach, however, is that if you have an underlying
(a,_s,b) and expect a (a,b) from it, you won't, because it translates to a
(a,null,b), which then translates to a (a).
Furthermore, there's a boundary condition on the very first pick, which causes
a (_s,a,b) to result in (_s,a,b), bypassing the filter, and thus, we wind up
with a resultant unfiltered (_s,a,b) which orc breaks on.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
