[
https://issues.apache.org/jira/browse/HIVE-15852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864733#comment-15864733
]
Ashutosh Chauhan commented on HIVE-15852:
-----------------------------------------
Gains of not creating empty files are huge even on HDFS, I don't think we want
to loose those.
We should fix tablesampling. I am not aware of Pig's assumption for bucketing.
Can you describe bit more what is that assumption? Anyway, even if it has that
seems like assuming something in implementation, since there is no interface
contract anywhere that # of buckets = # of files.
> Tablesampling on Tez in low-record case throws ArrayIndexOutOfBoundsException
> -----------------------------------------------------------------------------
>
> Key: HIVE-15852
> URL: https://issues.apache.org/jira/browse/HIVE-15852
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Affects Versions: 2.1.1
> Reporter: Thomas Poepping
>
> Due to HIVE-13040 ( https://issues.apache.org/jira/browse/HIVE-13040 ), which
> doesn't create empty files to represent empty buckets when Hive is on Tez, a
> couple things are broken.
> First of all, if there are empty buckets (which is possible with large
> datasets in the partitioned-bucketed case), tablesampling will not work if
> you're referencing a bucket number higher than the number of files.
> e.g. In some partition 'p', there are three rows. The table 't' is clustered
> into ten buckets. With maximal hashing, only three bucket files will be
> created. If we do select * from t tablesample (bucket x out of 10) where
> <selecting from p> (where x > 3), an ArrayIndexOutOfBoundsException will be
> thrown because Hive assumes there are only three buckets.
> Second, other applications (such as Pig) may be making assumptions about the
> number of files equaling the number of buckets.
> Possible fixes:
> * Revert HIVE-13040
> * Change how tablesampling is implemented to accept possibility that number
> of files != number of buckets
> ** Would require coordination across projects to change assumptions
> Things to consider:
> * what performance gains are there from not creating empty files?
> * if the gains are large, are we willing to lose them? (by reverting
> HIVE-13040)
> * _how else can we avoid creating unnecessary files, while still maintaining
> invariants other applications expect?_
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
