Sergey Shelukhin created HIVE-16223:
---------------------------------------
Summary: deterministic file naming for bucketing in Hive
Key: HIVE-16223
URL: https://issues.apache.org/jira/browse/HIVE-16223
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Bucketing in Hive is currently very fragile.
1) Some places determine bucket number from file name.
2) Some places determine bucket number from a file's "index" in a sorted list
of file in the directory.
3) It is possible to import files into a bucketed table without any regard for
either.
On top of that, weird rename paths (like _copy_1), subdirectories (e.g. from
Tez union, or just tables read with recursive input enabled), repeated inserts
into the same table, etc. can mess with either scheme.
Therefore I propose we include bucket index and count explicitly in the file
name (e.g. 000003_0_bucket_3of32). It will alleviate the above, and also may
simplify some pieces of code that try to account for missing bucket files,
multiple files, etc.
This will require changes to load table logic that is used in ctas, insert,
load, import etc.; change in logic when getting buckets, as well as when
altering table bucketing (to rename the files).
Users will still be able to use old-style bucketing by specifying a non-strict
config setting (not on by default).
The conversion of existing tables is the biggest issue. Perhaps the existing
tables can be "grandfathered" into the non-strict bucketing, with some warnings
asking the users to convert, and a command to do so in alter table/analyze
table.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
