[
https://issues.apache.org/jira/browse/HIVE-22938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051326#comment-17051326
]
Marton Bod edited comment on HIVE-22938 at 3/4/20, 3:33 PM:
------------------------------------------------------------
Removing the empty bucket creation logic breaks 400+ tests. At this point with
MR being deprecated, the cost/benefit ratio of removing this logic from
Hive-on-MR and adjusting all tests accordingly (and running the risk of
introducing unforeseen regressions) is not very compelling, so closing this
ticket.
was (Author: marton bod):
Removing the empty code part breaks 400+ tests. At this point with MR being
deprecated, the cost/benefit ratio of removing this logic from Hive-on-MR and
adjusting all tests accordingly (and running the risk of introducing unforeseen
regressions) is not very compelling, so closing this ticket.
> Investigate possibility of removing empty bucket file creation mechanism in
> Hive-on-MR
> --------------------------------------------------------------------------------------
>
> Key: HIVE-22938
> URL: https://issues.apache.org/jira/browse/HIVE-22938
> Project: Hive
> Issue Type: Task
> Reporter: Marton Bod
> Assignee: Marton Bod
> Priority: Major
> Attachments: HIVE-22938.1.patch
>
>
> As a follow-up to HIVE-22918, this ticket is to investigate whether the empty
> bucket file creation mechanism can be removed safely when using MR as the
> engine.
> For a bucketed table of N buckets, each insert will generate N bucket files
> in the delta directory, regardless of how many actual buckets are written to.
> As an example, if a table has 500 buckets, and we insert a single record, 499
> empty bucket files are generated alongside the single bucket that contains
> the actual data. This makes the operation substantially slower in some cases.
> This behaviour only seems to happen when using MR as the execution engine.
> Some components/parts of the code might depend on this behaviour though, so
> it needs to be verified that removing this logic does not interfere with
> anything.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
