[
https://issues.apache.org/jira/browse/HIVE-22938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046776#comment-17046776
]
Marton Bod commented on HIVE-22938:
-----------------------------------
As an addition: even for MR, the current state now is that when inserting into
full ACID tables the empty bucket files are only generated inside the staging
directory and they aren't moved to the final delta directory folder. The empty
bucket files are only kept for MM/insert-only tables.
> Investigate possibility of removing empty bucket file creation mechanism in
> Hive-on-MR
> --------------------------------------------------------------------------------------
>
> Key: HIVE-22938
> URL: https://issues.apache.org/jira/browse/HIVE-22938
> Project: Hive
> Issue Type: Task
> Reporter: Marton Bod
> Priority: Major
>
> As a follow-up to HIVE-22918, this ticket is to investigate whether the empty
> bucket file creation mechanism can be removed safely when using MR as the
> engine.
> For a bucketed table of N buckets, each insert will generate N bucket files
> in the delta directory, regardless of how many actual buckets are written to.
> As an example, if a table has 500 buckets, and we insert a single record, 499
> empty bucket files are generated alongside the single bucket that contains
> the actual data. This makes the operation substantially slower in some cases.
> This behaviour only seems to happen when using MR as the execution engine.
> Some components/parts of the code might depend on this behaviour though, so
> it needs to be verified that removing this logic does not interfere with
> anything.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
