[
https://issues.apache.org/jira/browse/HIVE-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134270#comment-16134270
]
Mariappan Asokan commented on HIVE-6476:
----------------------------------------
Sushanth,
Here are the changes I made in the uploaded patch:
* Moved some common code to new private methods ({{moveFiles()}} and
{{isReservedName()}})
* When dynamic partitioning is used, the table can be mutable
* For dynamic partitioning, when a partition does not exist, the optimization
to move the entire directory to the target location is still in effect.
However, when a partition exists, newly added files to the partition are moved
to the existing target directory one at a time
* The unique name generation logic is applied to only non-directory files
* In addition to deleting the newly created partitions in the metadata server,
all newly added files will be deleted when a commit fails
* Fixed a minor bug: when there is no new file to move ({{firstChild}} ==
{{null}} in {{moveTaskOutputs()}}), no action will be taken to avoid null
pointer dereferencing.
* Created tests that test different cases: appending new records such that all
records go to existing partitions, some records to existing partitions and
others to new partitions, and all records to new partitions
* Deleted an existing test that tested the failure of dynamic partitioning and
append
Please provide your feedback. Thanks.
> Support Append with Dynamic Partitioning
> ----------------------------------------
>
> Key: HIVE-6476
> URL: https://issues.apache.org/jira/browse/HIVE-6476
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog, Metastore, Query Processor, Thrift API
> Reporter: Sushanth Sowmyan
> Assignee: Mariappan Asokan
> Attachments: HIVE-6476.1.patch
>
>
> Currently, we do not support mixing dynamic partitioning and append in the
> same job. One reason is that we need exhaustive testing of corner cases for
> that, and a second reason is the behaviour of add_partitions. To support
> dynamic partitioning with append, we'd have to have a
> add_partitions_if_not_exist call, rather than an add_partitions call.
> Thus, the current implementation in HIVE-6475 assumes immutability for all
> dynamic partitioning jobs, irrespective of whether or not the table is marked
> as mutable or not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
