[
https://issues.apache.org/jira/browse/HIVE-20911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anishek reassigned HIVE-20911:
------------------------------
Assignee: anishek
> External Table Replication for Hive
> -----------------------------------
>
> Key: HIVE-20911
> URL: https://issues.apache.org/jira/browse/HIVE-20911
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 4.0.0
> Reporter: anishek
> Assignee: anishek
> Priority: Critical
> Fix For: 4.0.0
>
>
> External tables are not replicated currently as part of hive replication. As
> part of this jira we want to enable that.
> Approach:
> * Target cluster will have a top level base directory config that will be
> used to copy all data relevant to external tables. This will be provided via
> the *with* clause in the *repl load* command. This base path will be prefixed
> to the path of the same external table on source cluster.
> * Since changes to directories on the external table can happen without hive
> knowing it, hence we cant capture the relevant events when ever new data is
> added or removed, we will have to copy the data from the source path to
> target path for external tables every time we run incremental replication.
> ** this will require incremental *repl dump* to now create an additional
> file *\_external\_tables\_info* with data in the following form
> {code}
> OpearationType,tableName,base64Encoded(tableDataLocation)
> {code}
> where OpeartionType can be one in (ADD, REMOVE)
> ** *repl load* will look up all the external tables on target and remove
> tables listed with REMOVE type in the above file.
> ** For the remaining tables it will create tasks for the corresponding paths
> from source to target along with the existing tasks for incremental load.
> * New External tables will be created with data copied as part of regular
> tasks wile incremental load, applying the base directory prefix
> * Bootstrap will also create / copy these external tables as part of their
> regular workflow, applying the base directory prefix
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
