[
https://issues.apache.org/jira/browse/ATLAS-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Bonte updated ATLAS-3416:
--------------------------------
Description:
This will delete hive_table entities which are present in the hive_db but are
not the part of the incremental export request.
This enhances functionality provided by
[ATLAS-3324|https://issues.apache.org/jira/browse/ATLAS-3324].
Background:
Requirement for import API is that if request contains "replicatedFrom"
option, in this case import API should find & delete tables which are not
imported but are part of database entity.
E.g. Consider t1...t5 under db1 replicated from source to destination
using table level incremental export from source.
Now suppose subsequent import request at destination contains only
t1...t3, in such case t4 & t5 must be deleted at destination.
Solution approach:
Check if import request contains "replicatedFrom" option.
Extract exportRequest from imported zip.
Identify table entities which are imported using exportRequest.
Identify database entity using imported table entities.
Identify table entities which are part of database entity got from
previous step but are not imported.
Delete tables.
was:
This will delete hive_table entities which are present in the hive_db but are
not the part of the incremental export request.
This enhances functionality provided by
[ATLAS-3324|https://issues.apache.org/jira/browse/ATLAS-3324].
Detailed Background:
Requirement for import API is that if request contains "replicatedFrom"
option, in this case import API should find & delete tables which are not
imported but are part of database entity.
E.g. Consider t1...t5 under db1 replicated from source to destination
using table level incremental export from source.
Now suppose subsequent import request at destination contains only
t1...t3, in such case t4 & t5 must be deleted at destination.
Solution approach:
Check if import request contains "replicatedFrom" option.
Extract exportRequest from imported zip.
Identify table entities which are imported using exportRequest.
Identify database entity using imported table entities.
Identify table entities which are part of database entity got from
previous step but are not imported.
Delete tables.
> Import API: delete non-exported hive_table entities for table level
> replication
> -------------------------------------------------------------------------------
>
> Key: ATLAS-3416
> URL: https://issues.apache.org/jira/browse/ATLAS-3416
> Project: Atlas
> Issue Type: New Feature
> Reporter: Nikhil Bonte
> Assignee: Nikhil Bonte
> Priority: Major
> Fix For: trunk
>
> Attachments:
> Import-API-delete-non-exported-hive_table-for-table-v11.patch,
> Import-API-delete-non-exported-hive_table-for-table-v2-branch-0.8.patch,
> repl_exp_1.zip, repl_exp_2.zip
>
>
> This will delete hive_table entities which are present in the hive_db but are
> not the part of the incremental export request.
> This enhances functionality provided by
> [ATLAS-3324|https://issues.apache.org/jira/browse/ATLAS-3324].
> Background:
> Requirement for import API is that if request contains "replicatedFrom"
> option, in this case import API should find & delete tables which are not
> imported but are part of database entity.
> E.g. Consider t1...t5 under db1 replicated from source to destination
> using table level incremental export from source.
> Now suppose subsequent import request at destination contains only
> t1...t3, in such case t4 & t5 must be deleted at destination.
> Solution approach:
> Check if import request contains "replicatedFrom" option.
> Extract exportRequest from imported zip.
> Identify table entities which are imported using exportRequest.
> Identify database entity using imported table entities.
> Identify table entities which are part of database entity got from
> previous step but are not imported.
> Delete tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
