[
https://issues.apache.org/jira/browse/ATLAS-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikhil Bonte updated ATLAS-3416:
--------------------------------
Description:
This will delete hive_table entities which are present in the hive_db but are
not the part of the incremental export request.
This enhances functionality provided by ATLAS-3324.
*Background*:
Requirement for import API is that if request contains "replicatedFrom"
option, in this case import API should find & delete tables which are not
imported but are part of database entity.
E.g. Consider t1...t5 under db1 replicated from source to destination using
table level incremental export from source.
Now suppose subsequent import request at destination contains only t1...t3, in
such case t4 & t5 must be deleted at destination.
*Solution approach*:
* Check if import request contains "replicatedFrom" option.
* Extract exportRequest from imported zip.
* Identify table entities which are imported using exportRequest.
* Identify database entity using imported table entities.
* Identify table entities which are part of database entity obtained in
previous step but are not imported in current import request.
* Delete tables.
was:
This will delete hive_table entities which are present in the hive_db but are
not the part of the incremental export request.
This enhances functionality provided by ATLAS-3324.
*Background*:
Requirement for import API is that if request contains "replicatedFrom"
option, in this case import API should find & delete tables which are not
imported but are part of database entity.
E.g. Consider t1...t5 under db1 replicated from source to destination using
table level incremental export from source.
Now suppose subsequent import request at destination contains only t1...t3, in
such case t4 & t5 must be deleted at destination.
*Solution approach*:
* Check if import request contains "replicatedFrom" option.
* Extract exportRequest from imported zip.
* Identify table entities which are imported using exportRequest.
* Identify database entity using imported table entities.
* Identify table entities which are part of database entity got from previous
step but are not imported in current import request.
* Delete tables.
> Import API: delete non-exported hive_table entities for table level
> replication
> -------------------------------------------------------------------------------
>
> Key: ATLAS-3416
> URL: https://issues.apache.org/jira/browse/ATLAS-3416
> Project: Atlas
> Issue Type: New Feature
> Reporter: Nikhil Bonte
> Assignee: Nikhil Bonte
> Priority: Major
> Fix For: trunk
>
> Attachments:
> Import-API-delete-non-exported-hive_table-for-table-v11.patch,
> Import-API-delete-non-exported-hive_table-for-table-v2-branch-0.8.patch,
> repl_exp_1.zip, repl_exp_2.zip
>
>
> This will delete hive_table entities which are present in the hive_db but are
> not the part of the incremental export request.
> This enhances functionality provided by ATLAS-3324.
> *Background*:
> Requirement for import API is that if request contains "replicatedFrom"
> option, in this case import API should find & delete tables which are not
> imported but are part of database entity.
> E.g. Consider t1...t5 under db1 replicated from source to destination using
> table level incremental export from source.
> Now suppose subsequent import request at destination contains only t1...t3,
> in such case t4 & t5 must be deleted at destination.
> *Solution approach*:
> * Check if import request contains "replicatedFrom" option.
> * Extract exportRequest from imported zip.
> * Identify table entities which are imported using exportRequest.
> * Identify database entity using imported table entities.
> * Identify table entities which are part of database entity obtained in
> previous step but are not imported in current import request.
> * Delete tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
