-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72039/
-----------------------------------------------------------

(Updated Feb. 13, 2020, 1:02 p.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Changes
-------

* Addressed review comments.
* Included ATLAS-3603.


Summary (updated)
-----------------

ATLAS-3595, ATLAS-3603 Migration import: support zip file for data migration


Bugs: ATLAS-3595
    https://issues.apache.org/jira/browse/ATLAS-3595


Repository: atlas


Description (updated)
-------

## ATLAS-3595
Data migration service added under ATLAS-2461 works with json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:

* Property value for migration should contain full path of file, below is the 
example. 
* Data migration service will first check migration property if value refers to 
a zip file.
* If true, it will call ZipMigrationImporter to import data in zip file.
* ZipMigrationImporter internally uses existing ImportService to ingest zip.
* Audit service & Notification service will also be started as a part of zip 
import.


E.g. properties to enable zip data migration:

* atlas.migration.data.filename=/path/fileName.zip
* atlas.migration.mode.workers=5
* atlas.migration.mode.batch.size=20


## ATLAS-3603
Migration import should start when the data (zip) to be ingested is available 
at the pre-configured location.

Approach:

ZipMigrationImporter will first check if zip is available, accessible & 
completely written.

If yes, migration will start ingesting zip.
if not, will poll the location periodically. when existence is detected, import 
will proceed as usual.

It uses the file size comparison approach. Once modify event is received, file 
size is compared in an interval. If sizes match, it means file is available and 
not being written.


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 170136195 
  
repository/src/main/java/org/apache/atlas/repository/migration/FileWatcher.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrationImporter.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/MigrationImportTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/72039/diff/7/

Changes: https://reviews.apache.org/r/72039/diff/6-7/


Testing
-------

PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1614/


Thanks,

Nikhil Bonte

Reply via email to