-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71025/
-----------------------------------------------------------
(Updated Feb. 17, 2020, 5:54 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and
Sarath Subramanian.
Changes
-------
Updates include:
- Additional documentation.
- Support for new zip file format.
Bugs: ATLAS-3320
https://issues.apache.org/jira/browse/ATLAS-3320
Repository: atlas
Description (updated)
-------
**Approach**
- Use existing producer-consumer (PC) framework.
- Modify _BulkImporterImpl_ to implement _WorkItemConsumer_.
- Add support for configuring number of workers and batch size within
_AtlasImportRequest_.
- Existing import implementation continues to function as before. This is
maintained for backward compatibility.
- New implementation supports additional more memory efficient zip format
(_ZipDirect_). This drastically reduces memory requirement during import.
- The new import strategy, _MigrationImport_ uses the _bulkLoading_ mode of
_JanusGraph_ thereby achieving high ingest rates.
_AtlasImportRequest_
```
{
"options": {
"numWorkers": 8,
"batchSize": 25
}
}
```
Support for ZipDirect format:
_AtlasImportRequest_
```
{
"options": {
"numWorkers": 8,
"batchSize": 25,
"format": "zipDirect",
"migration": "true"
}
}
```
**CURL**
```
curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H
"Cache-Control: no-cache" -F request=@./import-options.json -F
data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import
```
Diffs (updated)
-----
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
4acb371f1
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
0b3ede93f
intg/src/main/java/org/apache/atlas/pc/WorkItemConsumer.java 9ba4bf4e3
intg/src/main/java/org/apache/atlas/pc/WorkItemManager.java a7ba67cb0
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java
bbe0dc5ba
repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java
55990f780
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
27001e3a9
repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java
351b47536
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
ca0bc415c
repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
39ea3f82e
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
c536f3b86
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
fdf117a25
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
54c32c5e8
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
746193188
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/ImportStrategy.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
c14850f43
repository/src/test/java/org/apache/atlas/repository/impexp/MigrationImportTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
0ffc3d595
repository/src/test/resources/zip-direct-1.zip PRE-CREATION
repository/src/test/resources/zip-direct-2.zip PRE-CREATION
Diff: https://reviews.apache.org/r/71025/diff/5/
Changes: https://reviews.apache.org/r/71025/diff/4-5/
Testing
-------
**Unit tests**
Existing tests.
**Functional tests**
- Verified import for pre-1.0 and post-1.0 exported ZIP files.
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1292
**Volume tests**
- Measure performance with large data.
+----------+----------+----------+------------------------+
| File | Before | After | Configuration |
+----------+----------+----------+------------------------+
| smalldb | 6 min | 2 min | Shards: 4, Threads: 8 |
| (2.2 MB) | | | |
+----------+----------+----------+------------------------+
| largedb | 3 hrs | 10 mins | Shards: 4, Threads: 16 |
| (40 MB) | | | |
+----------+----------+----------+------------------------+
Thanks,
Ashutosh Mestry
