----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71025/ -----------------------------------------------------------
(Updated March 2, 2020, 6:57 p.m.) Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Changes ------- Updates include: - Reduces size of patch by breaking in to smaller implementations. Bugs: ATLAS-3320 https://issues.apache.org/jira/browse/ATLAS-3320 Repository: atlas Description ------- **Approach** - Use existing producer-consumer (PC) framework. - Modify _BulkImporterImpl_ to implement _WorkItemConsumer_. - Add support for configuring number of workers and batch size within _AtlasImportRequest_. - Existing import implementation continues to function as before. This is maintained for backward compatibility. - New implementation supports additional more memory efficient zip format (_ZipDirect_). This drastically reduces memory requirement during import. - The new import strategy, _MigrationImport_ uses the _bulkLoading_ mode of _JanusGraph_ thereby achieving high ingest rates. _AtlasImportRequest_ ``` { "options": { "numWorkers": 8, "batchSize": 25 } } ``` Support for ZipDirect format: _AtlasImportRequest_ ``` { "options": { "numWorkers": 8, "batchSize": 25, "format": "zipDirect", "migration": "true" } } ``` **CURL** ``` curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@./import-options.json -F data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import ``` Diffs (updated) ----- graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 4acb371f1 intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 3362bf158 repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java bbe0dc5ba repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java 55990f780 repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 1964ade9a repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java f552525a4 repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 39ea3f82e repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 30f5e5a7c repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java fdf117a25 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 54c32c5e8 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java 2f3aad06b repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/ImportStrategy.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumerBuilder.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/StatusReporter.java PRE-CREATION Diff: https://reviews.apache.org/r/71025/diff/7/ Changes: https://reviews.apache.org/r/71025/diff/6-7/ Testing ------- **Unit tests** Existing tests. **Functional tests** - Verified import for pre-1.0 and post-1.0 exported ZIP files. **Pre-commit** https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1292 **Volume tests** - Measure performance with large data. +----------+----------+----------+------------------------+ | File | Before | After | Configuration | +----------+----------+----------+------------------------+ | smalldb | 6 min | 2 min | Shards: 4, Threads: 8 | | (2.2 MB) | | | | +----------+----------+----------+------------------------+ | largedb | 3 hrs | 10 mins | Shards: 4, Threads: 16 | | (40 MB) | | | | +----------+----------+----------+------------------------+ Thanks, Ashutosh Mestry