----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71025/ -----------------------------------------------------------
(Updated Jan. 23, 2020, 5:30 p.m.) Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Changes ------- Updates include: - Rebased with latest version. - Added new method that uses batch size to commit. - Refactoring. Bugs: ATLAS-3320 https://issues.apache.org/jira/browse/ATLAS-3320 Repository: atlas Description ------- **Approach** - Use existing producer-consumer (PC) framework. - Modify _BulkImporterImpl_ to implement _WorkItemConsumer_. - Add support for configuring number of workers and batch size within _AtlasImportRequest_. _AtlasImportRequest_ ``` { "options": { "numWorkers": 8, "batchSize": 25 } } ``` **CURL** ``` curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@./import-options.json -F data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import ``` Diffs (updated) ----- graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 4acb371f1 intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 0b3ede93f intg/src/main/java/org/apache/atlas/pc/WorkItemConsumer.java 9ba4bf4e3 intg/src/main/java/org/apache/atlas/pc/WorkItemManager.java a7ba67cb0 repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java bbe0dc5ba repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java 55990f780 repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java 928c70dba repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java 25284e92f repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java 54c32c5e8 Diff: https://reviews.apache.org/r/71025/diff/4/ Changes: https://reviews.apache.org/r/71025/diff/3-4/ Testing ------- **Unit tests** Existing tests. **Functional tests** - Verified import for pre-1.0 and post-1.0 exported ZIP files. **Pre-commit** https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1292 **Volume tests** - Measure performance with large data. +----------+----------+----------+------------------------+ | File | Before | After | Configuration | +----------+----------+----------+------------------------+ | smalldb | 6 min | 2 min | Shards: 4, Threads: 8 | | (2.2 MB) | | | | +----------+----------+----------+------------------------+ | largedb | 3 hrs | 10 mins | Shards: 4, Threads: 16 | | (40 MB) | | | | +----------+----------+----------+------------------------+ Thanks, Ashutosh Mestry