----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71025/#review219785 -----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/graph/IFullTextMapper.java Lines 34 (patched) <https://reviews.apache.org/r/71025/#comment307984> methods defined here looks more of like helper methods than interface methods. - Sarath Subramanian On March 4, 2020, 10:09 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71025/ > ----------------------------------------------------------- > > (Updated March 4, 2020, 10:09 p.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3320 > https://issues.apache.org/jira/browse/ATLAS-3320 > > > Repository: atlas > > > Description > ------- > > **Approach** > - Use existing producer-consumer (PC) framework. > - Modify _BulkImporterImpl_ to implement _WorkItemConsumer_. > - Add support for configuring number of workers and batch size within > _AtlasImportRequest_. > - Existing import implementation continues to function as before. This is > maintained for backward compatibility. > - New implementation supports additional more memory efficient zip format > (_ZipDirect_). This drastically reduces memory requirement during import. > - The new import strategy, _MigrationImport_ uses the _bulkLoading_ mode of > _JanusGraph_ thereby achieving high ingest rates. > > _AtlasImportRequest_ > ``` > { > "options": { > "numWorkers": 8, > "batchSize": 25 > } > } > ``` > Support for ZipDirect format: > _AtlasImportRequest_ > ``` > { > "options": { > "numWorkers": 8, > "batchSize": 25, > "format": "zipDirect", > "migration": "true" > } > } > ``` > > > **CURL** > ``` > curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H > "Cache-Control: no-cache" -F request=@./import-options.json -F > data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import > ``` > > > Diffs > ----- > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 4acb371f1 > intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java > 3362bf158 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > bbe0dc5ba > > repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java > 0f2b4bfae > > repository/src/main/java/org/apache/atlas/repository/graph/IFullTextMapper.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java > 1964ade9a > > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java > cb5a7acd0 > > repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java > f552525a4 > > repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java > 39ea3f82e > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > d7020a702 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 30f5e5a7c > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java > 54c32c5e8 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2f3aad06b > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IAtlasEntityChangeNotifier.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/EntityChangeNotifierNop.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/FullTextMapperV2Nop.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/ImportStrategy.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumerBuilder.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/TestModules.java 06e0ebc6c > > > Diff: https://reviews.apache.org/r/71025/diff/12/ > > > Testing > ------- > > **Unit tests** > Existing tests. > > **Functional tests** > - Verified import for pre-1.0 and post-1.0 exported ZIP files. > > **Pre-commit** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1712/ > > **Volume tests** > - Measure performance with large data. > > +----------+----------+----------+------------------------+ > | File | Before | After | Configuration | > +----------+----------+----------+------------------------+ > | smalldb | 6 min | 2 min | Shards: 4, Threads: 8 | > | (2.2 MB) | | | | > +----------+----------+----------+------------------------+ > | largedb | 3 hrs | 10 mins | Shards: 4, Threads: 16 | > | (40 MB) | | | | > +----------+----------+----------+------------------------+ > > > Thanks, > > Ashutosh Mestry > >
