> On April 1, 2018, 7:25 a.m., Madhan Neethiraj wrote: > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java > > Lines 61 (patched) > > <https://reviews.apache.org/r/66253/diff/7/?file=1990770#file1990770line61> > > > > wait upto 120 times the avgCommitTimeSeconds?
There is safety factor. Normally all commits adhere to the regular times. Occasionally I see delays from ZK/Solr where the commit time goes out of wack. This is just to account for those cases. - Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66253/#review200270 ----------------------------------------------------------- On April 3, 2018, 7:44 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66253/ > ----------------------------------------------------------- > > (Updated April 3, 2018, 7:44 p.m.) > > > Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath > Subramanian, and Vishal Suvagia. > > > Bugs: ATLAS-2460 > https://issues.apache.org/jira/browse/ATLAS-2460 > > > Repository: atlas > > > Description > ------- > > **Background** > This implementation deals with the 'import into' part of the data migration > process. > > It assumes: > - Export from older cluster is done. > - Generated file has been moved to newer cluster. > > **Implementation** > > During _Atlas_ server startup, the configuration parameter is checked, if > that parameter exists, all services except _DataMigrationService_ is started. > Migration is started. Atlas server is available in _MIGRATION_ mode. It > processes REST calls made only to the _AdminResource_. > > Here's are the udpates: > - New configuration parameter has been added: > _atlas.migration.mode.filename=<name of the file to be imported>_ > This configuration parameter is set by Ambari as part of its migration > orchestration. > - _DataMigrationService_: New service that performs async migration as soon > as Atlas server starts up. > - _MigrationProgressService_: Added. Get progress of import. > - _AdminResource.getStatus()_ Now supplies additional status about migration. > - _ServiceState_ Modified to carry additional status _MIGRATION_. This status > is set by looking at the configuration parameter above. > - _Services_ modified for special handling of _DataMigrationService_. > > > **CURL** > Check status using: > ``` > curl -X GET -u admin:admin -H "Content-Type: application/json" -H > "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status > ``` > > **Migration Status** > The above URL in migration mode yields JSON like: > ``` > {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}} > ``` > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 > common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 > common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java > 31d20855 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 6820a93c > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java > a0060200 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java > PRE-CREATION > > graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java > 44090097 > intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java > 1ea961d8 > intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a > pom.xml bfbb9535 > repository/pom.xml b1d6b1f9 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 5672d9dc > > repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java > 66762001 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java > 5bec16ed > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java > 1a04418a > > repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java > 8257faa1 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java > ac35860d > repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION > repository/src/test/resources/stocks-2.zip PRE-CREATION > repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION > repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION > repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION > webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java > 6681a372 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 1b3f2c86 > webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java > f1760e7f > webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java > 3fe8d18c > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > aab2bb8f > > > Diff: https://reviews.apache.org/r/66253/diff/11/ > > > Testing > ------- > > **Unit tests** > Unit tests for _AtlasGraphSONReader_ added. > > **Functional tests** > Steps to test file-based import: > - Place the exported file say _/root/atlas-data_ > - Add to _Atlas_ Ambari's customer property: > _atlas.migration.mode.filename=/root/atlas-data_ > - Ambari will prompt for a restart. Restart Atlas. > - On the server view the progress in the logs using: _tail -f > /var/log/atlas/application.log_ > - Use the CURL call mentioned above and view the status and the progress of > the import. > > Steps to test directory-based import: > - Place the exported files say _/root/atlas-data_ > - Add to _Atlas_ Ambari's customer property: > _atlas.migration.mode.filename=/root/atlas-data_ > - Ambari will prompt for a restart. Restart Atlas. > - On the server view the progress in the logs using: _tail -f > /var/log/atlas/application.log_ > - Use the CURL call mentioned above and view the status and the progress of > the import. > > **Performance Tests** > Single threaded average commits: 600,000 per minute. > Number of threads: 8, average commit: 2 million per minute. > > > Thanks, > > Ashutosh Mestry > >
