----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/66253/#review200270 -----------------------------------------------------------
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java Lines 233 (patched) <https://reviews.apache.org/r/66253/#comment280940> Please review how the following types in Atlas are handled here: - BigInteger - BigDecimal - Date graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java Lines 61 (patched) <https://reviews.apache.org/r/66253/#comment280939> wait upto 120 times the avgCommitTimeSeconds? repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java Line 92 (original), 94 (patched) <https://reviews.apache.org/r/66253/#comment280938> When in HA mode, is it necessary to inialize typeDefStore here when migration is enabled? Shouldn't it be initialized only in active instance, in AtlasTypeDefStoreInitializer.instanceIsActive()? Running typeDefStore initialzation from multiple instances can result in incorrect store contents. webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java Lines 90 (patched) <https://reviews.apache.org/r/66253/#comment280937> missing "else" before "if" webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java Lines 94 (patched) <https://reviews.apache.org/r/66253/#comment280936> this 'if' block is a duplicate of one above, in #90. - Madhan Neethiraj On April 1, 2018, 6:21 a.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/66253/ > ----------------------------------------------------------- > > (Updated April 1, 2018, 6:21 a.m.) > > > Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath > Subramanian, and Vishal Suvagia. > > > Bugs: ATLAS-2460 > https://issues.apache.org/jira/browse/ATLAS-2460 > > > Repository: atlas > > > Description > ------- > > **Background** > This implementation deals with the 'import into' part of the data migration > process. > > It assumes: > - Export from older cluster is done. > - Generated file has been moved to newer cluster. > > **Implementation** > > During _Atlas_ server startup, the configuration parameter is checked, if > that parameter exists, all services except _DataMigrationService_ is started. > Migration is started. Atlas server is available in _MIGRATION_ mode. It > processes REST calls made only to the _AdminResource_. > > Here's are the udpates: > - New configuration parameter has been added: > _atlas.migration.mode.filename=<name of the file to be imported>_ > This configuration parameter is set by Ambari as part of its migration > orchestration. > - _DataMigrationService_: New service that performs async migration as soon > as Atlas server starts up. > - _MigrationProgressService_: Added. Get progress of import. > - _AdminResource.getStatus()_ Now supplies additional status about migration. > - _ServiceState_ Modified to carry additional status _MIGRATION_. This status > is set by looking at the configuration parameter above. > - _Services_ modified for special handling of _DataMigrationService_. > > > **CURL** > Check status using: > ``` > curl -X GET -u admin:admin -H "Content-Type: application/json" -H > "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status > ``` > > **Migration Status** > The above URL in migration mode yields JSON like: > ``` > {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}} > ``` > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 > common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 > common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 > > graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java > 31d20855 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 6820a93c > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java > a0060200 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java > PRE-CREATION > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java > PRE-CREATION > > graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java > 44090097 > intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java > 1ea961d8 > intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java > PRE-CREATION > intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a > pom.xml bfbb9535 > repository/pom.xml b1d6b1f9 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > 5672d9dc > > repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java > 351b4753 > > repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java > 66762001 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java > 5bec16ed > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java > 1a04418a > > repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java > 8257faa1 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java > ac35860d > repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION > repository/src/test/resources/stocks-2.zip PRE-CREATION > repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION > repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION > repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION > webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java > 6681a372 > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 1b3f2c86 > webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java > f1760e7f > webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java > 3fe8d18c > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > aab2bb8f > > > Diff: https://reviews.apache.org/r/66253/diff/8/ > > > Testing > ------- > > **Unit tests** > Unit tests for _AtlasGraphSONReader_ added. > > **Functional tests** > Steps to test file-based import: > - Place the exported file say _/root/atlas-data_ > - Add to _Atlas_ Ambari's customer property: > _atlas.migration.mode.filename=/root/atlas-data_ > - Ambari will prompt for a restart. Restart Atlas. > - On the server view the progress in the logs using: _tail -f > /var/log/atlas/application.log_ > - Use the CURL call mentioned above and view the status and the progress of > the import. > > Steps to test directory-based import: > - Place the exported files say _/root/atlas-data_ > - Add to _Atlas_ Ambari's customer property: > _atlas.migration.mode.filename=/root/atlas-data_ > - Ambari will prompt for a restart. Restart Atlas. > - On the server view the progress in the logs using: _tail -f > /var/log/atlas/application.log_ > - Use the CURL call mentioned above and view the status and the progress of > the import. > > > Thanks, > > Ashutosh Mestry > >
