-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72384/
-----------------------------------------------------------
(Updated April 24, 2020, 5 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and
Sarath Subramanian.
Changes
-------
Updates include:
- Removed formatting for getting edge-label for STRUCT attributes.
Bugs: ATLAS-3737
https://issues.apache.org/jira/browse/ATLAS-3737
Repository: atlas
Description
-------
**Approach**
Please refer to JIRA for details.
**Benefits with this Approach**
- Memory usage remains constant.
- Cores, based on numWorkers, are utilized.
**Changes to Migration Import**
- Instantiates 2 _EntityStoreV2_, one with non-bulk _AtlasGraph_ and other with
bulk loading _AtlasGraph_.
- For entities that causes schema change (handled by _IllegalStateException_ or
_IllegalArgumentException_), the same entity is attempted with non-bulk
instantiated _EntityStoreV2_.
Diffs (updated)
-----
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
0176ba705
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java
24859389b
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
25042c19c
repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java
5548740d0
repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
172e187e4
repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java
417c96cd8
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
bff54ede3
repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java
dfaa739e8
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
9c57d54de
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerDelegate.java
512eb4636
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
540defca9
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java
a32b67124
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java
e614a4e41
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java
d3d9367fb
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
379150b7b
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
f393e515b
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
7425ac601
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
a4d732af0
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java
05878f405
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
04da41e1a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
36bee301d
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityStateChecker.java
2718a0627
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java
3b9694851
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/UniqAttrBasedEntityResolver.java
14ddc5d7a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
ff55e40eb
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java
ecce1b0e7
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
e0a457334
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumerBuilder.java
69d33b21e
repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
f8b7eda71
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java
7567daa8f
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java
90e714cb5
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
ce9056b54
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
38228a8ec
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityTestBase.java
368951dfc
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2Test.java
a40cf85a1
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java
37acbe5de
Diff: https://reviews.apache.org/r/72384/diff/7/
Changes: https://reviews.apache.org/r/72384/diff/6-7/
Testing
-------
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1830/
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1847/
**Volume testing**
Verified with large volume of data.
Import parameters:
numWorkers: 36
batchSize: 750
Heap: 16
Solr Heap: 60
Hbase memory tuning.
+--------------------------+---------------------+-----------------+
| Number of entities | | |
| (in millions) | Before | After |
+--------------------------+---------------------+-----------------+
| 6.21 | 5 hrs 26 m | 4 hrs 12 mins |
+--------------------------+---------------------+-----------------+
| 8.5 | 6 hrs 15 m | 4 hr 40 mins |
+--------------------------+---------------------+-----------------+
Thanks,
Ashutosh Mestry
