> On April 22, 2020, 11:39 p.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > > Lines 1032 (patched) > > <https://reviews.apache.org/r/72384/diff/5/?file=2224640#file2224640line1032> > > > > Processing relationship-defs for each entity-def would result in each > > relationship-def to be processed twice - one for entity-def at each end. I > > suggest the following: > > > > if (typeDef instanceof AtlasEntityDef) { > > createEdgeLabels(management, (AtlasEntityDef) typeDef); > > } else if (typeDef instanceof AtlasRelationshipDef) { > > createEdgeLabels(management, (AtlasRelationshipDef) typeDef); > > }
With this, I am not able to get relationships with r: > On April 22, 2020, 11:39 p.m., Madhan Neethiraj wrote: > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > > Lines 1045 (patched) > > <https://reviews.apache.org/r/72384/diff/5/?file=2224640#file2224640line1045> > > > > Use AtlasAttribute.getRelationshipEdgeLabel() to get the > > relationship-label name. I tried several approaches but I am not able to get the relationship edge lable for STRUCT attributes. - Ashutosh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72384/#review220436 ----------------------------------------------------------- On April 22, 2020, 4:58 a.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72384/ > ----------------------------------------------------------- > > (Updated April 22, 2020, 4:58 a.m.) > > > Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, > and Sarath Subramanian. > > > Bugs: ATLAS-3737 > https://issues.apache.org/jira/browse/ATLAS-3737 > > > Repository: atlas > > > Description > ------- > > **Approach** > Please refer to JIRA for details. > > **Benefits with this Approach** > - Memory usage remains constant. > - Cores, based on numWorkers, are utilized. > > **Changes to Migration Import** > - Instantiates 2 _EntityStoreV2_, one with non-bulk _AtlasGraph_ and other > with bulk loading _AtlasGraph_. > - For entities that causes schema change (handled by _IllegalStateException_ > or _IllegalArgumentException_), the same entity is attempted with non-bulk > instantiated _EntityStoreV2_. > > > Diffs > ----- > > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java > 0176ba705 > > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/query/NativeJanusGraphQuery.java > 24859389b > > repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java > 25042c19c > > repository/src/main/java/org/apache/atlas/discovery/EntityLineageService.java > 5548740d0 > > repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java > 172e187e4 > > repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java > 417c96cd8 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java > bff54ede3 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > dfaa739e8 > > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java > 9c57d54de > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerDelegate.java > 512eb4636 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java > 540defca9 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/HardDeleteHandlerV1.java > a32b67124 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v1/SoftDeleteHandlerV1.java > e614a4e41 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java > d3d9367fb > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > 379150b7b > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > f393e515b > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java > 7425ac601 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java > a4d732af0 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociator.java > 05878f405 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 04da41e1a > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java > 36bee301d > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityStateChecker.java > 2718a0627 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IDBasedEntityResolver.java > 3b9694851 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/UniqAttrBasedEntityResolver.java > 14ddc5d7a > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java > ff55e40eb > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java > ecce1b0e7 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java > e0a457334 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumerBuilder.java > 69d33b21e > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java > 734add6d7 > > repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java > f8b7eda71 > > repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java > 7567daa8f > > repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java > 90e714cb5 > > repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java > ce9056b54 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java > 38228a8ec > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityTestBase.java > 368951dfc > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2Test.java > a40cf85a1 > > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/ClassificationAssociatorTest.java > 37acbe5de > > > Diff: https://reviews.apache.org/r/72384/diff/5/ > > > Testing > ------- > > **Pre-commit** > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1830/ > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1847/ > > **Volume testing** > Verified with large volume of data. > > Import parameters: > numWorkers: 36 > batchSize: 750 > Heap: 16 > Solr Heap: 60 > Hbase memory tuning. > > > +--------------------------+---------------------+-----------------+ > | Number of entities | | | > | (in millions) | Before | After | > +--------------------------+---------------------+-----------------+ > | 6.21 | 5 hrs 26 m | 4 hrs 12 mins | > +--------------------------+---------------------+-----------------+ > | 8.5 | 6 hrs 15 m | 4 hr 40 mins | > +--------------------------+---------------------+-----------------+ > > > Thanks, > > Ashutosh Mestry > >
