----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51092/#review163145 -----------------------------------------------------------
Ship it! Ship It! - Madhan Neethiraj On Jan. 26, 2017, 4:53 p.m., Jeff Hagelberg wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51092/ > ----------------------------------------------------------- > > (Updated Jan. 26, 2017, 4:53 p.m.) > > > Review request for atlas and David Kantor. > > > Bugs: ATLAS-1114 > https://issues.apache.org/jira/browse/ATLAS-1114 > > > Repository: atlas > > > Description > ------- > > Apply performance fixes for create/update entities from IBM fork to Atlas. > During our performance profiling, we found a number of performance hotspots > in JProfiler. Our main findings were > > - multiple queries were being executed for each instance being > created/updated to find matches by unique attribute. > - one query was being executed for each instance being created/updated to > find the corresponding vertex if there is one > - Calculating the value of the full text property was taking a > significant portion of the time to create/update entities, mainly due to its > calls to getVertexForGUID > > The changes we put in do the following: > > - batch lookups by guid when create/update entities. Execute one > AtlasGraphQuery to find them all. > - batch lookups by unique attribute when create/update entities. Execute > one AtlasGraphQuery per class to find unique attribute matches. > - find all existing vertices up front during create/update entity. Use > those vertices during the graph mapping process to avoid running unnecessary > graph queries > - reuse reference vertices from instance to graph mapping when computing > full text property > > Also, resolved all test failures in webapp. I disentagled the three > competing versions of the hive model that the various tests were trying to > use. Now they all pass. I tried to follow the path of least resistence. We > really should clean this up more, there is really no need for threee > different versions of hive_table and its related classes. > > > Diffs > ----- > > > repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java > 50a007bf1bf7e8c07a57b327b3aa3b907dd5f660 > > repository/src/main/java/org/apache/atlas/repository/graph/DeleteHandler.java > 9eb086f79c556ed01ed24628f298f7c665e4a985 > > repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java > 911b1adbad92a76ce15f49ce023b56aeca8b4f94 > > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java > 0c80aeddbae35c80cdf8b83cea6aefadf6454a20 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java > 889236ca805142a93d9d4e63789fb0cc9aea05aa > > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java > 4e55bbcab91b8572d13345cec61e8df2f195ee4f > > repository/src/main/java/org/apache/atlas/repository/graph/VertexLookupContext.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java > 35a489f2d77578a72a2e73a37bdf094af25a166e > repository/src/main/java/org/apache/atlas/util/AttributeValueMap.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/util/IndexedInstance.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/graph/GraphHelperTest.java > a7dc13db72fb4ff268312c106df1b6c41f46962f > typesystem/src/test/resources/atlas-application.properties > 108630b485e712179dc80d001dbce97551b37516 > webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java > 17c8237569746bb77f75692d50ce115c21b80c7c > > webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java > 1774611285956a7f187bd4353778fe34a6893c69 > > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java > 4a3db8874468bc8373625afe93df0a0938d00e39 > webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java > 51be64c0dccecd56e720f3633ba46bbaf5c37f5d > > webapp/src/test/java/org/apache/atlas/web/resources/DataSetLineageJerseyResourceIT.java > 8334e4f9dd2eedf48e516ee34a5b2981489ce0da > > webapp/src/test/java/org/apache/atlas/web/resources/EntityDiscoveryJerseyResourceIT.java > 2bbe10a0827b2035a6c74e6b3aa3520b7e12d571 > > webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java > f084053a03f9940d2f9015d4a15ef9c085553ae5 > > webapp/src/test/java/org/apache/atlas/web/resources/EntityV2JerseyResourceIT.java > 74338fd7aee2d81f54f59b0be15bd249852fbd0b > > webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java > b004cb52cc996763dbc4c24cd80ab545c5749358 > > Diff: https://reviews.apache.org/r/51092/diff/ > > > Testing > ------- > > Ran complete build on linux, all tests passed > > > Thanks, > > Jeff Hagelberg > >
