Jeff, I get following test failures while running “mvn clean package” with this patch. Can you please look into these?
I am reviewing the patch. Optimizations look good; I will post my comments shortly. Thanks, Madhan Results : Failed tests: org.apache.atlas.discovery.DataSetLineageServiceTest.testGetSchema(org.apache.atlas.discovery.DataSetLineageServiceTest) Run 1: DataSetLineageServiceTest.testGetSchema:288->assertColumn:313 expected:<hive_column_v1> but was:<hive_column> Run 2: DataSetLineageServiceTest.testGetSchema:288->assertColumn:313 expected:<hive_column_v1> but was:<hive_column> Run 3: DataSetLineageServiceTest.testGetSchema:288->assertColumn:313 expected:<hive_column_v1> but was:<hive_column> Run 4: DataSetLineageServiceTest.testGetSchema:288->assertColumn:313 expected:<hive_column_v1> but was:<hive_column> org.apache.atlas.discovery.DataSetLineageServiceTest.testGetSchemaForEntity(org.apache.atlas.discovery.DataSetLineageServiceTest) Run 1: DataSetLineageServiceTest.testGetSchemaForEntity:305->assertColumn:313 expected:<hive_column_v1> but was:<hive_column> Run 2: DataSetLineageServiceTest.testGetSchemaForEntity:305->assertColumn:313 expected:<hive_column_v1> but was:<hive_column> Run 3: DataSetLineageServiceTest.testGetSchemaForEntity:305->assertColumn:313 expected:<hive_column_v1> but was:<hive_column> Run 4: DataSetLineageServiceTest.testGetSchemaForEntity:305->assertColumn:313 expected:<hive_column_v1> but was:<hive_column> org.apache.atlas.discovery.DataSetLineageServiceTest.testSearchByDSLQueries(org.apache.atlas.discovery.DataSetLineageServiceTest) Run 1: PASS Run 2: PASS Run 3: PASS Run 4: PASS Run 5: PASS Run 6: PASS Run 7: PASS Run 8: PASS Run 9: PASS Run 10: PASS Run 11: PASS Run 12: PASS Run 13: DataSetLineageServiceTest.testSearchByDSLQueries:123 » Discovery Invalid expre... Run 14: DataSetLineageServiceTest.testSearchByDSLQueries:123 » Discovery Invalid expre... Run 15: DataSetLineageServiceTest.testSearchByDSLQueries:123 » Discovery Invalid expre... Run 16: DataSetLineageServiceTest.testSearchByDSLQueries:123 » Discovery Invalid expre... Run 17: PASS Run 18: PASS Run 19: PASS Run 20: PASS Run 21: PASS Run 22: PASS Run 23: PASS Run 24: PASS Run 25: PASS Run 26: PASS Run 27: PASS Tests run: 584, Failures: 3, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Apache Atlas Server Build Tools .................... SUCCESS [ 0.683 s] [INFO] apache-atlas ....................................... SUCCESS [ 2.473 s] [INFO] Apache Atlas Integration ........................... SUCCESS [ 17.953 s] [INFO] Apache Atlas Common ................................ SUCCESS [ 3.162 s] [INFO] Apache Atlas Typesystem ............................ SUCCESS [ 45.467 s] [INFO] Apache Atlas Client ................................ SUCCESS [ 4.616 s] [INFO] Apache Atlas Server API ............................ SUCCESS [ 1.181 s] [INFO] Apache Atlas Notification .......................... SUCCESS [ 20.127 s] [INFO] Apache Atlas Graph Database Projects ............... SUCCESS [ 0.097 s] [INFO] Apache Atlas Graph Database API .................... SUCCESS [ 0.535 s] [INFO] Graph Database Common Code ......................... SUCCESS [ 0.472 s] [INFO] Shaded version of Apache hbase client .............. SUCCESS [ 6.432 s] [INFO] Apache Atlas Titan 0.5.4 Graph DB Impl ............. SUCCESS [01:16 min] [INFO] Apache Atlas Graph Database Implementation Dependencies SUCCESS [ 0.178 s] [INFO] Shaded version of Apache hbase server .............. SUCCESS [ 14.259 s] [INFO] Apache Atlas Repository ............................ FAILURE [08:51 min] [INFO] Apache Atlas Authorization ......................... SKIPPED On 1/24/17, 2:08 PM, "Jeff Hagelberg" <[email protected] on behalf of [email protected]> wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51092/#review162859 ----------------------------------------------------------- Specifically, the changes in webapp (and the other changes added in diff #5) fix the following test failures that I have been seeing consistently with my changes. I ended up applying them to the latest Atlas first, to ensure they passed there, and then merging them in with these changes. EntityJerseyResouErceIT.testCompleteUpdate:739 » AtlasService Metadata service ... EntityJerseyResourceIT.testEntityDeduping:197 expected:<1> but was:<0> org.apache.atlas.web.resources.EntityJerseyResourceIT.testEntityInvalidValue(org.apache.atlas.web.resources.EntityJerseyResourceIT) Run 1: EntityJerseyResourceIT.testEntityInvalidValue:280 expected:<Bad Request> but was:<Internal Server Error> Run 2: EntityJerseyResourceIT.testEntityInvalidValue:280 expected:<Bad Request> but was:<Internal Server Error> EntityJerseyResourceIT.testPartialUpdate:673 » AtlasService Metadata service A... ntityJerseyResourceIT.testUTF8:640->BaseResourceIT.createType:177->BaseResourceIT.createType:182 » AtlasService EntityV2JerseyResourceIT.testAddNullPropertyValue:274 NullPointer EntityV2JerseyResourceIT.testDeleteEntities:643->BaseResourceIT.createEntity:238->BaseResourceIT.modifyEntity:228 expected object to not be null EntityV2JerseyResourceIT.testDeleteEntityByUniqAttribute:666->createHiveDB:368->BaseResourceIT.createEntity:238->BaseResourceIT.modifyEntity:228 expected object to not be null EntityV2JerseyResourceIT.testGetEntityByAttribute:222 » AtlasService Metadata ... EntityV2JerseyResourceIT.testSubmitEntity:102->createDBAndTable:378->createHiveDB:368->BaseResourceIT.createEntity:238->BaseResourceIT.modifyEntity:228 expected object to not be null EntityV2JerseyResourceIT.testSubmitEntityWithBadDateFormat:230->BaseResourceIT.createEntity:238->BaseResourceIT.modifyEntity:228 expected object to not be null EntityV2JerseyResourceIT.testUTF8:552->BaseResourceIT.createEntity:238->BaseResourceIT.modifyEntity:228 expected object to not be null NotificationHookConsumerIT.testDeleteByQualifiedName:187->BaseResourceIT.waitFor:578 » NotificationHookConsumerIT.testUpdateEntityFullUpdate:216->BaseResourceIT.waitFor:578 » NotificationHookConsumerIT.testUpdateEntityPartial:130->BaseResourceIT.waitFor:573 » AtlasService NotificationHookConsumerIT.testUpdatePartialUpdatingQualifiedName:160->BaseResourceIT.waitFor:578 » DataSetLineageJerseyResourceIT.testSchema:152 » JSON JSONObject["type"] not fo... DataSetLineageJerseyResourceIT.testSchemaForEntity:171 » JSON JSONObject["type... EntityLineageJerseyResourceIT>DataSetLineageJerseyResourceIT.testSchema:152 » JSON EntityLineageJerseyResourceIT>DataSetLineageJerseyResourceIT.testSchemaForEntity:171 » JSON EntityDiscoveryJerseyResourceIT.testSearchByDSL:66 » AtlasService Metadata ser... EntityDiscoveryJerseyResourceIT.testSearchDSLLimits:87 » AtlasService Metadata... EntityDiscoveryJerseyResourceIT.testSearchUsingDSL:128 » AtlasService Metadata.. EntityV2JerseyResourceIT.testEntityDeduping:119 expected:<1> but was:<0> - Jeff Hagelberg On Jan. 24, 2017, 9:58 p.m., Jeff Hagelberg wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51092/ > ----------------------------------------------------------- > > (Updated Jan. 24, 2017, 9:58 p.m.) > > > Review request for atlas and David Kantor. > > > Repository: atlas > > > Description > ------- > > Apply performance fixes for create/update entities from IBM fork to Atlas. During our performance profiling, we found a number of performance hotspots in JProfiler. Our main findings were > > - multiple queries were being executed for each instance being created/updated to find matches by unique attribute. > - one query was being executed for each instance being created/updated to find the corresponding vertex if there is one > - Calculating the value of the full text property was taking a significant portion of the time to create/update entities, mainly due to its calls to getVertexForGUID > > The changes we put in do the following: > > - batch lookups by guid when create/update entities. Execute one AtlasGraphQuery to find them all. > - batch lookups by unique attribute when create/update entities. Execute one AtlasGraphQuery per class to find unique attribute matches. > - find all existing vertices up front during create/update entity. Use those vertices during the graph mapping process to avoid running unnecessary graph queries > - reuse reference vertices from instance to graph mapping when computing full text property > > Also, resolved all test failures in webapp. I disentagled the three competing versions of the hive model that the various tests were trying to use. Now they all pass. I tried to follow the path of least resistence. We really should clean this up more, there is really no need for threee different versions of hive_table and its related classes. > > > Diffs > ----- > > repository/src/main/java/org/apache/atlas/repository/audit/InMemoryEntityAuditRepository.java 50a007bf1bf7e8c07a57b327b3aa3b907dd5f660 > repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapper.java 911b1adbad92a76ce15f49ce023b56aeca8b4f94 > repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedMetadataRepository.java 0c80aeddbae35c80cdf8b83cea6aefadf6454a20 > repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 889236ca805142a93d9d4e63789fb0cc9aea05aa > repository/src/main/java/org/apache/atlas/repository/graph/TypedInstanceToGraphMapper.java 4e55bbcab91b8572d13345cec61e8df2f195ee4f > repository/src/main/java/org/apache/atlas/repository/graph/VertexLookupContext.java PRE-CREATION > repository/src/main/java/org/apache/atlas/services/DefaultMetadataService.java 35a489f2d77578a72a2e73a37bdf094af25a166e > repository/src/main/java/org/apache/atlas/util/AttributeValueMap.java PRE-CREATION > repository/src/main/java/org/apache/atlas/util/IndexedInstance.java PRE-CREATION > repository/src/test/java/org/apache/atlas/discovery/DataSetLineageServiceTest.java a0ee26c34517a441cfb7513d98852bd8d13ecca9 > repository/src/test/java/org/apache/atlas/repository/graph/GraphHelperTest.java a7dc13db72fb4ff268312c106df1b6c41f46962f > typesystem/src/test/resources/atlas-application.properties 108630b485e712179dc80d001dbce97551b37516 > webapp/src/main/java/org/apache/atlas/web/resources/EntityResource.java 17c8237569746bb77f75692d50ce115c21b80c7c > webapp/src/test/java/org/apache/atlas/notification/EntityNotificationIT.java 1774611285956a7f187bd4353778fe34a6893c69 > webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerIT.java 4a3db8874468bc8373625afe93df0a0938d00e39 > webapp/src/test/java/org/apache/atlas/web/resources/BaseResourceIT.java 51be64c0dccecd56e720f3633ba46bbaf5c37f5d > webapp/src/test/java/org/apache/atlas/web/resources/DataSetLineageJerseyResourceIT.java 8334e4f9dd2eedf48e516ee34a5b2981489ce0da > webapp/src/test/java/org/apache/atlas/web/resources/EntityDiscoveryJerseyResourceIT.java 2bbe10a0827b2035a6c74e6b3aa3520b7e12d571 > webapp/src/test/java/org/apache/atlas/web/resources/EntityJerseyResourceIT.java f084053a03f9940d2f9015d4a15ef9c085553ae5 > webapp/src/test/java/org/apache/atlas/web/resources/EntityV2JerseyResourceIT.java 74338fd7aee2d81f54f59b0be15bd249852fbd0b > webapp/src/test/java/org/apache/atlas/web/resources/MetadataDiscoveryJerseyResourceIT.java b004cb52cc996763dbc4c24cd80ab545c5749358 > > Diff: https://reviews.apache.org/r/51092/diff/ > > > Testing > ------- > > Ran complete build on linux, all tests passed > > > Thanks, > > Jeff Hagelberg > >
