----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69587/#review211414 -----------------------------------------------------------
Ship it! Current implementation provides lot of value in getting metrics per request. The numbers are cumulative. If same operation is called then the time taken becomes additive. Per request time can be calculated using: timeTaken/count. While this is helpful, it would be worthwhile having per request metrics as well. This will help in understanding call to call variations. - Ashutosh Mestry On Dec. 18, 2018, 7:49 p.m., Madhan Neethiraj wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69587/ > ----------------------------------------------------------- > > (Updated Dec. 18, 2018, 7:49 p.m.) > > > Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt, > Kapildeo Nayak, Mehul Parikh, Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3002 > https://issues.apache.org/jira/browse/ATLAS-3002 > > > Repository: atlas > > > Description > ------- > > - added instrumentation to collect time taken for sub-tasks during entity > create/update - like walkEntityGraph, resolveReferences, preCreateOrUpdate, > mapAttributesAndClassifications, fullTextMapping, graphCommit, > entityNotification, entityAudit > - updated default atlas-log4j.xml to include loggers for METRICS > - optimization: updated Hive hook to avoid including quertText in > hive_column_lineage.name attribute value, as this can unnecessarily bloat the > message size for large queries > - optimziation: updated fullTextMapper to skip going through object-id > collections; this saves cycles for example while processing hive_table > entities with large number of columns > - here is the sample metrics: > ```json > { > "walkEntityGraph": { "count": 1, "timeTaken": 1 }, > "resolveReferences": { "count": 1, "timeTaken": 670 }, > "findByUniqueAttributes": { "count": 217, "timeTaken": 668 }, > "findByTypeAndPropertyName": { "count": 217, "timeTaken": 641 }, > "findBySuperTypeAndPropertyName": { "count": 74, "timeTaken": 27 }, > "createOrUpdate": { "count": 1, "timeTaken": 1637 }, > "preCreateOrUpdate": { "count": 1, "timeTaken": 720 }, > "mapAttributesAndClassifications": { "count": 1, "timeTaken": 90 }, > "fullTextMapping": { "count": 1, "timeTaken": 195 }, > "notification-getReferenceable": { "count": 1, "timeTaken": 37 }, > "entityAudit": { "count": 1, "timeTaken": 31 }, > "entityNotification": { "count": 2, "timeTaken": 8 }, > "graphCommit": { "count": 75, "timeTaken": 315 } > } > ``` > > > Diffs > ----- > > > addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java > 2ccfff4d1 > common/src/main/java/org/apache/atlas/utils/AtlasPerfMetrics.java > PRE-CREATION > distro/src/conf/atlas-log4j.xml c183871eb > notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java > 4bec91709 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > 4c436779e > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java > dfacb3817 > > repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java > 8ca8c9a0b > > repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java > 08ccd9c73 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java > a8c3363d5 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java > 6580beecd > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > d97b74d9d > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java > 25770a334 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 6c830bafe > server-api/src/main/java/org/apache/atlas/RequestContext.java 099d713f6 > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > e0a60a133 > > webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java > b5e7ed871 > > webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java > b95594831 > > > Diff: https://reviews.apache.org/r/69587/diff/1/ > > > Testing > ------- > > - verified the instumentation output in metric.log > - pre-commit tests run: > https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/869 > > > Thanks, > > Madhan Neethiraj > >
