-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69576/
-----------------------------------------------------------
Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt,
Kapildeo Nayak, Mehul Parikh, Nikhil Bonte, Nixon Rodrigues, and Sarath
Subramanian.
Bugs: ATLAS-3002
https://issues.apache.org/jira/browse/ATLAS-3002
Repository: atlas
Description
-------
- added instrumentation to collect time taken for sub-tasks during entity
create/update - like walkEntityGraph, resolveReferences, preCreateOrUpdate,
mapAttributesAndClassifications, fullTextMapping, graphCommit,
entityNotification, entityAudit
- updated default atlas-log4j.xml to include loggers for METRICS
- optimization: updated Hive hook to avoid including quertText in
hive_column_lineage.name attribute value, as this can unnecessarily bloat the
message size for large queries
- optimziation: updated fullTextMapper to skip going through object-id
collections; this saves cycles for example while processing hive_table entities
with large number of columns
- here is the sample metrics:
```json
{
"walkEntityGraph": { "count": 1, "timeTaken": 1 },
"resolveReferences": { "count": 1, "timeTaken": 670 },
"findByUniqueAttributes": { "count": 217, "timeTaken": 668 },
"findByTypeAndPropertyName": { "count": 217, "timeTaken": 641 },
"findBySuperTypeAndPropertyName": { "count": 74, "timeTaken": 27 },
"createOrUpdate": { "count": 1, "timeTaken": 1637 },
"preCreateOrUpdate": { "count": 1, "timeTaken": 720 },
"mapAttributesAndClassifications": { "count": 1, "timeTaken": 90 },
"fullTextMapping": { "count": 1, "timeTaken": 195 },
"notification-getReferenceable": { "count": 1, "timeTaken": 37 },
"entityAudit": { "count": 1, "timeTaken": 31 },
"entityNotification": { "count": 2, "timeTaken": 8 },
"graphCommit": { "count": 75, "timeTaken": 315 }
}
```
Diffs
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
84f9c9704
distro/src/conf/atlas-log4j.xml a0f96296e
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
73bc31581
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java
c6a4bbeae
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
2a1881b03
repository/src/main/java/org/apache/atlas/repository/converters/AtlasInstanceConverter.java
9bde5db69
repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java
5b5158ce2
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityChangeNotifier.java
92d95f992
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java
12e8bb1f2
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
f578ded07
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
65c5d41fb
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java
e818ed52a
server-api/src/main/java/org/apache/atlas/RequestContextV1.java b85769cd4
server-api/src/main/java/org/apache/atlas/metrics/Metrics.java e0f4e4948
webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
926acfb67
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
165ea91b7
Diff: https://reviews.apache.org/r/69576/diff/1/
Testing
-------
- verified the instumentation output in metric.log
- pre-commit tests run:
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/868/
Thanks,
Madhan Neethiraj
