> On May 26, 2020, 10:45 p.m., Madhan Neethiraj wrote: > > Sid - I suggest to consider the following alternate approach to address > > this issue: > > > > public class EntityGraphMapper{ > > private AtlasEntityHeader constructHeader(AtlasEntity entity, final > > AtlasEntityType type, AtlasVertex vertex) { > > /* > > * replace current implementation with a call to > > entityRetriever.toAtlasEntityHeaderWithClassifications() > > * > > AtlasEntityHeader header = new > > AtlasEntityHeader(entity.getTypeName()); > > > > header.setGuid(getIdFromVertex(vertex)); > > header.setStatus(entity.getStatus()); > > header.setIsIncomplete(entity.getIsIncomplete()); > > > > for (AtlasAttribute attribute : type.getUniqAttributes().values()) { > > header.setAttribute(attribute.getName(), > > entity.getAttribute(attribute.getName())); > > } > > > > return header; > > * > > */ > > > > return > > entityRetriever.toAtlasEntityHeaderWithClassifications(vertex); > > } > > } > > > > public class EntityMutationResponse { > > JsonIgnore > > public void addEntity(EntityOperation op, AtlasEntityHeader header) { > > // if an entity is already included in CREATE, update the header, > > to capture propagated classifications > > if (op == EntityOperation.UPDATE || op == > > EntityOperation.PARTIAL_UPDATE) { > > if (entityHeaderExists(getCreatedEntities(), header.getGuid())) > > { > > // return; > > op = EntityOperaton.CREATE; > > } > > } > > > > ... > > } > > }
As you suggested I have changed the code. The issue is, we cache the entity at repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java:285 and 313 which is not getting updated with classification. At org.apache.atlas.repository.store.graph.v2.AtlasEntityChangeNotifier#notifyPropagatedEntities when we call fullTextMapperV2.getAndCacheEntity(guid) we still get the old one without classification. So along with this change I added code to constructHeader so that it will set classification to entity from entityHeader. Now the updated solution works. - Sidharth ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72540/#review220873 ----------------------------------------------------------- On May 28, 2020, 10:58 p.m., Sidharth Mishra wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72540/ > ----------------------------------------------------------- > > (Updated May 28, 2020, 10:58 p.m.) > > > Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath > Subramanian. > > > Bugs: ATLAS-3806 > https://issues.apache.org/jira/browse/ATLAS-3806 > > > Repository: atlas > > > Description > ------- > > Classifications information missing in notification events during entity > create/update > > > Diffs > ----- > > > intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java > d83f0e0f1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2ed524ff5 > > webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java > a677b315c > > > Diff: https://reviews.apache.org/r/72540/diff/2/ > > > Testing > ------- > > For add new ctas notification looks like (Just shown for ctas entity): > { > "version": { > "version": "1.0.0", > "versionParts": [ > 1 > ] > }, > "msgCompressionKind": "NONE", > "msgSplitIdx": 1, > "msgSplitCount": 1, > "msgSourceIP": "172.27.168.128", > "msgCreatedBy": "", > "msgCreationTime": 1590176023734, > "message": { > "type": "ENTITY_NOTIFICATION_V2", > "entity": { > "typeName": "hive_table", > "attributes": { > "owner": "anonymous", > "createTime": 1590176022000, > "qualifiedName": "sid_hive_db.sid_hive_table_ctas7@bulk", > "name": "sid_hive_table_ctas7" > }, > "guid": "9a0f46e1-99ad-4b1d-8818-ee60778b5619", > "status": "ACTIVE", > "displayText": "sid_hive_table_ctas7", > "classificationNames": [ > "ETL", > "PII" > ], > "classifications": [ > { > "typeName": "ETL", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > }, > { > "typeName": "PII", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > } > ] > }, > "operationType": "CLASSIFICATION_ADD", > "eventTime": 1590176022456 > } > } > > > Add new tag: > --------------- > { > "version": { > "version": "1.0.0", > "versionParts": [ > 1 > ] > }, > "msgCompressionKind": "NONE", > "msgSplitIdx": 1, > "msgSplitCount": 1, > "msgSourceIP": "172.27.168.128", > "msgCreatedBy": "", > "msgCreationTime": 1590181680675, > "message": { > "type": "ENTITY_NOTIFICATION_V2", > "entity": { > "typeName": "hive_table", > "attributes": { > "owner": "anonymous", > "createTime": 1590048670000, > "qualifiedName": "sid_hive_db.sid_hive_table_1@bulk", > "name": "sid_hive_table_1" > }, > "guid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "status": "ACTIVE", > "displayText": "sid_hive_table_1", > "classificationNames": [ > "ETL", > "PII", > "Fact" > ], > "classifications": [ > { > "typeName": "ETL", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > }, > { > "typeName": "PII", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > }, > { > "typeName": "Fact", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > } > ] > }, > "operationType": "CLASSIFICATION_ADD", > "eventTime": 1590181673717 > } > } > { > "version": { > "version": "1.0.0", > "versionParts": [ > 1 > ] > }, > "msgCompressionKind": "NONE", > "msgSplitIdx": 1, > "msgSplitCount": 1, > "msgSourceIP": "172.27.168.128", > "msgCreatedBy": "", > "msgCreationTime": 1590181680710, > "message": { > "type": "ENTITY_NOTIFICATION_V2", > "entity": { > "typeName": "hive_process", > "attributes": { > "qualifiedName": > "sid_hive_db.sid_hive_table_ctas3@bulk:1590120919000", > "name": "create table sid_hive_table_ctas3 as select * from > sid_hive_table_1" > }, > "guid": "36cc1a73-9c55-4a66-9d59-fa1a2f535791", > "status": "ACTIVE", > "displayText": "create table sid_hive_table_ctas3 as select * > from sid_hive_table_1", > "classificationNames": [ > "ETL", > "PII", > "Fact" > ], > "classifications": [ > { > "typeName": "ETL", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > }, > { > "typeName": "PII", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > }, > { > "typeName": "Fact", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > } > ] > }, > "operationType": "CLASSIFICATION_ADD", > "eventTime": 1590181673717 > } > } > > { > "version": { > "version": "1.0.0", > "versionParts": [ > 1 > ] > }, > "msgCompressionKind": "NONE", > "msgSplitIdx": 1, > "msgSplitCount": 1, > "msgSourceIP": "172.27.168.128", > "msgCreatedBy": "", > "msgCreationTime": 1590181680714, > "message": { > "type": "ENTITY_NOTIFICATION_V2", > "entity": { > "typeName": "hive_table", > "attributes": { > "owner": "anonymous", > "createTime": 1590120919000, > "qualifiedName": "sid_hive_db.sid_hive_table_ctas3@bulk", > "name": "sid_hive_table_ctas3" > }, > "guid": "ab302745-a843-4164-8166-017f85c7e15e", > "status": "ACTIVE", > "displayText": "sid_hive_table_ctas3", > "classificationNames": [ > "ETL", > "PII", > "Fact" > ], > "classifications": [ > { > "typeName": "ETL", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > }, > { > "typeName": "PII", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > }, > { > "typeName": "Fact", > "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811", > "entityStatus": "ACTIVE", > "propagate": true, > "validityPeriods": [], > "removePropagationsOnEntityDelete": false > } > ] > }, > "operationType": "CLASSIFICATION_ADD", > "eventTime": 1590181673717 > } > } > > > Thanks, > > Sidharth Mishra > >