> On May 26, 2020, 10:45 p.m., Madhan Neethiraj wrote:
> > Sid - I suggest to consider the following alternate approach to address
> > this issue:
> >
> > public class EntityGraphMapper{
> > private AtlasEntityHeader constructHeader(AtlasEntity entity, final
> > AtlasEntityType type, AtlasVertex vertex) {
> > /*
> > * replace current implementation with a call to
> > entityRetriever.toAtlasEntityHeaderWithClassifications()
> > *
> > AtlasEntityHeader header = new
> > AtlasEntityHeader(entity.getTypeName());
> >
> > header.setGuid(getIdFromVertex(vertex));
> > header.setStatus(entity.getStatus());
> > header.setIsIncomplete(entity.getIsIncomplete());
> >
> > for (AtlasAttribute attribute : type.getUniqAttributes().values()) {
> > header.setAttribute(attribute.getName(),
> > entity.getAttribute(attribute.getName()));
> > }
> >
> > return header;
> > *
> > */
> >
> > return
> > entityRetriever.toAtlasEntityHeaderWithClassifications(vertex);
> > }
> > }
> >
> > public class EntityMutationResponse {
> > JsonIgnore
> > public void addEntity(EntityOperation op, AtlasEntityHeader header) {
> > // if an entity is already included in CREATE, update the header,
> > to capture propagated classifications
> > if (op == EntityOperation.UPDATE || op ==
> > EntityOperation.PARTIAL_UPDATE) {
> > if (entityHeaderExists(getCreatedEntities(), header.getGuid()))
> > {
> > // return;
> > op = EntityOperaton.CREATE;
> > }
> > }
> >
> > ...
> > }
> > }
As you suggested I have changed the code. The issue is, we cache the entity at
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java:285
and 313 which is not getting updated with classification. At
org.apache.atlas.repository.store.graph.v2.AtlasEntityChangeNotifier#notifyPropagatedEntities
when we call fullTextMapperV2.getAndCacheEntity(guid) we still get the old one
without classification. So along with this change I added code to
constructHeader so that it will set classification to entity from entityHeader.
Now the updated solution works.
- Sidharth
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72540/#review220873
-----------------------------------------------------------
On May 28, 2020, 10:58 p.m., Sidharth Mishra wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72540/
> -----------------------------------------------------------
>
> (Updated May 28, 2020, 10:58 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath
> Subramanian.
>
>
> Bugs: ATLAS-3806
> https://issues.apache.org/jira/browse/ATLAS-3806
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Classifications information missing in notification events during entity
> create/update
>
>
> Diffs
> -----
>
>
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
> d83f0e0f1
>
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
> 2ed524ff5
>
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
> a677b315c
>
>
> Diff: https://reviews.apache.org/r/72540/diff/2/
>
>
> Testing
> -------
>
> For add new ctas notification looks like (Just shown for ctas entity):
> {
> "version": {
> "version": "1.0.0",
> "versionParts": [
> 1
> ]
> },
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.168.128",
> "msgCreatedBy": "",
> "msgCreationTime": 1590176023734,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes": {
> "owner": "anonymous",
> "createTime": 1590176022000,
> "qualifiedName": "sid_hive_db.sid_hive_table_ctas7@bulk",
> "name": "sid_hive_table_ctas7"
> },
> "guid": "9a0f46e1-99ad-4b1d-8818-ee60778b5619",
> "status": "ACTIVE",
> "displayText": "sid_hive_table_ctas7",
> "classificationNames": [
> "ETL",
> "PII"
> ],
> "classifications": [
> {
> "typeName": "ETL",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> },
> {
> "typeName": "PII",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> }
> ]
> },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1590176022456
> }
> }
>
>
> Add new tag:
> ---------------
> {
> "version": {
> "version": "1.0.0",
> "versionParts": [
> 1
> ]
> },
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.168.128",
> "msgCreatedBy": "",
> "msgCreationTime": 1590181680675,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes": {
> "owner": "anonymous",
> "createTime": 1590048670000,
> "qualifiedName": "sid_hive_db.sid_hive_table_1@bulk",
> "name": "sid_hive_table_1"
> },
> "guid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "status": "ACTIVE",
> "displayText": "sid_hive_table_1",
> "classificationNames": [
> "ETL",
> "PII",
> "Fact"
> ],
> "classifications": [
> {
> "typeName": "ETL",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> },
> {
> "typeName": "PII",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> },
> {
> "typeName": "Fact",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> }
> ]
> },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1590181673717
> }
> }
> {
> "version": {
> "version": "1.0.0",
> "versionParts": [
> 1
> ]
> },
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.168.128",
> "msgCreatedBy": "",
> "msgCreationTime": 1590181680710,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_process",
> "attributes": {
> "qualifiedName":
> "sid_hive_db.sid_hive_table_ctas3@bulk:1590120919000",
> "name": "create table sid_hive_table_ctas3 as select * from
> sid_hive_table_1"
> },
> "guid": "36cc1a73-9c55-4a66-9d59-fa1a2f535791",
> "status": "ACTIVE",
> "displayText": "create table sid_hive_table_ctas3 as select *
> from sid_hive_table_1",
> "classificationNames": [
> "ETL",
> "PII",
> "Fact"
> ],
> "classifications": [
> {
> "typeName": "ETL",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> },
> {
> "typeName": "PII",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> },
> {
> "typeName": "Fact",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> }
> ]
> },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1590181673717
> }
> }
>
> {
> "version": {
> "version": "1.0.0",
> "versionParts": [
> 1
> ]
> },
> "msgCompressionKind": "NONE",
> "msgSplitIdx": 1,
> "msgSplitCount": 1,
> "msgSourceIP": "172.27.168.128",
> "msgCreatedBy": "",
> "msgCreationTime": 1590181680714,
> "message": {
> "type": "ENTITY_NOTIFICATION_V2",
> "entity": {
> "typeName": "hive_table",
> "attributes": {
> "owner": "anonymous",
> "createTime": 1590120919000,
> "qualifiedName": "sid_hive_db.sid_hive_table_ctas3@bulk",
> "name": "sid_hive_table_ctas3"
> },
> "guid": "ab302745-a843-4164-8166-017f85c7e15e",
> "status": "ACTIVE",
> "displayText": "sid_hive_table_ctas3",
> "classificationNames": [
> "ETL",
> "PII",
> "Fact"
> ],
> "classifications": [
> {
> "typeName": "ETL",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> },
> {
> "typeName": "PII",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> },
> {
> "typeName": "Fact",
> "entityGuid": "410cb0f3-6df7-4523-ba23-8cd65752a811",
> "entityStatus": "ACTIVE",
> "propagate": true,
> "validityPeriods": [],
> "removePropagationsOnEntityDelete": false
> }
> ]
> },
> "operationType": "CLASSIFICATION_ADD",
> "eventTime": 1590181673717
> }
> }
>
>
> Thanks,
>
> Sidharth Mishra
>
>
