[
https://issues.apache.org/jira/browse/ATLAS-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988893#comment-14988893
]
Shwetha G S commented on ATLAS-158:
-----------------------------------
Attaching the patch here as I won't have permission to update the patch on
review board. Changes done:
1. Removed dependency of repository module in notification. Moved listeners to
common module
2. Added atlas.kafka.hook.group.id in distro/src/conf/application.properties -
required for atlas deployment. The consumer group id changed to 'atlas' as the
value should represent the consumer so that any debugging from kafka end can
relate to the actual consumer. Also, removed the property
atlas.kafka.entities.group.id to force users to set this value
3. Replaced TraitInfo with IStruct
4. Changed EntityNotification interface. I think the new interface makes sense
for the following reasons:
a. The entity returned IReferenceableInstance already has getTraits(). So,
EntityNotification.getAllTraits() can represent the flattened traits(including
super types)
b. Traits as map doesn't work - Assume traits t1, t2, t3. traits t2 and t3
extend t1. If traits t2 and t3 are associated with the entity, getAllTraits()
should return t1, t2, t1, t3. So, it has to be a list and not map
c. For trait_add/delete operation, the traits added/deleted won't be a
single trait, it will be multiple traits because of inheritance. And just the
trait name is not enough as the same super trait may be shared across multiple
traits. In general, its better to just give the whole entity and all its latest
traits even in case of entity/trait add/update/delete. This will work even for
ranger as ranger maintains resource-tag mapping and they can just replace all
entity mappings.
5. Deleted AbstractNotificationConsumerTest and EntityNotificationImplTest as
they weren't compiling with the new changes. They didn't really test anything
much. If you want, you can add them back
The patch compiles and all the tests pass except for
NotificationHookConsumerIT. Looks like kafka doesn't stop/start clean. Pending
are:
1. Debug NotificationHookConsumerIT failure
2. Add UT for EntityNotificationImpl.getAllTraits()
3. There are no tests at all for the end to end flow of entity notification.
Please add ITs for all operation types
4. Add documentation
I think without the ITs, its risky to commit the patch
2.
> Provide Atlas Entity Change Notification
> ----------------------------------------
>
> Key: ATLAS-158
> URL: https://issues.apache.org/jira/browse/ATLAS-158
> Project: Atlas
> Issue Type: Task
> Reporter: Tom Beerbower
> Assignee: Tom Beerbower
> Fix For: 0.6-incubating
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
