-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
-----------------------------------------------------------

Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon 
Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3313
    https://issues.apache.org/jira/browse/ATLAS-3313


Repository: atlas


Description
-------

Currently Atlas server processes notifications from Kafka topic named 
ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The 
name of the Kafka topic to which an Atlas Hook publishes notifications can be 
configured via configuration atlas.notification.hook.topic.name. However, since 
Atlas server processes notifications from a single topic, all hooks 
(Hive/HBase/Spark/Storm/..) are required to publish to the same topic.

To enable different hooks to publish to different Kafka topics, Atlas server 
should support processing notifications from multiple Kafka topics.


Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
 (comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES


Diffs
-----

  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316 
  intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
5c840c3 
  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
b04aba9 
  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
449eb6f 
  
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
 263903b 
  
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
 c963830 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java 
d5a4412 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa 
  repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
b56019e 
  
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
 1f8e810 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
 fb3ff26 
  
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
 3e35511 


Diff: https://reviews.apache.org/r/71006/diff/1/


Testing
-------

Tested by publishing create hive table JSON using kafka-console-producer on 
custom topics configured above. I was able to search the entities in atlas. 
Also tested basic use case by creating database and table from hive console. 

mvn clean package 
Some test cases are failing on my local (will work on fixing the same).


Thanks,

Saqeeb Shaikh

Reply via email to