-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71006/
-----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Sridhar K, Madhan Neethiraj, Nixon
Rodrigues, and Sarath Subramanian.
Bugs: ATLAS-3313
https://issues.apache.org/jira/browse/ATLAS-3313
Repository: atlas
Description
-------
Currently Atlas server processes notifications from Kafka topic named
ATLAS_HOOK (name configurable name via atlas.notification.hook.topic.name). The
name of the Kafka topic to which an Atlas Hook publishes notifications can be
configured via configuration atlas.notification.hook.topic.name. However, since
Atlas server processes notifications from a single topic, all hooks
(Hive/HBase/Spark/Storm/..) are required to publish to the same topic.
To enable different hooks to publish to different Kafka topics, Atlas server
should support processing notifications from multiple Kafka topics.
Add following properties in atlas:
atlas.notification.hook.consumer.topic.names=ATLAS_HIVE_HOOK1,ATLAS_HIVE_HOOK2,ATLAS_HOOK
(comma separated without space)
atlas.notification.entities.consumer.topic.names=ATLAS_ENTITIES
Diffs
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 3ff1316
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetrics.java c011ad9
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java
5c840c3
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java
b04aba9
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
449eb6f
notification/src/test/java/org/apache/atlas/kafka/KafkaNotificationMockTest.java
263903b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java
c963830
repository/src/main/java/org/apache/atlas/util/AtlasMetricsCounter.java
d5a4412
repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java f658caa
repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java
b56019e
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
1f8e810
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java
fb3ff26
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerTest.java
3e35511
Diff: https://reviews.apache.org/r/71006/diff/1/
Testing
-------
Tested by publishing create hive table JSON using kafka-console-producer on
custom topics configured above. I was able to search the entities in atlas.
Also tested basic use case by creating database and table from hive console.
mvn clean package
Some test cases are failing on my local (will work on fixing the same).
Thanks,
Saqeeb Shaikh