Deep Singh created ATLAS-4335:
---------------------------------
Summary: Hook Notifications through Rest Interface
Key: ATLAS-4335
URL: https://issues.apache.org/jira/browse/ATLAS-4335
Project: Atlas
Issue Type: New Feature
Components: atlas-core
Reporter: Deep Singh
Assignee: Deep Singh
In Data-lake <-> Workload-cluster scenario, versions of the services available
on clusters are not always the same. Especially when Kafka versions are
different on the clusters, many a time the Kafka client (on workload-cluster)
becomes incompatible with the Kafka broker (on data-lake). This leads to
failure in the delivery of Atlas Hook notification as the mechanism is
dependent on the Kafka client's capability to deliver messages.
To resolve such an issue there is no other way but to make Kafka client
compatible with the broker by upgrading or degrading the client or the broker.
Often this is not straight-fwd and the easiest thing to do.
We need another mechanism independent of the Kafka client to deliver hook
notifications to the Atlas consumer.
There are various REST API available today to perform CURD on Atlas entities.
Those APIs process notifications in real-time as they arrive, since event
processing is slow and time-consuming these APIs cannot give the throughput
which hooks require.
We need another REST interface that is lightweight and does not process
messages as they arrive instead, it could dump the messages to the local kafka.
Since Kafka will be available locally in the cluster, the interface will never
face the problem of version mismatch of client-broker, as faced by service
hooks running on a different cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
