Adam Rempter created ATLAS-3305:
-----------------------------------
Summary: Unable to scale atlas kafka consumers
Key: ATLAS-3305
URL: https://issues.apache.org/jira/browse/ATLAS-3305
Project: Atlas
Issue Type: Bug
Components: atlas-core, atlas-intg
Affects Versions: 2.0.0, 1.1.0
Reporter: Adam Rempter
We wanted to scale kafka consumers for atlas, as we are getting many lineage
messages and processing them just with one consumer is not enough.
There is parameter atlas.notification.hook.numthreads to scale consumers in
NotificationHookConsumer.
But the method:
notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
is always returning one element list, which effectively always starts one
consumer
List<NotificationConsumer<T>> consumers =
Collections.singletonList(kafkaConsumer);
Log incorrectly says that nuber of consumers has been created:
LOG.info("<== KafkaNotification.createConsumers(notificationType={},
numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers,
autoCommitEnabled)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
