[
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986690#comment-16986690
]
Adam Rempter commented on ATLAS-3305:
-------------------------------------
[~madhan], [~sarath], could you guys have a look at patch I attached and
suggest if we need any more improvements to it?
By default patch sets one consumer per topic - so the same like it is at the
moment.
Maybe add more clear docs like [~bolke] suggested to ensure that admin is aware
of how atlas will process hook messages?
Now Atlas supports multiple topics, so it could be another addition to improve
performance of atlas consumer. Btw I think it relaxes consistency also...
And consistency can be guaranteed on producer side, eg:
[https://www.javaworld.com/article/3066873/big-data-messaging-with-kafka-part-2.html]
Thanks,
Adam
> Unable to scale atlas kafka consumers
> -------------------------------------
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-intg
> Affects Versions: 1.1.0, 2.0.0
> Reporter: Adam Rempter
> Priority: Major
> Labels: performance
> Attachments: ATLAS-3305_multiple_kafka_consumers.patch,
> multiple_consumers_perf.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage
> messages and processing them just with one consumer is not enough.
>
> There is parameter atlas.notification.hook.numthreads to scale consumers in
> NotificationHookConsumer.
> But the method:
>
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>
> is always returning one element list, which effectively always starts one
> consumer
> List<NotificationConsumer<T>> consumers =
> Collections.singletonList(kafkaConsumer);
>
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={},
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers,
> autoCommitEnabled)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
