[ 
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986690#comment-16986690
 ] 

Adam Rempter commented on ATLAS-3305:
-------------------------------------

[~madhan], [~sarath], could you guys have a look at patch I attached and 
suggest if we need any more improvements to it?

By default patch sets one consumer per topic - so the same like it is at the 
moment.

Maybe add more clear docs like [~bolke] suggested to ensure that admin is aware 
of how atlas will process hook messages?

 

Now Atlas supports multiple topics, so it could be another addition to improve 
performance of atlas consumer. Btw I think it relaxes consistency also...

And consistency can be guaranteed on producer side, eg:

[https://www.javaworld.com/article/3066873/big-data-messaging-with-kafka-part-2.html]

 

Thanks,
Adam

 

> Unable to scale atlas kafka consumers
> -------------------------------------
>
>                 Key: ATLAS-3305
>                 URL: https://issues.apache.org/jira/browse/ATLAS-3305
>             Project: Atlas
>          Issue Type: Bug
>          Components:  atlas-core, atlas-intg
>    Affects Versions: 1.1.0, 2.0.0
>            Reporter: Adam Rempter
>            Priority: Major
>              Labels: performance
>         Attachments: ATLAS-3305_multiple_kafka_consumers.patch, 
> multiple_consumers_perf.png
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage 
> messages and processing them just with one consumer is not enough. 
>  
> There is parameter atlas.notification.hook.numthreads to scale consumers in  
> NotificationHookConsumer.
> But the method:
>  
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>  
> is always returning one element list, which effectively always starts one 
> consumer
> List<NotificationConsumer<T>> consumers = 
> Collections.singletonList(kafkaConsumer);
>  
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={}, 
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, 
> autoCommitEnabled)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to