[jira] [Updated] (ATLAS-3305) Unable to scale atlas kafka consumers
[
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Rempter updated ATLAS-3305:
Attachment: multiple_consumers_perf.png
> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
> Labels: performance
> Attachments: ATLAS-3305_multiple_kafka_consumers.patch,
> multiple_consumers_perf.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage
> messages and processing them just with one consumer is not enough.
>
> There is parameter atlas.notification.hook.numthreads to scale consumers in
> NotificationHookConsumer.
> But the method:
>
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>
> is always returning one element list, which effectively always starts one
> consumer
> List> consumers =
> Collections.singletonList(kafkaConsumer);
>
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={},
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers,
> autoCommitEnabled)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ATLAS-3305) Unable to scale atlas kafka consumers
[
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Rempter updated ATLAS-3305:
Attachment: ATLAS-3305_multiple_kafka_consumers.patch
> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
> Labels: performance
> Attachments: ATLAS-3305_multiple_kafka_consumers.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage
> messages and processing them just with one consumer is not enough.
>
> There is parameter atlas.notification.hook.numthreads to scale consumers in
> NotificationHookConsumer.
> But the method:
>
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>
> is always returning one element list, which effectively always starts one
> consumer
> List> consumers =
> Collections.singletonList(kafkaConsumer);
>
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={},
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers,
> autoCommitEnabled)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
