[
https://issues.apache.org/jira/browse/ATLAS-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Yamijala updated ATLAS-515:
-----------------------------------
Attachment: ATLAS-515.patch
The attached patch is a more complete version of the POC.
Couple of points to call out:
* The topic creation is protected with a flag, by default turned off for
backwards compatibility.
* It introduces configuration parameters for both {{ATLAS_HOOK}} and
{{ATLAS_ENTITIES}} separately for number of replicas. I kept them separate to
provide more flexibility.
* I take a very failsafe route to create the topics. For {{ATLAS_HOOK}}, I do
it both from the producer (Hook) and consumer (Atlas Server) - whichever comes
first. For {{ATLAS_ENTITIES}}, since the server is the producer, it can be
safely done only from there.
* This relies on the fact that topic creation is atomic and if concurrent calls
happen for creation of same topic, one of them would fail. All exceptions are
handled by the code and logged.
* There is one particular scenario on secure Kafka that needs further
explanation, that is not handled in this patch. I will comment on that
separately.
> Ability to initialize Kafka topics with more than 1 replica
> -----------------------------------------------------------
>
> Key: ATLAS-515
> URL: https://issues.apache.org/jira/browse/ATLAS-515
> Project: Atlas
> Issue Type: Sub-task
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.7-incubating
>
> Attachments: ATLAS-515-poc.patch, ATLAS-515.patch
>
>
> Atlas uses two Kafka topics - ATLAS_HOOK and ATLAS_ENTITIES for communicating
> with its upstream and downstream dependencies. If not explicitly created,
> these topics are setup with a single replica. From a HA perspective, this
> becomes a problem. This JIRA is to request for a mechanism for creating
> topics with more than 1 replica.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
