[
https://issues.apache.org/jira/browse/ATLAS-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313956#comment-15313956
]
Hemanth Yamijala commented on ATLAS-515:
----------------------------------------
As part of ATLAS-801, I am investigating options to fix this JIRA. (The intent
of ATLAS-801 is to minimize the possibility of data loss due to Kafka
unavailability, thus the relation with this JIRA.)
Broadly, there are two ways to do this:
* As part of Atlas server startup
* As part of Hook startup
If topics are not created up-front, the first message *produced* creates the
topic, provided {{auto.create.topics.enable}} is set to true. (Default is
true.) The number of replicas set up will be according to
{{default.replication.factor}}. (Default is 1.)
For ATLAS_HOOK, it is the hooks that produce messages. Also, I think it may be
safer to assume that hooks like Hive will be activated first rather than Atlas,
because of the relative importance of host components over Atlas. Hence, IMO,
it is important to try and create topics as part of Hook startup.
One thing that helps here is that topic creation is atomic. Hence, even if
multiple hooks try to create topics at the same time, one of them will cleanly
succeed and others fail. (I confirmed this offline with one of the Kafka
committers.)
So, the logic can roughly be:
{code}
Hook startup:
if (topicExists("ATLAS_HOOK")):
replicas = ApplicationProperties.get().get("num-replicas")
try {
createTopic("ATLAS_HOOK", numPartitions=1, replicas)
} catch (Exception e) {
log(e)
}
{code}
Same thing must be done for ATLAS_ENTITIES in the Atlas server as the server is
producer for this topic.
One issue to be aware of is that, the number of replicas configured in Atlas
configuration must be less than or equal to number of brokers available. If
not, this throws an exception and defaults would be used again. Likewise,
changes to number of replicas will not be managed once the topic is created.
This would need to be created manually using instructions like
[these|http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor],
which in general appears an involved process, hence I am reluctant to support
this from within Atlas.
The other option of creating all topics via the Atlas server would work as well
- provided hooks are not launched first. This has the advantage of running
under a more controlled environment like the Atlas server. The issue is the
restriction we are imposing that if the topic is already created, we can't
modify it.
Thoughts?
> Ability to initialize Kafka topics with more than 1 replica
> -----------------------------------------------------------
>
> Key: ATLAS-515
> URL: https://issues.apache.org/jira/browse/ATLAS-515
> Project: Atlas
> Issue Type: Sub-task
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.7-incubating
>
>
> Atlas uses two Kafka topics - ATLAS_HOOK and ATLAS_ENTITIES for communicating
> with its upstream and downstream dependencies. If not explicitly created,
> these topics are setup with a single replica. From a HA perspective, this
> becomes a problem. This JIRA is to request for a mechanism for creating
> topics with more than 1 replica.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
