[
https://issues.apache.org/jira/browse/SAMZA-42?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942569#comment-13942569
]
Martin Kleppmann commented on SAMZA-42:
---------------------------------------
I noticed that KafkaCheckpointManager checks whether it is responsible for
partition 0, and only creates the topic if that is the case. Using the fact
that partition 0 always exists and is always assigned to exactly one container,
that provides a simple way of performing a startup task in just one container.
Is that approach sufficient, or are there use cases that are not covered by the
partition-0 trick?
> Add a job setup phase to Samza
> ------------------------------
>
> Key: SAMZA-42
> URL: https://issues.apache.org/jira/browse/SAMZA-42
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
>
> We have several use cases for doing things once at the beginning of a Samza
> job's execution (before containers start). Examples:
> * Validate or create checkpoint topic (if using KafkaCheckpointManager)
> * Validate or create state topic (if using LoggedStore)
> Right now, we have to do this in the container, which means that there's a
> race condition when running on YARN, as each container will try to create the
> same topic.
> Initially, I thought this logic could be put in the YARN AM, but then we'd
> have to put corresponding logic in the LocalJobFactory. This gets problematic
> if we implement SAMZA-41, since there would no longer be a central place to
> do a "before job starts" operation with the LocalJobFactory. If we don't do
> SAMZA-41, then we should be fine putting this logic in the YARN AM and
> LocalJobFactory.
> Alternatively, we could put this logic in JobRunner. One downside to this is
> that it would mean the JobRunner would need full access to the grid that it
> was trying to execute on (not just the RM) so that it could talk to
> Kafka/ZooKeeper (for example). I think this is actually fine, since we always
> execute our jobs from a spot that has access to the full grid.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
