[
https://issues.apache.org/jira/browse/SAMZA-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768557#comment-15768557
]
Navina Ramesh commented on SAMZA-1064:
--------------------------------------
We had a round of discussions based on this initial design. Here are some notes:
* Action Items (or things not yet addressed in the design):
** Split config from JobModel message in the Coordination Stream
** Job Model distribution mechanism should be defined clearly
** Add a note about Yarn: In yarn , the leader starts the processors. Hence, it
can tell the container what version of job model to pick up from the CS.
* Design document should call out the following:
** rolling bounce is not supported in this design
** what defines the lifetime of a Samza job attempt?
** Debounce time should be larger than sessionTimeout (debounceTime >>
sessionTimeout)
** JobModel (and its version) should be treated as a resource that can be
updated by only one processor. Hence, we should use conditional updates to
fence off JobModel resource
** Figure out how to clean up the incomplete and/ old barriers
* Questions to resolve:
** Is coordination stream a suitable option for job model distribution? What
kind of guarantees do we want wrt jobmodel distribution (atomicity/durability?)
** Should JM be written as a single message to the coordinator stream ? or
multiple small container model messages? (1 msg / multipart message in
Coordinator Stream). If it is the latter, how do we guarantee atomicity of
writes? Do we need marker messages to indicate the start/stop of the batch of
job model messages?
** Do we need job attempts in the tree? If yes, who creates the persistent
subtrees
** Apart from performance, what is the benefit of having a /leader znode as a
lock? Does it really provide fencing for the leader processor?
> Standalone Samza with Zookeeper for Coordination
> ------------------------------------------------
>
> Key: SAMZA-1064
> URL: https://issues.apache.org/jira/browse/SAMZA-1064
> Project: Samza
> Issue Type: New Feature
> Reporter: Navina Ramesh
> Attachments: Samza Standalone-0.md, Samza Standalone-0.pdf,
> image_0.png, image_1.png
>
>
> In this use-case, we propose using Zookeeper for coordinating processor
> liveness and task distribution in a Samza job. Additionally, it opens up the
> possibility of allowing a flexible number of participating processors (no
> fixed container count).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
