[
https://issues.apache.org/jira/browse/SAMZA-881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181405#comment-15181405
]
Yi Pan (Data Infrastructure) commented on SAMZA-881:
----------------------------------------------------
Hi, [[email protected]], the proposal looks good to me. I am attaching a
few comments below:
# When we describe mutual exclusiveness in partition assignment, please exclude
broadcast stream in the discussion
# Be consistent w/ terms: in page 2, “leader container” vs “leader process”
# In the architecture graph, it would be nice to label the text on the edges w/
execution order
# Where is the container liveness management module in the design of
JobCoordinator? W/ SAMZA-871, requesting for direct heart beat between the
containers and AM (i.e. essentially followers and leaders in the new design), I
think that we should have a separate pluggable module for this, in addition to
ContainerProcessManager, which is just interface to allocate/request processes.
# Case 3.2 is the prototype implemented in SAMZA-516, right? We should call it
out.
Thanks a lot for the design!
> Re-think the Samza Job Coordinator
> ----------------------------------
>
> Key: SAMZA-881
> URL: https://issues.apache.org/jira/browse/SAMZA-881
> Project: Samza
> Issue Type: Improvement
> Reporter: Jagadish
> Assignee: Jagadish
> Attachments: SamzaJobCoordinatorRe-designProposal.pdf
>
>
> Currently, the only way to run Samza containers in distributed mode is using
> Yarn. However, there has been interest to run Samza on top of other resource
> managers with the recent explosion in the # of such systems. Users have also
> requested us to run Samza as a library, and to run Samza on Docker containers
> managed by Kubernetes.
> We must re-think the JobCoordinator functionality as follows:
> 1. ID assignment: Provide an ID to each SamzaContainer.
> 2. JobModel agreement: Ensure containers agree on a JobModel.
> 3. Re-start the SamzaContainer when the job model changes.
> This will arguably require some leader election (depending on how users
> choose to run Samza).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
