Jagadish created SAMZA-881:
------------------------------
Summary: Re-think the Samza Job Coordinator
Key: SAMZA-881
URL: https://issues.apache.org/jira/browse/SAMZA-881
Project: Samza
Issue Type: Improvement
Reporter: Jagadish
Assignee: Jagadish
Currently, the only way to run Samza containers in distributed mode is using
Yarn. However, there has been interest to run Samza on top of other resource
managers with the recent explosion in the # of such systems. Users have also
requested us to run Samza as a library, and to run Samza on Docker containers
managed by Kubernetes.
We must re-think the JobCoordinator functionality as follows:
1. ID assignment: Provide an ID to each SamzaContainer.
2. JobModel agreement: Ensure containers agree on a JobModel.
3. Re-start the SamzaContainer when the job model changes.
This will arguably require some leader election (depending on how users choose
to run Samza).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
