Bharath Kumarasubramanian created SAMZA-2610:
------------------------------------------------
Summary: Handle Metadata changes for AM HA orchestration
Key: SAMZA-2610
URL: https://issues.apache.org/jira/browse/SAMZA-2610
Project: Samza
Issue Type: New Feature
Reporter: Bharath Kumarasubramanian
Assignee: Bharath Kumarasubramanian
*Goal*
# Ensure the auto-sizing decisions are handled during AM HA.
# Ensure start points are handled during AM HA.
# Ensure the application containers post AM HA run with consistent job model.
*Description*
Job Model & Config Changes
With AM HA, the new AM will regenerate job model and may have to spin up
containers. The new job model may differ from the job model that some of the
running containers have. As a result, we can have a state where within the
application some containers run with one view of job model while other run with
different job model.
Scenarios where job model changes can happen include but not limited to
# Change in partition count of the input stream
# New topics that match the regex topic generator
# Auto-scaling decision or any medium that updates configuration in the
coordinator stream
Startpoint Handling
With AM HA, the new AM generates job model, performs startpoint fanouts and
launches containers (as needed). However, not all containers get launched as
some of them maybe alive from the previous attempt.
In those scenarios, either
* We want to make sure we restart all containers and they see & act on the
startpoints consistently.
* Do no-op on the startpoint and have startpoint get picked up on job restart
from external request
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
