Yi Pan (Data Infrastructure) created SAMZA-1482:
---------------------------------------------------
Summary: Restart or fail Samza jobs in YARN when input topic
partition changes
Key: SAMZA-1482
URL: https://issues.apache.org/jira/browse/SAMZA-1482
Project: Samza
Issue Type: Bug
Reporter: Yi Pan (Data Infrastructure)
Currently, after a Samza job is started, it works only on a set of fixed input
topic partitions at the start-up time. When input topic partitions are
expanded, we often lose the messages sent in the new partitions, until we
restart the job.
SAMZA-882 added a input stream partition count monitor inside the
JobCoordinator. This ticket is targeted to use this monitor metrics and trigger
the following actions in YARN:
# for stateless jobs, shutdown the JobCoordinator w/ UNDEFINED status code s.t.
YARN will restart the whole job
# for stateful jobs, shutdown the JobCoordinator w/ FAILED status code s.t.
YARN will stop the whole job
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
