Navina Ramesh created SAMZA-882:
-----------------------------------
Summary: Detect partition count changes in input streams
Key: SAMZA-882
URL: https://issues.apache.org/jira/browse/SAMZA-882
Project: Samza
Issue Type: Bug
Reporter: Navina Ramesh
Assignee: Navina Ramesh
This is a known issue where any change in the partition count in the upstream
affects the Samza job and it needs to be restarted. In such scenarios, we
experience data loss or incorrect processing because the application logic
depends on the partitioning strategy. It is worsened by the fact that we don't
even have a good mechanism to detect such a change.
As a first-step towards detection, I propose that we modify the stream metadata
cache maintained in Samza such that when there a change in partition count, we
increment a gauge metric. This way we can at least attach a hook to monitor
when this happens and take necessary actions.
However, in the long-term, we need to come up with a better strategy for
handling this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
