Bharath Kumarasubramanian created SAMZA-2393:
------------------------------------------------
Summary: Side input streams should respect offset reset and
broadcast properties
Key: SAMZA-2393
URL: https://issues.apache.org/jira/browse/SAMZA-2393
Project: Samza
Issue Type: Improvement
Reporter: Bharath Kumarasubramanian
Side inputs are treated like every other stream (mostly) in Samza today.
However, some of the properties of the stream like offset.reset policies,
broadcast etc don't apply to side inputs. As part of this ticket, we want to
explore the option of supporting these for side input streams too.
The current limitations are we don't have checkpointing for side inputs and use
the local checkpoint on the disk to register the starting offsets. We need to
tie down the offset manager into side inputs consumers which is capable of
handling offset reset policies.
Also, side input consumer updates the chooser every time when it chooses the
message before waiting for the messages to be processed by the side input
processor. This limits us to use priorities in side inputs and also implement
broadcast correctly. We would need to do something similar to the RunLoop to
support the above.
As part of this ticket, we need to explore if we can use a new run loop
instance for side inputs or extract some of the runloop logic to make it
applicable for side inputs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
