[
https://issues.apache.org/jira/browse/SAMZA-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545970#comment-14545970
]
Yi Pan (Data Infrastructure) commented on SAMZA-676:
----------------------------------------------------
Hi, [~closeuris], sorry to chime in late here. Here are my points in addition:
# Instead of changing SystemConsumer API, why not change the implementation of
SystemConsumers s.t. we allow M:1 mapping between SystemConsumer instance to
Task instance? That should satisfy the multi-subscriber use case as well.
SystemConsumers.register() method has to be changed but that is internal API.
# I do see the requirement to change / break the current MessageChooser API in
order to accommodate the multi-subscriber per SSP in a clean way. The
MessageChooser.choose API only returns the IncomingMessageEnvelope, which makes
it difficult to identify which task this message should go to w/o knowing from
which consumer this message is coming from. The only option w/o breaking the
MessageChooser API seems to require a hell-lot of bookkeeping in the
SystemConsumers to remember the lastest offset each SystemConsumer is at and
decorate/deliver the incoming message w/ the SystemConsumer that expects the
exact offset. Yet, there could be multiple deliveries if all consumers are at
the exact offset and moving at exact the same speed. So, I would propose the
add a MulticastMessageChoose API and make it configurable s.t. it is required
when broadcast topic is configured.
# I like [~naveenatceg]'s idea of making the grouper configurable better,
because it would allow the implementation of SystemStreamPartitionGrouper to
determine which SSPs are global w/o changing the SystemStreamPartitionGrouper
API and the logic can still be within this group() function, instead of
modifying JobCoordinator to add the global topics later.
Just my a few cents.
> Implement Broadcast Stream
> --------------------------
>
> Key: SAMZA-676
> URL: https://issues.apache.org/jira/browse/SAMZA-676
> Project: Samza
> Issue Type: Improvement
> Components: container
> Reporter: Yan Fang
> Assignee: Yan Fang
> Attachments: BroadcastStreamDesign.md, BroadcastStreamDesign.pdf
>
>
> There are a lot of discussion in SAMZA-353 about assigning the same SSP to
> multiple taskNames. This ticket is a subset of the discussion. Only focus on
> the broadcast stream implementation.
> The goal is to assign one SSP to all the taskNames.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
