[
https://issues.apache.org/jira/browse/SAMZA-123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981404#comment-13981404
]
Chris Riccomini commented on SAMZA-123:
---------------------------------------
1. I agree with [~jkreps] about the C2P topic. I'd either like to roll the C2P
mapping into the Checkpoint object (and topic), or have a generic topic as
[~jghoman] describes. I think having a generic topic is a big change that needs
to be thought through, through (see 6, below). What else would we put in there,
what would it be used for, etc.
2. The section on verification is vague:
bq. Second, the AM can diff the previously C2P log entry with the result of the
current set of mappings and warn (or fail) if these values disagree to some
extent.
You then go on to say that we can't verify because the logic is job dependent.
What exactly is the proposed logic for this? I think we should think through
the impact of adding/removing/moving SSPs on state for every strategy.
3. I agree with [~sriramsub]'s comments on the mailing list: we should not add
configs to make this fully pluggable until we're more convinced that all of
this is a good idea. We did this for the MessageChooser, and I think it's the
right approach here as well.
4. I like @sriram's naming suggestion:
bq. My vote for the naming would be taskName if it is a string and taskId if we
can map all use cases to ids.
5. What exactly is the use case for GroupIntoNSets? If we decouple both
checkpointing and state from the number of TaskInstances we have, why can't we
just use GroupBySSP, and use the yarn.container.count to control the "N"?
6. Per-[~martinkl], we shouldn't use the word "state" again:
bq. how about we just call it the job state log
It's just confusing, since we already use it for all the state management
stuff.
That said, the "job state" that a job has consists of is the input/output
offsets for the job, its configuration, and its SSP cohort mappings. We've been
treating these separately in the framework, but perhaps it makes sense to treat
them the same way? Storing config between job runs allows us to verify config
changes are "safe". It also allows us to do one-time mutations of the "job
state" via CLI, or on job-start time (see SAMZA-180). In this model, I'm not
sure if it would make sense to use KAFKA-1000, or just keep the offsets in the
"job state" topic--I'd have to think about it more.
> Move topic partition grouping to the AM and generalize
> ------------------------------------------------------
>
> Key: SAMZA-123
> URL: https://issues.apache.org/jira/browse/SAMZA-123
> Project: Samza
> Issue Type: Sub-task
> Components: container
> Affects Versions: 0.6.0
> Reporter: Jakob Homan
> Assignee: Jakob Homan
> Attachments: SAMZA-123-design-doc.md, SAMZA-123-design-doc.pdf
>
>
> Currently the AM sends a set of all the topics and partitions to the
> container, which then groups them by partition and assigns each set to a task
> instance. By moving the grouping to the AM, we can assign arbitrary groups to
> task instances, which will allow more partitioning strategies, as discussed
> in SAMZA-71.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
