[
https://issues.apache.org/jira/browse/SAMZA-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209960#comment-14209960
]
Chris Riccomini commented on SAMZA-448:
---------------------------------------
bq. do you want to commit after refactoring?
I want to break the commit down into two commits, unless people object. I've
already opened up SAMZA-465 as a follow on, which will eliminate the checkpoint
manager. Again, this is just to keep things as incremental as possible. Note
that this *does* mean master will be a little uglier/more unstable than usual
in the mean-time.
bq. Added job.container.count config, and removed yarn.container.count.
The reason for doing this is that the JobCoordinator, itself, needs to know
about container count now (to create the JobModel). This is independent of
YARN. As we move towards supporting things beyond YARN (SAMZA-375), it seemed
prudent to get rid of the YARN-specific naming.
I believe that the correct design for the JobCoordinator is to invert the
control between it and the scheduler. Right now, we have the JobCoordinator
embedded in the YARN AM. I think the correct design is actually to have the
JobCoordinator as the parent, and embed the YARN AM code as a pluggable
interface inside the coordinator. This would allow the coordinator to dictate
when new containers should get requested, what racks they should be on, etc. We
can then implement the interface for YARN, and also for Mesos.
I'll have a look at your feedback today, and try and update accordingly.
> Pass config from JobRunner to JobCoordinator via ConfigStream
> -------------------------------------------------------------
>
> Key: SAMZA-448
> URL: https://issues.apache.org/jira/browse/SAMZA-448
> Project: Samza
> Issue Type: Sub-task
> Components: container
> Affects Versions: 0.9.0
> Reporter: Chris Riccomini
> Assignee: Chris Riccomini
> Fix For: 0.9.0
>
> Attachments: SAMZA-448-0.patch
>
>
> SAMZA-438 modified the AM/local job factories (job coordinator) to pass
> configuration to SamzaContainer via an HTTP server. This ticket is to modify
> the other half of the pipeline: the config passing from the JobRunner to the
> job coordinator to be done via a ConfigStream, rather than an environment
> variable.
> The goal of this ticket is to make the change as transparent as possible.
> {{control-job.sh}} won't be introduced in this ticket. Instead, we'll
> continue using the ConfigFactories that we have, but pass the resolved config
> to the coordinator using the ConfigStream rather than the environment
> variable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
