[
https://issues.apache.org/jira/browse/SAMZA-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169568#comment-14169568
]
Chris Riccomini commented on SAMZA-348:
---------------------------------------
bq. How does the control-job.sh script use the SystemConsumer/SystemProducer?
A third solution to this problem would be to define the ConfigStream in code.
If we move toward doing wiring in code (rather than config) as part of
SAMZA-40, we could also define the ConfigStream in code. At first, this might
sound rather distasteful, since it will require a full rebuild of code in order
to change a ConfigStream, but if you bend your thinking a bit, the ConfigStream
is actually closer to wiring than config, so it might be OK to require this.
This would also require the control-job.sh script to download the job that it's
controlling, or to have the job's jars on its classpath, so that it would know
how to talk to the ConfigStream as well.
I like this approach, except for the complexity it introduces in
control-job.sh. Not sure if we can solve this, though. If not, then
SystemFactory.getConfig still seems the least distasteful.
> Configure Samza jobs through a stream
> -------------------------------------
>
> Key: SAMZA-348
> URL: https://issues.apache.org/jira/browse/SAMZA-348
> Project: Samza
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Chris Riccomini
> Labels: project
> Attachments: DESIGN-SAMZA-348-0.md, DESIGN-SAMZA-348-0.pdf,
> DESIGN-SAMZA-348-1.md, DESIGN-SAMZA-348-1.pdf
>
>
> Samza's existing config setup is problematic for a number of reasons:
> # It's completely immutable once a job starts. This prevents any dynamic
> reconfiguration and auto-scaling. It is debatable whether we want these
> feature or not, but our existing implementation actively prevents it. See
> SAMZA-334 for discussion.
> # We pass existing configuration through environment variables. YARN exports
> environment variables in a shell script, which limits the size to the varargs
> length on the machine. This is usually ~128KB. See SAMZA-333 and SAMZA-337
> for details.
> # User-defined configuration (the Config object) and programmatic
> configuration (checkpoints and TaskName:State mappings (see SAMZA-123)) are
> handled differently. It's debatable whether this makes sense.
> In SAMZA-123, [~jghoman] and I propose implementing a ConfigLog. This log
> would replace both the checkpoint topic and the existing config environment
> variables in SamzaContainer and Samza's YARN AM.
> I'd like to keep this ticket's scope limited to just the implementation of
> the ConfigLog, and not re-designing how Samza's config is used in the code
> (SAMZA-40). We should, however, discuss how this feature would affect dynamic
> reconfiguration/auto-scaling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
