[
https://issues.apache.org/jira/browse/SAMZA-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13852612#comment-13852612
]
Chris Riccomini commented on SAMZA-109:
---------------------------------------
Proposed changes:
1. Eliminate LocalJobFactory.
2. Create ProcessJobFactory.
3. Create ThreadJobFactory.
4. Make ProcessJobFactory behave like YarnJobFactory.
5. Forcibly assign the log4j.xml file (if it exists), and the
samza.log.dir/samza.container.name Java system properties.
6. Add CommandBuilder.setLogDir.
7. Make ShellCommandBuilder.buildEnvironment do a string replace for task.opts
(ENV_JAVA_OPTS) to replace $SAMZA_LOG_DIR with log dir, and
$SAMZA_CONTAINER_NAME with container name.
8. Update SamzaAppMasterTaskManager, and ProcessJobFactory to use the new
CommandBuilder.setLogDir method.
(3) is useful for debugging via IDE, so we should keep it.
(4) means that ProcessJob and YarnJob should both save a job ID file somewhere
(configurable). The ProcessJob's job ID file will contain the PID of the
SamzaContainer. The YarnJob's job ID file will contain the application attempt
ID for the Samza job in the YARN cluster. We should update kill-job.sh to take
a job ID, but to default to a standard location (e.g.
bin/../samza-job-name.pid). The YarnJob will use the job ID to kill the job in
the YARN cluster. The ProcessJob will use the PID to kill the job locally.
We'll need to update the StreamJob interface to support this.
(5) would make run-class.sh replace:
{code}
if [ -z "$JAVA_OPTS" ]; then
JAVA_OPTS="-Xmx768M -XX:+PrintGCDateStamps -Xloggc:$SAMZA_LOG_DIR/gc.log
-Dsamza.log.dir=$SAMZA_LOG_DIR -Dsamza.container.name=$SAMZA_CONTAINER_NAME"
if [ -f $base_dir/lib/log4j.xml ]; then
JAVA_OPTS="$JAVA_OPTS -Dlog4j.configuration=file:$base_dir/lib/log4j.xml"
fi
fi
{code}
with:
{code}
if [ -z "$JAVA_OPTS" ]; then
JAVA_OPTS="-Xmx768M -XX:+PrintGCDateStamps -Xloggc:$SAMZA_LOG_DIR/gc.log"
fi
if [ -f $base_dir/lib/log4j.xml ]; then
JAVA_OPTS="$JAVA_OPTS -Dlog4j.configuration=file:$base_dir/lib/log4j.xml"
fi
JAVA_OPTS="$JAVA_OPTS -Dsamza.log.dir=$SAMZA_LOG_DIR
-Dsamza.container.name=$SAMZA_CONTAINER_NAME"
{code}
These changes address the problems in the following way:
1. Changing task.opts will no longer disable logging. Log4j.xml will still be
applied, and the log dir and container name will both be available as system
properties for log4j.xml if you're using the daily rolling appender. If you
wish to use refer to the container name or log path, you can just use the
environment variables, and they will get string-replaced as expected. Changing
task.opts WOULD disable gc.log files, but gc.log could be re-enabled using the
environment variable due to change (7) above.
2. There is now a clear distinction between ProcessJob and ThreadJob, since
they each have their own factory now. No one should really ever use ThreadJob,
except for unit tests, and IDE debugging.
3. You now have a way to kill jobs in dev-mode using the updated kill-job.sh
script.
> Make task.opts easier to use
> ----------------------------
>
> Key: SAMZA-109
> URL: https://issues.apache.org/jira/browse/SAMZA-109
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
>
> task.opts is a bit tricky to use right now.
> The most common thing that I see people want to do are:
> * Change the Xmx setting for their Samza containers.
> * Turn on a remote debugger.
> * Change garbage collector settings.
> Most of the time they do this in dev-mode (using LocalJobFactory).
> Changing task.opts is tricky for a number of reasons.
> 1. Changing task.opts at all eliminates all of the defaults. This means, if
> you set -Xmx, you lose all the defaults set in run-class.sh when no JVM_OPTS
> is set (log settings, Java system properties, etc).
> 2. LocalJobFactory uses ThreadJob by default, which doesn't even pay
> attention to task.opts (since it starts the container as a thread, not a new
> process).
> 3. If you manage to figure out how to switch LocalJobFactory to use
> ProcessJob (undocumented), you end up with the SamzaContainer running as a
> separate process with no way to kill it (the LocalJobRunner starts the
> ProcessJob, then exits).
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
