[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969231#comment-15969231
]
Carlo Curino commented on YARN-6363:
------------------------------------
I have fixed and tested the commandline verifying both new and old options, and
nodes/no-nodes as follows:
{code}
# MANUALLY prepare copying source and data-files (with something like):
# cp -r <path-to-source>/hadoop-dist/target/hadoop-${COMMON_VERSION} $BASEDIR
# cp <path-to-source>/hadoop-tools/hadoop-sls/src/test/resources/*.json
$BASEDIR/data
BASEDIR=/home/ccurino/hadoop-deployed
COMMON_VERSION=3.0.0-alpha3-SNAPSHOT
export
HADOOP_CONF_DIR=$BASEDIR/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/sample-conf/
export
YARN_CONF_DIR=$BASEDIR/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/sample-conf/
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export
PATH=$PATH:${BASEDIR}/hadoop-${COMMON_VERSION}/bin:${BASEDIR}/hadoop-${COMMON_VERSION}/sbin:${BASEDIR}/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/bin
# NO NODES OLD OPTIONS
timeout -s 9 60s slsrun.sh
--input-rumen=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --input-sls=$BASEDIR/data/inputsls.json
--output-dir=/tmp/sls-out2
# NO NODES NEW OPTIONS
timeout -s 9 60s slsrun.sh --tracetype=RUMEN
--tracelocation=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SLS
--tracelocation=$BASEDIR/data/inputsls.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SYNTH
--tracelocation=$BASEDIR/data/syn.json --output-dir=/tmp/sls-out2
# WITH NODES OLD OPTIONS
timeout -s 9 60s slsrun.sh
--input-rumen=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--nodes=$BASEDIR/data/nodes.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --input-sls=$BASEDIR/data/inputsls.json
--nodes=$BASEDIR/data/nodes.json --output-dir=/tmp/sls-out2
# WITH NODES NEW OPTIONS
timeout -s 9 60s slsrun.sh --tracetype=RUMEN
--tracelocation=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--nodes=$BASEDIR/data/nodes.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SLS
--tracelocation=$BASEDIR/data/inputsls.json --nodes=$BASEDIR/data/nodes.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SYNTH
--tracelocation=$BASEDIR/data/syn.json --nodes=$BASEDIR/data/nodes.json
--output-dir=/tmp/sls-out2
{code}
I ran and checked all the configurations below, using default configs from
share/tools/sls/sample-conf, and the data from sample-data and
src/test/resources/*.json.
Manually inspecting the runs things look good, RM comes up with Webapp, nodes
are loaded as expected, and apps get to run for all but RUMEN configurations
(more below).
The new TestSLSRunner also introduces more checks (with both schedulers). In
future JIRAs we can further extend this to leverage YARN-6451 to have a deeper
introspection/check of these runs.
The RUMEN version seems to have a problem parsing the rumen example trace
(nodes are loaded only from --nodes, and jobs are never loaded).
Turns out this problem is also present in trunk (running the TestSLSRunner does
bring the RM up, but doesn't run any job nor loads nodes)---I am opening a
separate JIRA to track this existing bug YARN-6482.
(Per [~subru] offline comment, I am converting the attached overview document
in additions to the SLS documentation, will upload soon)
> Extending SLS: Synthetic Load Generator
> ---------------------------------------
>
> Key: YARN-6363
> URL: https://issues.apache.org/jira/browse/YARN-6363
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch,
> YARN-6363.v10.patch, YARN-6363.v11.patch, YARN-6363.v1.patch,
> YARN-6363.v2.patch, YARN-6363.v3.patch, YARN-6363.v4.patch,
> YARN-6363.v5.patch, YARN-6363.v6.patch, YARN-6363.v7.patch, YARN-6363.v9.patch
>
>
> This JIRA tracks the introduction of a synthetic load generator in the SLS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
