[jira] [Commented] (YARN-2672) Improve Gridmix (synthetic generator + reservation support)
[
https://issues.apache.org/jira/browse/YARN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658648#comment-14658648
]
Hadoop QA commented on YARN-2672:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 31s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 8m 18s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 13s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 23s | The applied patch generated
7 release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 31s | The applied patch generated
177 new checkstyle issues (total was 126, now 300). |
| {color:red}-1{color} | whitespace | 0m 2s | The patch has 14 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 1m 1s | The patch appears to introduce 5
new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | tools/hadoop tests | 11m 11s | Tests failed in
hadoop-gridmix. |
| | | 50m 12s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-gridmix |
| Failed unit tests | hadoop.mapred.gridmix.TestGridmixMemoryEmulation |
| | hadoop.mapred.gridmix.TestSleepJob |
| | hadoop.mapred.gridmix.TestHighRamJob |
| Timed out tests | org.apache.hadoop.mapred.gridmix.TestGridmixSubmission |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748880/YARN-2672.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4ab49a4 |
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/8773/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8773/artifact/patchprocess/diffcheckstylehadoop-gridmix.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8773/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/8773/artifact/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html
|
| hadoop-gridmix test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8773/artifact/patchprocess/testrun_hadoop-gridmix.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8773/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8773/console |
This message was automatically generated.
Improve Gridmix (synthetic generator + reservation support)
---
Key: YARN-2672
URL: https://issues.apache.org/jira/browse/YARN-2672
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler, fairscheduler, resourcemanager
Reporter: Carlo Curino
Assignee: Carlo Curino
Labels: BB2015-05-TBR
Attachments: YARN-2672.1.patch, YARN-2672.patch
This JIRA proposes an enhancement of Gridmix that contains:
1) a synthetic generator to produce load without the need of a trace, but
based on distributions
2) include negotiation of reservations (to test YARN-1051).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2672) Improve Gridmix (synthetic generator + reservation support)
[
https://issues.apache.org/jira/browse/YARN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524972#comment-14524972
]
Hadoop QA commented on YARN-2672:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12674061/YARN-2672.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7633/console |
This message was automatically generated.
Improve Gridmix (synthetic generator + reservation support)
---
Key: YARN-2672
URL: https://issues.apache.org/jira/browse/YARN-2672
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler, fairscheduler, resourcemanager
Reporter: Carlo Curino
Assignee: Carlo Curino
Attachments: YARN-2672.patch
This JIRA proposes an enhancement of Gridmix that contains:
1) a synthetic generator to produce load without the need of a trace, but
based on distributions
2) include negotiation of reservations (to test YARN-1051).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2672) Improve Gridmix (synthetic generator + reservation support)
[
https://issues.apache.org/jira/browse/YARN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525033#comment-14525033
]
Hadoop QA commented on YARN-2672:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12674061/YARN-2672.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7651/console |
This message was automatically generated.
Improve Gridmix (synthetic generator + reservation support)
---
Key: YARN-2672
URL: https://issues.apache.org/jira/browse/YARN-2672
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler, fairscheduler, resourcemanager
Reporter: Carlo Curino
Assignee: Carlo Curino
Attachments: YARN-2672.patch
This JIRA proposes an enhancement of Gridmix that contains:
1) a synthetic generator to produce load without the need of a trace, but
based on distributions
2) include negotiation of reservations (to test YARN-1051).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2672) Improve Gridmix (synthetic generator + reservation support)
[ https://issues.apache.org/jira/browse/YARN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166032#comment-14166032 ] Carlo Curino commented on YARN-2672: The patch contains: * a simple synthetic generator, that allows to control avg and stdev for many common parameters (#maps, #reducers, map-time, red-time, IOs for in/out/shuffle, deadlines and duration of jobs for reservation). It is easy to define (see syn.json) different workloads, and job classes with various properties (size/frequency). Not very tested is also the possibility to generate jobs at different rates (by controlling the weighting of subsequent time ranges). This is generally useful. * Extensions to experiment with reservations, where we can specify with what probability a job class would be run with reservation, and submit the corresponding ReservationRequeust via YARN-1051, and upon acceptance launch the job in it. The patch is *rough*, but as few people are starting to experiment with YARN-1051 we thought it was important to toss the code out and let folks experiments/improve/provide feedback. Improve Gridmix (synthetic generator + reservation support) --- Key: YARN-2672 URL: https://issues.apache.org/jira/browse/YARN-2672 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler, fairscheduler, resourcemanager Reporter: Carlo Curino Assignee: Carlo Curino Attachments: YARN-2672.patch This JIRA proposes an enhancement of Gridmix that contains: 1) a synthetic generator to produce load without the need of a trace, but based on distributions 2) include negotiation of reservations (to test YARN-1051). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2672) Improve Gridmix (synthetic generator + reservation support)
[
https://issues.apache.org/jira/browse/YARN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166055#comment-14166055
]
Carlo Curino commented on YARN-2672:
Quick how to use:
I usually run it with something like this:
{code:title=gridmix.sh|borderStyle=solid}
#!/bin/bash
TRACE=${1:-syn.json}
LOCATION=${2:-/user/hadoop/gridmix100g}
. env.sh
date
hadoop fs -rm -r $LOCATION/gridmix
hadoop fs -rm /user/hadoop/$TRACE
hadoop fs -put $TRACE /user/hadoop
export
HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_YARN_HOME/share/hadoop/tools/lib/*:/home/hadoop/commons-math3-3.0.jar
echo launching gridmix
hadoop jar
$HADOOP_COMMON_HOME/share/hadoop/tools/lib/hadoop-gridmix-3.0.0-SNAPSHOT.jar
-libjars
$HADOOP_COMMON_HOME/share/hadoop/tools/lib/hadoop-rumen-3.0.0-SNAPSHOT.jar \
-Dgridmix.job-producer.is.synthetic=true \
-Dgridmix.job-submission.policy=REPLAY \
-Dgridmix.job.type=LOADJOB \
-Dgridmix.job-submission.default-queue=default \
-Dgridmix.sleep.fake-locations=3 \
-Dgridmix.compression-emulation.enable=false \
-Dgridmix.job.seq=1 \
-Dgridmix.client.submit.threads=20 \
-Dgridmix.client.pending.queue.depth=10 \
-Dmapreduce.map.java.opts=-Xmx2000m \
-Dmapreduce.reduce.java.opts=-Xmx4000m \
$LOCATION /user/hadoop/$TRACE
{code}
The syn.json looks something like this:
{code:title=syn.json|borderStyle=solid}
{
description : tiny jobs workload,
num_jobs : 1000,
rand_seed : 2,
workloads : [
{
workload_name : tiny-test,
workload_weight: 0.5,
description : Sort jobs,
queue_name : dedicated,
job_classes : [
{
class_name : class_1,
class_weight : 1.0,
mtasks_avg : 5,
mtasks_stddev : 1,
rtasks_avg : 5,
rtasks_stddev : 1,
in_avg : 1048500,
in_stddev : 17466,
shuffle_avg : 104085000,
shuffle_stddev : 162666,
out_avg : 10485700,
out_stddev : 1876000,
dur_avg : 600,
dur_stddev : 60,
mtime_avg : 3,
mtime_stddev : 60,
rtime_avg : 3,
rtime_stddev : 6,
map_max_memory_avg : 1024,
map_max_memory_stddev : 0.001,
reduce_max_memory_avg : 1024,
reduce_max_memory_stddev : 0.001,
bytes_per_map_record : 512,
bytes_per_shuffle_record : 512,
bytes_per_reduce_record : 1024,
chance_of_reservation : 1.0,
deadline_factor_avg : 10.0,
deadline_factor_stddev : 0.001,
gang_size : 1
}
],
time_distribution : [
{ time : 1, jobs : 100 },
{ time : 3600, jobs : 0 }
]
}
]
}
{code}
The distributions of each parameter are *Normal* with average X_avg and
standard deviation X_stddev.
Improve Gridmix (synthetic generator + reservation support)
---
Key: YARN-2672
URL: https://issues.apache.org/jira/browse/YARN-2672
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler, fairscheduler, resourcemanager
Reporter: Carlo Curino
Assignee: Carlo Curino
Attachments: YARN-2672.patch
This JIRA proposes an enhancement of Gridmix that contains:
1) a synthetic generator to produce load without the need of a trace, but
based on distributions
2) include negotiation of reservations (to test YARN-1051).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2672) Improve Gridmix (synthetic generator + reservation support)
[
https://issues.apache.org/jira/browse/YARN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166111#comment-14166111
]
Hadoop QA commented on YARN-2672:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674061/YARN-2672.patch
against trunk revision e532ed8.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5353//console
This message is automatically generated.
Improve Gridmix (synthetic generator + reservation support)
---
Key: YARN-2672
URL: https://issues.apache.org/jira/browse/YARN-2672
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler, fairscheduler, resourcemanager
Reporter: Carlo Curino
Assignee: Carlo Curino
Attachments: YARN-2672.patch
This JIRA proposes an enhancement of Gridmix that contains:
1) a synthetic generator to produce load without the need of a trace, but
based on distributions
2) include negotiation of reservations (to test YARN-1051).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
