[ https://issues.apache.org/jira/browse/YARN-7732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351129#comment-16351129 ]
genericqa commented on YARN-7732: --------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 2s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 9s{color} | {color:orange} hadoop-tools/hadoop-sls: The patch generated 1 new + 50 unchanged - 1 fixed = 51 total (was 51) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 40s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 37s{color} | {color:red} hadoop-tools/hadoop-sls generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 7s{color} | {color:green} hadoop-sls in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 48m 53s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-tools/hadoop-sls | | | int value cast to float and then passed to Math.round in org.apache.hadoop.yarn.sls.synthetic.SynthJob.getTaskAttemptInfo(TaskType, int, int) At SynthJob.java:and then passed to Math.round in org.apache.hadoop.yarn.sls.synthetic.SynthJob.getTaskAttemptInfo(TaskType, int, int) At SynthJob.java:[line 345] | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7732 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909038/YARN-7732.04.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 7223e1d20bd1 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 5072388 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/19584/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt | | findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/19584/artifact/out/new-findbugs-hadoop-tools_hadoop-sls.html | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/19584/testReport/ | | asflicense | https://builds.apache.org/job/PreCommit-YARN-Build/19584/artifact/out/patch-asflicense-problems.txt | | Max. process+thread count | 467 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-sls U: hadoop-tools/hadoop-sls | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19584/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Support Generic AM Simulator from SynthGenerator > ------------------------------------------------ > > Key: YARN-7732 > URL: https://issues.apache.org/jira/browse/YARN-7732 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler-load-simulator > Reporter: Young Chen > Assignee: Young Chen > Priority: Minor > Attachments: YARN-7732-YARN-7798.01.patch, > YARN-7732-YARN-7798.02.patch, YARN-7732.01.patch, YARN-7732.02.patch, > YARN-7732.03.patch, YARN-7732.04.patch > > > Extract the MapReduce specific set-up in the SLSRunner into the > MRAMSimulator, and enable support for pluggable AMSimulators. > Previously, the AM set up in SLSRunner had the MRAMSimulator type hard coded, > for example startAMFromSynthGenerator() calls this: > > {code:java} > runNewAM(SLSUtils.DEFAULT_JOB_TYPE, user, jobQueue, oldJobId, > jobStartTimeMS, jobFinishTimeMS, containerList, reservationId, > job.getDeadline(), getAMContainerResource(null)); > {code} > where SLSUtils.DEFAULT_JOB_TYPE = "mapreduce" > The container set up was also only suitable for mapreduce: > > {code:java} > Version:1.0 StartHTML:000000286 EndHTML:000012564 StartFragment:000003634 > EndFragment:000012474 StartSelection:000003700 EndSelection:000012464 > SourceURL:https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java > > // map tasks > for (int i = 0; i < job.getNumberMaps(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.MAP, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size()))) > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add(new ContainerSimulator(containerResource, > containerLifeTime, hostname, DEFAULT_MAPPER_PRIORITY, "map")); > } > // reduce tasks > for (int i = 0; i < job.getNumberReduces(); i++) { > TaskAttemptInfo tai = job.getTaskAttemptInfo(TaskType.REDUCE, i, 0); > RMNode node = > nmMap.get(keyAsArray.get(rand.nextInt(keyAsArray.size()))) > .getNode(); > String hostname = "/" + node.getRackName() + "/" + node.getHostName(); > long containerLifeTime = tai.getRuntime(); > Resource containerResource = > Resource.newInstance((int) tai.getTaskInfo().getTaskMemory(), > (int) tai.getTaskInfo().getTaskVCores()); > containerList.add( > new ContainerSimulator(containerResource, containerLifeTime, > hostname, DEFAULT_REDUCER_PRIORITY, "reduce")); > } > {code} > > In addition, the syn.json format supported only mapreduce (the parameters > were very specific: mtime, rtime, mtasks, rtasks, etc..). > This patch aims to introduce a new syn.json format that can describe generic > jobs, and the SLS setup required to support the synth generation of generic > jobs. > See syn_generic.json for an equivalent of the previous syn.json in the new > format. > Using the new generic format, we describe a StreamAMSimulator simulates a > long running streaming service that maintains N number of containers for the > lifetime of the AM. See syn_stream.json. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org