date:20181004

[jira] [Commented] (YARN-8644) Improve unit test for RMAppImpl.FinalTransition

2018-10-04 Thread Hadoop QA (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639259#comment-16639259
 ] 

Hadoop QA commented on YARN-8644:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 42s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 28s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 3 new + 132 unchanged - 4 fixed = 135 total (was 136) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 35s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 23s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}118m 25s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8644 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12942460/YARN-8644.010.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 63a71b2081b1 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 619e490 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/22066/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| whitespace |

[jira] [Commented] (YARN-8788) mvn package -Pyarn-ui fails on JDK9

2018-10-04 Thread Akira Ajisaka (JIRA)



[ 
https://issues.apache.org/jira/browse/YARN-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639234#comment-16639234
 ] 

Akira Ajisaka commented on YARN-8788:
-

Attached the patch in the pull request to trigger precommit jenkins job.

> mvn package -Pyarn-ui fails on JDK9
> ---
>
> Key: YARN-8788
> URL: https://issues.apache.org/jira/browse/YARN-8788
> Project: Hadoop YARN
>  Issue Type: Bug
> Environment: Java 9.0.4, CentOS 7.5
>Reporter: Akira Ajisaka
>Assignee: Vidura Bhathiya Mudalige
>Priority: Major
>  Labels: newbie
> Attachments: 421.patch
>
>
> {{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk.
> {noformat}
> [ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run 
> (default) on project hadoop-yarn-ui: Execution default of goal 
> ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was 
> encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run: 
> java.lang.ExceptionInInitializerError: null
> [ERROR] -
> [ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] = 
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
> [ERROR] urls[1] = 
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
> [ERROR] urls[2] = 
> file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> (snip)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8788) mvn package -Pyarn-ui fails on JDK9

2018-10-04 Thread Akira Ajisaka (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated YARN-8788:

Attachment: 421.patch

> mvn package -Pyarn-ui fails on JDK9
> ---
>
> Key: YARN-8788
> URL: https://issues.apache.org/jira/browse/YARN-8788
> Project: Hadoop YARN
>  Issue Type: Bug
> Environment: Java 9.0.4, CentOS 7.5
>Reporter: Akira Ajisaka
>Assignee: Vidura Bhathiya Mudalige
>Priority: Major
>  Labels: newbie
> Attachments: 421.patch
>
>
> {{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk.
> {noformat}
> [ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run 
> (default) on project hadoop-yarn-ui: Execution default of goal 
> ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was 
> encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run: 
> java.lang.ExceptionInInitializerError: null
> [ERROR] -
> [ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] = 
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
> [ERROR] urls[1] = 
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
> [ERROR] urls[2] = 
> file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> (snip)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-8849) DynoYARN: A simulation and testing infrastructure for YARN clusters

2018-10-04 Thread Keqiu Hu (JIRA)



 [ 
https://issues.apache.org/jira/browse/YARN-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keqiu Hu reassigned YARN-8849:
--

Assignee: Keqiu Hu

> DynoYARN: A simulation and testing infrastructure for YARN clusters
> ---
>
> Key: YARN-8849
> URL: https://issues.apache.org/jira/browse/YARN-8849
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Arun Suresh
>Assignee: Keqiu Hu
>Priority: Major
>
> Traditionally, YARN workload simulation is performed using SLS (Scheduler 
> Load Simulator) which is packaged with YARN. It Essentially, starts a full 
> fledged *ResourceManager*, but runs simulators for the *NodeManager* and the 
> *ApplicationMaster* Containers. These simulators are lightweight and run in a 
> threadpool. The NM simulators do not open any external ports and send 
> (in-process) heartbeats to the ResourceManager.
> There are a couple of drawbacks with using the SLS:
> * It might be difficult to simulate really large clusters without having 
> access to a very beefy box - since the NMs are launched as tasks in a 
> threadpool, and each NM has to send periodic heartbeats to the RM.
> * Certain features (like YARN-1011) requires changes to the NodeManager - 
> aspects such as queuing and selectively killing containers have to be 
> incorporate into the existing NM Simulator which might make the simulator a 
> bit heavy weight - there is a need for locking and synchronization.
> * Since the NM and AM are simulations, only the Scheduler is faithfully 
> tested - it does not really perform an end-2-end test of a cluster.
> Therefore, drawing inspiration from 
> [Dynamometer|https://github.com/linkedin/dynamometer], we propose a framework 
> for YARN deployable YARN cluster - *DynoYARN* - for testing, with the 
> following features:
> * The NM already has hooks to plug-in custom *ContainerExecutor* and 
> *NodeResourceMonitor*. If we can plug-in a custom *ContainersMonitorImpl*'s 
> Monitoring thread (and other modules like the LocalizationService), We can 
> probably inject an Executor that does not actually launch containers and a 
> Node and Container resource monitor that reports synthetic pre-specified 
> Utilization metrics back to the RM.
> * Since we are launching fake containers, we cannot run normal AM containers. 
> We can therefore, use *Unmanaged AM*'s to launch synthetic jobs.
> Essentially, a test workflow would look like this:
> * Launch a DynoYARN cluster.
> * Use the Unmanaged AM feature to directly negotiate with the DynaYARN 
> Resource Manager for container tokens.
> * Use the container tokens from the RM to directly ask the DynoYARN Node 
> Managers to start fake containers.
> * The DynoYARN NodeManagers will start the fake containers and report to the 
> DynoYARN Resource Manager synthetically generated resource utilization for 
> the containers (which will be injected via the *ContainerLaunchContext* and 
> parsed by the plugged-in Container Executor).
> * The Scheduler will use the utilization report to schedule containers - we 
> will be able to test allocation of *Opportunistic* containers based on 
> resource utilization.
> * Since the DynoYARN Node Managers run the actual code paths, all preemption 
> and queuing logic will be faithfully executed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8849) DynoYARN: A simulation and testing infrastructure for YARN clusters

2018-10-04 Thread Arun Suresh (JIRA)

[
https://issues.apache.org/jira/browse/YARN-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arun Suresh updated YARN-8849:
--
Description:
Traditionally, YARN workload simulation is performed using SLS (Scheduler Load
Simulator) which is packaged with YARN. It Essentially, starts a full fledged
*ResourceManager*, but runs simulators for the *NodeManager* and the
*ApplicationMaster* Containers. These simulators are lightweight and run in a
threadpool. The NM simulators do not open any external ports and send
(in-process) heartbeats to the ResourceManager.

There are a couple of drawbacks with using the SLS:
* It might be difficult to simulate really large clusters without having access
to a very beefy box - since the NMs are launched as tasks in a threadpool, and
each NM has to send periodic heartbeats to the RM.
* Certain features (like YARN-1011) requires changes to the NodeManager -
aspects such as queuing and selectively killing containers have to be
incorporate into the existing NM Simulator which might make the simulator a bit
heavy weight - there is a need for locking and synchronization.
* Since the NM and AM are simulations, only the Scheduler is faithfully tested
- it does not really perform an end-2-end test of a cluster.

Therefore, drawing inspiration from
[Dynamometer|https://github.com/linkedin/dynamometer], we propose a framework
for YARN deployable YARN cluster - *DynoYARN* - for testing, with the following
features:
* The NM already has hooks to plug-in custom *ContainerExecutor* and
*NodeResourceMonitor*. If we can plug-in a custom *ContainersMonitorImpl*'s
Monitoring thread (and other modules like the LocalizationService), We can
probably inject an Executor that does not actually launch containers and a Node
and Container resource monitor that reports synthetic pre-specified Utilization
metrics back to the RM.
* Since we are launching fake containers, we cannot run normal AM containers.
We can therefore, use *Unmanaged AM*'s to launch synthetic jobs.

Essentially, a test workflow would look like this:
* Launch a DynoYARN cluster.
* Use the Unmanaged AM feature to directly negotiate with the DynaYARN Resource
Manager for container tokens.
* Use the container tokens from the RM to directly ask the DynoYARN Node
Managers to start fake containers.
* The DynoYARN NodeManagers will start the fake containers and report to the
DynoYARN Resource Manager synthetically generated resource utilization for the
containers (which will be injected via the *ContainerLaunchContext* and parsed
by the plugged-in Container Executor).
* The Scheduler will use the utilization report to schedule containers - we
will be able to test allocation of *Opportunistic* containers based on resource
utilization.
* Since the DynoYARN Node Managers run the actual code paths, all preemption
and queuing logic will be faithfully executed.

was:
Traditionally, YARN workload simulation is performed using SLS (Scheduler Load
Simulator) which is packaged with YARN. It Essentially, starts a full fledged
*ResourceManager*, but runs simulators for the *NodeManager* and the
*ApplicationMaster* Containers. These simulators are lightweight and run in a
threadpool. The NM simulators do not open any external ports and send
(in-process) heartbeats to the ResourceManager.

1 2 >

1 - 100 of 113 matches

Mail list logo