[jira] [Resolved] (YARN-4547) LeafQueue#getApplications() is read-only interface, but it provides reference to caller

2016-02-16 Thread Rohith Sharma K S (JIRA) 

 [ 
https://issues.apache.org/jira/browse/YARN-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-4547.
-
Resolution: Done

This change is handled along with YARN-4617. Closing it as Done

> LeafQueue#getApplications() is read-only interface, but it provides reference 
> to caller
> ---
>
> Key: YARN-4547
> URL: https://issues.apache.org/jira/browse/YARN-4547
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>
> The below API is read-only interface, but returning reference to the caller. 
> This causing caller to modify the orderingPolicy entities. If required 
> reference of ordering policy, caller can use 
> {{LeagQueue#getOrderingPolicy()#getSchedulableEntities()}}
> The returning object should be clone of 
> orderingPolicy.getSchedulableEntities()
> {code}
>   /**
>* Obtain (read-only) collection of active applications.
>*/
>   public Collection getApplications() {
> return orderingPolicy.getSchedulableEntities();
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Hadoop-Yarn-trunk #1774

2016-02-16 Thread Apache Jenkins Server 
See

Hadoop-Yarn-trunk-Java8 - Build # 1064 - Failure

2016-02-16 Thread Apache Jenkins Server 
See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/1064/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 4181 lines...]
[INFO] 
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-yarn-server-common ---
[INFO] Building jar: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Yarn-trunk-Java8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/target/hadoop-yarn-server-common-3.0.0-SNAPSHOT-test-sources.jar
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ 
hadoop-yarn-server-common ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
hadoop-yarn-server-common ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-yarn-server-common ---
[INFO] Skipping javadoc generation
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (depcheck) @ 
hadoop-yarn-server-common ---
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:checkstyle (default-cli) @ 
hadoop-yarn-server-common ---
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop YARN  SUCCESS [  7.404 s]
[INFO] Apache Hadoop YARN API  SUCCESS [02:09 min]
[INFO] Apache Hadoop YARN Common . SUCCESS [03:22 min]
[INFO] Apache Hadoop YARN Server . SUCCESS [  0.092 s]
[INFO] Apache Hadoop YARN Server Common .. FAILURE [ 17.682 s]
[INFO] Apache Hadoop YARN NodeManager  SKIPPED
[INFO] Apache Hadoop YARN Web Proxy .. SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService .. SKIPPED
[INFO] Apache Hadoop YARN ResourceManager  SKIPPED
[INFO] Apache Hadoop YARN Server Tests ... SKIPPED
[INFO] Apache Hadoop YARN Client . SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager . SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage  SKIPPED
[INFO] Apache Hadoop YARN Applications ... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ... SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher .. SKIPPED
[INFO] Apache Hadoop YARN Site ... SKIPPED
[INFO] Apache Hadoop YARN Registry ... SKIPPED
[INFO] Apache Hadoop YARN Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 05:59 min
[INFO] Finished at: 2016-02-17T03:45:30+00:00
[INFO] Final Memory: 80M/1120M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-checkstyle-plugin:2.15:checkstyle (default-cli) 
on project hadoop-yarn-server-common: An error has occurred in Checkstyle 
report generation. Failed during checkstyle execution: Unable to process 
configuration file at location: checkstyle/checkstyle.xml: Cannot create 
file-based resource:invalid code lengths set -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :hadoop-yarn-server-common
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Sending e-mails to: yarn-dev@hadoop.apache.org
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

Build failed in Jenkins: Hadoop-Yarn-trunk-Java8 #1064

2016-02-16 Thread Apache Jenkins Server 
See 

Changes:

[junping_du] HDFS-9456. BlockPlacementPolicyWithNodeGroup should override

--
[...truncated 3988 lines...]
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.conf.TestHAUtil
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.03 sec - in 
org.apache.hadoop.yarn.conf.TestHAUtil
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.TestRecordFactory
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.405 sec - in 
org.apache.hadoop.yarn.TestRecordFactory
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.util.TestWebAppUtils
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.487 sec - in 
org.apache.hadoop.yarn.webapp.util.TestWebAppUtils
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.TestParseRoute
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.244 sec - in 
org.apache.hadoop.yarn.webapp.TestParseRoute
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.test.TestWebAppTests
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.708 sec - in 
org.apache.hadoop.yarn.webapp.test.TestWebAppTests
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.TestWebApp
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.381 sec - in 
org.apache.hadoop.yarn.webapp.TestWebApp
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.hamlet.TestParseSelector
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.167 sec - in 
org.apache.hadoop.yarn.webapp.hamlet.TestParseSelector
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.hamlet.TestHamlet
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.593 sec - in 
org.apache.hadoop.yarn.webapp.hamlet.TestHamlet
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.hamlet.TestHamletImpl
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.591 sec - in 
org.apache.hadoop.yarn.webapp.hamlet.TestHamletImpl
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.TestSubViews
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.085 sec - in 
org.apache.hadoop.yarn.webapp.TestSubViews
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.view.TestInfoBlock
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.991 sec - in 
org.apache.hadoop.yarn.webapp.view.TestInfoBlock
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.view.TestCommonViews
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.073 sec - in 
org.apache.hadoop.yarn.webapp.view.TestCommonViews
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.view.TestTwoColumnCssPage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.107 sec - in 
org.apache.hadoop.yarn.webapp.view.TestTwoColumnCssPage
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.view.TestHtmlPage
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.997 sec - in 
org.apache.hadoop.yarn.webapp.view.TestHtmlPage
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.view.TestTwoColumnLayout
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.509 sec - in 
org.apache.hadoop.yarn.webapp.view.TestTwoColumnLayout
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.yarn.webapp.view.TestHtmlBlock
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.032 sec - in 
org.apache.hadoop.yarn.webapp.view.TestHtmlBlock
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running

[jira] [Created] (YARN-4697) NM aggregation thread pool is not bound by limits

2016-02-16 Thread Haibo Chen (JIRA) 
Haibo Chen created YARN-4697:


 Summary: NM aggregation thread pool is not bound by limits
 Key: YARN-4697
 URL: https://issues.apache.org/jira/browse/YARN-4697
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: nodemanager
Reporter: Haibo Chen
Assignee: Haibo Chen


In the LogAggregationService.java we create a threadpool to upload logs from 
the nodemanager to HDFS if log aggregation is turned on. This is a cached 
threadpool which based on the javadoc is an ulimited pool of threads.
In the case that we have had a problem with log aggregation this could cause a 
problem on restart. The number of threads created at that point could be huge 
and will put a large load on the NameNode and in worse case could even bring it 
down due to file descriptor issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (YARN-4696) EntityGroupFSTimelineStore to work in the absence of an RM

2016-02-16 Thread Steve Loughran (JIRA) 
Steve Loughran created YARN-4696:


 Summary: EntityGroupFSTimelineStore to work in the absence of an RM
 Key: YARN-4696
 URL: https://issues.apache.org/jira/browse/YARN-4696
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: 2.8.0
Reporter: Steve Loughran


{{EntityGroupFSTimelineStore}} now depends on an RM being up and running; the 
configuration pointing to it. This is a new change, and impacts testing where 
you have historically been able to test without an RM running.

The sole purpose of the probe is to automatically determine if an app is 
running; it falls back to "unknown" if not. If the RM connection was optional, 
the "unknown" codepath could be called directly, relying on age of file as a 
metric of completion

Options

# add a flag to disable RM connect
# skip automatically if RM not defined/set to 0.0.0.0
# disable retries on yarn client IPC; if it fails, tag app as unknown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (YARN-4695) EntityGroupFSTimelineStore to not log errors during shutdown

2016-02-16 Thread Steve Loughran (JIRA) 
Steve Loughran created YARN-4695:


 Summary: EntityGroupFSTimelineStore to not log errors during 
shutdown
 Key: YARN-4695
 URL: https://issues.apache.org/jira/browse/YARN-4695
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: 2.8.0
Reporter: Steve Loughran


# The {{EntityGroupFSTimelineStore}} threads log exceptions that get raised 
during their execution.
# the service stops by interrupting all its workers
# as a result, the workers all log exceptions at error *even during a managed 
shutdown*
# this creates distracting noise in logs




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)