[jira] [Created] (YARN-8516) Compilation error for branch-2.8
Rohith Sharma K S created YARN-8516:
---
Summary: Compilation error for branch-2.8
Key: YARN-8516
URL: https://issues.apache.org/jira/browse/YARN-8516
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
branch-2.8 compilation is failing with below error
{noformat}
INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 6.142 s
[INFO] Finished at: 2018-07-11T08:28:24+05:30
[INFO] Final Memory: 64M/790M
[INFO]
[WARNING] The requested profile "yarn-ui" could not be activated because it
does not exist.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-yarn-server-nodemanager: Compilation failure
[ERROR]
/Users/rsharmaks/Repos/Apache/Commit_Repos/branch-2.8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java:[333,12]
no suitable method found for
warn(java.lang.String,org.apache.hadoop.yarn.api.records.ContainerId,org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl,org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationState)
[ERROR] method org.apache.commons.logging.Log.warn(java.lang.Object) is not
applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] method
org.apache.commons.logging.Log.warn(java.lang.Object,java.lang.Throwable) is
not applicable
[ERROR] (actual and formal argument lists differ in length)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7806) DS will hang if ATSv2 back end is unavailable.
Rohith Sharma K S created YARN-7806: --- Summary: DS will hang if ATSv2 back end is unavailable. Key: YARN-7806 URL: https://issues.apache.org/jira/browse/YARN-7806 Project: Hadoop YARN Issue Type: Bug Environment: DS publishes container start/stop events using sync API. If back end is not down for some reasons, then DS will hang till container start/stop events are published. By default, retry is 30 and interval is 1sec. To publish single entity using sync API will take 1 minutes to come out. In case of DS, if number of containers are 10 then 10minutes for start event and 10minutes for stop event. Overall 20 minutes will wait. DS should publish container events using asyn api. Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7765) [Atsv2] App collector failed to authenticate with HBase in secure cluster
Rohith Sharma K S created YARN-7765:
---
Summary: [Atsv2] App collector failed to authenticate with HBase
in secure cluster
Key: YARN-7765
URL: https://issues.apache.org/jira/browse/YARN-7765
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Secure cluster is deployed and all YARN services are started successfully. When
application is submitted, app collectors which is started as aux-service
throwing below exception. But this exception is *NOT* observed from RM
TimelineCollector.
{noformat}
2018-01-17 11:04:48,017 FATAL ipc.RpcClientImpl (RpcClientImpl.java:run(684)) -
SASL authentication failed. The most likely cause is missing or invalid
credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
{noformat}
cc :/ [~vrushalic] [~varun_saxena]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8006) Make Hbase-2 profile as default for YARN-7055 branch
Rohith Sharma K S created YARN-8006: --- Summary: Make Hbase-2 profile as default for YARN-7055 branch Key: YARN-8006 URL: https://issues.apache.org/jira/browse/YARN-8006 Project: Hadoop YARN Issue Type: Sub-task Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S In last weekly call folks discussed that we should have separate branch with hbase-2 as profile by default. Trunk default profile is hbase-1 which runs all the tests under hbase-1 profile. But for hbase-2 profile tests are not running. As per the discussion, lets keep YARN-7055 branch for hbase-2 profile as default. Any server side patches can be given to this branch as well which runs tests for hbase-2 profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7212) [Atsv2] TimelineSchemaCreator fails to create flowrun table causes RegionServer down!
[
https://issues.apache.org/jira/browse/YARN-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved YARN-7212.
-
Resolution: Done
Definitely NO. I am closing this JIRA as done as part of YARN-7346
> [Atsv2] TimelineSchemaCreator fails to create flowrun table causes
> RegionServer down!
> -
>
> Key: YARN-7212
> URL: https://issues.apache.org/jira/browse/YARN-7212
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Priority: Major
>
> *Hbase-2.0* officially support *hadoop-alpha* compilations. So I was trying
> to build and test with HBase-2.0. But table schema creation fails and causes
> RegionServer to shutdown with following error
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> org.apache.hadoop.hbase.Tag.asList([BII)Ljava/util/List;
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.getCurrentAggOp(FlowScanner.java:250)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.nextInternal(FlowScanner.java:226)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.next(FlowScanner.java:145)
> at
> org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:132)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
> at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:973)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2252)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2672)
> {noformat}
> Since HBase-2.0 community is ready to release Hadoop-3.x compatible versions,
> ATSv2 also need to support HBase-2.0 versions. For this, we need to take up a
> task of test and validate HBase-2.0 issues!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8023) REST API doesn't show new application
[
https://issues.apache.org/jira/browse/YARN-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved YARN-8023.
-
Resolution: Invalid
> REST API doesn't show new application
> -
>
> Key: YARN-8023
> URL: https://issues.apache.org/jira/browse/YARN-8023
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Affects Versions: 2.7.3
> Environment: Release label:emr-5.5.0
> Hadoop distribution:Amazon 2.7.3
> Applications:Spark 2.1.0, Hive 2.1.1, Hue 3.12.0
>Reporter: Airton Sampaio de Sobral
>Priority: Major
> Attachments: Screen Shot 2018-03-10 at 5.46.13 PM.png
>
>
> After killing an application using the HADOOP UI, and creating a new one with
> the same configuration, the brand new one doesn't appear on the API route:
> /api/v1/applications?status=running
> I've tried the applications path as well (/api/v1/applications) without
> success.
> This happens randomly and it seems that after a really long time, the new
> instance appears on the API.
> On the UI the new application instance appears, and it's working fine. (Print
> screen on attachment).
> On the API it shows the last instance of the application as it's running, but
> it's dead by one hour!
> {code:java}
> /api/v1/applications?status=running
> {
> "id" : "application_1511385973584_0087",
> "name" : "AdActionPaymentKafkaToJDBC",
> "attempts" : [ {
> "attemptId" : "1",
> "startTime" : "2018-01-16T19:08:32.275GMT",
> "endTime" : "1969-12-31T23:59:59.999GMT",
> "lastUpdated" : "2018-01-16T19:08:34.016GMT",
> "duration" : 0,
> "sparkUser" : "hadoop",
> "completed" : false,
> "endTimeEpoch" : -1,
> "startTimeEpoch" : 1516129712275,
> "lastUpdatedEpoch" : 1516129714016
> }
> {code}
>
> Update:
> After two hours, the application appeared on the API response:
>
> {code:java}
> {
> "id" : "application_1511385973584_0154",
> "name" : "AdActionPaymentKafkaToJDBC",
> "attempts" : [ {
> "attemptId" : "1",
> "startTime" : "2018-03-10T21:08:30.557GMT",
> "endTime" : "1969-12-31T23:59:59.999GMT",
> "lastUpdated" : "2018-03-10T21:08:32.310GMT",
> "duration" : 0,
> "sparkUser" : "hadoop",
> "completed" : false,
> "endTimeEpoch" : -1,
> "startTimeEpoch" : 1520716110557,
> "lastUpdatedEpoch" : 1520716112310
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8124) Service Application Master log file can't be found.
Rohith Sharma K S created YARN-8124:
---
Summary: Service Application Master log file can't be found.
Key: YARN-8124
URL: https://issues.apache.org/jira/browse/YARN-8124
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
It is observed that service am log file can't be found in log folder. When
inspected, _yarnservice-log4j.properties_ has entry for
log4j.appender.amlog.File=*${LOG_DIR}/serviceam.log* where LOG_DIR is not
resolving.
When changed above value to log4j.appender.amlog.File=*./serviceam.log*, able
to see the log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8133) Doc link broken for yarn-service from overview page.
Rohith Sharma K S created YARN-8133: --- Summary: Doc link broken for yarn-service from overview page. Key: YARN-8133 URL: https://issues.apache.org/jira/browse/YARN-8133 Project: Hadoop YARN Issue Type: Bug Components: yarn-native-services Affects Versions: 3.1.0 Reporter: Rohith Sharma K S I see that documentation link broken from overview page. Any link clicking from http://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html page causing an error. It looks like Overview page, redirecting with .md page which doesn't exist. It should redirect to *.html page -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8145) yarn rmadmin -getGroups doesn't return updated groups for user
Rohith Sharma K S created YARN-8145: --- Summary: yarn rmadmin -getGroups doesn't return updated groups for user Key: YARN-8145 URL: https://issues.apache.org/jira/browse/YARN-8145 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S Post YARN-8062, we sees that still some cache clearing issue for yarn rmadmin -getGroups which causes getGroup results in stale data for sometime. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8156) Increase the default value of yarn.timeline-service.app-collector.linger-period.ms
Rohith Sharma K S created YARN-8156: --- Summary: Increase the default value of yarn.timeline-service.app-collector.linger-period.ms Key: YARN-8156 URL: https://issues.apache.org/jira/browse/YARN-8156 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S The default value of *yarn.timeline-service.app-collector.linger-period.ms* is 1 sec which is very low. In production cluster, we observe that most of the entity information is not published because of 1sec linger period. I would suggest to increase the default value to either 10sec or 30sec or 60 sec. As far as I have seen, 10sec should be sufficient since node heartbeat period is 3 sec by default, all NM gets request to stop the application. cc:/ [~haibo.chen] [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
Rohith Sharma K S created YARN-8155: --- Summary: Improve the logging in NMTimelinePublisher and TimelineCollectorWebService Key: YARN-8155 URL: https://issues.apache.org/jira/browse/YARN-8155 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S We see that NM logs are filled with larger stack trace of NotFoundException if collector is removed from one of the NM and other NMs are still publishing the entities. This Jira is to improve the logging in NM so that we log with informative message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8126) [Follow up] Support auto-spawning of admin configured services during bootstrap of rm
Rohith Sharma K S created YARN-8126: --- Summary: [Follow up] Support auto-spawning of admin configured services during bootstrap of rm Key: YARN-8126 URL: https://issues.apache.org/jira/browse/YARN-8126 Project: Hadoop YARN Issue Type: Sub-task Reporter: Rohith Sharma K S YARN-8048 adds support auto-spawning of admin configured services during bootstrap of rm. This JIRA is to follow up some of the comments discussed in YARN-8048. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8131) Provice CLI option to DS for publishing entities into sub application
Rohith Sharma K S created YARN-8131: --- Summary: Provice CLI option to DS for publishing entities into sub application Key: YARN-8131 URL: https://issues.apache.org/jira/browse/YARN-8131 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S Post YARN-6936, TimelineV2Client exposes API to publish entities into sub application table. We should add this CLI option in DS so that API can be tested. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8073) TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration
Rohith Sharma K S created YARN-8073: --- Summary: TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration Key: YARN-8073 URL: https://issues.apache.org/jira/browse/YARN-8073 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S Post YARN-6736, RM support writing into ats v1 and v2 by new configuration setting _yarn.timeline-service.versions_. Couple of issues observed in deployment are # TimelineClientImpl doesn't honor newly added configuration rather it still get version number from _yarn.timeline-service.version_. This causes not writing into v1.5 API's even though _yarn.timeline-service.versions has 1.5 value._ # Similar line from 1st point, TimelineUtils#timelineServiceV1_5Enabled doesn't honor timeline-service.versions. # JobHistoryEventHandler#serviceInit(), line no 271 check for version number rather than calling YarnConfiguration#timelineServiceV2Enabled cc :/ [~agresch] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8053) Exclude hadoop-distcp dependencies in hbase-server for timelineservice-hbase-client package.
Rohith Sharma K S created YARN-8053:
---
Summary: Exclude hadoop-distcp dependencies in hbase-server for
timelineservice-hbase-client package.
Key: YARN-8053
URL: https://issues.apache.org/jira/browse/YARN-8053
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Assignee: Rohith Sharma K S
It is observed that when we change the version number of hadoop leading build
failure because of dependency resolution conflicts. We see below error which
tells that hbase-server has dependency on hadoop-distcp. We also need to
exclude hadoop-distcp from exclusion list.
{code}
07:42:36 2018/03/19 14:42:36 INFO: [ERROR] Failed to execute goal on
project hadoop-yarn-server-timelineservice-hbase-client: Could not resolve
dependencies for project
org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:jar:3.0.0.3.0.0.0-1059:
Could not find artifact org.apache.hadoop:hadoop-distcp:jar:3.0.0.3.0.0.0-1059
in public
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8063) DistributedShellTimelinePlugin wrongly check for entityId instead of entityType
Rohith Sharma K S created YARN-8063:
---
Summary: DistributedShellTimelinePlugin wrongly check for entityId
instead of entityType
Key: YARN-8063
URL: https://issues.apache.org/jira/browse/YARN-8063
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Assignee: Rohith Sharma K S
DistributedShellTimelinePlugin#getTimelineEntityGroupId compare with entityId
rather than entityType. This causes to fail to getTimelineEntityGroupId.
{code}
public Set getTimelineEntityGroupId(String entityId,
String entityType) {
if (ApplicationMaster.DSEntity.DS_CONTAINER.toString().equals(entityId)) {
ContainerId containerId = ContainerId.fromString(entityId);
ApplicationId appId = containerId.getApplicationAttemptId()
.getApplicationId();
return toEntityGroupId(appId.toString());
}
return null;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8048) Support auto-spawning of admin configured services during bootstrap of rm/apiserver
Rohith Sharma K S created YARN-8048: --- Summary: Support auto-spawning of admin configured services during bootstrap of rm/apiserver Key: YARN-8048 URL: https://issues.apache.org/jira/browse/YARN-8048 Project: Hadoop YARN Issue Type: Sub-task Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S Goal is to support auto-spawning of admin configured services during bootstrap of resourcemanager/apiserver. *Requirement:* Some of the services might required to be consumed by yarn itself ex: Hbase for atsv2. Instead of depending on user installed HBase or sometimes user may not required to install HBase at all, in such conditions running HBase app on YARN will help for ATSv2. Before YARN cluster is started, admin configure these services spec and place it in common location in HDFS. At the time of RM/apiServer bootstrap, these services will be submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3163) admin support for YarnAuthorizationProvider
[ https://issues.apache.org/jira/browse/YARN-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3163. - Resolution: Won't Fix > admin support for YarnAuthorizationProvider > --- > > Key: YARN-3163 > URL: https://issues.apache.org/jira/browse/YARN-3163 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Sunil G >Assignee: Sunil G >Priority: Major > > Runtime configuration support for YarnAuthorizationProvider. Using admin > commands, one should be able to set and get permission from the > YarnAuthorizationProvider. This mechanism will help users without updating > config files and firing reload commands. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3162) persistence support for YarnAuthorizationProvider
[ https://issues.apache.org/jira/browse/YARN-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3162. - Resolution: Won't Fix > persistence support for YarnAuthorizationProvider > - > > Key: YARN-3162 > URL: https://issues.apache.org/jira/browse/YARN-3162 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Sunil G >Assignee: Rohith Sharma K S >Priority: Major > > As discussed in YARN-3100, admin support can be a good addition for > YarnAuthorizationProvider. Hence sync up between memory store and config file > will be of higher importantance. This JIRA will focus on a persistence > storage for ACLs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7701) Both RM are in standby in secure cluster
[
https://issues.apache.org/jira/browse/YARN-7701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved YARN-7701.
-
Resolution: Cannot Reproduce
I tried in trunk to reproduce same, but could not get this. The reason is
YARN-6061 and YARN-3742 are fixed in trunk which triggers event to transition
to standby.
I am closing as can't reproduce in trunk!
> Both RM are in standby in secure cluster
>
>
> Key: YARN-7701
> URL: https://issues.apache.org/jira/browse/YARN-7701
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0, 2.8.3, 3.0.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Critical
> Attachments: YARN-7701.01.patch
>
>
> Both RM were running perfectly fine for many days and switched multiple
> times. At some point of time when RM is switched from ACTIVE -> STANDBY, UGI
> information got either changed or to subject new user got added.
> As a result UGI#getShortUserName() is returning wrong user which result in
> fail to transition to ACTIVE with AccessControlException!
> {code}Caused by: org.apache.hadoop.security.AccessControlException: User
> odsuser doesn't have permission to call 'refreshAdminAcls'
> {code}
> _odsuser_ user is application submitted user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5512) Finished containers for running application should be displayed on container table
[ https://issues.apache.org/jira/browse/YARN-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-5512. - Resolution: Implemented I am not sure which is the sub task under YARN-3368 fixes this issue. But I see completed container details in UI2. So I am closing as implemented. > Finished containers for running application should be displayed on container > table > -- > > Key: YARN-5512 > URL: https://issues.apache.org/jira/browse/YARN-5512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > > In offline discussion with [~vinodkv], one of the point on yarn-web-ui > improvement is, > Currently yarn-web-ui attempt page displays running container details. But > these container disappear once it got finished. Earlier there was no > mechanism to track finished container details. Now, once ATSv2 is ready, > finished containers are being published from NodeManager and can be read. It > would be good if finished containers details also displayed. > In new RM web ui , better if we can consider this also. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7977) Do ACLs check for flow activity entities
Rohith Sharma K S created YARN-7977: --- Summary: Do ACLs check for flow activity entities Key: YARN-7977 URL: https://issues.apache.org/jira/browse/YARN-7977 Project: Hadoop YARN Issue Type: Sub-task Components: timelinereader Reporter: Rohith Sharma K S Verify ACLs while retrieving flow activity entities -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7979) Do ACLs check for application entities
Rohith Sharma K S created YARN-7979: --- Summary: Do ACLs check for application entities Key: YARN-7979 URL: https://issues.apache.org/jira/browse/YARN-7979 Project: Hadoop YARN Issue Type: Sub-task Components: timelinereader Reporter: Rohith Sharma K S Verify ACLs for application entities -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7978) Do ACLs check while retriving flowrun entities
Rohith Sharma K S created YARN-7978: --- Summary: Do ACLs check while retriving flowrun entities Key: YARN-7978 URL: https://issues.apache.org/jira/browse/YARN-7978 Project: Hadoop YARN Issue Type: Sub-task Components: timelinereader Reporter: Rohith Sharma K S Verify ACLs while retrieving flowrun entities -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7981) Do ACLs check for sub app entities
Rohith Sharma K S created YARN-7981: --- Summary: Do ACLs check for sub app entities Key: YARN-7981 URL: https://issues.apache.org/jira/browse/YARN-7981 Project: Hadoop YARN Issue Type: Sub-task Components: timelinereader Reporter: Rohith Sharma K S ACLs check while retrieving sub app entities. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7980) Do ACLs check for generic entities
Rohith Sharma K S created YARN-7980: --- Summary: Do ACLs check for generic entities Key: YARN-7980 URL: https://issues.apache.org/jira/browse/YARN-7980 Project: Hadoop YARN Issue Type: Sub-task Components: timelinereader Reporter: Rohith Sharma K S This JIRA is to verify ACLs for generic entities. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7982) Do ACLs check while retrieving entity-types per application
Rohith Sharma K S created YARN-7982:
---
Summary: Do ACLs check while retrieving entity-types per
application
Key: YARN-7982
URL: https://issues.apache.org/jira/browse/YARN-7982
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Rohith Sharma K S
REST end point {{/apps/$appid/entity-types}} retrieves all the entity-types for
given application. This need to be guarded with ACL check
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7983) Add table for storing timeline domain informations
Rohith Sharma K S created YARN-7983: --- Summary: Add table for storing timeline domain informations Key: YARN-7983 URL: https://issues.apache.org/jira/browse/YARN-7983 Project: Hadoop YARN Issue Type: Sub-task Reporter: Rohith Sharma K S Create a timeline domain table in HBase for storing domain information -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
Rohith Sharma K S created YARN-8964:
---
Summary: UI2 should use clusters/{cluster name} for all ATSv2 REST
APIs
Key: YARN-8964
URL: https://issues.apache.org/jira/browse/YARN-8964
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Rohith Sharma K S
UI2 makes a REST call to TimelineReader without cluster name. It is advised to
make a REST call with clusters/{cluster name} so that remote TimelineReader
daemon could serve for different clusters.
*Example*:
*Current*: /ws/v2/timeline/flows/
*Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
*yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
could be used to get cluster-id
cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9028) TLSv2.0 ats-hbase app flooding Isilon OneFS hdfs.log and ats-hbase region server logs
[ https://issues.apache.org/jira/browse/YARN-9028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-9028. - Resolution: Invalid > TLSv2.0 ats-hbase app flooding Isilon OneFS hdfs.log and ats-hbase region > server logs > - > > Key: YARN-9028 > URL: https://issues.apache.org/jira/browse/YARN-9028 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 >Affects Versions: 3.0.1 >Reporter: Kirankumar Bhusanurmath >Priority: Major > > HDp3.0.1 deployed on DellEMC Isilon OneFS 8.1.2 > The ats-hbase app is up and running but flooding the OneFS hdfs and ats-hbase > region server logs with below message. > OneFS log > 2018-11-16T04:29:44Z <30.6> kbhusan-y93o5ew-1 hdfs[2711]: [hdfs] RPC V9 > complete user: yarn-ats exception: java.io.FileNotFoundException cause: File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.1542088499176 > STATUS_OBJECT_NAME_NOT_FOUND = 0xC034 Path: > atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.1542088499176> > 2018-11-16T04:29:45Z <30.6> kbhusan-y93o5ew-1 hdfs[2711]: [hdfs] RPC V9 > complete user: yarn-ats exception: java.io.FileNotFoundException cause: File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.meta.1542185701850.meta > STATUS_OBJECT_NAME_NOT_FOUND = 0xC034 Path: > atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.meta.1542185701850.meta> > > hbase-yarn-ats-regionserver-RDUVNODE334518.log. > 2018-11-13 03:23:30,178 WARN [Close-WAL-Writer-22] > asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1541663220372/rduvnode334518.west.isilon.com%2C17020%2C1541663220372.1541742426412 > failed, retry = 16586 > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1541663220372/rduvnode334518.west.isilon.com%2C17020%2C1541663220372.1541742426412 > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy18.complete(Unknown Source) > > (each logs are more then 250MB unable to upload > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9029) yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle
[ https://issues.apache.org/jira/browse/YARN-9029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-9029. - Resolution: Invalid This is vendor specific deployment issue. Please contact the vendor! > yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle > > > Key: YARN-9029 > URL: https://issues.apache.org/jira/browse/YARN-9029 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 >Affects Versions: 3.0.1 > Environment: Ambari 2.7.1 > HDP 3.0.1 > Isilon OneFS 8.1.2 >Reporter: Kirankumar Bhusanurmath >Priority: Major > > *Background on ats-hbase start success*:: We had setStoragePolicy exception > on OneFS for yarn-ats hbase region server and below messages respectively on > HDP and OneFS cluster. > *HDP Host* *è* */var/log/hadoop-yarn/embedded-yarn-ats-hbase/* > 2018-11-02 05:07:21,027 WARN [StoreOpener-1588230740-1] util.CommonFSUtils: > Unable to set storagePolicy=HOT for > path=hdfs://kb-hdp-1.west.isilon.com:8020/atsv2/hbase/data/data/hbase/meta/1588230740/info. > DEBUG log level might have more details. > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > *OneFS HDFS log for (unsecured)* > 2018-11-01T10:25:19Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 > user: yarn-ats exception: org.apache.hadoop.ipc.RpcNoSuchMethodException > cause: Unknown RPC: setStoragePolicy > > *OneFS HDFS log for (secured)* > 2018-11-01T12:39:56Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 > user: > yarn-ats-hbase/[krb-hdp301.solarch.emc@solarch.emc.com|mailto:krb-hdp301.solarch.emc@solarch.emc.com] > exception: > org.apache.hadoop.ipc.RpcNoSuchMethodException cause: Unknown RPC: > setStoragePolicy > > > As a resolution, we set below 2 properties in custom hbase-site.xml file and > setstoragePolicy exception got resolved. > *Custom yarn-hbase-site.xml* > 'hbase.wal.storage.policy' to ‘NONE’ > ‘hbase.hstore.block.storage.policy’ to ‘NONE’ > > > * > *NOW*, we have a new issue that is OneFS HDFS log and > /var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-.log > gets flooded with below error messages.*OneFS HDFS Log* > --- > 2018-11-06T06:25:43Z <30.6> kbhusan-edl9tdc-1 hdfs[2542]: [hdfs] RPC V9 > complete user: > yarn-ats-hbase/[rduvnode343393.west.isilon@west.isilon.com|mailto:rduvnode343393.west.isilon@west.isilon.com] > exception: > java.io.FileNotFoundException cause: File not found: > /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta > STATUS_OBJECT_NAME_NOT_FOUND = 0xC034 Path: > atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta> > --- > > */var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-.log* > -- > 2018-11-06 04:01:12,863 INFO [LruBlockCacheStatsExecutor] > hfile.LruBlockCache: totalSize=1.12 MB, freeSize=1.25 GB, max=1.25 GB, > blockCount=24, accesses=287, hits=263, hitRatio=91.64%, , > cachingAccesses=287, cachingHits=263, cachingHitsRatio=91.64%, > evictions=419, evicted=0, evictedPerRun=0.0 > > 2018-11-06 04:01:48,763 WARN > [LeaseRenewer:[yarn-...@kb-hdp301-2.west.isilon.com|mailto:yarn-...@kb-hdp301-2.west.isilon.com]:8020] > impl.LeaseRenewer: > Failed to renew lease for [DFSClient_NONMAPREDUCE_-533598397_1] for 30 > seconds. Will retry shortly ... > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > Client auth method 'SIMPLE', mechanism '' is not supported on zone:1 > auth:'kerberos_only' > > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) > > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > > “ > “ > “ > “ > 2018-11-06 12:51:02,843 WARN [Close-WAL-Writer-1] > asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file > /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541488822956/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541488822956.1541492429802 > failed, retry = 637 > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File > not
[jira] [Created] (YARN-9031) YARN UI2 should make use of "yarn.log.server.web-service.url" to fetch container logs
Rohith Sharma K S created YARN-9031: --- Summary: YARN UI2 should make use of "yarn.log.server.web-service.url" to fetch container logs Key: YARN-9031 URL: https://issues.apache.org/jira/browse/YARN-9031 Project: Hadoop YARN Issue Type: Improvement Reporter: Rohith Sharma K S Post YARN-5742, ATSv2.0 introduces log servlet which could be used for log fetching. Many cluster deployment ATS1.5 may not exist which causes log fetching failure in UI2. # UI2 should be able to fetch logs from ATS2.0 log servlet. # Its better to take log server url based on the property "yarn.log.server.web-service.url" # Each ATSv2 log URL takes clusterid as query params. UI2 should send clusterid query param if required. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9142) UI cluster nodes page is broken
Rohith Sharma K S created YARN-9142: --- Summary: UI cluster nodes page is broken Key: YARN-9142 URL: https://issues.apache.org/jira/browse/YARN-9142 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S Attachments: ClusterNodePage.png It is observed in trunk build YARN cluster node pages is broken even though data exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9042) Javadoc error in deviceplugin package
Rohith Sharma K S created YARN-9042:
---
Summary: Javadoc error in deviceplugin package
Key: YARN-9042
URL: https://issues.apache.org/jira/browse/YARN-9042
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Many java doc errors are in deviceplugin
{noformat}
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:29:
error: bad HTML entity
[ERROR] * This is a spec used to prepare & run container.
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:35:
error: bad HTML entity
[ERROR] * The volume & device mounts describes key isolation requirements
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: domain
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: bus
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: slot
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: func
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
Rohith Sharma K S created YARN-9044: --- Summary: LogsCLI should contact ATSv2 for "-am" option Key: YARN-9044 URL: https://issues.apache.org/jira/browse/YARN-9044 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not configured. Rather LogsCLI should contact ATSv2 for AM container info. Alternative to above one can use *yarn logs -containerId * to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9034) ApplicationCLI should have option to take clusterId
Rohith Sharma K S created YARN-9034: --- Summary: ApplicationCLI should have option to take clusterId Key: YARN-9034 URL: https://issues.apache.org/jira/browse/YARN-9034 Project: Hadoop YARN Issue Type: Sub-task Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S Post YARN-8303, LogsCLI provide an option to input clusterid which could be used for fetching data from atsv2. ApplicationCLI also should have this option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9149) yarn container -status misses logUrl when integrated with ATSv2
Rohith Sharma K S created YARN-9149:
---
Summary: yarn container -status misses logUrl when integrated with
ATSv2
Key: YARN-9149
URL: https://issues.apache.org/jira/browse/YARN-9149
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Post YARN-8303, yarn client can be integrated with ATSv2. But log url and start
and end time is printing data is wrong!
{code}
Container Report :
Container-Id : container_1545035586969_0001_01_01
Start-Time : 0
Finish-Time : 0
State : COMPLETE
Execution-Type : GUARANTEED
LOG-URL : null
Host : localhost:25006
NodeHttpAddress : localhost:25008
Diagnostics :
{code}
TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*. This
need set for proper log url based on yarn.log.server.web-service.url
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9242) Revert YARN-8270 from branch-3.1
[
https://issues.apache.org/jira/browse/YARN-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved YARN-9242.
-
Resolution: Done
I have reverted YARN-8270 in branch-3.1 and branch-3.1.2. Also updated Fix
Version in YARN-8270.
Closing as Done!
> Revert YARN-8270 from branch-3.1
>
>
> Key: YARN-9242
> URL: https://issues.apache.org/jira/browse/YARN-9242
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
>
> It is observed that in hadoop-3.1-RC0, NodeManager are unable to initialize
> TimelineCollectorWebService!
> Primary reason is HADOOP-15657 is not present in hadoop-3.1 branch!
> Following error is seen NM logs
> {noformat}
> Caused by: org.apache.hadoop.metrics2.MetricsException: Unsupported metric
> field putEntitiesFailureLatency of type
> org.apache.hadoop.metrics2.lib.MutableQuantiles
> at
> org.apache.hadoop.metrics2.lib.MutableMetricsFactory.newForField(MutableMetricsFactory.java:87)
> {noformat}
> We need to revert YARN-8270 from branch-3.1!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0
Rohith Sharma K S created YARN-9256: --- Summary: Make ATSv2 compilation default with hbase.profile=2.0 Key: YARN-9256 URL: https://issues.apache.org/jira/browse/YARN-9256 Project: Hadoop YARN Issue Type: Task Reporter: Rohith Sharma K S By default Hadoop compiles with hbase.profile one which corresponds to hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by default in trunk. This JIRA is to discuss for any concerns. cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9804) Update ATSv2 document for latest feature supports
Rohith Sharma K S created YARN-9804: --- Summary: Update ATSv2 document for latest feature supports Key: YARN-9804 URL: https://issues.apache.org/jira/browse/YARN-9804 Project: Hadoop YARN Issue Type: Sub-task Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S Revisit ATSv2 documents and update for GA features. And also for the road map. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5760) [ATSv2] Create HBase connection only if an app collector is publishing from NM
[ https://issues.apache.org/jira/browse/YARN-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-5760. - Resolution: Won't Fix No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [ATSv2] Create HBase connection only if an app collector is publishing from NM > -- > > Key: YARN-5760 > URL: https://issues.apache.org/jira/browse/YARN-5760 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Major > > Irrespective of NM handling an app or not, we initialize > HBaseTimelineWriterImpl in TimelineCollectorManager. > This in turn calls ConnectionFactory#createConnection to manage connections > with HBase. > But it seems this opens up a connection with Zookeeper (i.e. as soon as NM > starts up) instead of opening connection when atleast one app arrives for > publishing and closing it if no apps are being published from this NM. > This leads to unnecessary connections to Zookeeper. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4504) Retrospect on defaults for created time while querying
[ https://issues.apache.org/jira/browse/YARN-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-4504. - Resolution: Won't Fix No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > Retrospect on defaults for created time while querying > -- > > Key: YARN-4504 > URL: https://issues.apache.org/jira/browse/YARN-4504 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4261) fix the order of timelinereader in yarn/yarn.cmd
[ https://issues.apache.org/jira/browse/YARN-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-4261. - Resolution: Won't Fix No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > fix the order of timelinereader in yarn/yarn.cmd > > > Key: YARN-4261 > URL: https://issues.apache.org/jira/browse/YARN-4261 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena >Priority: Trivial > Labels: YARN-5355 > > The order of the timelinereader command is not correct in yarn/yarn.cmd. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5071) address HBase compatibility issues with trunk
[ https://issues.apache.org/jira/browse/YARN-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-5071. - Resolution: Won't Fix No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. Currently we support only hbase-1.2.4/5 hbase-2 versions. We have separate packages for both the versions. I guess this shouldn't be much issue as per current code in trunk. > address HBase compatibility issues with trunk > - > > Key: YARN-5071 > URL: https://issues.apache.org/jira/browse/YARN-5071 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Labels: YARN-5355 > > The trunk is now adding or planning to add more and more > backward-incompatible changes. Some examples include > - remove v.1 metrics classes (HADOOP-12504) > - update jersey version (HADOOP-9613) > - target java 8 by default (HADOOP-11858) > This poses big challenges for the timeline service v.2 as we have a > dependency on hbase which depends on an older version of hadoop. > We need to find a way to solve/contain/manage these risks before it is too > late. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3865) Backward compatibility of reader with ATSv1
[ https://issues.apache.org/jira/browse/YARN-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3865. - Resolution: Won't Fix No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > Backward compatibility of reader with ATSv1 > --- > > Key: YARN-3865 > URL: https://issues.apache.org/jira/browse/YARN-3865 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Major > Labels: YARN-5355 > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4343) Need to support Application History Server on ATSV2
[ https://issues.apache.org/jira/browse/YARN-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-4343. - Resolution: Duplicate This functionality exist in latest trunk code. I don't remember which JIRA but [~abmodi] did it. If ATSv2 is enabled, then completed application which is not present in RM is fetched from ATSv2. > Need to support Application History Server on ATSV2 > --- > > Key: YARN-4343 > URL: https://issues.apache.org/jira/browse/YARN-4343 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Major > Labels: YARN-5355 > > AHS is used by the CLI and Webproxy(REST), if the application related > information is not found in RM then it tries to fetch from AHS and show -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4069) For long running apps (> 2 days), populate flow activity table
[ https://issues.apache.org/jira/browse/YARN-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-4069. - Resolution: Won't Fix No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > For long running apps (> 2 days), populate flow activity table > -- > > Key: YARN-4069 > URL: https://issues.apache.org/jira/browse/YARN-4069 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Varun Saxena >Priority: Major > Labels: YARN-5355 > > YARN-4063 / YARN-3901 contain the work being done to populate the flow > activity and flow run tables. > The flow activity table is updated each time a yarn application is created > and finishes. So if an application runs for more than 3 days, day1 has an > entry for the flow for start time, day3 has an entry for the flow for end > time but day2 has no entry for that flow. > Filing the jira to ensure that for long running apps, the flow activity table > does get a snapshot time entered for each day that an application is running > in that flow. > It may be the case that for ALL apps (long running or not) the same update > may be done in the flow activity table. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3032) [Collector implementation] Implement timeline collector functionality to serve ATS readers' requests for live apps
[ https://issues.apache.org/jira/browse/YARN-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3032. - Resolution: Won't Fix Closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Collector implementation] Implement timeline collector functionality to > serve ATS readers' requests for live apps > -- > > Key: YARN-3032 > URL: https://issues.apache.org/jira/browse/YARN-3032 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Naganarasimha G R >Priority: Major > Labels: YARN-5355 > > Per design in YARN-2928, implement the functionality in ATS writer to serve > data for live apps coming from ATS readers. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3818) [Aggregation] Queue-level Aggregation on Application States table
[ https://issues.apache.org/jira/browse/YARN-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3818. - Resolution: Won't Fix As part of ATSv2 JIRA clean up as discussed in weekly call, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Aggregation] Queue-level Aggregation on Application States table > - > > Key: YARN-3818 > URL: https://issues.apache.org/jira/browse/YARN-3818 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Junping Du >Priority: Major > Labels: YARN-5355 > > Queue level aggregation represents summary info of a specific queue, it > should include summary info of accumulated and statistic means on > applications that belongs to a queue (logically or physically). -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3196) [Compatibility] Make TS next gen be compatible with the current TS
[ https://issues.apache.org/jira/browse/YARN-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3196. - Resolution: Won't Fix As part of ATSv2 JIRA clean up as discussed in weekly call, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Compatibility] Make TS next gen be compatible with the current TS > -- > > Key: YARN-3196 > URL: https://issues.apache.org/jira/browse/YARN-3196 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du >Priority: Major > Labels: YARN-5355 > > File a jira to make sure that we don't forget to be compatible with the > current TS, such that we can smoothly move users to new TS. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3817) [Aggregation] Flow and User level aggregation on Application States table
[ https://issues.apache.org/jira/browse/YARN-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3817. - Resolution: Won't Fix As part of ATSv2 JIRA clean up as discussed in weekly call, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Aggregation] Flow and User level aggregation on Application States table > - > > Key: YARN-3817 > URL: https://issues.apache.org/jira/browse/YARN-3817 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Li Lu >Priority: Major > Labels: YARN-5355 > Attachments: Detail Design for Flow and User Level Aggregation.pdf, > YARN-3817-poc-v1-rebase.patch, YARN-3817-poc-v1.patch > > > We need time-based flow/user level aggregation to present flow/user related > states to end users. > Flow level represents summary info of a specific flow. User level aggregation > represents summary info of a specific user, it should include summary info of > accumulated and statistic means (by two levels: application and flow), like: > number of Flows, applications, resource consumption, resource means per app > or flow, etc. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3115) [Collector wireup] Work-preserving restarting of per-node timeline collector
[ https://issues.apache.org/jira/browse/YARN-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3115. - Resolution: Won't Fix As part of ATSv2 JIRA clean up as discussed in weekly call, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Collector wireup] Work-preserving restarting of per-node timeline collector > > > Key: YARN-3115 > URL: https://issues.apache.org/jira/browse/YARN-3115 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du >Priority: Major > Labels: YARN-5355 > > YARN-3030 makes the per-node aggregator work as the aux service of a NM. It > contains the states of the per-app aggregators corresponding to the running > AM containers on this NM. While NM is restarted in work-preserving mode, this > information of per-node aggregator needs to be carried on over restarting too. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3166) [Source organization] Decide detailed package structures for timeline service v2 components
[
https://issues.apache.org/jira/browse/YARN-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved YARN-3166.
-
Resolution: Won't Fix
In ATSv2 weekly call, discussed for cleaning up JIRA which are not going
implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still
same design approach exists.
> [Source organization] Decide detailed package structures for timeline service
> v2 components
> ---
>
> Key: YARN-3166
> URL: https://issues.apache.org/jira/browse/YARN-3166
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Li Lu
>Assignee: Li Lu
>Priority: Major
> Labels: YARN-5355
>
> Open this JIRA to track all discussions on detailed package structures for
> timeline services v2. This JIRA is for discussion only.
> For our current timeline service v2 design, aggregator (previously called
> "writer") implementation is in hadoop-yarn-server's:
> {{org.apache.hadoop.yarn.server.timelineservice.aggregator}}
> In YARN-2928's design, the next gen ATS reader is also a server. Maybe we
> want to put reader related implementations into hadoop-yarn-server's:
> {{org.apache.hadoop.yarn.server.timelineservice.reader}}
> Both readers and aggregators will expose features that may be used by YARN
> and other 3rd party components, such as aggregator/reader APIs. For those
> features, maybe we would like to expose their interfaces to
> hadoop-yarn-common's {{org.apache.hadoop.yarn.timelineservice}}?
> Let's use this JIRA as a centralized place to track all related discussions.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3545) Investigate the concurrency issue with the map of timeline collector
[ https://issues.apache.org/jira/browse/YARN-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3545. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > Investigate the concurrency issue with the map of timeline collector > > > Key: YARN-3545 > URL: https://issues.apache.org/jira/browse/YARN-3545 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Li Lu >Priority: Major > Labels: YARN-5355, oct16-medium > Attachments: YARN-3545-YARN-2928.000.patch > > > See the discussion in YARN-3390 for details. Let's continue the discussion > here. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3352) [Event producers] Change distributed shell to use TIMELINE_SERVICE_VERSION
[ https://issues.apache.org/jira/browse/YARN-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3352. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Event producers] Change distributed shell to use TIMELINE_SERVICE_VERSION > -- > > Key: YARN-3352 > URL: https://issues.apache.org/jira/browse/YARN-3352 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu >Priority: Major > Labels: YARN-5355 > > After YARN-3034, we have a new global configuration for active timeline > service version. We may want to use that new setting in distributed shell, > instead of a customized command-line setting. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3038) [Collector wireup] Handle timeline collector failure scenarios
[ https://issues.apache.org/jira/browse/YARN-3038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3038. - Resolution: Won't Fix Closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Collector wireup] Handle timeline collector failure scenarios > -- > > Key: YARN-3038 > URL: https://issues.apache.org/jira/browse/YARN-3038 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena >Priority: Major > Labels: YARN-5355 > > Per design in YARN-2928, consider various ATS writer failure scenarios, and > implement proper handling. > For example, ATS writers may fail and exit due to OOM. It should be retried a > certain number of times in that case. We also need to tie fatal ATS writer > failures (after exhausting all retries) to the application failure, and so on. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3033) [Collector wireup] Implement NM starting the standalone timeline collector daemon
[ https://issues.apache.org/jira/browse/YARN-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3033. - Resolution: Won't Fix Closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Collector wireup] Implement NM starting the standalone timeline collector > daemon > - > > Key: YARN-3033 > URL: https://issues.apache.org/jira/browse/YARN-3033 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Li Lu >Priority: Major > Labels: YARN-5355 > Attachments: MappingandlaunchingApplevelTimelineaggregators.pdf > > > Per design in YARN-2928, implement node managers starting the ATS writer > companion. In YARN-2928, we already have an auxiliary service based solution. > Per discussion below, the bulk of that approach has actually been done as > part of YARN-3030. In this ticket we can work on the remaining tasks, for > example: > # any needed change for configuration, esp. running it inside the NM (e.g. > the number of servlet threads) > # set up a start script that starts the per-node aggregator as a standalone > daemon > # for the standalone mode, implement a service that receives requests to set > up and tear down the app-level data -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5281) Explore supporting a simpler back-end implementation for ATS v2
[ https://issues.apache.org/jira/browse/YARN-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-5281. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > Explore supporting a simpler back-end implementation for ATS v2 > --- > > Key: YARN-5281 > URL: https://issues.apache.org/jira/browse/YARN-5281 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Priority: Major > Labels: YARN-5355 > > During the merge discussion [~kasha] raised the question whether we would > support simpler backend for users to try out, in addition to the HBase > implementation. > The understanding is that this would not be meant to scale, but it could > simplify initial adoption and early usage. > I'm filing this jira to gather the merits and challenges of such approach in > one place. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3118) [Data Serving] Clustering of ATS reader instances
[ https://issues.apache.org/jira/browse/YARN-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3118. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Data Serving] Clustering of ATS reader instances > - > > Key: YARN-3118 > URL: https://issues.apache.org/jira/browse/YARN-3118 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sunil Govindan >Priority: Major > Labels: YARN-5355 > > YARN-3047 introduces the ATS reader basically as a single daemon. As a > follow-up, we should consider clustering of ATS reader instances to be able > to handle more traffic volume (large clusters, many use cases, etc.). > It doesn't have to be in phase 1 (maybe for phase 2?). -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-5304. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Vrushali C >Priority: Major > Labels: YARN-5355 > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3815) [Aggregation] Application/Flow/User/Queue Level Aggregations
[ https://issues.apache.org/jira/browse/YARN-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3815. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > [Aggregation] Application/Flow/User/Queue Level Aggregations > > > Key: YARN-3815 > URL: https://issues.apache.org/jira/browse/YARN-3815 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Junping Du >Priority: Critical > Labels: YARN-5355 > Attachments: Timeline Service Nextgen Flow, User, Queue Level > Aggregations (v1).pdf, aggregation-design-discussion.pdf, > hbase-schema-proposal-for-aggregation.pdf > > > Per previous discussions in some design documents for YARN-2928, the basic > scenario is the query for stats can happen on: > - Application level, expect return: an application with aggregated stats > - Flow level, expect return: aggregated stats for a flow_run, flow_version > and flow > - User level, expect return: aggregated stats for applications submitted by > user > - Queue level, expect return: aggregated stats for applications within the > Queue > Application states is the basic building block for all other level > aggregations. We can provide Flow/User/Queue level aggregated statistics info > based on application states (a dedicated table for application states is > needed which is missing from previous design documents like HBase/Phoenix > schema design). -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3741) consider nulling member maps/sets of TimelineEntity
[ https://issues.apache.org/jira/browse/YARN-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3741. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > consider nulling member maps/sets of TimelineEntity > --- > > Key: YARN-3741 > URL: https://issues.apache.org/jira/browse/YARN-3741 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Vrushali C >Priority: Major > Labels: YARN-5355 > > Currently there are multiple collection members of TimelineEntity that are > always instantiated, regardless of whether they are used or not: info, > configs, metrics, events, isRelatedToEntities, and relatesToEntities. > Since TimelineEntities will be created very often and in lots of cases many > of these members will be empty, creating these empty collections is wasteful > in terms of garbage collector pressure. > It would be good to start out with null members, and instantiate these > collections only if they are actually used. Of course, we need to make that > contract very clear and refactor all client code to handle that scenario. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3650) Consider concurrency situations for TimelineWriter
[ https://issues.apache.org/jira/browse/YARN-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3650. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > Consider concurrency situations for TimelineWriter > -- > > Key: YARN-3650 > URL: https://issues.apache.org/jira/browse/YARN-3650 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Priority: Major > Labels: YARN-5355 > > [~jrottinghuis] brought up an interesting point in YARN-3411. Filing jira to > track to discuss and handle the following: > For TimelineWriter and its implementations, is there an expectation set around > concurrency? Is any synchronization expected / needed to ensure visibility > when calls happen from different threads? > How about entities, are they expected to be immutable once passed to the > write method? > Similarly for the constructor, we're assuming that the configuration object > will not be modified while we're constructing a TimelineWriter? -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3616) determine how to generate YARN container events
[ https://issues.apache.org/jira/browse/YARN-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3616. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > determine how to generate YARN container events > --- > > Key: YARN-3616 > URL: https://issues.apache.org/jira/browse/YARN-3616 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Naganarasimha G R >Priority: Major > Labels: YARN-5355 > > The initial design called for the node manager to write YARN container events > to take advantage of the distributed writes. RM acting as a sole writer of > all YARN container events would have significant scalability problems. > Still, there are some types of events that are not captured by the NM. The > current implementation has both: RM writing container events and NM writing > container events. > We need to sort this out, and decide how we can write all needed container > events in a scalable manner. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3622) Enable application client to communicate with new timeline service
[ https://issues.apache.org/jira/browse/YARN-3622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-3622. - Resolution: Won't Fix In ATSv2 weekly call, discussed for cleaning up JIRA which are not going implement. Hence, closing the JIRA as Won't Fix. Feel free to reopen if still same design approach exists. > Enable application client to communicate with new timeline service > -- > > Key: YARN-3622 > URL: https://issues.apache.org/jira/browse/YARN-3622 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Major > Labels: YARN-5355 > > YARN application has client and AM. We have the story to make TimelineClient > work inside AM for v2, but not for client. TimelineClient inside app client > needs to be taken care of too. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9796) Fix ASF license issue in branch-3.2
Rohith Sharma K S created YARN-9796:
---
Summary: Fix ASF license issue in branch-3.2
Key: YARN-9796
URL: https://issues.apache.org/jira/browse/YARN-9796
Project: Hadoop YARN
Issue Type: Task
Reporter: Rohith Sharma K S
branch-3.2 jenkins fails dues to ASF license issue
{code:java}
Lines that start with ? in the ASF License report indicate files that do
not have an Apache license header:
!?
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/yarn.lock
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7055) YARN Timeline Service v.2: beta 1 / GA
[ https://issues.apache.org/jira/browse/YARN-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-7055. - Fix Version/s: 3.2.1 Release Note: Application Timeline Server v2 is ready for production. It is GA from 3.2.1 release on wards. Resolution: Fixed > YARN Timeline Service v.2: beta 1 / GA > -- > > Key: YARN-7055 > URL: https://issues.apache.org/jira/browse/YARN-7055 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Priority: Major > Fix For: 3.2.1 > > Attachments: TSv2 next steps.pdf > > > This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service > v.2. > YARN-2928 was alpha1, YARN-5355 was alpha2. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9388) HBaseTimelineSchemaCreator skipExistingTable should honor custom hbase tablenames
[ https://issues.apache.org/jira/browse/YARN-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-9388. - Resolution: Duplicate This is fixed as part of YARN-9387. Closing as duplicate > HBaseTimelineSchemaCreator skipExistingTable should honor custom hbase > tablenames > - > > Key: YARN-9388 > URL: https://issues.apache.org/jira/browse/YARN-9388 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > > HBaseTimelineSchemaCreator provides option to configure custom table names > for timelineservice tables. The option skipExistingTable ignores the tables > already created with custom tablenames and recreates the tables with default > table names. The custom table names has to be persisted so that > skipExistingTable, HBaseTimelineWriterImpl, HBaseTimelineReaderImpl can use > this. It currently expects to pass all custom table names every time > TimelineSchemaCreator is used. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted
Rohith Sharma K S created YARN-9820:
---
Summary: RM logs InvalidStateTransitionException when app is
submitted
Key: YARN-9820
URL: https://issues.apache.org/jira/browse/YARN-9820
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
It is observed that RM logs InvalidStateTransitionException. Not sure what is
the impact but its better to handle it.
{noformat}
2019-09-08 12:40:46,327 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED on
event = LAUNCHED
2019-09-08 12:40:46,327 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the
launch time for applicationId: application_1567926390667_0001, attemptId:
appattempt_1567926390667_0001_01launchTime: 1567926646327
2019-09-08 12:40:46,328 INFO
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating
info for app: application_1567926390667_0001
2019-09-08 12:40:46,332 ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App:
application_1567926390667_0001 can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
APP_UPDATE_SAVED at ACCEPTED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
