[jira] [Commented] (YARN-9311) TestRMRestart hangs due to a deadlock
[
https://issues.apache.org/jira/browse/YARN-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1674#comment-1674
]
Rohith Sharma K S commented on YARN-9311:
-
I just went through YARN-8448 & YARN-8449 patch. This test is failed in the
patch but ignored!
Instead of starting in new thread, this event can be skipped for waiting. This
{code} !(event instanceof RMStateStoreProxyCAEvent){code} should work. Can you
verify?
> TestRMRestart hangs due to a deadlock
> -
>
> Key: YARN-9311
> URL: https://issues.apache.org/jira/browse/YARN-9311
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9311-001.patch, jstackdata, jstackdata1
>
>
> {{TestRMRestart#testRMStateStoreDispatcherDrainedOnRMStop}} hangs as
> {{MockRM}} start runs in an infinite loop at {{handleStoreEvent}}
> {code}
> [INFO] Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.468
> s - in org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9299) TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors
[ https://issues.apache.org/jira/browse/YARN-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1633#comment-1633 ] Rohith Sharma K S commented on YARN-9299: - thanks [~Prabhu Joseph] for the patch. +1 and will commit it later of today > TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors > -- > > Key: YARN-9299 > URL: https://issues.apache.org/jira/browse/YARN-9299 > Project: Hadoop YARN > Issue Type: Test >Affects Versions: 3.1.2 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9299-001.patch > > > TestTimelineReaderWhitelistAuthorizationFilter positive test cases does not > check if there is any Error in HttpResponse. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8625) Aggregate Resource Allocation for each job is not present in ATS
[ https://issues.apache.org/jira/browse/YARN-8625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1631#comment-1631 ] Rohith Sharma K S commented on YARN-8625: - [~Prabhu Joseph] These metrics holds good for running applications. ATS doesn't have these data. Always ApplicationHistoryManagerOnTimelineStore constructs with empty resources. > Aggregate Resource Allocation for each job is not present in ATS > > > Key: YARN-8625 > URL: https://issues.apache.org/jira/browse/YARN-8625 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 >Affects Versions: 2.7.4 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: 0001-YARN-8625.patch, 0002-YARN-8625.patch > > > Aggregate Resource Allocation shown on RM UI for finished job is very useful > metric to understand how much resource a job has consumed. But this does not > get stored in ATS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7266) Timeline Server event handler threads locked
[ https://issues.apache.org/jira/browse/YARN-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777650#comment-16777650 ] Rohith Sharma K S commented on YARN-7266: - Thanks [~Prabhu Joseph] for details.. I would like others folks to look at the patch. [~eyang]/[~billie.rinaldi] Could you please help to review this patch? Few comments.. # We need not to verify for TimelineAbout class. Expect call coming for DAO classes only. # New file ContextFactory loads default class. That is not required.. JobContext could be created here itself using builder or direct newInstance invocation. # ContextFactory class is NOT fully *Singleton* class. There is high chances multiple instance will be created. Probably try lazy initialization. # jaxb.properties should be kept under resource folder, not in src folder path. # Test also need change accordingly. > Timeline Server event handler threads locked > > > Key: YARN-7266 > URL: https://issues.apache.org/jira/browse/YARN-7266 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2, timelineserver >Affects Versions: 2.7.3 >Reporter: Venkata Puneet Ravuri >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-7266-001.patch, YARN-7266-002.patch, > YARN-7266-003.patch > > > Event handlers for Timeline Server seem to take a lock while parsing HTTP > headers of the request. This is causing all other threads to wait and slowing > down the overall performance of Timeline server. We have resourcemanager > metrics enabled to send to timeline server. Because of the high load on > ResourceManager, the metrics to be sent are getting backlogged and in turn > increasing heap footprint of Resource Manager (due to pending metrics). > This is the complete stack trace of a blocked thread on timeline server:- > "2079644967@qtp-1658980982-4560" #4632 daemon prio=5 os_prio=0 > tid=0x7f6ba490a000 nid=0x5eb waiting for monitor entry > [0x7f6b9142c000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:82) > - waiting to lock <0x0005c0621860> (a java.lang.Class for > com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector) > at > com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:168) > at > com.sun.xml.bind.v2.runtime.reflect.Accessor$FieldReflection.optimize(Accessor.java:282) > at > com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.(SingleElementNodeProperty.java:94) > at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) > at > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:183) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:532) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:551) > at > com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.(ArrayElementProperty.java:112) > at > com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.(ArrayElementNodeProperty.java:62) > at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) > at > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:183) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:532) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.(JAXBContextImpl.java:347) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170) > at > com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145) > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at javax.xml.bind.ContextFinder.newInstance(Unknown Source) > at javax.xml.bind.ContextFinder.newInstance(Unknown Source) > at javax.xml.bind.ContextFinder.find(Unknown Source) > at javax.xml.bind.JAXBContext.newInstance(Unknown Source) > at
[jira] [Commented] (YARN-7266) Timeline Server event handler threads locked
[ https://issues.apache.org/jira/browse/YARN-7266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774745#comment-16774745 ] Rohith Sharma K S commented on YARN-7266: - Thanks [~Prabhu Joseph] for the patch! Why are we checking for TimelineAbout.class instead of others? This could lead to race condition? Instead of creating a contextfactory, we can bind JAXBContextResolver. Any issue with it? > Timeline Server event handler threads locked > > > Key: YARN-7266 > URL: https://issues.apache.org/jira/browse/YARN-7266 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2, timelineserver >Affects Versions: 2.7.3 >Reporter: Venkata Puneet Ravuri >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-7266-001.patch, YARN-7266-002.patch, > YARN-7266-003.patch > > > Event handlers for Timeline Server seem to take a lock while parsing HTTP > headers of the request. This is causing all other threads to wait and slowing > down the overall performance of Timeline server. We have resourcemanager > metrics enabled to send to timeline server. Because of the high load on > ResourceManager, the metrics to be sent are getting backlogged and in turn > increasing heap footprint of Resource Manager (due to pending metrics). > This is the complete stack trace of a blocked thread on timeline server:- > "2079644967@qtp-1658980982-4560" #4632 daemon prio=5 os_prio=0 > tid=0x7f6ba490a000 nid=0x5eb waiting for monitor entry > [0x7f6b9142c000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:82) > - waiting to lock <0x0005c0621860> (a java.lang.Class for > com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector) > at > com.sun.xml.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:168) > at > com.sun.xml.bind.v2.runtime.reflect.Accessor$FieldReflection.optimize(Accessor.java:282) > at > com.sun.xml.bind.v2.runtime.property.SingleElementNodeProperty.(SingleElementNodeProperty.java:94) > at sun.reflect.GeneratedConstructorAccessor52.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) > at > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:183) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:532) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:551) > at > com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.(ArrayElementProperty.java:112) > at > com.sun.xml.bind.v2.runtime.property.ArrayElementNodeProperty.(ArrayElementNodeProperty.java:62) > at sun.reflect.GeneratedConstructorAccessor19.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown > Source) > at java.lang.reflect.Constructor.newInstance(Unknown Source) > at > com.sun.xml.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:128) > at > com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:183) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:532) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl.(JAXBContextImpl.java:347) > at > com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170) > at > com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145) > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at javax.xml.bind.ContextFinder.newInstance(Unknown Source) > at javax.xml.bind.ContextFinder.newInstance(Unknown Source) > at javax.xml.bind.ContextFinder.find(Unknown Source) > at javax.xml.bind.JAXBContext.newInstance(Unknown Source) > at javax.xml.bind.JAXBContext.newInstance(Unknown Source) > at > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.buildModelAndSchemas(WadlGeneratorJAXBGrammarGenerator.java:412) > at > com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.createExternalGrammar(WadlGeneratorJAXBGrammarGenerator.java:352) > at > com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:115) > at >
[jira] [Commented] (YARN-4327) RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters
[ https://issues.apache.org/jira/browse/YARN-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771594#comment-16771594 ] Rohith Sharma K S commented on YARN-4327: - This appears to be configuration issue. What is the value set for *hadoop.http.filter.initializers*? It must be org.apache.hadoop.security.AuthenticationFilterInitializer only. > RM can not renew TIMELINE_DELEGATION_TOKEN in secure clusters > -- > > Key: YARN-4327 > URL: https://issues.apache.org/jira/browse/YARN-4327 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, security, timelineserver >Affects Versions: 2.7.1 > Environment: hadoop 2.7.1hdfs,yarn, mrhistoryserver, ATS all use > kerberos security. > conf like this: > > hadoop.security.authorization > true > Is service-level authorization enabled? > > > hadoop.security.authentication > kerberos > Possible values are simple (no authentication), and kerberos > > >Reporter: zhangshilong >Priority: Major > > bin hadoop 2.7.1 > ATS conf like this: > > yarn.timeline-service.http-authentication.type > simple > > > yarn.timeline-service.http-authentication.kerberos.principal > HTTP/_h...@xxx.com > > > yarn.timeline-service.http-authentication.kerberos.keytab > /etc/hadoop/keytabs/xxx.keytab > > > yarn.timeline-service.principal > xxx/_h...@xxx.com > > > yarn.timeline-service.keytab > /etc/hadoop/keytabs/xxx.keytab > > > yarn.timeline-service.best-effort > true > > > yarn.timeline-service.enabled > true > > > I'd like to allow everyone to access ATS from HTTP as RM,HDFS. > client can submit job to RM and add TIMELINE_DELEGATION_TOKEN to AM > Context, but RM can not renew TIMELINE_DELEGATION_TOKEN and make application > to failure. > RM logs: > 2015-11-03 11:58:38,191 WARN > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: > Unable to add the application to the delegation token renewer. > java.io.IOException: Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, > Service: 10.12.38.4:8188, Ident: (owner=yarn-test, renewer=yarn-test, > realUser=, issueDate=1446523118046, maxDate=1447127918046, sequenceNumber=9, > masterKeyId=2) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:439) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:78) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:847) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:828) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: HTTP status [500], message [Null user] > at > org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:169) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:287) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:212) > at > org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:414) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$3.run(TimelineClientImpl.java:396) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$3.run(TimelineClientImpl.java:378) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$5.run(TimelineClientImpl.java:451) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:183) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:466) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.renewDelegationToken(TimelineClientImpl.java:400) > at >
[jira] [Commented] (YARN-9311) TestRMRestart hangs due to a deadlock
[
https://issues.apache.org/jira/browse/YARN-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16771570#comment-16771570
]
Rohith Sharma K S commented on YARN-9311:
-
It appear recent code/test-code change caused this issue. This test case is
running more than 3 years..
> TestRMRestart hangs due to a deadlock
> -
>
> Key: YARN-9311
> URL: https://issues.apache.org/jira/browse/YARN-9311
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9311-001.patch, jstackdata, jstackdata1
>
>
> {{TestRMRestart#testRMStateStoreDispatcherDrainedOnRMStop}} hangs as
> {{MockRM}} start runs in an infinite loop at {{handleStoreEvent}}
> {code}
> [INFO] Running org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
> [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.468
> s - in org.apache.hadoop.yarn.server.resourcemanager.TestMoveApplication
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9233) RM may report allocated container which is killed (but not acquired by AM ) to AM which can cause spark AM confused
[ https://issues.apache.org/jira/browse/YARN-9233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770893#comment-16770893 ] Rohith Sharma K S commented on YARN-9233: - thanks [~BilwaST] for updating patch.. few comments # what is the difference between for normal container and master container in this flow? why special condition for master container? # isAquired flag is not required. Instead, compare with getState() == acquired ? > RM may report allocated container which is killed (but not acquired by AM ) > to AM which can cause spark AM confused > --- > > Key: YARN-9233 > URL: https://issues.apache.org/jira/browse/YARN-9233 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9233-001.patch, YARN-9233-002.patch, > YARN-9233-003.patch > > > After the RM kills an allocated (Allocated state) container for various > reasons, it will go through the state transition process to the FINISHED > state just like other state containers. Currently RM doesn't consider if > container is acquired by the AM. Hence All the containers transitioned to > FINISH state are added to justFinishedContainers list. Therefore the > container that is not obtained by the AM and is killed by the rm will also > return through the AM heartbeat. So AM re-applies for more resources than > needed which would eventually cause number of containers to exceed the > maximum limit -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8404) Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down
[ https://issues.apache.org/jira/browse/YARN-8404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-8404: Labels: atsv2 (was: ) > Timeline event publish need to be async to avoid Dispatcher thread leak in > case ATS is down > --- > > Key: YARN-8404 > URL: https://issues.apache.org/jira/browse/YARN-8404 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.0, 3.0.2 >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Blocker > Labels: atsv2 > Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.4 > > Attachments: YARN-8404.01.patch > > > It is observed that if ATS1/1.5 daemon is not running, RM recovery is delayed > as long as timeline client get timed out for each applications. By default, > timed out will take around 5 mins. If completed applications are more then > amount of time RM will wait is *(number of completed applications in a > cluster * 5 minutes)* which is kind of hanged. > Primary reason for this behavior is YARN-3044 YARN-4129 which refactor > existing system metric publisher. This refactoring made appFinished event as > synchronous which was asynchronous earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8455) Add basic ACL check for all ATS v2 REST APIs
[ https://issues.apache.org/jira/browse/YARN-8455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-8455: Labels: atsv2 (was: ) > Add basic ACL check for all ATS v2 REST APIs > > > Key: YARN-8455 > URL: https://issues.apache.org/jira/browse/YARN-8455 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Labels: atsv2 > Fix For: 3.2.0, 3.1.1 > > Attachments: YARN-8455.001.patch, YARN-8455.002.patch, > YARN-8455.003.patch, YARN-8455.004.patch > > > YARN-8319 filter check for flows pages. The same behavior need to be added > for all other REST API as long as ATS provides support for ACLs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8512) ATSv2 entities are not published to HBase from second attempt onwards
[
https://issues.apache.org/jira/browse/YARN-8512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8512:
Labels: atsv2 (was: )
> ATSv2 entities are not published to HBase from second attempt onwards
> -
>
> Key: YARN-8512
> URL: https://issues.apache.org/jira/browse/YARN-8512
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 2.10.0, 3.2.0, 3.0.3
>Reporter: Yesha Vora
>Assignee: Rohith Sharma K S
>Priority: Major
> Labels: atsv2
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8512.01.patch, YARN-8512.02.patch,
> YARN-8512.03.patch
>
>
> It is observed that if 1st attempt master container is died and 2nd attempt
> master container is launched in a NM where old containers are running but not
> master container.
> ||Attempt||NM1||NM2||Action||
> |attempt-1|master container i.e container-1-1|container-1-2|master container
> died|
> |attempt-2|NA|container-1-2 and master container container-2-1|NA|
> In the above scenario, NM doesn't identifies flowContext and will get log
> below
> {noformat}
> 2018-07-10 00:44:38,285 WARN storage.HBaseTimelineWriterImpl
> (HBaseTimelineWriterImpl.java:write(170)) - Found null for one of:
> flowName=null appId=application_1531175172425_0001 userId=hbase
> clusterId=yarn-cluster . Not proceeding with writing to hbase
> 2018-07-10 00:44:38,560 WARN storage.HBaseTimelineWriterImpl
> (HBaseTimelineWriterImpl.java:write(170)) - Found null for one of:
> flowName=null appId=application_1531175172425_0001 userId=hbase
> clusterId=yarn-cluster . Not proceeding with writing to hbase
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8630) ATSv2 REST APIs should honor filter-entity-list-by-user in non-secure cluster when ACls are enabled
[ https://issues.apache.org/jira/browse/YARN-8630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-8630: Labels: atsv2 (was: ) > ATSv2 REST APIs should honor filter-entity-list-by-user in non-secure cluster > when ACls are enabled > --- > > Key: YARN-8630 > URL: https://issues.apache.org/jira/browse/YARN-8630 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Labels: atsv2 > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8630.01.patch > > > It is observed that ATSv2 REST endpoints are not honoring > *yarn.webapp.filter-entity-list-by-user* in non-secure cluster when ACLs are > enabled. > The issue can be seen if static web app filter is not configured in > non-secure cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8486) yarn.webapp.filter-entity-list-by-user should honor limit filter for TS reader flows api
[ https://issues.apache.org/jira/browse/YARN-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-8486: Labels: atsv2 (was: ) > yarn.webapp.filter-entity-list-by-user should honor limit filter for TS > reader flows api > > > Key: YARN-8486 > URL: https://issues.apache.org/jira/browse/YARN-8486 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Charan Hebri >Assignee: Rohith Sharma K S >Priority: Major > Labels: atsv2 > > Post YARN-8319, flows restrict entities per user. If limit is applied to the > flows then returned values are inconsistent. Reason is if back end returned > values are 10 and contains no data for user1, then flows api returns empty. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8492) ATSv2 HBase tests are failing with ClassNotFoundException
[
https://issues.apache.org/jira/browse/YARN-8492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8492:
Labels: atsv2 test (was: test)
> ATSv2 HBase tests are failing with ClassNotFoundException
> -
>
> Key: YARN-8492
> URL: https://issues.apache.org/jira/browse/YARN-8492
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Labels: atsv2, test
> Fix For: 3.2.0
>
> Attachments: YARN-8492.01.patch, YARN-8492.02.patch
>
>
> It is seen in recent QA report that ATSv2 Hbase tests are failing with
> ClassNotFoundException.
> This looks to be regression from hadoop common patch or any other patch. We
> need to figure out which JIRA broke this and fix tests failure.
> hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
>
> hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
>
> hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
> hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
> hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown
>
> hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
>
> hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
>
> hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
>
> hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity
>
> {noformat}
> ERROR]
> org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
> Time elapsed: 0.102 s <<< ERROR!
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/crypto/key/KeyProviderTokenIssuer
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps.setupBeforeClass(TestHBaseTimelineStorageApps.java:97)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.crypto.key.KeyProviderTokenIssuer
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8591) [ATSv2] NPE while checking for entity acl in non-secure cluster
[
https://issues.apache.org/jira/browse/YARN-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8591:
Labels: atsv2 (was: )
> [ATSv2] NPE while checking for entity acl in non-secure cluster
> ---
>
> Key: YARN-8591
> URL: https://issues.apache.org/jira/browse/YARN-8591
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader, timelineserver
>Reporter: Akhil PB
>Assignee: Rohith Sharma K S
>Priority: Major
> Labels: atsv2
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8591.01.patch
>
>
> {code:java}
> GET
> http://ctr-e138-1518143905142-417433-01-04.hwx.site:8198/ws/v2/timeline/apps/application_1532578985272_0002/entities/YARN_CONTAINER?fields=ALL&_=1532670071899{code}
> {code:java}
> 2018-07-27 05:32:03,468 WARN webapp.GenericExceptionHandler
> (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:196)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:624)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:474)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilter.doFilter(TimelineReaderWhitelistAuthorizationFilter.java:85)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:98)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
> at
>
[jira] [Updated] (YARN-8950) Compilation fails with dependency convergence error for hbase.profile=2.0
[
https://issues.apache.org/jira/browse/YARN-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-8950:
Labels: atsv2 (was: )
> Compilation fails with dependency convergence error for hbase.profile=2.0
> -
>
> Key: YARN-8950
> URL: https://issues.apache.org/jira/browse/YARN-8950
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Labels: atsv2
> Fix For: 3.2.0, 3.1.2, 3.3.0
>
> Attachments: YARN-8950.01.patch, YARN-8950.01.patch,
> YARN-8950.01.patch, with-patch-compile-pass.out,
> without-patch-compile-fail.out
>
>
> Dependency check for hbase-client package fails when source code is compiled
> with *-Dhbase.profile=2.0*
> {noformat}
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (depcheck) @
> hadoop-yarn-server-timelineservice-hbase-client ---
> [WARNING]
> Dependency convergence error for
> org.eclipse.jetty:jetty-http:9.3.24.v20180605 paths to dependency are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:3.3.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-common:3.3.0-SNAPSHOT
> +-org.eclipse.jetty:jetty-server:9.3.24.v20180605
> +-org.eclipse.jetty:jetty-http:9.3.24.v20180605
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:3.3.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:2.0.0-beta-1
> +-org.apache.hbase:hbase-http:2.0.0-beta-1
> +-org.eclipse.jetty:jetty-http:9.3.19.v20170502
> [WARNING]
> Dependency convergence error for
> org.eclipse.jetty:jetty-security:9.3.24.v20180605 paths to dependency are:
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:3.3.0-SNAPSHOT
> +-org.apache.hadoop:hadoop-common:3.3.0-SNAPSHOT
> +-org.eclipse.jetty:jetty-servlet:9.3.24.v20180605
> +-org.eclipse.jetty:jetty-security:9.3.24.v20180605
> and
> +-org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-client:3.3.0-SNAPSHOT
> +-org.apache.hbase:hbase-server:2.0.0-beta-1
> +-org.apache.hbase:hbase-http:2.0.0-beta-1
> +-org.eclipse.jetty:jetty-security:9.3.19.v20170502
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
> failed with message:
> Failed while enforcing releasability. See above detailed error message.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9034: Labels: atsv2 (was: ) > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Labels: atsv2 > Fix For: 3.3.0 > > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch, YARN-9034.04.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9251) Build failure for -Dhbase.profile=2.0
[
https://issues.apache.org/jira/browse/YARN-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-9251:
Labels: atsv2 (was: )
> Build failure for -Dhbase.profile=2.0
> -
>
> Key: YARN-9251
> URL: https://issues.apache.org/jira/browse/YARN-9251
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Labels: atsv2
> Fix For: 3.3.0
>
> Attachments: HADOOP-16088.01.patch
>
>
> Post HADOOP-14178, hadoop build failure due to incorrect pom.xml.
> {noformat}
> HW12723:hadoop rsharmaks$ mvn clean install -DskipTests -DskipShade
> -Dhbase.profile=2.0
> [INFO] Scanning for projects...
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar is
> missing. @ line 485, column 21
> @
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR] The project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-tests:3.3.0-SNAPSHOT
>
> (/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml)
> has 1 error
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar
> is missing. @ line 485, column 21
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> {noformat}
> cc:/ [~ajisakaa]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
[ https://issues.apache.org/jira/browse/YARN-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9044: Labels: atsv2 (was: ) > LogsCLI should contact ATSv2 for "-am" option > - > > Key: YARN-9044 > URL: https://issues.apache.org/jira/browse/YARN-9044 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Labels: atsv2 > Fix For: 3.3.0 > > Attachments: YARN-9044.01.patch, YARN-9044.01.patch, > YARN-9044.02.patch > > > *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not > configured. Rather LogsCLI should contact ATSv2 for AM container info. > Alternative to above one can use *yarn logs -containerId * > to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9242) Revert YARN-8270 from branch-3.1 and branch-3.1.2
[
https://issues.apache.org/jira/browse/YARN-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-9242:
Labels: atsv2 (was: )
> Revert YARN-8270 from branch-3.1 and branch-3.1.2
> -
>
> Key: YARN-9242
> URL: https://issues.apache.org/jira/browse/YARN-9242
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Labels: atsv2
>
> It is observed that in hadoop-3.1-RC0, NodeManager are unable to initialize
> TimelineCollectorWebService!
> Primary reason is HADOOP-15657 is not present in hadoop-3.1 branch!
> Following error is seen NM logs
> {noformat}
> Caused by: org.apache.hadoop.metrics2.MetricsException: Unsupported metric
> field putEntitiesFailureLatency of type
> org.apache.hadoop.metrics2.lib.MutableQuantiles
> at
> org.apache.hadoop.metrics2.lib.MutableMetricsFactory.newForField(MutableMetricsFactory.java:87)
> {noformat}
> We need to revert YARN-8270 from branch-3.1!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9233) RM may report allocated container which is killed (but not acquired by AM ) to AM which can cause spark AM confused
[ https://issues.apache.org/jira/browse/YARN-9233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762412#comment-16762412 ] Rohith Sharma K S commented on YARN-9233: - Its better to skip event from RMContinerImpl to RMAppAttemptImpl if getState is allocated. > RM may report allocated container which is killed (but not acquired by AM ) > to AM which can cause spark AM confused > --- > > Key: YARN-9233 > URL: https://issues.apache.org/jira/browse/YARN-9233 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9233-001.patch, YARN-9233-002.patch > > > After the RM kills an allocated (Allocated state) container for various > reasons, it will go through the state transition process to the FINISHED > state just like other state containers. Currently RM doesn't consider if > container is acquired by the AM. Hence All the containers transitioned to > FINISH state are added to justFinishedContainers list. Therefore the > container that is not obtained by the AM and is killed by the rm will also > return through the AM heartbeat. So AM re-applies for more resources than > needed which would eventually cause number of containers to exceed the > maximum limit -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9233) RM may report allocated container which is killed (but not acquired by AM ) to AM which can cause spark AM confused
[ https://issues.apache.org/jira/browse/YARN-9233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762403#comment-16762403 ] Rohith Sharma K S commented on YARN-9233: - Thanks [~BilwaST] for the patch! bq. Therefore the container that is not obtained by the AM and is killed by the rm will also return through the AM heartbeat. So AM re-applies for more resources than needed which would eventually cause number of containers to exceed the maximum limit Which AM asking for more resource. Is this Yarn amrmclient or MR or spakr? This should be bug from ApplicationMaster! > RM may report allocated container which is killed (but not acquired by AM ) > to AM which can cause spark AM confused > --- > > Key: YARN-9233 > URL: https://issues.apache.org/jira/browse/YARN-9233 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9233-001.patch, YARN-9233-002.patch > > > After the RM kills an allocated (Allocated state) container for various > reasons, it will go through the state transition process to the FINISHED > state just like other state containers. Currently RM doesn't consider if > container is acquired by the AM. Hence All the containers transitioned to > FINISH state are added to justFinishedContainers list. Therefore the > container that is not obtained by the AM and is killed by the rm will also > return through the AM heartbeat. So AM re-applies for more resources than > needed which would eventually cause number of containers to exceed the > maximum limit -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8219) Add application launch time to ATSV2
[ https://issues.apache.org/jira/browse/YARN-8219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762349#comment-16762349 ] Rohith Sharma K S commented on YARN-8219: - Verified test in local and it doesn't related to this patch. Going ahead with committing patch.. > Add application launch time to ATSV2 > > > Key: YARN-8219 > URL: https://issues.apache.org/jira/browse/YARN-8219 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Reporter: Kanwaljeet Sachdev >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8219.001.patch, YARN-8219.002.patch > > > YARN-7088 publishes application launch time to RMStore and also adds it to > the YARN UI. It would be a nice enhancement to have the launchTime event > published into the ATSV2. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9224) TimelineDelegationTokenIdentifier.Renewer contacts KDC for every renew/cancel token operation
[
https://issues.apache.org/jira/browse/YARN-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762354#comment-16762354
]
Rohith Sharma K S commented on YARN-9224:
-
[~tarunparimi] thanks for the patch! Skimming through patch, approach looks
good but it appears concurrency issue exist. Did you verify this patch in real
cluster?
> TimelineDelegationTokenIdentifier.Renewer contacts KDC for every renew/cancel
> token operation
> -
>
> Key: YARN-9224
> URL: https://issues.apache.org/jira/browse/YARN-9224
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3
>Reporter: Tarun Parimi
>Priority: Major
> Attachments: YARN-9224.001.patch, YARN-9224.002.patch
>
>
> In a production cluster, we have observed the active RM principal making
> excessive requests to the KDC server. Being a service principal, this
> shouldn't be the case normally.
> On capturing tcpdump for the connections between RM and KDC, we saw that
> these excessive requests were for the SPNEGO service
> HTTP/ats-host.example.com .
> The requests were also matching in frequency with the below log entry in RM.
> {code:java}
> 2019-01-09T03:41:56.048-0500 INFO
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl: Timeline service
> address: http://ats-host.example.com:8188/ws/v1/timeline/
> {code}
> On looking at the code in TimelineDelegationTokenIdentifier.java, it seems
> this kdc request for SPNEGO is done as we are creating a new timeline client
> instance every time.
> {code:java}
> @SuppressWarnings("unchecked")
> @Override
> public long renew(Token token, Configuration conf) throws IOException,
> InterruptedException {
> TimelineClient client = TimelineClient.createTimelineClient();
> try {
> client.init(conf);
> client.start();
> return client.renewDelegationToken(
> (Token) token);
> } catch (YarnException e) {
> throw new IOException(e);
> } finally {
> client.stop();
> }
> }
> @SuppressWarnings("unchecked")
> @Override
> public void cancel(Token token, Configuration conf) throws IOException,
> InterruptedException {
> TimelineClient client = TimelineClient.createTimelineClient();
> try {
> client.init(conf);
> client.start();
> client.cancelDelegationToken(
> (Token) token);
> } catch (YarnException e) {
> throw new IOException(e);
> } finally {
> client.stop();
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8219) Add application launch time to ATSV2
[ https://issues.apache.org/jira/browse/YARN-8219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761437#comment-16761437 ] Rohith Sharma K S commented on YARN-8219: - TestRMRestart timed out.. Can you confirm is it related to patch or not? > Add application launch time to ATSV2 > > > Key: YARN-8219 > URL: https://issues.apache.org/jira/browse/YARN-8219 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Reporter: Kanwaljeet Sachdev >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8219.001.patch, YARN-8219.002.patch > > > YARN-7088 publishes application launch time to RMStore and also adds it to > the YARN UI. It would be a nice enhancement to have the launchTime event > published into the ATSV2. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7976) [atsv2 read acls] REST API to list domain/domains
[
https://issues.apache.org/jira/browse/YARN-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16761436#comment-16761436
]
Rohith Sharma K S commented on YARN-7976:
-
I am thinking should we hold this JIRA until entity acls are verified. Exposing
REST API brings a feeling that acls exist end to end but its not the case.
> [atsv2 read acls] REST API to list domain/domains
> -
>
> Key: YARN-7976
> URL: https://issues.apache.org/jira/browse/YARN-7976
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-7976.001.patch, YARN-7976.002.patch
>
>
> Provide REST API to list domains and domain in TimelineReaderWebService.
> /domains and /domain/\{domainId}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8219) Add application launch time to ATSV2
[ https://issues.apache.org/jira/browse/YARN-8219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760582#comment-16760582 ] Rohith Sharma K S commented on YARN-8219: - thanks [~abmodi] for the clarification! I do see the same behavior from RMAppImpl class. Patch looks good.. Kicking of jenkins to see latest reports > Add application launch time to ATSV2 > > > Key: YARN-8219 > URL: https://issues.apache.org/jira/browse/YARN-8219 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineserver >Reporter: Kanwaljeet Sachdev >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8219.001.patch, YARN-8219.002.patch > > > YARN-7088 publishes application launch time to RMStore and also adds it to > the YARN UI. It would be a nice enhancement to have the launchTime event > published into the ATSV2. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6735) Have a way to turn off container metrics from NMs
[ https://issues.apache.org/jira/browse/YARN-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760564#comment-16760564 ] Rohith Sharma K S commented on YARN-6735: - +1 lgtm > Have a way to turn off container metrics from NMs > - > > Key: YARN-6735 > URL: https://issues.apache.org/jira/browse/YARN-6735 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-6735.001.patch, YARN-6735.002.patch, > YARN-6735.003.patch > > > Have a way to turn off emitting system metrics from NMs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7976) [atsv2 read acls] REST API to list domain/domains
[
https://issues.apache.org/jira/browse/YARN-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760550#comment-16760550
]
Rohith Sharma K S commented on YARN-7976:
-
[~abmodi] Does writer side implementation is done?
> [atsv2 read acls] REST API to list domain/domains
> -
>
> Key: YARN-7976
> URL: https://issues.apache.org/jira/browse/YARN-7976
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-7976.001.patch, YARN-7976.002.patch
>
>
> Provide REST API to list domains and domain in TimelineReaderWebService.
> /domains and /domain/\{domainId}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3488) AM get timeline service info from RM rather than Application specific configuration.
[ https://issues.apache.org/jira/browse/YARN-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760525#comment-16760525 ] Rohith Sharma K S commented on YARN-3488: - Thanks [~abmodi] for the patch. Overall patch looks good to me. One comment on optimization! # While processing AM registration, instead of checking for timeline service check, it could it be cached during service initialization and used again. > AM get timeline service info from RM rather than Application specific > configuration. > > > Key: YARN-3488 > URL: https://issues.apache.org/jira/browse/YARN-3488 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications >Reporter: Junping Du >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > Attachments: YARN-3488.001.patch, YARN-3488.002.patch > > > Since v1 timeline service, we have MR configuration to enable/disable putting > history event to timeline service. For today's v2 timeline service ongoing > effort, currently we have different methods/structures between v1 and v2 for > consuming TimelineClient, so application have to be aware of which version > timeline service get used there. > There are basically two options here: > First option is as current way in DistributedShell or MR to let application > has specific configuration to point out that if enabling ATS and which > version could be, like: MRJobConfig.MAPREDUCE_JOB_EMIT_TIMELINE_DATA, etc. > The other option is to let application to figure out timeline related info > from YARN/RM, it can be done through registerApplicationMaster() in > ApplicationMasterProtocol with return value for service "off", "v1_on", or > "v2_on". > We prefer the latter option because application owner doesn't have to aware > RM/YARN infrastructure details. Please note that we should keep compatible > (consistent behavior with the same setting) with released configurations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9149) yarn container -status misses logUrl when integrated with ATSv2
[
https://issues.apache.org/jira/browse/YARN-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760518#comment-16760518
]
Rohith Sharma K S commented on YARN-9149:
-
+1 lgtm
> yarn container -status misses logUrl when integrated with ATSv2
> ---
>
> Key: YARN-9149
> URL: https://issues.apache.org/jira/browse/YARN-9149
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-9149.001.patch, YARN-9149.002.patch,
> YARN-9149.003.patch
>
>
> Post YARN-8303, yarn client can be integrated with ATSv2. But log url and
> start and end time is printing data is wrong!
> {code}
> Container Report :
> Container-Id : container_1545035586969_0001_01_01
> Start-Time : 0
> Finish-Time : 0
> State : COMPLETE
> Execution-Type : GUARANTEED
> LOG-URL : null
> Host : localhost:25006
> NodeHttpAddress : localhost:25008
> Diagnostics :
> {code}
> # TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*.
> This need set for proper log url based on yarn.log.server.web-service.url
> # TimelineEntityV2Converter#convertToContainerReport parses start/end time
> wrongly. Comparison should happen with entityType but below code is doing
> entityId
> {code}
> if (events != null) {
> for (TimelineEvent event : events) {
> if (event.getId().equals(
> ContainerMetricsConstants.CREATED_IN_RM_EVENT_TYPE)) {
> createdTime = event.getTimestamp();
> } else if (event.getId().equals(
> ContainerMetricsConstants.FINISHED_IN_RM_EVENT_TYPE)) {
> finishedTime = event.getTimestamp();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0
[ https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9256: Attachment: YARN-9256.01.patch > Make ATSv2 compilation default with hbase.profile=2.0 > - > > Key: YARN-9256 > URL: https://issues.apache.org/jira/browse/YARN-9256 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Priority: Major > Attachments: YARN-9256.01.patch > > > By default Hadoop compiles with hbase.profile one which corresponds to > hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by > default in trunk. > This JIRA is to discuss for any concerns. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0
[ https://issues.apache.org/jira/browse/YARN-9256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9256: Target Version/s: 3.3.0 > Make ATSv2 compilation default with hbase.profile=2.0 > - > > Key: YARN-9256 > URL: https://issues.apache.org/jira/browse/YARN-9256 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Priority: Major > > By default Hadoop compiles with hbase.profile one which corresponds to > hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by > default in trunk. > This JIRA is to discuss for any concerns. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9256) Make ATSv2 compilation default with hbase.profile=2.0
Rohith Sharma K S created YARN-9256: --- Summary: Make ATSv2 compilation default with hbase.profile=2.0 Key: YARN-9256 URL: https://issues.apache.org/jira/browse/YARN-9256 Project: Hadoop YARN Issue Type: Task Reporter: Rohith Sharma K S By default Hadoop compiles with hbase.profile one which corresponds to hbase.version=1.4 for ATSv2. Change compilation to hbase.profile=2.0 by default in trunk. This JIRA is to discuss for any concerns. cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9251) Build failure for -Dhbase.profile=2.0
[
https://issues.apache.org/jira/browse/YARN-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756840#comment-16756840
]
Rohith Sharma K S commented on YARN-9251:
-
Thanks [~ajisakaa] for committing patch.
> Build failure for -Dhbase.profile=2.0
> -
>
> Key: YARN-9251
> URL: https://issues.apache.org/jira/browse/YARN-9251
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16088.01.patch
>
>
> Post HADOOP-14178, hadoop build failure due to incorrect pom.xml.
> {noformat}
> HW12723:hadoop rsharmaks$ mvn clean install -DskipTests -DskipShade
> -Dhbase.profile=2.0
> [INFO] Scanning for projects...
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar is
> missing. @ line 485, column 21
> @
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR] The project
> org.apache.hadoop:hadoop-yarn-server-timelineservice-hbase-tests:3.3.0-SNAPSHOT
>
> (/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/pom.xml)
> has 1 error
> [ERROR] 'dependencies.dependency.version' for org.mockito:mockito-all:jar
> is missing. @ line 485, column 21
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> {noformat}
> cc:/ [~ajisakaa]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9242) Revert YARN-8270 from branch-3.1 and branch-3.1.2
[
https://issues.apache.org/jira/browse/YARN-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-9242:
Summary: Revert YARN-8270 from branch-3.1 and branch-3.1.2 (was: Revert
YARN-8270 from branch-3.1)
> Revert YARN-8270 from branch-3.1 and branch-3.1.2
> -
>
> Key: YARN-9242
> URL: https://issues.apache.org/jira/browse/YARN-9242
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
>
> It is observed that in hadoop-3.1-RC0, NodeManager are unable to initialize
> TimelineCollectorWebService!
> Primary reason is HADOOP-15657 is not present in hadoop-3.1 branch!
> Following error is seen NM logs
> {noformat}
> Caused by: org.apache.hadoop.metrics2.MetricsException: Unsupported metric
> field putEntitiesFailureLatency of type
> org.apache.hadoop.metrics2.lib.MutableQuantiles
> at
> org.apache.hadoop.metrics2.lib.MutableMetricsFactory.newForField(MutableMetricsFactory.java:87)
> {noformat}
> We need to revert YARN-8270 from branch-3.1!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9242) Revert YARN-8270 from branch-3.1
[
https://issues.apache.org/jira/browse/YARN-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved YARN-9242.
-
Resolution: Done
I have reverted YARN-8270 in branch-3.1 and branch-3.1.2. Also updated Fix
Version in YARN-8270.
Closing as Done!
> Revert YARN-8270 from branch-3.1
>
>
> Key: YARN-9242
> URL: https://issues.apache.org/jira/browse/YARN-9242
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
>
> It is observed that in hadoop-3.1-RC0, NodeManager are unable to initialize
> TimelineCollectorWebService!
> Primary reason is HADOOP-15657 is not present in hadoop-3.1 branch!
> Following error is seen NM logs
> {noformat}
> Caused by: org.apache.hadoop.metrics2.MetricsException: Unsupported metric
> field putEntitiesFailureLatency of type
> org.apache.hadoop.metrics2.lib.MutableQuantiles
> at
> org.apache.hadoop.metrics2.lib.MutableMetricsFactory.newForField(MutableMetricsFactory.java:87)
> {noformat}
> We need to revert YARN-8270 from branch-3.1!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader
[ https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753714#comment-16753714 ] Rohith Sharma K S commented on YARN-8270: - In branch-3.1, this was causing error while publishing events from NodeManager. The reason is HADOOP-15657 was't present in branch-3.1. I have reverted this change for unblocking release 3.1.2. Updated the 'Fix Version'! > Adding JMX Metrics for Timeline Collector and Reader > > > Key: YARN-8270 > URL: https://issues.apache.org/jira/browse/YARN-8270 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2, timelineserver >Reporter: Sushil Ks >Assignee: Sushil Ks >Priority: Major > Fix For: 3.2.0, 3.1.2 > > Attachments: YARN-8270.001.patch, YARN-8270.002.patch, > YARN-8270.003.patch, YARN-8270.004.patch, YARN-8270.005.patch > > > This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and > Timeline Reader, basically for Timeline Collector it tries to capture success > and failure latencies for *putEntities* and *putEntitiesAsync* from > *TimelineCollectorWebService* , similarly all the API's success and failure > latencies for fetching TimelineEntities from *TimelineReaderWebServices*. > This would actually help in monitoring and measuring performance for ATSv2 at > scale. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader
[ https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-8270: Fix Version/s: (was: 3.1.2) > Adding JMX Metrics for Timeline Collector and Reader > > > Key: YARN-8270 > URL: https://issues.apache.org/jira/browse/YARN-8270 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2, timelineserver >Reporter: Sushil Ks >Assignee: Sushil Ks >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8270.001.patch, YARN-8270.002.patch, > YARN-8270.003.patch, YARN-8270.004.patch, YARN-8270.005.patch > > > This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and > Timeline Reader, basically for Timeline Collector it tries to capture success > and failure latencies for *putEntities* and *putEntitiesAsync* from > *TimelineCollectorWebService* , similarly all the API's success and failure > latencies for fetching TimelineEntities from *TimelineReaderWebServices*. > This would actually help in monitoring and measuring performance for ATSv2 at > scale. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9242) Revert YARN-8270 from branch-3.1
[
https://issues.apache.org/jira/browse/YARN-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-9242:
Issue Type: Task (was: Bug)
> Revert YARN-8270 from branch-3.1
>
>
> Key: YARN-9242
> URL: https://issues.apache.org/jira/browse/YARN-9242
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
>
> It is observed that in hadoop-3.1-RC0, NodeManager are unable to initialize
> TimelineCollectorWebService!
> Primary reason is HADOOP-15657 is not present in hadoop-3.1 branch!
> Following error is seen NM logs
> {noformat}
> Caused by: org.apache.hadoop.metrics2.MetricsException: Unsupported metric
> field putEntitiesFailureLatency of type
> org.apache.hadoop.metrics2.lib.MutableQuantiles
> at
> org.apache.hadoop.metrics2.lib.MutableMetricsFactory.newForField(MutableMetricsFactory.java:87)
> {noformat}
> We need to revert YARN-8270 from branch-3.1!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Moved] (YARN-9242) Revert YARN-8270 from branch-3.1
[
https://issues.apache.org/jira/browse/YARN-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S moved HADOOP-16078 to YARN-9242:
--
Target Version/s: 3.1.2 (was: 3.1.2)
Key: YARN-9242 (was: HADOOP-16078)
Project: Hadoop YARN (was: Hadoop Common)
> Revert YARN-8270 from branch-3.1
>
>
> Key: YARN-9242
> URL: https://issues.apache.org/jira/browse/YARN-9242
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
>
> It is observed that in hadoop-3.1-RC0, NodeManager are unable to initialize
> TimelineCollectorWebService!
> Primary reason is HADOOP-15657 is not present in hadoop-3.1 branch!
> Following error is seen NM logs
> {noformat}
> Caused by: org.apache.hadoop.metrics2.MetricsException: Unsupported metric
> field putEntitiesFailureLatency of type
> org.apache.hadoop.metrics2.lib.MutableQuantiles
> at
> org.apache.hadoop.metrics2.lib.MutableMetricsFactory.newForField(MutableMetricsFactory.java:87)
> {noformat}
> We need to revert YARN-8270 from branch-3.1!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7761) [UI2] Clicking 'master container log' or 'Link' next to 'log' under application's appAttempt goes to Old UI's Log link
[ https://issues.apache.org/jira/browse/YARN-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752022#comment-16752022 ] Rohith Sharma K S commented on YARN-7761: - +1 lgtm > [UI2] Clicking 'master container log' or 'Link' next to 'log' under > application's appAttempt goes to Old UI's Log link > -- > > Key: YARN-7761 > URL: https://issues.apache.org/jira/browse/YARN-7761 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Sumana Sathish >Assignee: Akhil PB >Priority: Major > Attachments: YARN-7761.001.patch, YARN-7761.002.patch, > YARN-7761.003.patch > > > Clicking 'master container log' or 'Link' next to 'Log' under application's > appAttempt goes to Old UI's Log link -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9224) TimelineDelegationTokenIdentifier.Renewer contacts KDC for every renew/cancel token operation
[
https://issues.apache.org/jira/browse/YARN-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749766#comment-16749766
]
Rohith Sharma K S commented on YARN-9224:
-
It means that application submission rate per second is very high. When the
application is submitted/finished, RM renews/cancel ALL the tokens passed in
submissionContext. Since these tokens are maintained by RM, it is
responsibility of RM to renew/cancel. The renew is happening for different
application every time(Not for same application). It is directly proportional
to application submission rate.
> TimelineDelegationTokenIdentifier.Renewer contacts KDC for every renew/cancel
> token operation
> -
>
> Key: YARN-9224
> URL: https://issues.apache.org/jira/browse/YARN-9224
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.6.0, 2.7.3
>Reporter: Tarun Parimi
>Priority: Major
>
> In a production cluster, we have observed the active RM principal making
> excessive requests to the KDC server. Being a service principal, this
> shouldn't be the case normally.
> On capturing tcpdump for the connections between RM and KDC, we saw that
> these excessive requests were for the SPNEGO service
> HTTP/ats-host.example.com .
> The requests were also matching in frequency with the below log entry in RM.
> {code:java}
> 2019-01-09T03:41:56.048-0500 INFO
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl: Timeline service
> address: http://ats-host.example.com:8188/ws/v1/timeline/
> {code}
> On looking at the code in TimelineDelegationTokenIdentifier.java, it seems
> this kdc request for SPNEGO is done as we are creating a new timeline client
> instance every time.
> {code:java}
> @SuppressWarnings("unchecked")
> @Override
> public long renew(Token token, Configuration conf) throws IOException,
> InterruptedException {
> TimelineClient client = TimelineClient.createTimelineClient();
> try {
> client.init(conf);
> client.start();
> return client.renewDelegationToken(
> (Token) token);
> } catch (YarnException e) {
> throw new IOException(e);
> } finally {
> client.stop();
> }
> }
> @SuppressWarnings("unchecked")
> @Override
> public void cancel(Token token, Configuration conf) throws IOException,
> InterruptedException {
> TimelineClient client = TimelineClient.createTimelineClient();
> try {
> client.init(conf);
> client.start();
> client.cancelDelegationToken(
> (Token) token);
> } catch (YarnException e) {
> throw new IOException(e);
> } finally {
> client.stop();
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6695) Race condition in RM for publishing container events vs appFinished events causes NPE
[
https://issues.apache.org/jira/browse/YARN-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749560#comment-16749560
]
Rohith Sharma K S commented on YARN-6695:
-
Yes this is corner case and need to handle it. We can fix this issue as I have
already reviewed patch and given comments on current patch. Secondly, it is
better to update the documentation by removing
yarn.rm.system-metrics-publisher.emit-container-events configuration in
*Enabling Timeline Service v.2*
> Race condition in RM for publishing container events vs appFinished events
> causes NPE
> --
>
> Key: YARN-6695
> URL: https://issues.apache.org/jira/browse/YARN-6695
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Critical
> Attachments: YARN-6695.001.patch
>
>
> When RM publishes container events i.e by enabling
> *yarn.rm.system-metrics-publisher.emit-container-events*, there is race
> condition for processing events
> vs appFinished event that removes appId from collector list which cause NPE.
> Look at the below trace where appId is removed from collectors first and then
> corresponding events are processed.
> {noformat}
> 2017-06-06 19:28:48,896 INFO capacity.ParentQueue
> (ParentQueue.java:removeApplication(472)) - Application removed - appId:
> application_1496758895643_0005 user: root leaf-queue of parent: root
> #applications: 0
> 2017-06-06 19:28:48,921 INFO collector.TimelineCollectorManager
> (TimelineCollectorManager.java:remove(190)) - The collector service for
> application_1496758895643_0005 was removed
> 2017-06-06 19:28:48,922 ERROR metrics.TimelineServiceV2Publisher
> (TimelineServiceV2Publisher.java:putEntity(451)) - Error when publishing
> entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e01_1496758895643_0005_01_02']
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:72)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:480)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:469)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:201)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:127)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6695) Race condition in RM for publishing container events vs appFinished events causes NPE
[
https://issues.apache.org/jira/browse/YARN-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736704#comment-16736704
]
Rohith Sharma K S commented on YARN-6695:
-
[~eyang] Publishing container events from RM is disabled by default i.e
*yarn.rm.system-metrics-publisher.emit-container-events* is set to *false*. Are
you enabled this configuration? And we don't recommend to enable this
configuration since it overloads RM with lot of events. If you can attach stack
trace would be help full.
Reg the patch, I am not a fan of catching NPE! Instead lets do explicit null
check and log with right message something similar to
NMTimelinePublisher#putEntity.
> Race condition in RM for publishing container events vs appFinished events
> causes NPE
> --
>
> Key: YARN-6695
> URL: https://issues.apache.org/jira/browse/YARN-6695
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Critical
> Attachments: YARN-6695.001.patch
>
>
> When RM publishes container events i.e by enabling
> *yarn.rm.system-metrics-publisher.emit-container-events*, there is race
> condition for processing events
> vs appFinished event that removes appId from collector list which cause NPE.
> Look at the below trace where appId is removed from collectors first and then
> corresponding events are processed.
> {noformat}
> 2017-06-06 19:28:48,896 INFO capacity.ParentQueue
> (ParentQueue.java:removeApplication(472)) - Application removed - appId:
> application_1496758895643_0005 user: root leaf-queue of parent: root
> #applications: 0
> 2017-06-06 19:28:48,921 INFO collector.TimelineCollectorManager
> (TimelineCollectorManager.java:remove(190)) - The collector service for
> application_1496758895643_0005 was removed
> 2017-06-06 19:28:48,922 ERROR metrics.TimelineServiceV2Publisher
> (TimelineServiceV2Publisher.java:putEntity(451)) - Error when publishing
> entity TimelineEntity[type='YARN_CONTAINER',
> id='container_e01_1496758895643_0005_01_02']
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:72)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:480)
> at
> org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:469)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:201)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:127)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6735) Have a way to turn off container metrics from NMs
[ https://issues.apache.org/jira/browse/YARN-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732893#comment-16732893 ] Rohith Sharma K S commented on YARN-6735: - I see that method reportContainerResourceUsage doesn't added this check. Was it intentional? > Have a way to turn off container metrics from NMs > - > > Key: YARN-6735 > URL: https://issues.apache.org/jira/browse/YARN-6735 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-6735.001.patch, YARN-6735.002.patch > > > Have a way to turn off emitting system metrics from NMs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6149) Allow port range to be specified while starting NM Timeline collector manager.
[ https://issues.apache.org/jira/browse/YARN-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732867#comment-16732867 ] Rohith Sharma K S commented on YARN-6149: - +1 lgtm, I will commit it later of today if no more objections > Allow port range to be specified while starting NM Timeline collector manager. > -- > > Key: YARN-6149 > URL: https://issues.apache.org/jira/browse/YARN-6149 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-6149.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8567) Fetching yarn logs fails for long running application if it is not present in timeline store
[
https://issues.apache.org/jira/browse/YARN-8567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732806#comment-16732806
]
Rohith Sharma K S edited comment on YARN-8567 at 1/3/19 9:36 AM:
-
+1, patch doesn't seem apply.. would you rebase it and add a tests?
was (Author: rohithsharma):
+1, patch doesn't seem apply.. would you rebase it?
> Fetching yarn logs fails for long running application if it is not present in
> timeline store
>
>
> Key: YARN-8567
> URL: https://issues.apache.org/jira/browse/YARN-8567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.7.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Labels: log-aggregation
> Attachments: YARN-8567.001.patch
>
>
> Using yarn logs command for a long running application which has been running
> longer than the configured timeline service ttl
> {{yarn.timeline-service.ttl-ms }} fails with the following exception.
> {code:java}
> Exception in thread "main"
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: The entity
> for application application_152347939332_1 doesn't exist in the timeline
> store
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplication(ApplicationHistoryManagerOnTimelineStore.java:670)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainers(ApplicationHistoryManagerOnTimelineStore.java:219)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.getContainers(ApplicationHistoryClientService.java:211)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationHistoryProtocolPBServiceImpl.getContainers(ApplicationHistoryProtocolPBServiceImpl.java:172)
> at
> org.apache.hadoop.yarn.proto.ApplicationHistoryProtocol$ApplicationHistoryProtocolService$2.callBlockingMethod(ApplicationHistoryProtocol.java:201)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2309)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationHistoryProtocolPBClientImpl.getContainers(ApplicationHistoryProtocolPBClientImpl.java:183)
> at
> org.apache.hadoop.yarn.client.api.impl.AHSClientImpl.getContainers(AHSClientImpl.java:151)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getContainers(YarnClientImpl.java:720)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerReportsFromRunningApplication(LogsCLI.java:1089)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainersLogRequestForRunningApplication(LogsCLI.java:1064)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:976)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:300)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:107)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:327)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8567) Fetching yarn logs fails for long running application if it is not present in timeline store
[
https://issues.apache.org/jira/browse/YARN-8567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732806#comment-16732806
]
Rohith Sharma K S commented on YARN-8567:
-
+1, patch doesn't seem apply.. would you rebase it?
> Fetching yarn logs fails for long running application if it is not present in
> timeline store
>
>
> Key: YARN-8567
> URL: https://issues.apache.org/jira/browse/YARN-8567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 2.7.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Labels: log-aggregation
> Attachments: YARN-8567.001.patch
>
>
> Using yarn logs command for a long running application which has been running
> longer than the configured timeline service ttl
> {{yarn.timeline-service.ttl-ms }} fails with the following exception.
> {code:java}
> Exception in thread "main"
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: The entity
> for application application_152347939332_1 doesn't exist in the timeline
> store
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplication(ApplicationHistoryManagerOnTimelineStore.java:670)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainers(ApplicationHistoryManagerOnTimelineStore.java:219)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.getContainers(ApplicationHistoryClientService.java:211)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationHistoryProtocolPBServiceImpl.getContainers(ApplicationHistoryProtocolPBServiceImpl.java:172)
> at
> org.apache.hadoop.yarn.proto.ApplicationHistoryProtocol$ApplicationHistoryProtocolService$2.callBlockingMethod(ApplicationHistoryProtocol.java:201)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2309)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationHistoryProtocolPBClientImpl.getContainers(ApplicationHistoryProtocolPBClientImpl.java:183)
> at
> org.apache.hadoop.yarn.client.api.impl.AHSClientImpl.getContainers(AHSClientImpl.java:151)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getContainers(YarnClientImpl.java:720)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerReportsFromRunningApplication(LogsCLI.java:1089)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainersLogRequestForRunningApplication(LogsCLI.java:1064)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:976)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:300)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:107)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:327)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9149) yarn container -status misses logUrl when integrated with ATSv2
[
https://issues.apache.org/jira/browse/YARN-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725975#comment-16725975
]
Rohith Sharma K S commented on YARN-9149:
-
Sure.. assigned to you!
> yarn container -status misses logUrl when integrated with ATSv2
> ---
>
> Key: YARN-9149
> URL: https://issues.apache.org/jira/browse/YARN-9149
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
>
> Post YARN-8303, yarn client can be integrated with ATSv2. But log url and
> start and end time is printing data is wrong!
> {code}
> Container Report :
> Container-Id : container_1545035586969_0001_01_01
> Start-Time : 0
> Finish-Time : 0
> State : COMPLETE
> Execution-Type : GUARANTEED
> LOG-URL : null
> Host : localhost:25006
> NodeHttpAddress : localhost:25008
> Diagnostics :
> {code}
> # TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*.
> This need set for proper log url based on yarn.log.server.web-service.url
> # TimelineEntityV2Converter#convertToContainerReport parses start/end time
> wrongly. Comparison should happen with entityType but below code is doing
> entityId
> {code}
> if (events != null) {
> for (TimelineEvent event : events) {
> if (event.getId().equals(
> ContainerMetricsConstants.CREATED_IN_RM_EVENT_TYPE)) {
> createdTime = event.getTimestamp();
> } else if (event.getId().equals(
> ContainerMetricsConstants.FINISHED_IN_RM_EVENT_TYPE)) {
> finishedTime = event.getTimestamp();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9149) yarn container -status misses logUrl when integrated with ATSv2
[
https://issues.apache.org/jira/browse/YARN-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S reassigned YARN-9149:
---
Assignee: Abhishek Modi (was: Rohith Sharma K S)
> yarn container -status misses logUrl when integrated with ATSv2
> ---
>
> Key: YARN-9149
> URL: https://issues.apache.org/jira/browse/YARN-9149
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
>
> Post YARN-8303, yarn client can be integrated with ATSv2. But log url and
> start and end time is printing data is wrong!
> {code}
> Container Report :
> Container-Id : container_1545035586969_0001_01_01
> Start-Time : 0
> Finish-Time : 0
> State : COMPLETE
> Execution-Type : GUARANTEED
> LOG-URL : null
> Host : localhost:25006
> NodeHttpAddress : localhost:25008
> Diagnostics :
> {code}
> # TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*.
> This need set for proper log url based on yarn.log.server.web-service.url
> # TimelineEntityV2Converter#convertToContainerReport parses start/end time
> wrongly. Comparison should happen with entityType but below code is doing
> entityId
> {code}
> if (events != null) {
> for (TimelineEvent event : events) {
> if (event.getId().equals(
> ContainerMetricsConstants.CREATED_IN_RM_EVENT_TYPE)) {
> createdTime = event.getTimestamp();
> } else if (event.getId().equals(
> ContainerMetricsConstants.FINISHED_IN_RM_EVENT_TYPE)) {
> finishedTime = event.getTimestamp();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9149) yarn container -status misses logUrl when integrated with ATSv2
[
https://issues.apache.org/jira/browse/YARN-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S reassigned YARN-9149:
---
Assignee: Rohith Sharma K S
> yarn container -status misses logUrl when integrated with ATSv2
> ---
>
> Key: YARN-9149
> URL: https://issues.apache.org/jira/browse/YARN-9149
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
>
> Post YARN-8303, yarn client can be integrated with ATSv2. But log url and
> start and end time is printing data is wrong!
> {code}
> Container Report :
> Container-Id : container_1545035586969_0001_01_01
> Start-Time : 0
> Finish-Time : 0
> State : COMPLETE
> Execution-Type : GUARANTEED
> LOG-URL : null
> Host : localhost:25006
> NodeHttpAddress : localhost:25008
> Diagnostics :
> {code}
> # TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*.
> This need set for proper log url based on yarn.log.server.web-service.url
> # TimelineEntityV2Converter#convertToContainerReport parses start/end time
> wrongly. Comparison should happen with entityType but below code is doing
> entityId
> {code}
> if (events != null) {
> for (TimelineEvent event : events) {
> if (event.getId().equals(
> ContainerMetricsConstants.CREATED_IN_RM_EVENT_TYPE)) {
> createdTime = event.getTimestamp();
> } else if (event.getId().equals(
> ContainerMetricsConstants.FINISHED_IN_RM_EVENT_TYPE)) {
> finishedTime = event.getTimestamp();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9149) yarn container -status misses logUrl when integrated with ATSv2
[
https://issues.apache.org/jira/browse/YARN-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-9149:
Description:
Post YARN-8303, yarn client can be integrated with ATSv2. But log url and start
and end time is printing data is wrong!
{code}
Container Report :
Container-Id : container_1545035586969_0001_01_01
Start-Time : 0
Finish-Time : 0
State : COMPLETE
Execution-Type : GUARANTEED
LOG-URL : null
Host : localhost:25006
NodeHttpAddress : localhost:25008
Diagnostics :
{code}
# TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*. This
need set for proper log url based on yarn.log.server.web-service.url
# TimelineEntityV2Converter#convertToContainerReport parses start/end time
wrongly. Comparison should happen with entityType but below code is doing
entityId
{code}
if (events != null) {
for (TimelineEvent event : events) {
if (event.getId().equals(
ContainerMetricsConstants.CREATED_IN_RM_EVENT_TYPE)) {
createdTime = event.getTimestamp();
} else if (event.getId().equals(
ContainerMetricsConstants.FINISHED_IN_RM_EVENT_TYPE)) {
finishedTime = event.getTimestamp();
}
}
}
{code}
was:
Post YARN-8303, yarn client can be integrated with ATSv2. But log url and start
and end time is printing data is wrong!
{code}
Container Report :
Container-Id : container_1545035586969_0001_01_01
Start-Time : 0
Finish-Time : 0
State : COMPLETE
Execution-Type : GUARANTEED
LOG-URL : null
Host : localhost:25006
NodeHttpAddress : localhost:25008
Diagnostics :
{code}
TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*. This
need set for proper log url based on yarn.log.server.web-service.url
> yarn container -status misses logUrl when integrated with ATSv2
> ---
>
> Key: YARN-9149
> URL: https://issues.apache.org/jira/browse/YARN-9149
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Major
>
> Post YARN-8303, yarn client can be integrated with ATSv2. But log url and
> start and end time is printing data is wrong!
> {code}
> Container Report :
> Container-Id : container_1545035586969_0001_01_01
> Start-Time : 0
> Finish-Time : 0
> State : COMPLETE
> Execution-Type : GUARANTEED
> LOG-URL : null
> Host : localhost:25006
> NodeHttpAddress : localhost:25008
> Diagnostics :
> {code}
> # TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*.
> This need set for proper log url based on yarn.log.server.web-service.url
> # TimelineEntityV2Converter#convertToContainerReport parses start/end time
> wrongly. Comparison should happen with entityType but below code is doing
> entityId
> {code}
> if (events != null) {
> for (TimelineEvent event : events) {
> if (event.getId().equals(
> ContainerMetricsConstants.CREATED_IN_RM_EVENT_TYPE)) {
> createdTime = event.getTimestamp();
> } else if (event.getId().equals(
> ContainerMetricsConstants.FINISHED_IN_RM_EVENT_TYPE)) {
> finishedTime = event.getTimestamp();
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9149) yarn container -status misses logUrl when integrated with ATSv2
Rohith Sharma K S created YARN-9149:
---
Summary: yarn container -status misses logUrl when integrated with
ATSv2
Key: YARN-9149
URL: https://issues.apache.org/jira/browse/YARN-9149
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Post YARN-8303, yarn client can be integrated with ATSv2. But log url and start
and end time is printing data is wrong!
{code}
Container Report :
Container-Id : container_1545035586969_0001_01_01
Start-Time : 0
Finish-Time : 0
State : COMPLETE
Execution-Type : GUARANTEED
LOG-URL : null
Host : localhost:25006
NodeHttpAddress : localhost:25008
Diagnostics :
{code}
TimelineEntityV2Converter#convertToContainerReport set logUrl as *null*. This
need set for proper log url based on yarn.log.server.web-service.url
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9143) Aux service system-class is loading from a wrong property
[
https://issues.apache.org/jira/browse/YARN-9143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16723765#comment-16723765
]
Rohith Sharma K S commented on YARN-9143:
-
thanks [~cheersyang] for the patch!
+1 lgtm.. pending jenkins
> Aux service system-class is loading from a wrong property
> -
>
> Key: YARN-9143
> URL: https://issues.apache.org/jira/browse/YARN-9143
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-9143.001.patch, YARN-9143.002.patch
>
>
> For example, if we define a aux service with name A, and it's class is
> pkg.class.A
> It should be loading from {{yarn.nodemanager.aux-services.A.system-classes}},
> but currently it is loading from
> {{yarn.nodemanager.aux-services.pkg.class.A.system-classes}} now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9142) UI cluster nodes page is broken
[ https://issues.apache.org/jira/browse/YARN-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722798#comment-16722798 ] Rohith Sharma K S edited comment on YARN-9142 at 12/17/18 9:17 AM: --- YARN-8854 could be the reason! cc:/ [~akhilpb] [~sunilg] Could you confirm this ? was (Author: rohithsharma): YARN-8854 could be the reason break! cc:/ [~akhilpb] [~sunilg] Could you confirm this ? > UI cluster nodes page is broken > --- > > Key: YARN-9142 > URL: https://issues.apache.org/jira/browse/YARN-9142 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Priority: Critical > Attachments: ClusterNodePage.png > > > It is observed in trunk build YARN cluster node pages is broken even though > data exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9142) UI cluster nodes page is broken
[ https://issues.apache.org/jira/browse/YARN-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722793#comment-16722793 ] Rohith Sharma K S commented on YARN-9142: - We can see cluster node page is rendering nothing! [^ClusterNodePage.png] > UI cluster nodes page is broken > --- > > Key: YARN-9142 > URL: https://issues.apache.org/jira/browse/YARN-9142 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Priority: Critical > Attachments: ClusterNodePage.png > > > It is observed in trunk build YARN cluster node pages is broken even though > data exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9142) UI cluster nodes page is broken
[ https://issues.apache.org/jira/browse/YARN-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722798#comment-16722798 ] Rohith Sharma K S commented on YARN-9142: - YARN-8854 could be the reason break! cc:/ [~akhilpb] [~sunilg] Could you confirm this ? > UI cluster nodes page is broken > --- > > Key: YARN-9142 > URL: https://issues.apache.org/jira/browse/YARN-9142 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Priority: Critical > Attachments: ClusterNodePage.png > > > It is observed in trunk build YARN cluster node pages is broken even though > data exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9142) UI cluster nodes page is broken
Rohith Sharma K S created YARN-9142: --- Summary: UI cluster nodes page is broken Key: YARN-9142 URL: https://issues.apache.org/jira/browse/YARN-9142 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S Attachments: ClusterNodePage.png It is observed in trunk build YARN cluster node pages is broken even though data exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9142) UI cluster nodes page is broken
[ https://issues.apache.org/jira/browse/YARN-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9142: Attachment: ClusterNodePage.png > UI cluster nodes page is broken > --- > > Key: YARN-9142 > URL: https://issues.apache.org/jira/browse/YARN-9142 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Priority: Critical > Attachments: ClusterNodePage.png > > > It is observed in trunk build YARN cluster node pages is broken even though > data exist. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9125) Carriage Return character in launch command cause node manager to become unhealthy
[ https://issues.apache.org/jira/browse/YARN-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721983#comment-16721983 ] Rohith Sharma K S commented on YARN-9125: - Thanks [~billie.rinaldi] for the confirmation. > Carriage Return character in launch command cause node manager to become > unhealthy > -- > > Key: YARN-9125 > URL: https://issues.apache.org/jira/browse/YARN-9125 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.9.1, 3.0.2, 3.1.1, 2.9.2, 3.0.3 >Reporter: Eric Yang >Assignee: Billie Rinaldi >Priority: Major > Attachments: YARN-9125-branch-3.0.01.patch, YARN-9125.1.patch > > > If launch command contains \n character, it can cause node manager to become > unhealthy. There is a protection check against this, but it doesn't seem to > be working as implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9125) Carriage Return character in launch command cause node manager to become unhealthy
[ https://issues.apache.org/jira/browse/YARN-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721968#comment-16721968 ] Rohith Sharma K S commented on YARN-9125: - [~billie.rinaldi] [~eyang] One doubt, fix is done in docker client. Doesn't it applies to non-docker but using LinuxCE? > Carriage Return character in launch command cause node manager to become > unhealthy > -- > > Key: YARN-9125 > URL: https://issues.apache.org/jira/browse/YARN-9125 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.9.1, 3.0.2, 3.1.1, 2.9.2, 3.0.3 >Reporter: Eric Yang >Assignee: Billie Rinaldi >Priority: Major > Attachments: YARN-9125-branch-3.0.01.patch, YARN-9125.1.patch > > > If launch command contains \n character, it can cause node manager to become > unhealthy. There is a protection check against this, but it doesn't seem to > be working as implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9114) [UI2] log service redirect url should also support user name
[ https://issues.apache.org/jira/browse/YARN-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718619#comment-16718619 ] Rohith Sharma K S commented on YARN-9114: - +1 > [UI2] log service redirect url should also support user name > > > Key: YARN-9114 > URL: https://issues.apache.org/jira/browse/YARN-9114 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp, yarn-ui-v2 >Reporter: Sunil Govindan >Assignee: Akhil PB >Priority: Major > Attachments: YARN-9114.001.patch > > > Add username to yarn-log-service end point -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707455#comment-16707455
]
Rohith Sharma K S commented on YARN-9016:
-
bq. Also not sure why some unit tests are getting skipped while building
This could be an issue with build! cc: [~sunilg]
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707440#comment-16707440
]
Rohith Sharma K S commented on YARN-9016:
-
bq. So the TimelineSchemaCreator can be moved to the
hadoop-yarn-server-timelineservice module and each backend can have their
specific implementation, what do you think?
That's better! In future, each backend could implement this.
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706902#comment-16706902
]
Rohith Sharma K S commented on YARN-9016:
-
Ignore my previous doubt, I was referring older tar.gz :-(
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706889#comment-16706889
]
Rohith Sharma K S commented on YARN-9016:
-
I did full build and newly added package jars doesn't seems appearing in
tar.gz! Is there any profile to be enabled?
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2
[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706854#comment-16706854
]
Rohith Sharma K S commented on YARN-9016:
-
Going through this patch and design, I see one major change required i.e
regarding schema creation.
# {code}hadoop
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator{code}
Currently, TimelineSchemaCreator corresponds to HBase schema creator. We should
not change these syntax rather lets create interface or abstract class as
TimelineSchemaCreator. Each backend should implement this class. This way we
can add any schema creation logic plug-able based on backend.
# yarn.timeline-service.store-type-> yarn.timeline-service.document-store-type?
# I think all configuration name prefix with 'document-store' i.e
yarn.timeline-service.document-store
Going through patch still and I will be keep posting if any comments.
> DocumentStore as a backend for ATSv2
>
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
>
>
> yarn.timeline-service.writer.class/name>
>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
>
>
>yarn.timeline-service.reader.class/name>
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
>
>
>yarn.timeline-service.documentstore.db-name
>YOUR_DATABASE_NAME
> {code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
>
>
>yarn.timeline-service.store-type
>COSMOS_DB
>
>
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
>
>
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
[ https://issues.apache.org/jira/browse/YARN-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9044: Attachment: YARN-9044.02.patch > LogsCLI should contact ATSv2 for "-am" option > - > > Key: YARN-9044 > URL: https://issues.apache.org/jira/browse/YARN-9044 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9044.01.patch, YARN-9044.01.patch, > YARN-9044.02.patch > > > *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not > configured. Rather LogsCLI should contact ATSv2 for AM container info. > Alternative to above one can use *yarn logs -containerId * > to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
[ https://issues.apache.org/jira/browse/YARN-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698422#comment-16698422 ] Rohith Sharma K S commented on YARN-9044: - [~suma.shivaprasad] [~sunilg] Could you please review this patch? > LogsCLI should contact ATSv2 for "-am" option > - > > Key: YARN-9044 > URL: https://issues.apache.org/jira/browse/YARN-9044 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9044.01.patch, YARN-9044.01.patch > > > *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not > configured. Rather LogsCLI should contact ATSv2 for AM container info. > Alternative to above one can use *yarn logs -containerId * > to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
[ https://issues.apache.org/jira/browse/YARN-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9044: Attachment: YARN-9044.01.patch > LogsCLI should contact ATSv2 for "-am" option > - > > Key: YARN-9044 > URL: https://issues.apache.org/jira/browse/YARN-9044 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9044.01.patch, YARN-9044.01.patch > > > *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not > configured. Rather LogsCLI should contact ATSv2 for AM container info. > Alternative to above one can use *yarn logs -containerId * > to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
[ https://issues.apache.org/jira/browse/YARN-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S reassigned YARN-9044: --- Assignee: Rohith Sharma K S > LogsCLI should contact ATSv2 for "-am" option > - > > Key: YARN-9044 > URL: https://issues.apache.org/jira/browse/YARN-9044 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9044.01.patch > > > *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not > configured. Rather LogsCLI should contact ATSv2 for AM container info. > Alternative to above one can use *yarn logs -containerId * > to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
[ https://issues.apache.org/jira/browse/YARN-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9044: Attachment: YARN-9044.01.patch > LogsCLI should contact ATSv2 for "-am" option > - > > Key: YARN-9044 > URL: https://issues.apache.org/jira/browse/YARN-9044 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Priority: Major > Attachments: YARN-9044.01.patch > > > *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not > configured. Rather LogsCLI should contact ATSv2 for AM container info. > Alternative to above one can use *yarn logs -containerId * > to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9034: Attachment: YARN-9034.04.patch > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch, YARN-9034.04.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695578#comment-16695578 ] Rohith Sharma K S commented on YARN-9034: - I found test error and working on correcting it. > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695514#comment-16695514 ] Rohith Sharma K S commented on YARN-9034: - UT failure is unrelated to this patch. I checked the UT logs and all tests are passed from TestYARNCLi.java. One test is skipped from TestAMRMClient. This may the reason jenkin is showing red. No failures in test run! Reg the checkstyle about "Line is longer than 80 characters", those are required as-is in test to compare the help messages. These could be ignored. Whitespace could be fixed during committing patch using "--whitespace=" in git apply command. > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9044) LogsCLI should contact ATSv2 for "-am" option
Rohith Sharma K S created YARN-9044: --- Summary: LogsCLI should contact ATSv2 for "-am" option Key: YARN-9044 URL: https://issues.apache.org/jira/browse/YARN-9044 Project: Hadoop YARN Issue Type: Bug Reporter: Rohith Sharma K S *yarn logs -applicationId appId -am 1* contact ATS1.5 even though it is not configured. Rather LogsCLI should contact ATSv2 for AM container info. Alternative to above one can use *yarn logs -containerId * to fetch logs. But -am option should also work along with ATSv2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9042) Javadoc error in deviceplugin package
Rohith Sharma K S created YARN-9042:
---
Summary: Javadoc error in deviceplugin package
Key: YARN-9042
URL: https://issues.apache.org/jira/browse/YARN-9042
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rohith Sharma K S
Many java doc errors are in deviceplugin
{noformat}
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:29:
error: bad HTML entity
[ERROR] * This is a spec used to prepare & run container.
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/DeviceRuntimeSpec.java:35:
error: bad HTML entity
[ERROR] * The volume & device mounts describes key isolation requirements
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: domain
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: bus
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: slot
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
[ERROR] ^
[ERROR]
/Users/rsharmaks/Repos/Apache/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/deviceplugin/Device.java:56:
error: unknown tag: func
[ERROR]* PCI Bus ID in format ]:]]:][][.[]].
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8936) Bump up Atsv2 hbase versions
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-8936: Summary: Bump up Atsv2 hbase versions (was: Update ATSv2 hbase.two.version to 2.0.2) > Bump up Atsv2 hbase versions > > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch, > YARN-8936.0002.patch, YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9034: Attachment: YARN-9034.03.patch > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch, > YARN-9034.03.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9034: Attachment: YARN-9034.02.patch > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch, YARN-9034.02.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9034) ApplicationCLI should have option to take clusterId
[ https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-9034: Attachment: YARN-9034.01.patch > ApplicationCLI should have option to take clusterId > --- > > Key: YARN-9034 > URL: https://issues.apache.org/jira/browse/YARN-9034 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S >Priority: Major > Attachments: YARN-9034.01.patch > > > Post YARN-8303, LogsCLI provide an option to input clusterid which could be > used for fetching data from atsv2. ApplicationCLI also should have this > option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8936) Update ATSv2 hbase.two.version to 2.0.2
[ https://issues.apache.org/jira/browse/YARN-8936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692763#comment-16692763 ] Rohith Sharma K S commented on YARN-8936: - I am OK to bump up to latest stable version. Will commit it later of today if no more objections > Update ATSv2 hbase.two.version to 2.0.2 > --- > > Key: YARN-8936 > URL: https://issues.apache.org/jira/browse/YARN-8936 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.2.0, 3.1.2, 3.3.0 >Reporter: Rohith Sharma K S >Assignee: Vrushali C >Priority: Major > Attachments: YARN-8936-yarn-7055.0001.patch, YARN-8936.0001.patch, > YARN-8936.0002.patch, YARN-8936.0003.patch > > > Hadoop trunk uses hbase.two.version as 2.0.0-beta-1. HBase has release stable > Hbase-2.0.2 version and this could be used in Hadoop-3.3/3.2/3.1 branches. > cc:/ [~vrushalic] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6931) Make the aggregation interval in AppLevelTimelineCollector configurable
[ https://issues.apache.org/jira/browse/YARN-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692755#comment-16692755 ] Rohith Sharma K S commented on YARN-6931: - It is good candidate for branch-2. We can backport it > Make the aggregation interval in AppLevelTimelineCollector configurable > --- > > Key: YARN-6931 > URL: https://issues.apache.org/jira/browse/YARN-6931 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Abhishek Modi >Priority: Minor > Fix For: 3.2.0 > > Attachments: YARN-6931.001.patch, YARN-6931.002.patch > > > We do application-level metrics aggregation in AppLevelTimelineCollector, but > the interval is hardcoded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9034) ApplicationCLI should have option to take clusterId
Rohith Sharma K S created YARN-9034: --- Summary: ApplicationCLI should have option to take clusterId Key: YARN-9034 URL: https://issues.apache.org/jira/browse/YARN-9034 Project: Hadoop YARN Issue Type: Sub-task Reporter: Rohith Sharma K S Assignee: Rohith Sharma K S Post YARN-8303, LogsCLI provide an option to input clusterid which could be used for fetching data from atsv2. ApplicationCLI also should have this option. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8964) UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
[
https://issues.apache.org/jira/browse/YARN-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690215#comment-16690215
]
Rohith Sharma K S commented on YARN-8964:
-
Skimming through patch, it looks reasonable. But I want once [~sunilg] to take
a look at it once.
> UI2 should use clusters/{cluster name} for all ATSv2 REST APIs
> --
>
> Key: YARN-8964
> URL: https://issues.apache.org/jira/browse/YARN-8964
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rohith Sharma K S
>Assignee: Akhil PB
>Priority: Major
> Attachments: YARN-8964.001.patch
>
>
> UI2 makes a REST call to TimelineReader without cluster name. It is advised
> to make a REST call with clusters/{cluster name} so that remote
> TimelineReader daemon could serve for different clusters.
> *Example*:
> *Current*: /ws/v2/timeline/flows/
> *Change*: /ws/v2/timeline/*clusters/\{cluster name\}*/flows/
> *yarn.resourcemanager.cluster-id *is configured with cluster. So, this config
> could be used to get cluster-id
> cc:/ [~sunilg] [~akhilpb]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8834) Provide Java client for fetching Yarn specific entities from TimelineReader
[ https://issues.apache.org/jira/browse/YARN-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689403#comment-16689403 ] Rohith Sharma K S commented on YARN-8834: - This patch wan't committed to branch-3.2/branch-3.1/branch-3.0. I cherry-picked to 3.2/3.1 branched only. I removing 3.0.3 from fix version. > Provide Java client for fetching Yarn specific entities from TimelineReader > --- > > Key: YARN-8834 > URL: https://issues.apache.org/jira/browse/YARN-8834 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Fix For: 3.1.2, 3.3.0, 3.2.1 > > Attachments: YARN-8834.001.patch, YARN-8834.002.patch, > YARN-8834.003.patch, YARN-8834.004.patch, YARN-8834.005.patch, > YARN-8834.006.patch > > > While reviewing YARN-8303, we felt that it is necessary to provide > TimelineReaderClient which wraps all the REST calls in it so that user can > just provide application or container ids along with filters.Currently > fetching entities from TimelineReader is only via REST call or somebody need > to write java client get entities. > It is good to provide TimelineReaderClient which fetch entities from > TimelineReaderServer. This will be more useful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8834) Provide Java client for fetching Yarn specific entities from TimelineReader
[ https://issues.apache.org/jira/browse/YARN-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-8834: Fix Version/s: (was: 3.0.3) > Provide Java client for fetching Yarn specific entities from TimelineReader > --- > > Key: YARN-8834 > URL: https://issues.apache.org/jira/browse/YARN-8834 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Fix For: 3.1.2, 3.3.0, 3.2.1 > > Attachments: YARN-8834.001.patch, YARN-8834.002.patch, > YARN-8834.003.patch, YARN-8834.004.patch, YARN-8834.005.patch, > YARN-8834.006.patch > > > While reviewing YARN-8303, we felt that it is necessary to provide > TimelineReaderClient which wraps all the REST calls in it so that user can > just provide application or container ids along with filters.Currently > fetching entities from TimelineReader is only via REST call or somebody need > to write java client get entities. > It is good to provide TimelineReaderClient which fetch entities from > TimelineReaderServer. This will be more useful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[ https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689340#comment-16689340 ] Rohith Sharma K S commented on YARN-8303: - +1 committing shortly > YarnClient should contact TimelineReader for application/attempt/container > report > - > > Key: YARN-8303 > URL: https://issues.apache.org/jira/browse/YARN-8303 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Attachments: YARN-8303.001.patch, YARN-8303.002.patch, > YARN-8303.003.patch, YARN-8303.004.patch, YARN-8303.005.patch, > YARN-8303.006.patch, YARN-8303.poc.patch > > > YarnClient get app/attempt/container information from RM. If RM doesn't have > then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result > empty. > YarnClient is used by many users which result in empty information for > app/attempt/container report. > Proposal is to have adapter from yarn client so that app/attempt/container > reports can be generated from AHSv2Client which does REST API to > TimelineReader and get the entity and convert it into app/attempt/container > report. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9031) YARN UI2 should make use of "yarn.log.server.web-service.url" to fetch container logs
Rohith Sharma K S created YARN-9031: --- Summary: YARN UI2 should make use of "yarn.log.server.web-service.url" to fetch container logs Key: YARN-9031 URL: https://issues.apache.org/jira/browse/YARN-9031 Project: Hadoop YARN Issue Type: Improvement Reporter: Rohith Sharma K S Post YARN-5742, ATSv2.0 introduces log servlet which could be used for log fetching. Many cluster deployment ATS1.5 may not exist which causes log fetching failure in UI2. # UI2 should be able to fetch logs from ATS2.0 log servlet. # Its better to take log server url based on the property "yarn.log.server.web-service.url" # Each ATSv2 log URL takes clusterid as query params. UI2 should send clusterid query param if required. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[ https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689304#comment-16689304 ] Rohith Sharma K S commented on YARN-8303: - +1 lgtm.. [~abmodi] test is failing, could you correct test case ? > YarnClient should contact TimelineReader for application/attempt/container > report > - > > Key: YARN-8303 > URL: https://issues.apache.org/jira/browse/YARN-8303 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Attachments: YARN-8303.001.patch, YARN-8303.002.patch, > YARN-8303.003.patch, YARN-8303.004.patch, YARN-8303.005.patch, > YARN-8303.006.patch, YARN-8303.poc.patch > > > YarnClient get app/attempt/container information from RM. If RM doesn't have > then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result > empty. > YarnClient is used by many users which result in empty information for > app/attempt/container report. > Proposal is to have adapter from yarn client so that app/attempt/container > reports can be generated from AHSv2Client which does REST API to > TimelineReader and get the entity and convert it into app/attempt/container > report. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9028) TLSv2.0 ats-hbase app flooding Isilon OneFS hdfs.log and ats-hbase region server logs
[ https://issues.apache.org/jira/browse/YARN-9028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689032#comment-16689032 ] Rohith Sharma K S commented on YARN-9028: - This is vendor specific deployment issue. Please contact the vendor! > TLSv2.0 ats-hbase app flooding Isilon OneFS hdfs.log and ats-hbase region > server logs > - > > Key: YARN-9028 > URL: https://issues.apache.org/jira/browse/YARN-9028 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 >Affects Versions: 3.0.1 >Reporter: Kirankumar Bhusanurmath >Priority: Major > > HDp3.0.1 deployed on DellEMC Isilon OneFS 8.1.2 > The ats-hbase app is up and running but flooding the OneFS hdfs and ats-hbase > region server logs with below message. > OneFS log > 2018-11-16T04:29:44Z <30.6> kbhusan-y93o5ew-1 hdfs[2711]: [hdfs] RPC V9 > complete user: yarn-ats exception: java.io.FileNotFoundException cause: File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.1542088499176 > STATUS_OBJECT_NAME_NOT_FOUND = 0xC034 Path: > atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.1542088499176> > 2018-11-16T04:29:45Z <30.6> kbhusan-y93o5ew-1 hdfs[2711]: [hdfs] RPC V9 > complete user: yarn-ats exception: java.io.FileNotFoundException cause: File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.meta.1542185701850.meta > STATUS_OBJECT_NAME_NOT_FOUND = 0xC034 Path: > atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.meta.1542185701850.meta> > > hbase-yarn-ats-regionserver-RDUVNODE334518.log. > 2018-11-13 03:23:30,178 WARN [Close-WAL-Writer-22] > asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1541663220372/rduvnode334518.west.isilon.com%2C17020%2C1541663220372.1541742426412 > failed, retry = 16586 > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1541663220372/rduvnode334518.west.isilon.com%2C17020%2C1541663220372.1541742426412 > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy18.complete(Unknown Source) > > (each logs are more then 250MB unable to upload > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9028) TLSv2.0 ats-hbase app flooding Isilon OneFS hdfs.log and ats-hbase region server logs
[ https://issues.apache.org/jira/browse/YARN-9028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-9028. - Resolution: Invalid > TLSv2.0 ats-hbase app flooding Isilon OneFS hdfs.log and ats-hbase region > server logs > - > > Key: YARN-9028 > URL: https://issues.apache.org/jira/browse/YARN-9028 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 >Affects Versions: 3.0.1 >Reporter: Kirankumar Bhusanurmath >Priority: Major > > HDp3.0.1 deployed on DellEMC Isilon OneFS 8.1.2 > The ats-hbase app is up and running but flooding the OneFS hdfs and ats-hbase > region server logs with below message. > OneFS log > 2018-11-16T04:29:44Z <30.6> kbhusan-y93o5ew-1 hdfs[2711]: [hdfs] RPC V9 > complete user: yarn-ats exception: java.io.FileNotFoundException cause: File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.1542088499176 > STATUS_OBJECT_NAME_NOT_FOUND = 0xC034 Path: > atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.1542088499176> > 2018-11-16T04:29:45Z <30.6> kbhusan-y93o5ew-1 hdfs[2711]: [hdfs] RPC V9 > complete user: yarn-ats exception: java.io.FileNotFoundException cause: File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.meta.1542185701850.meta > STATUS_OBJECT_NAME_NOT_FOUND = 0xC034 Path: > atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1542081292033/rduvnode334518.west.isilon.com%2C17020%2C1542081292033.meta.1542185701850.meta> > > hbase-yarn-ats-regionserver-RDUVNODE334518.log. > 2018-11-13 03:23:30,178 WARN [Close-WAL-Writer-22] > asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1541663220372/rduvnode334518.west.isilon.com%2C17020%2C1541663220372.1541742426412 > failed, retry = 16586 > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File > not found: > /atsv2/hbase/data/WALs/rduvnode334518.west.isilon.com,17020,1541663220372/rduvnode334518.west.isilon.com%2C17020%2C1541663220372.1541742426412 > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) > at com.sun.proxy.$Proxy18.complete(Unknown Source) > > (each logs are more then 250MB unable to upload > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9029) yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle
[ https://issues.apache.org/jira/browse/YARN-9029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-9029. - Resolution: Invalid This is vendor specific deployment issue. Please contact the vendor! > yarn-ats hbase expects OneFS(namenode and datanode) to have a yarn principle > > > Key: YARN-9029 > URL: https://issues.apache.org/jira/browse/YARN-9029 > Project: Hadoop YARN > Issue Type: Bug > Components: ATSv2 >Affects Versions: 3.0.1 > Environment: Ambari 2.7.1 > HDP 3.0.1 > Isilon OneFS 8.1.2 >Reporter: Kirankumar Bhusanurmath >Priority: Major > > *Background on ats-hbase start success*:: We had setStoragePolicy exception > on OneFS for yarn-ats hbase region server and below messages respectively on > HDP and OneFS cluster. > *HDP Host* *è* */var/log/hadoop-yarn/embedded-yarn-ats-hbase/* > 2018-11-02 05:07:21,027 WARN [StoreOpener-1588230740-1] util.CommonFSUtils: > Unable to set storagePolicy=HOT for > path=hdfs://kb-hdp-1.west.isilon.com:8020/atsv2/hbase/data/data/hbase/meta/1588230740/info. > DEBUG log level might have more details. > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > *OneFS HDFS log for (unsecured)* > 2018-11-01T10:25:19Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 > user: yarn-ats exception: org.apache.hadoop.ipc.RpcNoSuchMethodException > cause: Unknown RPC: setStoragePolicy > > *OneFS HDFS log for (secured)* > 2018-11-01T12:39:56Z <30.6> kbhusan-2oq1zs7-1 hdfs[2604]: [hdfs] RPC V9 > user: > yarn-ats-hbase/[krb-hdp301.solarch.emc@solarch.emc.com|mailto:krb-hdp301.solarch.emc@solarch.emc.com] > exception: > org.apache.hadoop.ipc.RpcNoSuchMethodException cause: Unknown RPC: > setStoragePolicy > > > As a resolution, we set below 2 properties in custom hbase-site.xml file and > setstoragePolicy exception got resolved. > *Custom yarn-hbase-site.xml* > 'hbase.wal.storage.policy' to ‘NONE’ > ‘hbase.hstore.block.storage.policy’ to ‘NONE’ > > > * > *NOW*, we have a new issue that is OneFS HDFS log and > /var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-.log > gets flooded with below error messages.*OneFS HDFS Log* > --- > 2018-11-06T06:25:43Z <30.6> kbhusan-edl9tdc-1 hdfs[2542]: [hdfs] RPC V9 > complete user: > yarn-ats-hbase/[rduvnode343393.west.isilon@west.isilon.com|mailto:rduvnode343393.west.isilon@west.isilon.com] > exception: > java.io.FileNotFoundException cause: File not found: > /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta > STATUS_OBJECT_NAME_NOT_FOUND = 0xC034 Path: > atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541478673646/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541478673646.meta.1541478709060.meta> > --- > > */var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-regionserver-.log* > -- > 2018-11-06 04:01:12,863 INFO [LruBlockCacheStatsExecutor] > hfile.LruBlockCache: totalSize=1.12 MB, freeSize=1.25 GB, max=1.25 GB, > blockCount=24, accesses=287, hits=263, hitRatio=91.64%, , > cachingAccesses=287, cachingHits=263, cachingHitsRatio=91.64%, > evictions=419, evicted=0, evictedPerRun=0.0 > > 2018-11-06 04:01:48,763 WARN > [LeaseRenewer:[yarn-...@kb-hdp301-2.west.isilon.com|mailto:yarn-...@kb-hdp301-2.west.isilon.com]:8020] > impl.LeaseRenewer: > Failed to renew lease for [DFSClient_NONMAPREDUCE_-533598397_1] for 30 > seconds. Will retry shortly ... > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): > Client auth method 'SIMPLE', mechanism '' is not supported on zone:1 > auth:'kerberos_only' > > at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1497) > > at org.apache.hadoop.ipc.Client.call(Client.java:1443) > > at org.apache.hadoop.ipc.Client.call(Client.java:1353) > > “ > “ > “ > “ > 2018-11-06 12:51:02,843 WARN [Close-WAL-Writer-1] > asyncfs.FanOutOneBlockAsyncDFSOutputHelper: complete file > /atsv2/hbase/data/WALs/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/],17020,1541488822956/[rduvnode343393.west.isilon.com|http://rduvnode343393.west.isilon.com/]%2C17020%2C1541488822956.1541492429802 > failed, retry = 637 > org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File > not
[jira] [Commented] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687700#comment-16687700 ] Rohith Sharma K S commented on YARN-8856: - +1lgtm > TestTimelineReaderWebServicesHBaseStorage tests failing with > NoClassDefFoundError > - > > Key: YARN-8856 > URL: https://issues.apache.org/jira/browse/YARN-8856 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Sushil Ks >Priority: Major > Attachments: YARN-8856.001.patch > > > TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds > with NoClassDefFoundError in the tests. Sample error and stacktrace to > follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[ https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687689#comment-16687689 ] Rohith Sharma K S commented on YARN-8303: - [~abmodi] We also need to newer option in help message i.e in method createPrintOpts other help message doesn't show up this name. Could you also make this change > YarnClient should contact TimelineReader for application/attempt/container > report > - > > Key: YARN-8303 > URL: https://issues.apache.org/jira/browse/YARN-8303 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Attachments: YARN-8303.001.patch, YARN-8303.002.patch, > YARN-8303.003.patch, YARN-8303.004.patch, YARN-8303.poc.patch > > > YarnClient get app/attempt/container information from RM. If RM doesn't have > then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result > empty. > YarnClient is used by many users which result in empty information for > app/attempt/container report. > Proposal is to have adapter from yarn client so that app/attempt/container > reports can be generated from AHSv2Client which does REST API to > TimelineReader and get the entity and convert it into app/attempt/container > report. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[
https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687590#comment-16687590
]
Rohith Sharma K S commented on YARN-8303:
-
Digging more details, it appears
NMTimelinePublisher#publishContainerCreatedEvent
{code}entityInfo.put(ContainerMetricsConstants.ALLOCATED_PRIORITY_INFO,
container.getPriority().toString());{code} is publishing String value
which is the reason for failure. So, lets not change the publisher rather lets
change in converter!
> YarnClient should contact TimelineReader for application/attempt/container
> report
> -
>
> Key: YARN-8303
> URL: https://issues.apache.org/jira/browse/YARN-8303
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Critical
> Attachments: YARN-8303.001.patch, YARN-8303.002.patch,
> YARN-8303.003.patch, YARN-8303.004.patch, YARN-8303.poc.patch
>
>
> YarnClient get app/attempt/container information from RM. If RM doesn't have
> then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result
> empty.
> YarnClient is used by many users which result in empty information for
> app/attempt/container report.
> Proposal is to have adapter from yarn client so that app/attempt/container
> reports can be generated from AHSv2Client which does REST API to
> TimelineReader and get the entity and convert it into app/attempt/container
> report.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[
https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687572#comment-16687572
]
Rohith Sharma K S commented on YARN-8303:
-
thanks [~abmodi] for the patch! I see some issues while testing this patch!
# All entity info will be in String. So we need to parse string to int.
{code}
2018-11-15 12:25:35,354 WARN impl.YarnClientImpl: Got an error while fetching
container report from ATSv2
java.lang.ClassCastException: java.lang.String cannot be cast to
java.lang.Integer
at
org.apache.hadoop.yarn.util.timeline.TimelineEntityV2Converter.convertToContainerReport(TimelineEntityV2Converter.java:97)
at
org.apache.hadoop.yarn.client.api.impl.AHSv2ClientImpl.getContainers(AHSv2ClientImpl.java:142)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getContainerReportFromHistory(YarnClientImpl.java:922)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getContainers(YarnClientImpl.java:872)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.listContainers(ApplicationCLI.java:1244)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:487)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:123)
{code}
Looks like similar issue exist all other converter which need to be relooked
based on variable type
> YarnClient should contact TimelineReader for application/attempt/container
> report
> -
>
> Key: YARN-8303
> URL: https://issues.apache.org/jira/browse/YARN-8303
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Critical
> Attachments: YARN-8303.001.patch, YARN-8303.002.patch,
> YARN-8303.003.patch, YARN-8303.004.patch, YARN-8303.poc.patch
>
>
> YarnClient get app/attempt/container information from RM. If RM doesn't have
> then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result
> empty.
> YarnClient is used by many users which result in empty information for
> app/attempt/container report.
> Proposal is to have adapter from yarn client so that app/attempt/container
> reports can be generated from AHSv2Client which does REST API to
> TimelineReader and get the entity and convert it into app/attempt/container
> report.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[ https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686556#comment-16686556 ] Rohith Sharma K S commented on YARN-8303: - [~abmodi] along with above comments, 2 of checkstyle warnings could be fixed also > YarnClient should contact TimelineReader for application/attempt/container > report > - > > Key: YARN-8303 > URL: https://issues.apache.org/jira/browse/YARN-8303 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Attachments: YARN-8303.001.patch, YARN-8303.002.patch, > YARN-8303.003.patch, YARN-8303.poc.patch > > > YarnClient get app/attempt/container information from RM. If RM doesn't have > then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result > empty. > YarnClient is used by many users which result in empty information for > app/attempt/container report. > Proposal is to have adapter from yarn client so that app/attempt/container > reports can be generated from AHSv2Client which does REST API to > TimelineReader and get the entity and convert it into app/attempt/container > report. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[
https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686554#comment-16686554
]
Rohith Sharma K S commented on YARN-8303:
-
Some comments
YarnClientImpl
# Most of the API's does't have *return* after ahs2client.() is called.
This causes always ATS1.5 API to call!
{code:java}
if (timelineV2ServiceEnabled) {
try {
ahsV2Client.getApplicationAttemptReport(appAttemptId);
} catch (Exception ex) {
LOG.warn("Failed to fetch application attempt report from "
+ "ATS v2", ex);
}
}
{code}
# Newly added method getContainerReportFromHistory has ambiguity. After
catching, again ahsV2Client is called!.
{code:java}
private List getContainerReportFromHistory(
ApplicationAttemptId applicationAttemptId)
throws IOException, YarnException {
List containersListFromAHS = null;
if (timelineV2ServiceEnabled) {
try {
containersListFromAHS = ahsV2Client.getContainers(applicationAttemptId);
} catch (Exception e) {
LOG.warn("Got an error while fetching container report from ATSv2", e);
if (historyServiceEnabled) {
containersListFromAHS = ahsV2Client.getContainers(
applicationAttemptId);
} else {
throw e;
}
}
} else if (historyServiceEnabled) {
containersListFromAHS = historyClient.getContainers(applicationAttemptId);
}
return containersListFromAHS;
}
{code}
> YarnClient should contact TimelineReader for application/attempt/container
> report
> -
>
> Key: YARN-8303
> URL: https://issues.apache.org/jira/browse/YARN-8303
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Critical
> Attachments: YARN-8303.001.patch, YARN-8303.002.patch,
> YARN-8303.003.patch, YARN-8303.poc.patch
>
>
> YarnClient get app/attempt/container information from RM. If RM doesn't have
> then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result
> empty.
> YarnClient is used by many users which result in empty information for
> app/attempt/container report.
> Proposal is to have adapter from yarn client so that app/attempt/container
> reports can be generated from AHSv2Client which does REST API to
> TimelineReader and get the entity and convert it into app/attempt/container
> report.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
