[ https://issues.apache.org/jira/browse/YARN-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Szilard Nemeth updated YARN-10850: ---------------------------------- Fix Version/s: 3.3.4 > TimelineService v2 lists containers for all attempts when filtering for one > --------------------------------------------------------------------------- > > Key: YARN-10850 > URL: https://issues.apache.org/jira/browse/YARN-10850 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader > Reporter: Benjamin Teke > Assignee: Benjamin Teke > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.4 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > When using the command > {code:java} > yarn container -list <app_attempt_id> > {code} > with an application attempt ID based on the help only the containers for that > attempt should be listed. > {code:java} > -list <Application Name or Attempt ID> List containers for application > attempt when application > attempt ID is provided. When > application name is provided, > then it finds the instances of > the application based on app's > own implementation, and > -appTypes option must be > specified unless it is the > default yarn-service type. With > app name, it supports optional > use of -version to filter > instances based on app version, > -components to filter instances > based on component names, > -states to filter instances > based on instance state. > {code} > When TimelineService v2 is enabled all of the containers for the application > are returned. > {code:java} > hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn > applicationattempt -list application_1625124233002_0007 > 21/07/01 09:32:23 INFO impl.TimelineReaderClientImpl: Initialized > TimelineReader > URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, > clusterId=yarn-cluster > 21/07/01 09:32:24 INFO client.AHSProxy: Connecting to Application History > server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200 > 21/07/01 09:32:24 INFO client.ConfiguredRMFailoverProxyProvider: Failing over > to rm2 > Total number of application attempts :2 > ApplicationAttempt-Id State > AM-Container-Id Tracking-URL > appattempt_1625124233002_0007_000001 FAILED > container_e43_1625124233002_0007_01_000001 > http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/ > appattempt_1625124233002_0007_000002 KILLED > container_e43_1625124233002_0007_02_000001 > http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/ > {code} > Querying the 2 app attempts produces the same output: > {code:java} > hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container > -list appattempt_1625124233002_0007_000001 > 21/07/01 09:32:35 INFO impl.TimelineReaderClientImpl: Initialized > TimelineReader > URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, > clusterId=yarn-cluster > 21/07/01 09:32:35 INFO client.AHSProxy: Connecting to Application History > server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200 > 21/07/01 09:32:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over > to rm2 > 21/07/01 09:32:36 INFO conf.Configuration: found resource resource-types.xml > at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml > Total number of containers :12 > Container-Id Start Time Finish > Time State Host Node Http Address > LOG-URL > container_e43_1625124233002_0007_02_000004 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000002.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000002.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa > container_e43_1625124233002_0007_02_000005 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000007.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000007.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa > container_e43_1625124233002_0007_02_000003 N/A > N/A null > ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa > container_e43_1625124233002_0007_02_000002 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa > container_e43_1625124233002_0007_02_000006 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000005.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000005.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa > container_e43_1625124233002_0007_02_000001 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa > container_e43_1625124233002_0007_01_000005 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000002.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000002.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa > container_e43_1625124233002_0007_01_000004 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa > container_e43_1625124233002_0007_01_000002 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000005.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000005.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa > container_e43_1625124233002_0007_01_000006 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa > container_e43_1625124233002_0007_01_000003 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000007.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000007.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa > container_e43_1625124233002_0007_01_000001 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa > {code} > {code:java} > hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container > -list appattempt_1625124233002_0007_000002 > 21/07/01 09:32:50 INFO impl.TimelineReaderClientImpl: Initialized > TimelineReader > URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, > clusterId=yarn-cluster > 21/07/01 09:32:51 INFO client.AHSProxy: Connecting to Application History > server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200 > 21/07/01 09:32:51 INFO client.ConfiguredRMFailoverProxyProvider: Failing over > to rm2 > 21/07/01 09:32:51 INFO conf.Configuration: found resource resource-types.xml > at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml > Total number of containers :12 > Container-Id Start Time Finish > Time State Host Node Http Address > LOG-URL > container_e43_1625124233002_0007_02_000004 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000002.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000002.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa > container_e43_1625124233002_0007_02_000005 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000007.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000007.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa > container_e43_1625124233002_0007_02_000003 N/A > N/A null > ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa > container_e43_1625124233002_0007_02_000002 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa > container_e43_1625124233002_0007_02_000006 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000005.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000005.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa > container_e43_1625124233002_0007_02_000001 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa > container_e43_1625124233002_0007_01_000005 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000002.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000002.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa > container_e43_1625124233002_0007_01_000004 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa > container_e43_1625124233002_0007_01_000002 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000005.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000005.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa > container_e43_1625124233002_0007_01_000006 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa > container_e43_1625124233002_0007_01_000003 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000007.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000007.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa > container_e43_1625124233002_0007_01_000001 N/A > N/A COMPLETE > ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 > ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 > http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa > {code} > The reason for this is that the [container collector > method|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java#L135] > in TimelineServiceV2 gets the *application ID* from the *attempt ID* and > collects the [containers for the > application|https://github.com/apache/hadoop/blob/a3edfddcf7822ea13bdf4858672eb82cea5e0b5f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java#L116]. > The other implementations however are using the application attempt ID: > [ApplicationHistoryManagerOnTimelineStore|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java#L221] > and > [ApplicationHistoryManagerImpl|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java#L235]. > TSv2 should behave similarly to v1. -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org