[jira] [Created] (YARN-8128) Document better the per-node per-app file limit in YARN log aggregation
Xuan Gong created YARN-8128: --- Summary: Document better the per-node per-app file limit in YARN log aggregation Key: YARN-8128 URL: https://issues.apache.org/jira/browse/YARN-8128 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5193) For long running services, aggregate logs when a container completes instead of when the app completes
[ https://issues.apache.org/jira/browse/YARN-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-5193. - Resolution: Won't Fix > For long running services, aggregate logs when a container completes instead > of when the app completes > -- > > Key: YARN-5193 > URL: https://issues.apache.org/jira/browse/YARN-5193 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Siddharth Seth >Priority: Major > > For a long running service, containers will typically not complete very > often. However, when a container completes - it would be useful to aggregate > the logs right then, instead of waiting for the app to complete. > This will allow the command line log tool to lookup containers for an app > from the log file index itself, instead of having to go and talk to YARN. > Talking to YARN really only works if ATS is enabled, and YARN is configured > to publish container information to ATS (That may not always be the case - > since this can overload ATS quite fast). > There's some added benefits like cleaning out local disk space early, instead > of waiting till the app completes. (There's probably a separate jira > somewhere about cleanup of container for long running services anyway) > cc [~vinodkv], [~xgong] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7952) Find a way to persist the log aggregation status
Xuan Gong created YARN-7952: --- Summary: Find a way to persist the log aggregation status Key: YARN-7952 URL: https://issues.apache.org/jira/browse/YARN-7952 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong In MAPREDUCE-6415, we have created a CLI to har the aggregated logs, and In YARN-4946: RM should write out Aggregated Log Completion file flag next to logs, we have a discussion on how we can get the log aggregation status: make a client call to RM or get it directly from the Distributed file system(HDFS). No matter which approach we would like to choose, we need to figure out a way to persist the log aggregation status first. This ticket is used to track the working progress for this purpose. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7951) Find a way to persist the log aggregation status
Xuan Gong created YARN-7951: --- Summary: Find a way to persist the log aggregation status Key: YARN-7951 URL: https://issues.apache.org/jira/browse/YARN-7951 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong In MAPREDUCE-6415, we have created a CLI to har the aggregated logs, and In YARN-4946: RM should write out Aggregated Log Completion file flag next to logs, we have a discussion on how we can get the log aggregation status: make a client call to RM or get it directly from the Distributed file system(HDFS). No matter which approach we would like to choose, we need to figure out a way to persist the log aggregation status first. This ticket is used to track the working progress for this purpose. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7891) LogAggregationIndexedFileController should support HAR file
Xuan Gong created YARN-7891: --- Summary: LogAggregationIndexedFileController should support HAR file Key: YARN-7891 URL: https://issues.apache.org/jira/browse/YARN-7891 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7883) Make HAR tool support IndexedLogAggregtionController
[ https://issues.apache.org/jira/browse/YARN-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-7883. - Resolution: Duplicate > Make HAR tool support IndexedLogAggregtionController > > > Key: YARN-7883 > URL: https://issues.apache.org/jira/browse/YARN-7883 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Major > > In https://issues.apache.org/jira/browse/MAPREDUCE-6415, we have created a > tool to combine aggregated logs into HAR files which currently only work for > TFileLogAggregationFileController. We should make it support > IndexedLogAggregtionController as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7883) Make HAR tool support IndexedLogAggregtionController
Xuan Gong created YARN-7883: --- Summary: Make HAR tool support IndexedLogAggregtionController Key: YARN-7883 URL: https://issues.apache.org/jira/browse/YARN-7883 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong In https://issues.apache.org/jira/browse/MAPREDUCE-6415, we have created a tool to combine aggregated logs into HAR files which currently only work for TFileLogAggregationFileController. We should make it support IndexedLogAggregtionController as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7697) NM goes down with OOM due to leak in log-aggregation
Xuan Gong created YARN-7697: --- Summary: NM goes down with OOM due to leak in log-aggregation Key: YARN-7697 URL: https://issues.apache.org/jira/browse/YARN-7697 Project: Hadoop YARN Issue Type: Bug Reporter: Santhosh B Gowda Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7598) Document how to use classpath isolation for aux-services in YARN
Xuan Gong created YARN-7598: --- Summary: Document how to use classpath isolation for aux-services in YARN Key: YARN-7598 URL: https://issues.apache.org/jira/browse/YARN-7598 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7571) YARN has multiple Queue Configuration Providers
Xuan Gong created YARN-7571: --- Summary: YARN has multiple Queue Configuration Providers Key: YARN-7571 URL: https://issues.apache.org/jira/browse/YARN-7571 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong After YARN-5734, we have multiple Queue Configuration Providers: * queue configuration XML file from local conf directory * OrgQueue: queue configuration from zk/leveldb backing store. * Remote Configuration Provider(YARN-1611) We should be careful if all of them are enabled by accident -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7570) Merge Scheduler configuration Store and ZK RM State Store
Xuan Gong created YARN-7570: --- Summary: Merge Scheduler configuration Store and ZK RM State Store Key: YARN-7570 URL: https://issues.apache.org/jira/browse/YARN-7570 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong In RMHA, we are using zookeeper based store for scheduler configuration updates which is separately from the ZK RM State Store. Given they have many similarities, it is better to merge the implementations together. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7569) Add ability to dump and upload scheduler configuration XML
Xuan Gong created YARN-7569: --- Summary: Add ability to dump and upload scheduler configuration XML Key: YARN-7569 URL: https://issues.apache.org/jira/browse/YARN-7569 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7568) Add audit log for add/remove queues
Xuan Gong created YARN-7568: --- Summary: Add audit log for add/remove queues Key: YARN-7568 URL: https://issues.apache.org/jira/browse/YARN-7568 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7567) Make sure that OrgQueue works in secure ZK environment
Xuan Gong created YARN-7567: --- Summary: Make sure that OrgQueue works in secure ZK environment Key: YARN-7567 URL: https://issues.apache.org/jira/browse/YARN-7567 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7566) [Umbrella] More improvements on YARN Queue Management
Xuan Gong created YARN-7566: --- Summary: [Umbrella] More improvements on YARN Queue Management Key: YARN-7566 URL: https://issues.apache.org/jira/browse/YARN-7566 Project: Hadoop YARN Issue Type: Task Reporter: Xuan Gong Assignee: Xuan Gong This serves as an umbrella ticket for tasks related to adding more improvements on YARN Queue Management. * YARN-5734: OrgQueue for easy CapacityScheduler queue configuration management * YARN-5724: Configuration-based Queue Management -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7558) YARN log command fails to get logs for running containers if the url authentication is enabled.
Xuan Gong created YARN-7558: --- Summary: YARN log command fails to get logs for running containers if the url authentication is enabled. Key: YARN-7558 URL: https://issues.apache.org/jira/browse/YARN-7558 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7417) re-factory IndexedFileAggregatedLogsBlock and TFileAggregatedLogsBlock to remove duplicate codes
Xuan Gong created YARN-7417: --- Summary: re-factory IndexedFileAggregatedLogsBlock and TFileAggregatedLogsBlock to remove duplicate codes Key: YARN-7417 URL: https://issues.apache.org/jira/browse/YARN-7417 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7381) Enable the configuration: yarn.nodemanager.log-container-debug-info.enabled
Xuan Gong created YARN-7381: --- Summary: Enable the configuration: yarn.nodemanager.log-container-debug-info.enabled Key: YARN-7381 URL: https://issues.apache.org/jira/browse/YARN-7381 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.9.0, 3.0.0, 3.1.0 Reporter: Xuan Gong Assignee: Xuan Gong Enable the configuration "yarn.nodemanager.log-container-debug-info.enabled", so we can aggregate launch_container.sh and directory.info -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7304) Merge YARN-5734 branch to branch-3.0
[ https://issues.apache.org/jira/browse/YARN-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-7304. - Resolution: Fixed > Merge YARN-5734 branch to branch-3.0 > > > Key: YARN-7304 > URL: https://issues.apache.org/jira/browse/YARN-7304 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7303) Merge YARN-5734 branch to trunk branch
[ https://issues.apache.org/jira/browse/YARN-7303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-7303. - Resolution: Fixed > Merge YARN-5734 branch to trunk branch > -- > > Key: YARN-7303 > URL: https://issues.apache.org/jira/browse/YARN-7303 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7305) Merge YARN-5734 branch to branch-2
[ https://issues.apache.org/jira/browse/YARN-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-7305. - Resolution: Fixed Fix Version/s: 2.9.0 > Merge YARN-5734 branch to branch-2 > -- > > Key: YARN-7305 > URL: https://issues.apache.org/jira/browse/YARN-7305 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7305) Merge YARN-5734 branch to branch-2
Xuan Gong created YARN-7305: --- Summary: Merge YARN-5734 branch to branch-2 Key: YARN-7305 URL: https://issues.apache.org/jira/browse/YARN-7305 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7304) Merge YARN-5734 branch to branch-3.0
Xuan Gong created YARN-7304: --- Summary: Merge YARN-5734 branch to branch-3.0 Key: YARN-7304 URL: https://issues.apache.org/jira/browse/YARN-7304 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7303) Merge YARN-5734 branch to trunk branch
Xuan Gong created YARN-7303: --- Summary: Merge YARN-5734 branch to trunk branch Key: YARN-7303 URL: https://issues.apache.org/jira/browse/YARN-7303 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7268) testCompareXmlAgainstConfigurationClass fails due to 1 missing property from yarn-default
[
https://issues.apache.org/jira/browse/YARN-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong resolved YARN-7268.
-
Resolution: Invalid
> testCompareXmlAgainstConfigurationClass fails due to 1 missing property from
> yarn-default
> -
>
> Key: YARN-7268
> URL: https://issues.apache.org/jira/browse/YARN-7268
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Yesha Vora
>
> {code}
> Error Message
> yarn-default.xml has 1 properties missing in class
> org.apache.hadoop.yarn.conf.YarnConfiguration
> Stacktrace
> java.lang.AssertionError: yarn-default.xml has 1 properties missing in class
> org.apache.hadoop.yarn.conf.YarnConfiguration
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.conf.TestConfigurationFieldsBase.testCompareXmlAgainstConfigurationClass(TestConfigurationFieldsBase.java:414)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Standard Output
> File yarn-default.xml (253 properties)
> yarn-default.xml has 1 properties missing in class
> org.apache.hadoop.yarn.conf.YarnConfiguration
> yarn.log-aggregation.file-controller.TFile.class
> ={code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7195) Document Log CLI in YarnCommands.md
Xuan Gong created YARN-7195: --- Summary: Document Log CLI in YarnCommands.md Key: YARN-7195 URL: https://issues.apache.org/jira/browse/YARN-7195 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7194) Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat
Xuan Gong created YARN-7194: --- Summary: Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat Key: YARN-7194 URL: https://issues.apache.org/jira/browse/YARN-7194 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7174) Add retry logic in LogsCLI when fetch running application logs
Xuan Gong created YARN-7174: --- Summary: Add retry logic in LogsCLI when fetch running application logs Key: YARN-7174 URL: https://issues.apache.org/jira/browse/YARN-7174 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7148) TestLogsCLI fails in trunk and branch-2
Xuan Gong created YARN-7148: --- Summary: TestLogsCLI fails in trunk and branch-2 Key: YARN-7148 URL: https://issues.apache.org/jira/browse/YARN-7148 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong The testcase failures(TestLogsCLI) should be related to YARN-6877 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7144) Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader.
Xuan Gong created YARN-7144: --- Summary: Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader. Key: YARN-7144 URL: https://issues.apache.org/jira/browse/YARN-7144 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7089) Mark the log-aggregation-controller APIs as public
Xuan Gong created YARN-7089: --- Summary: Mark the log-aggregation-controller APIs as public Key: YARN-7089 URL: https://issues.apache.org/jira/browse/YARN-7089 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-7072) Add a new log aggregation file format controller
Xuan Gong created YARN-7072: --- Summary: Add a new log aggregation file format controller Key: YARN-7072 URL: https://issues.apache.org/jira/browse/YARN-7072 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6922) Findbugs warning in YARN NodeManager
Xuan Gong created YARN-6922:
---
Summary: Findbugs warning in YARN NodeManager
Key: YARN-6922
URL: https://issues.apache.org/jira/browse/YARN-6922
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Xuan Gong
Several findbugs warning in YARN NodeManager package.
{code}
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
is a mutable collection which should be package protected
Bug type MS_MUTABLE_COLLECTION_PKGPROTECT (click for details)
In class
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics
Field
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainerMetrics.usageMetrics
At ContainerMetrics.java:[line 134]
{code}
{code}
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
makes inefficient use of keySet iterator instead of entrySet iterator
Bug type WMI_WRONG_MAP_ITERATOR (click for details)
In class
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer
In method
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.createStatus()
Field
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.pendingResources
At ContainerLocalizer.java:[line 357]
{code}
{code}
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
makes inefficient use of keySet iterator instead of entrySet iterator
Bug type WMI_WRONG_MAP_ITERATOR (click for details)
In class org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl
In method
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache()
Field
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.recentlyStoppedContainers
At NodeStatusUpdaterImpl.java:[line 719]
{code}
{code}
Hard coded reference to an absolute pathname in
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
Bug type DMI_HARDCODED_ABSOLUTE_FILENAME (click for details)
In class
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime
In method
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(ContainerRuntimeContext)
File name /sys/fs/cgroup
At DockerLinuxContainerRuntime.java:[line 490]
{code}
{code}
Useless object stored in variable removedNullContainers of method
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
Bug type UC_USELESS_OBJECT (click for details)
In class org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl
In method
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
Value removedNullContainers
Type java.util.HashSet
At NodeStatusUpdaterImpl.java:[line 642]
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6877) Create an abstract log reader for extendability
Xuan Gong created YARN-6877: --- Summary: Create an abstract log reader for extendability Key: YARN-6877 URL: https://issues.apache.org/jira/browse/YARN-6877 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Currently, TFile log reader is used to read aggregated log in YARN. We need to add an abstract layer, and pick up the correct log reader based on the configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6876) Create an abstract log writer for extendability
Xuan Gong created YARN-6876: --- Summary: Create an abstract log writer for extendability Key: YARN-6876 URL: https://issues.apache.org/jira/browse/YARN-6876 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Currently, TFile log writer is used to aggregated log in YARN. We need to add an abstract layer, and pick up the correct log writer based on the configuration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6875) New aggregated log file format for YARN log aggregation.
Xuan Gong created YARN-6875: --- Summary: New aggregated log file format for YARN log aggregation. Key: YARN-6875 URL: https://issues.apache.org/jira/browse/YARN-6875 Project: Hadoop YARN Issue Type: New Feature Reporter: Xuan Gong Assignee: Xuan Gong T-file is the underlying log format for the aggregated logs in YARN. We have seen several performance issues, especially for very large log files. We will introduce a new log format which have better performance for large log files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6509) Add a size threshold beyond which yarn logs will require a force option
Xuan Gong created YARN-6509: --- Summary: Add a size threshold beyond which yarn logs will require a force option Key: YARN-6509 URL: https://issues.apache.org/jira/browse/YARN-6509 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Xuan Gong An accidental fetch for a long running application can lead to scenario which the large size of log can fill up a disk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6440) re-visit handling QUEUE-STOP process
Xuan Gong created YARN-6440: --- Summary: re-visit handling QUEUE-STOP process Key: YARN-6440 URL: https://issues.apache.org/jira/browse/YARN-6440 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6431) make DELETE/STOP/CONVERT queues work in reservation system
Xuan Gong created YARN-6431: --- Summary: make DELETE/STOP/CONVERT queues work in reservation system Key: YARN-6431 URL: https://issues.apache.org/jira/browse/YARN-6431 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Xuan Gong Assignee: Xuan Gong Previous, we have made some enhancements on DELETE/STOP/CONVERT queues. We need to make sure that those enhancements work for reservation system as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6367) YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService
Xuan Gong created YARN-6367: --- Summary: YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService Key: YARN-6367 URL: https://issues.apache.org/jira/browse/YARN-6367 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6314) Potential infinite redirection on YARN log redirection web service
Xuan Gong created YARN-6314: --- Summary: Potential infinite redirection on YARN log redirection web service Key: YARN-6314 URL: https://issues.apache.org/jira/browse/YARN-6314 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong In YARN-6113, we have added a re-direct NM web service to get container logs which could cause the potential infinite redirection. It can happens when: * Call AHS web service to get a running/finished AM container log for a running application. * AHS web service would re-direct the request the specific NM given the application is still running. And the NM would handle the request * If the log file we requested has already been aggregated and deleted from NM, the NM would re-direct the request back to AHS. In this case, we would do step 2 and step 3 infinite times. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6313) yarn logs cli does not provide logs for a completed container even when the nm address is provided
Xuan Gong created YARN-6313: --- Summary: yarn logs cli does not provide logs for a completed container even when the nm address is provided Key: YARN-6313 URL: https://issues.apache.org/jira/browse/YARN-6313 Project: Hadoop YARN Issue Type: Sub-task Reporter: Siddharth Seth Assignee: Xuan Gong Running app. Completed container. Provide the appId, containerId, nodeId - yarn logs does not return the logs. Specific use case: Long Running app. One daemon crashed. Logs are not accessible without shutting down the app. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6270) WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting
Xuan Gong created YARN-6270: --- Summary: WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting Key: YARN-6270 URL: https://issues.apache.org/jira/browse/YARN-6270 Project: Hadoop YARN Issue Type: Bug Reporter: Sumana Sathish Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6220) Few TestSecureRMRegistryOperations UT fails
[
https://issues.apache.org/jira/browse/YARN-6220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong resolved YARN-6220.
-
Resolution: Cannot Reproduce
> Few TestSecureRMRegistryOperations UT fails
> ---
>
> Key: YARN-6220
> URL: https://issues.apache.org/jira/browse/YARN-6220
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Xuan Gong
> Attachments: testAlicePathRestrictedAnonAccess.log,
> testAnonNoWriteAccess.log, testAnonNoWriteAccessOffRoot.log,
> testAnonReadAccess.log, testDigestAccess.log,
> testUserHomedirsPermissionsRestricted.log,
> testUserZookeeperHomePathAccess.log, testZookeeperCanWriteUnderSystem.log
>
>
> 8 Tests from TestSecureRMRegistryOperations fails as below.
> * testAlicePathRestrictedAnonAccess
> * testAnonNoWriteAccess
> * testAnonNoWriteAccessOffRoot
> * testAnonReadAccess
> * testDigestAccess
> * testUserHomedirsPermissionsRestricted
> * testUserZookeeperHomePathAccess
> * testZookeeperCanWriteUnderSystem
> {code}
> java.lang.reflect.UndeclaredThrowableException: null
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:262)
> at java.util.concurrent.FutureTask.get(FutureTask.java:119)
> at
> org.apache.hadoop.registry.secure.TestSecureRMRegistryOperations$1.run(TestSecureRMRegistryOperations.java:107)
> at
> org.apache.hadoop.registry.secure.TestSecureRMRegistryOperations$1.run(TestSecureRMRegistryOperations.java:98)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hadoop.registry.secure.TestSecureRMRegistryOperations.startRMRegistryOperations(TestSecureRMRegistryOperations.java:97)
> at
> org.apache.hadoop.registry.secure.TestSecureRMRegistryOperations.testAnonNoWriteAccess(TestSecureRMRegistryOperations.java:148){code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6126) Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:"
Xuan Gong created YARN-6126: --- Summary: Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:" Key: YARN-6126 URL: https://issues.apache.org/jira/browse/YARN-6126 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6122) Add a service to fetch a given list of log files, to a single archive
Xuan Gong created YARN-6122: --- Summary: Add a service to fetch a given list of log files, to a single archive Key: YARN-6122 URL: https://issues.apache.org/jira/browse/YARN-6122 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6113) re-direct NM Web Service to get container logs for finished applications
Xuan Gong created YARN-6113: --- Summary: re-direct NM Web Service to get container logs for finished applications Key: YARN-6113 URL: https://issues.apache.org/jira/browse/YARN-6113 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong In NM web ui, when we try to get container logs for finished application, it would redirect to the log server based on the configuration: yarn.log.server.url. We should do the similar thing for NM WebService -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6109) Add an ability to convert ChildQueue to ParentQueue
Xuan Gong created YARN-6109: --- Summary: Add an ability to convert ChildQueue to ParentQueue Key: YARN-6109 URL: https://issues.apache.org/jira/browse/YARN-6109 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6108) Improve AHS webservice to accept NM address as a parameter to get container logs
Xuan Gong created YARN-6108: --- Summary: Improve AHS webservice to accept NM address as a parameter to get container logs Key: YARN-6108 URL: https://issues.apache.org/jira/browse/YARN-6108 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Currently, if we want to get container log for running application, we need to get NM web address from AHS which we need to enable yarn.timeline-service.generic-application-history.save-non-am-container-meta-info for non-am containers. But, in most of time, we will disable this configuration for ATS performance purpose. In this case, it is impossible for us to get the logs for non-am container in a running application. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6100) improve YARN webservice to output aggregated container logs
Xuan Gong created YARN-6100: --- Summary: improve YARN webservice to output aggregated container logs Key: YARN-6100 URL: https://issues.apache.org/jira/browse/YARN-6100 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6099) Improve webservice to list aggregated log files
Xuan Gong created YARN-6099: --- Summary: Improve webservice to list aggregated log files Key: YARN-6099 URL: https://issues.apache.org/jira/browse/YARN-6099 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6080) Create WebServiceUtils to have common functions used in RMWebService, NMWebService and AHSWebService
Xuan Gong created YARN-6080: --- Summary: Create WebServiceUtils to have common functions used in RMWebService, NMWebService and AHSWebService Key: YARN-6080 URL: https://issues.apache.org/jira/browse/YARN-6080 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Create WebServiceUtils to remove the duplicate code. Also, provide the pattern to create webService call which could be used by client. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5755) Enhancements to STOP queue handling
[ https://issues.apache.org/jira/browse/YARN-5755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-5755. - Resolution: Duplicate Fix Version/s: 3.0.0-alpha2 2.9.0 > Enhancements to STOP queue handling > --- > > Key: YARN-5755 > URL: https://issues.apache.org/jira/browse/YARN-5755 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0, 3.0.0-alpha2 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-6011) Add a new web service to list the files on a container in AHSWebService
Xuan Gong created YARN-6011: --- Summary: Add a new web service to list the files on a container in AHSWebService Key: YARN-6011 URL: https://issues.apache.org/jira/browse/YARN-6011 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5923) Unable to access logs for a running application if YARN_ACL_ENABLE is enabled
Xuan Gong created YARN-5923: --- Summary: Unable to access logs for a running application if YARN_ACL_ENABLE is enabled Key: YARN-5923 URL: https://issues.apache.org/jira/browse/YARN-5923 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5761) Separate QueueManager from Scheduler
Xuan Gong created YARN-5761: --- Summary: Separate QueueManager from Scheduler Key: YARN-5761 URL: https://issues.apache.org/jira/browse/YARN-5761 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Currently, in scheduler code, we are doing queue manager and scheduling work. We'd better separate the queue manager out of scheduler logic. In that case, it would be much easier and safer to extend. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5756) Add state-machine implementation for queues
Xuan Gong created YARN-5756: --- Summary: Add state-machine implementation for queues Key: YARN-5756 URL: https://issues.apache.org/jira/browse/YARN-5756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5755) Enhancements to STOP queue handling
Xuan Gong created YARN-5755: --- Summary: Enhancements to STOP queue handling Key: YARN-5755 URL: https://issues.apache.org/jira/browse/YARN-5755 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5746) The state of the parentQueue and its childQueues should be synchronized.
Xuan Gong created YARN-5746: --- Summary: The state of the parentQueue and its childQueues should be synchronized. Key: YARN-5746 URL: https://issues.apache.org/jira/browse/YARN-5746 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Xuan Gong Assignee: Xuan Gong The state of the parentQueue and its childQeues need to be synchronized. * If the state of the parentQueue becomes STOPPED, the state of its childQueue need to become STOPPED as well. * If we change the state of the queue to RUNNING, we should make sure the state of all its ancestor must be RUNNING. Otherwise, we need to fail this operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5724) [Umbrella] Better Queue Management in YARN
Xuan Gong created YARN-5724: --- Summary: [Umbrella] Better Queue Management in YARN Key: YARN-5724 URL: https://issues.apache.org/jira/browse/YARN-5724 Project: Hadoop YARN Issue Type: Task Reporter: Xuan Gong Assignee: Xuan Gong This serves as an umbrella ticket for tasks related to better queue management in YARN. Today's the only way to manage the queue is through admins editing configuration files and then issuing a refresh command. This will bring many inconveniences. For example, the users can not create / delete /modify their own queues without talking to site level admins. Even in today's approach (configuration-based), we still have several places needed to improve: * It is possible today to add or modify queues without restarting the RM, via a CS refresh. But for deleting queue, we have to restart the resourcemanager. * When a queue is STOPPED, resources allocated to the queue can be handled better. Currently, they'll only be used if the other queues are setup to go over their capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5556) Support for deleting queues without requiring a RM restart
Xuan Gong created YARN-5556: --- Summary: Support for deleting queues without requiring a RM restart Key: YARN-5556 URL: https://issues.apache.org/jira/browse/YARN-5556 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Xuan Gong Assignee: Xuan Gong Today, we could add or modify queues without restarting the RM, via a CS refresh. But for deleting queue, we have to restart the ResourceManager. We could support for deleting queues without requiring a RM restart -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
Xuan Gong created YARN-5470: --- Summary: Differentiate exactly match with regex in yarn log CLI Key: YARN-5470 URL: https://issues.apache.org/jira/browse/YARN-5470 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5442) TestYarnClient fails in trunk
Xuan Gong created YARN-5442: --- Summary: TestYarnClient fails in trunk Key: YARN-5442 URL: https://issues.apache.org/jira/browse/YARN-5442 Project: Hadoop YARN Issue Type: Test Reporter: Xuan Gong testReservationDelete(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 2.218 sec <<< FAILURE! java.lang.AssertionError: Exhausted attempts in checking if node capacity was added to the plan at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationDelete(TestYarnClient.java:1584) testUpdateReservation(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 2.181 sec <<< FAILURE! java.lang.AssertionError: Exhausted attempts in checking if node capacity was added to the plan at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testUpdateReservation(TestYarnClient.java:1300) testListReservationsByTimeIntervalContainingNoReservations(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 2.257 sec <<< FAILURE! java.lang.AssertionError: Exhausted attempts in checking if node capacity was added to the plan at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testListReservationsByTimeIntervalContainingNoReservations(TestYarnClient.java:1494) testCreateReservation(org.apache.hadoop.yarn.client.api.impl.TestYarnClient) Time elapsed: 2.29 sec <<< FAILURE! java.lang.AssertionError: Exhausted attempts in checking if node capacity was added to the plan at org.junit.Assert.fail(Assert.java:88) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.setupMiniYARNCluster(TestYarnClient.java:1222) at org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testCreateReservation(TestYarnClient.java:1257) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5440) Use AHSClient in YarnClient when TimelineServer is running
Xuan Gong created YARN-5440: --- Summary: Use AHSClient in YarnClient when TimelineServer is running Key: YARN-5440 URL: https://issues.apache.org/jira/browse/YARN-5440 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong In YarnClient, depends on whether we enable yarn.timeline-service.generic-application-history.enabled, the AHSClient will not be used. But the AHSClientService is enabled by default when we start the TimelineServer which means we are able to get history information for applications/applicationAttempts/containers by using ahsClient when the TimelineServer is running. So, we do not have to reply on this deprecated configuration to get history information by using YarnClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5369) Improve Yarn logs command to get container logs based on Node Id
Xuan Gong created YARN-5369:
---
Summary: Improve Yarn logs command to get container logs based on
Node Id
Key: YARN-5369
URL: https://issues.apache.org/jira/browse/YARN-5369
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
Assignee: Xuan Gong
It is helpful if we could have yarn logs --applicationId appId --nodeAddress
${nodeId} to get all the container logs which ran on the specific nm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5313) TestLogsCLI.testFetchApplictionLogsAsAnotherUser fails in trunk
Xuan Gong created YARN-5313: --- Summary: TestLogsCLI.testFetchApplictionLogsAsAnotherUser fails in trunk Key: YARN-5313 URL: https://issues.apache.org/jira/browse/YARN-5313 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong We have reverted HADOOP-12718 recently which caused this failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5251) Yarn CLI to obtain App logs for last 'n' bytes fails with 'java.io.IOException' and for 'n' bytes fails with NumberFormatException
Xuan Gong created YARN-5251:
---
Summary: Yarn CLI to obtain App logs for last 'n' bytes fails with
'java.io.IOException' and for 'n' bytes fails with NumberFormatException
Key: YARN-5251
URL: https://issues.apache.org/jira/browse/YARN-5251
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Sumana Sathish
Assignee: Xuan Gong
{code}
yarn logs -applicationId application_1465421211793_0017 -size 1024 >> appLog1
on finished application
2016-06-13 18:44:25,989 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #2 tokens
and #1 secret keys for NM use for launching container
2016-06-13 18:44:25,989 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of
containertok"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at
org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.readContainerLogs(AggregatedLogFormat.java:691)
at
org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.readAContainerLogsForALogType(AggregatedLogFormat.java:767)
at
org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLogs(LogCLIHelpers.java:354)
at
org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:830)
at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:231)
at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:264)
{code}
{code}
yarn logs -applicationId application_1465421211793_0004 -containerId
container_e07_1465421211793_0004_01_01 -logFiles syslog -size -1000
Exception in thread "main" java.io.IOException: The bytes were skipped are
different from the caller requested
at
org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.readContainerLogsForALogType(AggregatedLogFormat.java:838)
at
org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAContainerLogsForALogType(LogCLIHelpers.java:300)
at
org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAContainersLogsForALogTypeWithoutNodeId(LogCLIHelpers.java:224)
at
org.apache.hadoop.yarn.client.cli.LogsCLI.printContainerLogsForFinishedApplicationWithoutNodeId(LogsCLI.java:447)
at
org.apache.hadoop.yarn.client.cli.LogsCLI.fetchContainerLogs(LogsCLI.java:782)
at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:228)
at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:264)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5249) app logs with 'n' bytes via CLI fails with NumberFormatException
Xuan Gong created YARN-5249:
---
Summary: app logs with 'n' bytes via CLI fails with
NumberFormatException
Key: YARN-5249
URL: https://issues.apache.org/jira/browse/YARN-5249
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.9.0
Reporter: Sumana Sathish
Assignee: Xuan Gong
app logs with 'n' bytes via CLI fails with NumberFormatException for finished
appliction
{code}
yarn logs -applicationId application_1465421211793_0017 -size 1024 >> appLog1
on finished application
2016-06-13 18:44:25,989 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #2 tokens
and #1 secret keys for NM use for launching container
2016-06-13 18:44:25,989 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of
containertok"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:589)
at java.lang.Long.parseLong(Long.java:631)
at
org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.readContainerLogs(AggregatedLogFormat.java:691)
at
org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.readAContainerLogsForALogType(AggregatedLogFormat.java:767)
at
org.apache.hadoop.yarn.logaggregation.LogCLIHelpers.dumpAllContainersLogs(LogCLIHelpers.java:354)
at
org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:830)
at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:231)
at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:264)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5248) TestLogsCLI#testFetchApplictionLogs fails in trunk/branch-2
Xuan Gong created YARN-5248: --- Summary: TestLogsCLI#testFetchApplictionLogs fails in trunk/branch-2 Key: YARN-5248 URL: https://issues.apache.org/jira/browse/YARN-5248 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5237) Not all logs get aggregated with rolling log aggregation.
Xuan Gong created YARN-5237: --- Summary: Not all logs get aggregated with rolling log aggregation. Key: YARN-5237 URL: https://issues.apache.org/jira/browse/YARN-5237 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Steps to reproduce: 1) enable RM recovery 2) Run a sleep job 3) restart RM 4) kill the application We can not find that the logs for the first attempt -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5226) remove AHS enable check from LogsCLI#fetchAMContainerLogs
Xuan Gong created YARN-5226: --- Summary: remove AHS enable check from LogsCLI#fetchAMContainerLogs Key: YARN-5226 URL: https://issues.apache.org/jira/browse/YARN-5226 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5223) Container line in yarn logs output for a live application does not include the hostname
Xuan Gong created YARN-5223: --- Summary: Container line in yarn logs output for a live application does not include the hostname Key: YARN-5223 URL: https://issues.apache.org/jira/browse/YARN-5223 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5217) Close FileInputStream in NMWebServices#getLogs in branch-2.8
Xuan Gong created YARN-5217: --- Summary: Close FileInputStream in NMWebServices#getLogs in branch-2.8 Key: YARN-5217 URL: https://issues.apache.org/jira/browse/YARN-5217 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong In https://issues.apache.org/jira/browse/YARN-5199, we close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs. We should do the same thing in branch-2.8. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5200) Improve yarn logs to get Container List
Xuan Gong created YARN-5200: --- Summary: Improve yarn logs to get Container List Key: YARN-5200 URL: https://issues.apache.org/jira/browse/YARN-5200 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5199) Close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs
Xuan Gong created YARN-5199: --- Summary: Close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs Key: YARN-5199 URL: https://issues.apache.org/jira/browse/YARN-5199 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5191) Rename the “download=true” option for getLogs in NMWebServices and AHSWebServices
Xuan Gong created YARN-5191: --- Summary: Rename the “download=true” option for getLogs in NMWebServices and AHSWebServices Key: YARN-5191 URL: https://issues.apache.org/jira/browse/YARN-5191 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Rename the “download=true” option to instead be something like “format=octet-stream”, so that we are explicit -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5141) Get Container logs for the Running application from Yarn Logs CommandLine
Xuan Gong created YARN-5141: --- Summary: Get Container logs for the Running application from Yarn Logs CommandLine Key: YARN-5141 URL: https://issues.apache.org/jira/browse/YARN-5141 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Currently, we can only get container logs for Finished applications -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5115) Security risk by using CONTENT-DISPOSITION header
Xuan Gong created YARN-5115:
---
Summary: Security risk by using CONTENT-DISPOSITION header
Key: YARN-5115
URL: https://issues.apache.org/jira/browse/YARN-5115
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
In NMWebService/AHSWebservice, we have used CONTENT-DISPOSITION header for
show/download container logs. Looks like it has security risks. And people have
devised content-disposition hacking.
The HTTP 1.1 Standard (RFC 2616) also mentions the possible security side
effects of content disposition:
{code}
15.5 Content-Disposition Issues
RFC 1806 [35], from which the often implemented Content-Disposition
(see section 19.5.1) header in HTTP is derived, has a number of very
serious security considerations. Content-Disposition is not part of
the HTTP standard, but since it is widely implemented, we are
documenting its use and risks for implementors. See RFC 2183 [49]
(which updates RFC 1806) for details.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4913) Yarn logs should take a -out option to write to a directory
[ https://issues.apache.org/jira/browse/YARN-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-4913. - Resolution: Fixed Fix Version/s: 2.9.0 > Yarn logs should take a -out option to write to a directory > --- > > Key: YARN-4913 > URL: https://issues.apache.org/jira/browse/YARN-4913 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-4913.1.patch, YARN-4913.2.patch, YARN-4913.3.patch, > YARN-4913.4.patch, YARN-4913.5.1.patch, YARN-4913.5.patch, YARN-4913.6.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5108) TestFileSystemApplicationHistoryStore fails
Xuan Gong created YARN-5108: --- Summary: TestFileSystemApplicationHistoryStore fails Key: YARN-5108 URL: https://issues.apache.org/jira/browse/YARN-5108 Project: Hadoop YARN Issue Type: Test Reporter: Xuan Gong Running org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore Tests run: 8, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 20.188 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore) Time elapsed: 0.054 sec <<< ERROR! java.lang.NullPointerException: null at java.util.TreeMap.getEntry(TreeMap.java:347) at java.util.TreeMap.get(TreeMap.java:278) at org.apache.hadoop.fs.GlobalStorageStatistics.put(GlobalStorageStatistics.java:73) at org.apache.hadoop.fs.FileSystem.getStatistics(FileSystem.java:3598) at org.apache.hadoop.fs.FileSystem.initialize(FileSystem.java:214) at org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:101) at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70) at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.setup(TestFileSystemApplicationHistoryStore.java:64) testMissingApplicationAttemptHistoryData(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore) Time elapsed: 0.054 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.tearDown(TestFileSystemApplicationHistoryStore.java:89) testInitNonExistingWorkingDirectoryInSafeMode(org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore) Time elapsed: 0.517 sec <<< FAILURE! org.mockito.exceptions.verification.WantedButNotInvoked: Wanted but not invoked: rawLocalFileSystem.isDirectory(); -> at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHistoryStore.java:304) However, there were other interactions with this mock: -> at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.initAndStartStore(TestFileSystemApplicationHistoryStore.java:70) at org.apache.hadoop.yarn.server.applicationhistoryservice.TestFileSystemApplicationHistoryStore.testInitNonExistingWorkingDirectoryInSafeMode(TestFileSystemApplicationHistoryStore.java:304) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5101) YARN_APPLICATION_UPDATED event is parsed in ApplicationHistoryManagerOnTimelineStore#convertToApplicationReport with reversed order
Xuan Gong created YARN-5101: --- Summary: YARN_APPLICATION_UPDATED event is parsed in ApplicationHistoryManagerOnTimelineStore#convertToApplicationReport with reversed order Key: YARN-5101 URL: https://issues.apache.org/jira/browse/YARN-5101 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.8.0 Reporter: Xuan Gong Right now, the application events are parsed in in ApplicationHistoryManagerOnTimelineStore#convertToApplicationReport with timestamp descending order, which means the later events would be parsed first, and the previous same type of events would override the information. In https://issues.apache.org/jira/browse/YARN-4044, we have introduced YARN_APPLICATION_UPDATED events which might be submitted by RM multiple times in one application life cycle. This could cause problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5100) The YarnApplicationState is always running in ATS no matter the application is running or finishes.
Xuan Gong created YARN-5100: --- Summary: The YarnApplicationState is always running in ATS no matter the application is running or finishes. Key: YARN-5100 URL: https://issues.apache.org/jira/browse/YARN-5100 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5089) Improve Yarn log Command line "logFiles" option to support Java regex
Xuan Gong created YARN-5089: --- Summary: Improve Yarn log Command line "logFiles" option to support Java regex Key: YARN-5089 URL: https://issues.apache.org/jira/browse/YARN-5089 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5088) Improve Yarn log Command line to read the last K bytes for the log files
Xuan Gong created YARN-5088: --- Summary: Improve Yarn log Command line to read the last K bytes for the log files Key: YARN-5088 URL: https://issues.apache.org/jira/browse/YARN-5088 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5081) Replace RPC calls with WebService calls in LogsCLI
Xuan Gong created YARN-5081: --- Summary: Replace RPC calls with WebService calls in LogsCLI Key: YARN-5081 URL: https://issues.apache.org/jira/browse/YARN-5081 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Currently in LogsCLI, we still use YarnClient to get ContainerReport. We expect the users to enable yarn.timeline-service.generic-application-history.enabled to get finished container report which is not ideal. We can replace all RPC calls with WebService call, so the users do not need to change their configuration (to enable generic-application-history). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-5029) RM needs to send update event with YarnApplicationState as Running to ATS/AHS
Xuan Gong created YARN-5029: --- Summary: RM needs to send update event with YarnApplicationState as Running to ATS/AHS Key: YARN-5029 URL: https://issues.apache.org/jira/browse/YARN-5029 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Right now, Application in AHS/ATS is alway in ACCEPTED state until the application finishes/Fails/is killed. This is because RM did not send any other YarnApplicationState information, except FINISHED/FAILED/KILLED, to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4913) Yarn logs should take a -out option to write to a directory
[ https://issues.apache.org/jira/browse/YARN-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong resolved YARN-4913. - Resolution: Won't Fix > Yarn logs should take a -out option to write to a directory > --- > > Key: YARN-4913 > URL: https://issues.apache.org/jira/browse/YARN-4913 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4913.1.patch, YARN-4913.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4993) Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication.
Xuan Gong created YARN-4993: --- Summary: Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication. Key: YARN-4993 URL: https://issues.apache.org/jira/browse/YARN-4993 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4966) More improvement to get Container logs without specify nodeId
Xuan Gong created YARN-4966: --- Summary: More improvement to get Container logs without specify nodeId Key: YARN-4966 URL: https://issues.apache.org/jira/browse/YARN-4966 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Currently, for the finished application, we can get the container logs without specify node id, but we need to enable yarn.timeline-service.generic-application-history.enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4956) findbug issue on LevelDBCacheTimelineStore
Xuan Gong created YARN-4956:
---
Summary: findbug issue on LevelDBCacheTimelineStore
Key: YARN-4956
URL: https://issues.apache.org/jira/browse/YARN-4956
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Xuan Gong
{code}
Multithreaded correctness Warnings
Code Warning IS Inconsistent synchronization of
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.configuration;
locked 66% of time
Bug type IS2_INCONSISTENT_SYNC (click for details)
In class org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore
Field
org.apache.hadoop.yarn.server.timeline.LevelDBCacheTimelineStore.configuration
Synchronized 66% of the time
Unsynchronized access at LevelDBCacheTimelineStore.java:[line 82]
Synchronized access at LevelDBCacheTimelineStore.java:[line 117]
Synchronized access at LevelDBCacheTimelineStore.java:[line 122]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-4955) Add retry for SocketTimeoutException in TimelineClient
Xuan Gong created YARN-4955: --- Summary: Add retry for SocketTimeoutException in TimelineClient Key: YARN-4955 URL: https://issues.apache.org/jira/browse/YARN-4955 Project: Hadoop YARN Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong We saw this exception several times when we tried to getDelegationToken from ATS. java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: java.net.SocketTimeoutException: Read timed out at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$5.run(TimelineClientImpl.java:569) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:234) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:582) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:479) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken(YarnClientImpl.java:349) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:330) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:250) at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:291) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:290) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128) at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:194) at java.lang.Thread.run(Thread.java:745) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276) Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: java.net.SocketTimeoutException: Read timed out at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332) at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128) at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:285) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:166) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:475) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:467) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$5.run(TimelineClientImpl.java:567) ... 24 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at
[jira] [Created] (YARN-4920) ATS/NM should support a link to dowload/get the logs in text format
Xuan Gong created YARN-4920: --- Summary: ATS/NM should support a link to dowload/get the logs in text format Key: YARN-4920 URL: https://issues.apache.org/jira/browse/YARN-4920 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4919) Yarn logs should support a option to output logs as compressed archive
Xuan Gong created YARN-4919: --- Summary: Yarn logs should support a option to output logs as compressed archive Key: YARN-4919 URL: https://issues.apache.org/jira/browse/YARN-4919 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4913) Yarn logs should take a -out option to write to a directory
Xuan Gong created YARN-4913: --- Summary: Yarn logs should take a -out option to write to a directory Key: YARN-4913 URL: https://issues.apache.org/jira/browse/YARN-4913 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4905) Improve Yarn log Command line option to show log metadata
Xuan Gong created YARN-4905: --- Summary: Improve Yarn log Command line option to show log metadata Key: YARN-4905 URL: https://issues.apache.org/jira/browse/YARN-4905 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong Improve the Yarn log commandline to have "ls" command which can list containers for which we have logs, list files within each container, along with file size -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4904) YARN Log tooling enhancement
Xuan Gong created YARN-4904: --- Summary: YARN Log tooling enhancement Key: YARN-4904 URL: https://issues.apache.org/jira/browse/YARN-4904 Project: Hadoop YARN Issue Type: Task Reporter: Xuan Gong Assignee: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4817) Change Log Level to DEBUG for putDomain call in ATS 1.5
Xuan Gong created YARN-4817: --- Summary: Change Log Level to DEBUG for putDomain call in ATS 1.5 Key: YARN-4817 URL: https://issues.apache.org/jira/browse/YARN-4817 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong We have already changed the log level to DEBUG for putEntity call. Let us make it consistence for the putDomain call -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4815) ATS 1.5 timelineclinet impl try to create attempt directory for every event call
Xuan Gong created YARN-4815: --- Summary: ATS 1.5 timelineclinet impl try to create attempt directory for every event call Key: YARN-4815 URL: https://issues.apache.org/jira/browse/YARN-4815 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong ATS 1.5 timelineclinet impl, try to create attempt directory for every event call. Since per attempt only one call to create directory is enough, this is causing perf issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4814) ATS 1.5 timelineclient impl call flush after every event write
Xuan Gong created YARN-4814: --- Summary: ATS 1.5 timelineclient impl call flush after every event write Key: YARN-4814 URL: https://issues.apache.org/jira/browse/YARN-4814 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong ATS 1.5 timelineclient impl call flush after every event write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4680) TimerTasks leak in ATS V1.5 Writer
Xuan Gong created YARN-4680: --- Summary: TimerTasks leak in ATS V1.5 Writer Key: YARN-4680 URL: https://issues.apache.org/jira/browse/YARN-4680 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong Assignee: Xuan Gong We have seen TimerTasks leak which could cause application server done (such as oozie server done due to too many active threads) Although we have fixed some potentially leak situations in upper application level, such as https://issues.apache.org/jira/browse/MAPREDUCE-6618 https://issues.apache.org/jira/browse/MAPREDUCE-6621, we still can not guarantee that we fixed the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
