[jira] [Moved] (YARN-3157) Improvement for ConverterUtils
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt moved HADOOP-11562 to YARN-3157:
-
Affects Version/s: (was: 2.6.0)
2.6.0
Key: YARN-3157 (was: HADOOP-11562)
Project: Hadoop YARN (was: Hadoop Common)
Improvement for ConverterUtils
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: HADOOP-11562.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3157) Improvement for ConverterUtils
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3157:
---
Attachment: YARN-3157.patch
Updating patch with new JIRA id
Improvement for ConverterUtils
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: HADOOP-11562.patch, YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3157) Improvement for ConverterUtils
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3157:
---
Attachment: (was: HADOOP-11562.patch)
Improvement for ConverterUtils
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3157) Improvement for ConverterUtils
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3157:
---
Issue Type: Bug (was: Improvement)
Improvement for ConverterUtils
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3127) Apphistory url crashes when RM switches with ATS enabled
[
https://issues.apache.org/jira/browse/YARN-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307589#comment-14307589
]
Bibin A Chundatt commented on YARN-3127:
Thanks a lot Vinod and Naga for looking into the issue.
Vinod do you suggest the below part also should be gracefully handled (attempt
details not available) and not publish event to ATS during recovery?
{quote}
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore
public ApplicationAttemptReport getApplicationAttempt(
ApplicationAttemptId appAttemptId) throws YarnException, IOException {
ApplicationReportExt app = getApplication(
appAttemptId.getApplicationId(), ApplicationReportField.USER_AND_ACLS);
checkAccess(app);
TimelineEntity entity = timelineDataManager.getEntity(
AppAttemptMetricsConstants.ENTITY_TYPE,
appAttemptId.toString(), EnumSet.allOf(Field.class),
UserGroupInformation.getLoginUser());
if (entity == null) {
throw new ApplicationAttemptNotFoundException(
The entity for application attempt + appAttemptId +
doesn't exist in the timeline store);
} else {
return convertToApplicationAttemptReport(entity);
}
}
{quote}
Please do correct me if i am wrong.
Apphistory url crashes when RM switches with ATS enabled
Key: YARN-3127
URL: https://issues.apache.org/jira/browse/YARN-3127
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager, timelineserver
Affects Versions: 2.6.0
Environment: RM HA with ATS
Reporter: Bibin A Chundatt
Assignee: Naganarasimha G R
1.Start RM with HA and ATS configured and run some yarn applications
2.Once applications are finished sucessfully start timeline server
3.Now failover HA form active to standby
4.Access timeline server URL IP:PORT/applicationhistory
Result: Application history URL fails with below info
{quote}
2015-02-03 20:28:09,511 ERROR org.apache.hadoop.yarn.webapp.View: Failed to
read the applications.
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1643)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:80)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:67)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
...
Caused by:
org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: The
entity for application attempt appattempt_1422972608379_0001_01 doesn't
exist in the timeline store
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplicationAttempt(ApplicationHistoryManagerOnTimelineStore.java:151)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.generateApplicationReport(ApplicationHistoryManagerOnTimelineStore.java:499)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getAllApplications(ApplicationHistoryManagerOnTimelineStore.java:108)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:84)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:81)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
... 51 more
2015-02-03 20:28:09,512 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI: /applicationhistory
org.apache.hadoop.yarn.webapp.WebAppException: Error rendering block:
nestLevel=6 expected 5
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
{quote}
Behaviour with AHS with file based history store
-Apphistory url is working
-No attempt entries are shown for each application.
Based on inital analysis when RM switches ,application attempts from state
store are not replayed but only applications are.
So when /applicaitonhistory url is accessed it tries for all attempt id and
fails
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3149) Typo in message for invalid application id
[ https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308435#comment-14308435 ] Bibin A Chundatt commented on YARN-3149: Thank you Tsuyoshi for resubmitting the patch and Xuang for commiting. Result was strange. Typo in message for invalid application id -- Key: YARN-3149 URL: https://issues.apache.org/jira/browse/YARN-3149 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Priority: Trivial Fix For: 2.7.0 Attachments: YARN-3149.patch, YARN-3149.patch, screenshot-1.png Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3151) On Failover tracking url wrong in application cli for KILLED application
Bibin A Chundatt created YARN-3151:
--
Summary: On Failover tracking url wrong in application cli for
KILLED application
Key: YARN-3151
URL: https://issues.apache.org/jira/browse/YARN-3151
Project: Hadoop YARN
Issue Type: Bug
Components: client, resourcemanager
Affects Versions: 2.6.0
Environment: 2 RM HA
Reporter: Bibin A Chundatt
Priority: Minor
Run an application and kill the same after starting
Check {color:red} ./yarn application -list -appStates KILLED {color}
(empty line)
{quote}
Application-Id Tracking-URL
application_1423219262738_0001
http://IP:PORT/cluster/app/application_1423219262738_0001
{quote}
Shutdown the active RM1
Check the same command {color:red} ./yarn application -list -appStates KILLED
{color} after RM2 is active
{quote}
Application-Id Tracking-URL
application_1423219262738_0001 null
{quote}
Tracking url for application is shown as null
Expected : Same url before failover should be shown
ApplicationReport .getOriginalTrackingUrl() is null after failover
org.apache.hadoop.yarn.client.cli.ApplicationCLI
listApplications(SetString appTypes,
EnumSetYarnApplicationState appStates)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3127) Apphistory url crashes when RM switches with ATS enabled
Bibin A Chundatt created YARN-3127:
--
Summary: Apphistory url crashes when RM switches with ATS enabled
Key: YARN-3127
URL: https://issues.apache.org/jira/browse/YARN-3127
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager, timelineserver
Affects Versions: 2.6.0
Environment: RM HA with ATS
Reporter: Bibin A Chundatt
1.Start RM with HA and ATS configured and run some yarn applications
2.Once applications are finished sucessfully start timeline server
3.Now failover HA form active to standby
4.Access timeline server URL IP:PORT/applicationhistory
Result: Application history URL fails with below info
{quote}
2015-02-03 20:28:09,511 ERROR org.apache.hadoop.yarn.webapp.View: Failed to
read the applications.
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1643)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:80)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:67)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
...
Caused by:
org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: The
entity for application attempt appattempt_1422972608379_0001_01 doesn't
exist in the timeline store
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplicationAttempt(ApplicationHistoryManagerOnTimelineStore.java:151)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.generateApplicationReport(ApplicationHistoryManagerOnTimelineStore.java:499)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getAllApplications(ApplicationHistoryManagerOnTimelineStore.java:108)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:84)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:81)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
... 51 more
2015-02-03 20:28:09,512 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI: /applicationhistory
org.apache.hadoop.yarn.webapp.WebAppException: Error rendering block:
nestLevel=6 expected 5
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
{quote}
Behaviour with AHS with file based history store
-Apphistory url is working
-No attempt entries are shown for each application.
Based on inital analysis when RM switches ,application attempts from state
store are not replayed but only applications are.
So when /applicaitonhistory url is accessed it tries for all attempt id and
fails
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3164) rmadmin command usage prints incorrect command name
[
https://issues.apache.org/jira/browse/YARN-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315443#comment-14315443
]
Bibin A Chundatt commented on YARN-3164:
Findbug and Test failure seems not related to this commit only console message
gets updated with patch uploaded
rmadmin command usage prints incorrect command name
---
Key: YARN-3164
URL: https://issues.apache.org/jira/browse/YARN-3164
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3164.1.patch
/hadoop/bin{color:red} ./yarn rmadmin -transitionToActive {color}
transitionToActive: incorrect number of arguments
Usage:{color:red} HAAdmin {color} [-transitionToActive serviceId
[--forceactive]]
{color:red} ./yarn HAAdmin {color}
Error: Could not find or load main class HAAdmin
Expected it should be rmadmin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3157) Wrong format for application id / attempt id not handled completely
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3157:
---
Attachment: YARN-3157.1.patch
uploading after applying formatter
Wrong format for application id / attempt id not handled completely
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3157.1.patch, YARN-3157.patch, YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3164) rmadmin command usage prints incorrect command name
[
https://issues.apache.org/jira/browse/YARN-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3164:
---
Attachment: YARN-3164.1.patch
Patch added for the same. Please review the same.
rmadmin command usage prints incorrect command name
---
Key: YARN-3164
URL: https://issues.apache.org/jira/browse/YARN-3164
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3164.1.patch
/hadoop/bin{color:red} ./yarn rmadmin -transitionToActive {color}
transitionToActive: incorrect number of arguments
Usage:{color:red} HAAdmin {color} [-transitionToActive serviceId
[--forceactive]]
{color:red} ./yarn HAAdmin {color}
Error: Could not find or load main class HAAdmin
Expected it should be rmadmin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3157) Wrong format for application id / attempt id not handled completely
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3157:
---
Attachment: YARN-3157.2.patch
Wrong format for application id / attempt id not handled completely
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3157.1.patch, YARN-3157.2.patch, YARN-3157.patch,
YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3157) Refactor the exception handling in ConverterUtils#to*Id
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3157:
---
Attachment: YARN-3157.2.patch
Uploading same patch again. For container appattempt and application same issue
on similar conditions mentioned in defect happens . Please do review
Refactor the exception handling in ConverterUtils#to*Id
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3157.1.patch, YARN-3157.2.patch, YARN-3157.2.patch,
YARN-3157.patch, YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
{color:red}./yarn container -status container_e20_1423221031460_0003_01{color}
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:227)
at
org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3164) rmadmin command usage prints incorrect command name
[
https://issues.apache.org/jira/browse/YARN-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3164:
---
Attachment: YARN-3164.2.patch
rmadmin command usage prints incorrect command name
---
Key: YARN-3164
URL: https://issues.apache.org/jira/browse/YARN-3164
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3164.1.patch, YARN-3164.2.patch
/hadoop/bin{color:red} ./yarn rmadmin -transitionToActive {color}
transitionToActive: incorrect number of arguments
Usage:{color:red} HAAdmin {color} [-transitionToActive serviceId
[--forceactive]]
{color:red} ./yarn HAAdmin {color}
Error: Could not find or load main class HAAdmin
Expected it should be rmadmin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3164) rmadmin command usage prints incorrect command name
[
https://issues.apache.org/jira/browse/YARN-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319581#comment-14319581
]
Bibin A Chundatt commented on YARN-3164:
[~xgong] I did look at the method but same way was not able to do. please do
correct me if i am wrong
In HAAdmin the errOut is Initialized as below and used in function
{code}
public abstract class HAAdmin extends Configured implements Tool {
/** Output stream for errors, for use in tests */
protected PrintStream errOut = System.err;
protected PrintStream out = System.out;
}
private void printUsage(PrintStream errOut, String cmd) {
..
errOut.println(Usage: HAAdmin [ + cmd + + usage.args + ]);
}
{code}
Error out gets initialized in Setup of TestRMAdminCli. So when tried to set
Sys.err in test method didnt work out.
For {code}class DFSHAAdmin extends HAAdmin {code} also the same way its handled
.DFSHAAdmin have added method {color:red}setErrOut{color} for that too.
Another option is to set System.err in {color:red}Before{color} of
TestRMAdminCli before initializing *rmAdminCLIWithHAEnabled* variable. So
hoping that current implementation is fine what do you say?
rmadmin command usage prints incorrect command name
---
Key: YARN-3164
URL: https://issues.apache.org/jira/browse/YARN-3164
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3164.1.patch, YARN-3164.2.patch
/hadoop/bin{color:red} ./yarn rmadmin -transitionToActive {color}
transitionToActive: incorrect number of arguments
Usage:{color:red} HAAdmin {color} [-transitionToActive serviceId
[--forceactive]]
{color:red} ./yarn HAAdmin {color}
Error: Could not find or load main class HAAdmin
Expected it should be rmadmin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3109) Broken link in cluster apps for application when queue full and failed
[
https://issues.apache.org/jira/browse/YARN-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3109:
---
Description:
Application link in cluster/apps broken when queue full and failed
Configure capacity scheduler with default queue size as 1
Submit 2 mapreduce jobs to the default queue
Select application detail link in ID for
/cluster/app/application_1422467063659_0006
{quote}
property
nameyarn.scheduler.capacity.root.default.maximum-applications/name
value1/value
description /description
/property
{quote}
{quote}
15/01/29 14:29:43 ERROR webapp.Dispatcher: error handling URI:
/cluster/app/application_1422467063659_0006
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:84)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
[jira] [Created] (YARN-3109) Broken link in cluster apps for application when queue full an rejected
Bibin A Chundatt created YARN-3109:
--
Summary: Broken link in cluster apps for application when queue
full an rejected
Key: YARN-3109
URL: https://issues.apache.org/jira/browse/YARN-3109
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.6.0
Environment: Linux , 2 NM and 1 RM
Reporter: Bibin A Chundatt
Priority: Minor
Fix For: 2.6.0
Application link in cluster/apps broken when queue full and rejected
Configure capacity scheduler with default queue size as 1
Submit 2 mapreduce jobs to the default queue
Select application detail link in ID for
/cluster/app/application_1422467063659_0006
{quote}
property
nameyarn.scheduler.capacity.root.default.maximum-applications/name
value1/value
description /description
/property
{quote}
{quote}
15/01/29 14:29:43 ERROR webapp.Dispatcher: error handling URI:
/cluster/app/application_1422467063659_0006
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:84)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
[jira] [Updated] (YARN-3109) Broken link in cluster apps for application when queue full and failed
[
https://issues.apache.org/jira/browse/YARN-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3109:
---
Summary: Broken link in cluster apps for application when queue full and
failed (was: Broken link in cluster apps for application when queue full an
rejected)
Broken link in cluster apps for application when queue full and failed
--
Key: YARN-3109
URL: https://issues.apache.org/jira/browse/YARN-3109
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.6.0
Environment: Linux , 2 NM and 1 RM
Reporter: Bibin A Chundatt
Priority: Minor
Fix For: 2.6.0
Application link in cluster/apps broken when queue full and rejected
Configure capacity scheduler with default queue size as 1
Submit 2 mapreduce jobs to the default queue
Select application detail link in ID for
/cluster/app/application_1422467063659_0006
{quote}
property
nameyarn.scheduler.capacity.root.default.maximum-applications/name
value1/value
description /description
/property
{quote}
{quote}
15/01/29 14:29:43 ERROR webapp.Dispatcher: error handling URI:
/cluster/app/application_1422467063659_0006
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:84)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
[jira] [Created] (YARN-3110) Faulty link and state in ApplicationHistory when aplication is in unassigned state
Bibin A Chundatt created YARN-3110: -- Summary: Faulty link and state in ApplicationHistory when aplication is in unassigned state Key: YARN-3110 URL: https://issues.apache.org/jira/browse/YARN-3110 Project: Hadoop YARN Issue Type: Bug Components: applications, timelineserver Affects Versions: 2.6.0 Reporter: Bibin A Chundatt Priority: Minor Application state and History link wrong when Application is in unassigned state 1.Configure capacity schedular with queue size as 1 also max Absolute Max Capacity:10.0% (Current application state is Accepted and Unassigned from resource manager side) 2.Submit application to queue and check the state and link in Applcation history State= null and History link shown as N/A -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3110) Faulty link and state in ApplicationHistory when aplication is in unassigned state
[ https://issues.apache.org/jira/browse/YARN-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3110: --- Description: Application state and History link wrong when Application is in unassigned state 1.Configure capacity schedular with queue size as 1 also max Absolute Max Capacity:10.0% (Current application state is Accepted and Unassigned from resource manager side) 2.Submit application to queue and check the state and link in Application history State= null and History link shown as N/A in applicationhistoty page was: Application state and History link wrong when Application is in unassigned state 1.Configure capacity schedular with queue size as 1 also max Absolute Max Capacity:10.0% (Current application state is Accepted and Unassigned from resource manager side) 2.Submit application to queue and check the state and link in Applcation history State= null and History link shown as N/A Faulty link and state in ApplicationHistory when aplication is in unassigned state -- Key: YARN-3110 URL: https://issues.apache.org/jira/browse/YARN-3110 Project: Hadoop YARN Issue Type: Bug Components: applications, timelineserver Affects Versions: 2.6.0 Reporter: Bibin A Chundatt Assignee: Naganarasimha G R Priority: Minor Application state and History link wrong when Application is in unassigned state 1.Configure capacity schedular with queue size as 1 also max Absolute Max Capacity: 10.0% (Current application state is Accepted and Unassigned from resource manager side) 2.Submit application to queue and check the state and link in Application history State= null and History link shown as N/A in applicationhistoty page -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3110) Faulty link and state in ApplicationHistory when aplication is in unassigned state
[ https://issues.apache.org/jira/browse/YARN-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3110: --- Description: Application state and History link wrong when Application is in unassigned state 1.Configure capacity schedular with queue size as 1 also max Absolute Max Capacity:10.0% (Current application state is Accepted and Unassigned from resource manager side) 2.Submit application to queue and check the state and link in Application history State= null and History link shown as N/A in applicationhistory page was: Application state and History link wrong when Application is in unassigned state 1.Configure capacity schedular with queue size as 1 also max Absolute Max Capacity:10.0% (Current application state is Accepted and Unassigned from resource manager side) 2.Submit application to queue and check the state and link in Application history State= null and History link shown as N/A in applicationhistoty page Faulty link and state in ApplicationHistory when aplication is in unassigned state -- Key: YARN-3110 URL: https://issues.apache.org/jira/browse/YARN-3110 Project: Hadoop YARN Issue Type: Bug Components: applications, timelineserver Affects Versions: 2.6.0 Reporter: Bibin A Chundatt Assignee: Naganarasimha G R Priority: Minor Application state and History link wrong when Application is in unassigned state 1.Configure capacity schedular with queue size as 1 also max Absolute Max Capacity: 10.0% (Current application state is Accepted and Unassigned from resource manager side) 2.Submit application to queue and check the state and link in Application history State= null and History link shown as N/A in applicationhistory page -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3110) Faulty link and state in ApplicationHistory when aplication is in unassigned state
[
https://issues.apache.org/jira/browse/YARN-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3110:
---
Description:
Application state and History link wrong when Application is in unassigned state
1.Configure capacity schedular with queue size as 1 also max Absolute Max
Capacity:10.0%
(Current application state is Accepted and Unassigned from resource manager
side)
2.Submit application to queue and check the state and link in Application
history
State= null and History link shown as N/A in applicationhistory page
Kill the same application . In timeline server logs the below is show when
selecting application link.
{quote}
2015-01-29 15:39:50,956 ERROR org.apache.hadoop.yarn.webapp.View: Failed to
read the AM container of the application attempt
appattempt_1422467063659_0007_01.
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainer(ApplicationHistoryManagerOnTimelineStore.java:162)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getAMContainer(ApplicationHistoryManagerOnTimelineStore.java:184)
at
org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:160)
at
org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.yarn.server.webapp.AppBlock.render(AppBlock.java:156)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:67)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSController.app(AHSController.java:38)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter.doFilter(CrossOriginFilter.java:95)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
at
[jira] [Updated] (YARN-3110) Faulty link and state in ApplicationHistory when aplication is in unassigned state
[
https://issues.apache.org/jira/browse/YARN-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3110:
---
Description:
Application state and History link wrong when Application is in unassigned state
1.Configure capacity schedular with queue size as 1 also max Absolute Max
Capacity:10.0%
(Current application state is Accepted and Unassigned from resource manager
side)
2.Submit application to queue and check the state and link in Application
history
State= null and History link shown as N/A in applicationhistory page
Kill the same application in
{quote}
2015-01-29 15:39:50,956 ERROR org.apache.hadoop.yarn.webapp.View: Failed to
read the AM container of the application attempt
appattempt_1422467063659_0007_01.
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainer(ApplicationHistoryManagerOnTimelineStore.java:162)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getAMContainer(ApplicationHistoryManagerOnTimelineStore.java:184)
at
org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:160)
at
org.apache.hadoop.yarn.server.webapp.AppBlock$3.run(AppBlock.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.yarn.server.webapp.AppBlock.render(AppBlock.java:156)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:67)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
at
org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:845)
at
org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:56)
at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:82)
at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:212)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSController.app(AHSController.java:38)
at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter.doFilter(CrossOriginFilter.java:95)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at
[jira] [Updated] (YARN-3109) Broken link in cluster apps for application when queue full and failed
[
https://issues.apache.org/jira/browse/YARN-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3109:
---
Fix Version/s: (was: 2.6.0)
2.7.0
Broken link in cluster apps for application when queue full and failed
--
Key: YARN-3109
URL: https://issues.apache.org/jira/browse/YARN-3109
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.6.0
Environment: Linux , 2 NM and 1 RM
Reporter: Bibin A Chundatt
Priority: Minor
Fix For: 2.7.0
Application link in cluster/apps broken when queue full and failed
Configure capacity scheduler with default queue size as 1
Submit 2 mapreduce jobs to the default queue
Select application detail link in ID for
/cluster/app/application_1422467063659_0006
{quote}
property
nameyarn.scheduler.capacity.root.default.maximum-applications/name
value1/value
description /description
/property
{quote}
{quote}
15/01/29 14:29:43 ERROR webapp.Dispatcher: error handling URI:
/cluster/app/application_1422467063659_0006
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:84)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
[jira] [Resolved] (YARN-3109) Broken link in cluster apps for application when queue full and failed
[
https://issues.apache.org/jira/browse/YARN-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt resolved YARN-3109.
Resolution: Duplicate
Just found similar defect already raised .
Marking it as duplicate of YARN-2414
Broken link in cluster apps for application when queue full and failed
--
Key: YARN-3109
URL: https://issues.apache.org/jira/browse/YARN-3109
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.6.0
Environment: Linux , 2 NM and 1 RM
Reporter: Bibin A Chundatt
Priority: Minor
Fix For: 2.6.0
Application link in cluster/apps broken when queue full and failed
Configure capacity scheduler with default queue size as 1
Submit 2 mapreduce jobs to the default queue
Select application detail link in ID for
/cluster/app/application_1422467063659_0006
{quote}
property
nameyarn.scheduler.capacity.root.default.maximum-applications/name
value1/value
description /description
/property
{quote}
{quote}
15/01/29 14:29:43 ERROR webapp.Dispatcher: error handling URI:
/cluster/app/application_1422467063659_0006
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor49.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:153)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at
org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:84)
at
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at
com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:572)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:269)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:542)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:84)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
[jira] [Created] (YARN-3102) Decommisioned Nodes not listed in Web UI
Bibin A Chundatt created YARN-3102: -- Summary: Decommisioned Nodes not listed in Web UI Key: YARN-3102 URL: https://issues.apache.org/jira/browse/YARN-3102 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Environment: 2 Node Manager and 1 Resource Manager Reporter: Bibin A Chundatt Priority: Minor Configure yarn.resourcemanager.nodes.exclude-path in yarn-site.xml to yarn.exlude file In RM1 machine Add Yarn.exclude with NM1 Host Name Start the node as listed below NM1,NM2 Resource manager Now check Nodes decommisioned in /cluster/nodes Number of decommisioned node is listed as 1 but Table is empty in /cluster/nodes/decommissioned (detail of Decommision node not shown) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3157) Wrong format for application id / attempt id not handled completely
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3157:
---
Attachment: YARN-3157.patch
As per comments from Naga , have updated testcase and also handled for
containerid
Wrong format for application id / attempt id not handled completely
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3157.patch, YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3164) rmadmin command usage prints incorrect command name
[
https://issues.apache.org/jira/browse/YARN-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314167#comment-14314167
]
Bibin A Chundatt commented on YARN-3164:
Any problem in changing the message as below
{color:red} Usage: rmadmin {color}
rmadmin command usage prints incorrect command name
---
Key: YARN-3164
URL: https://issues.apache.org/jira/browse/YARN-3164
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
/hadoop/bin{color:red} ./yarn rmadmin -transitionToActive {color}
transitionToActive: incorrect number of arguments
Usage:{color:red} HAAdmin {color} [-transitionToActive serviceId
[--forceactive]]
{color:red} ./yarn HAAdmin {color}
Error: Could not find or load main class HAAdmin
Expected it should be rmadmin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3164) rmadmin command usage prints incorrect command name
[
https://issues.apache.org/jira/browse/YARN-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14315888#comment-14315888
]
Bibin A Chundatt commented on YARN-3164:
Have tested the same manually as of now and is working fine .
transitionToActive: incorrect number of arguments
Usage : rmadmin [-transitionToActive [--forceactive] serviceId]
rmadmin command usage prints incorrect command name
---
Key: YARN-3164
URL: https://issues.apache.org/jira/browse/YARN-3164
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3164.1.patch
/hadoop/bin{color:red} ./yarn rmadmin -transitionToActive {color}
transitionToActive: incorrect number of arguments
Usage:{color:red} HAAdmin {color} [-transitionToActive serviceId
[--forceactive]]
{color:red} ./yarn HAAdmin {color}
Error: Could not find or load main class HAAdmin
Expected it should be rmadmin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3157) Wrong format for application id / attempt id not handled completely
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14312161#comment-14312161
]
Bibin A Chundatt commented on YARN-3157:
Naga, Thank you for the comment. I had missed container check will add that too
{quote}
public static ContainerId toContainerId(String containerIdStr)
return ContainerId.fromString(containerIdStr);
{quote}
Need to add catch block here too.
Wrong format for application id / attempt id not handled completely
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3157) Refactor the exception handling in ConverterUtils#to*Id
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3157:
---
Attachment: YARN-3157.3.patch
attaching patch based on review comments
Refactor the exception handling in ConverterUtils#to*Id
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3157.1.patch, YARN-3157.2.patch, YARN-3157.2.patch,
YARN-3157.3.patch, YARN-3157.patch, YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
{color:red}./yarn container -status container_e20_1423221031460_0003_01{color}
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:227)
at
org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3157) Refactor the exception handling in ConverterUtils#to*Id
[
https://issues.apache.org/jira/browse/YARN-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318522#comment-14318522
]
Bibin A Chundatt commented on YARN-3157:
[~ozawa] Thank you for comming the patch and reviewing the same.
Refactor the exception handling in ConverterUtils#to*Id
---
Key: YARN-3157
URL: https://issues.apache.org/jira/browse/YARN-3157
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Fix For: 2.7.0
Attachments: YARN-3157.1.patch, YARN-3157.2.patch, YARN-3157.2.patch,
YARN-3157.3.patch, YARN-3157.patch, YARN-3157.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
{color:red}./yarn container -status container_e20_1423221031460_0003_01{color}
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.api.records.ContainerId.fromString(ContainerId.java:227)
at
org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:178)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3164) rmadmin command usage prints incorrect command name
[
https://issues.apache.org/jira/browse/YARN-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14317857#comment-14317857
]
Bibin A Chundatt commented on YARN-3164:
Thank you [~aw],[~leftnoteasy],[~xgong] for looking into the patch. I have done
the formatting required and have added test case in TestRMAdminCLI.
rmadmin command usage prints incorrect command name
---
Key: YARN-3164
URL: https://issues.apache.org/jira/browse/YARN-3164
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3164.1.patch, YARN-3164.2.patch
/hadoop/bin{color:red} ./yarn rmadmin -transitionToActive {color}
transitionToActive: incorrect number of arguments
Usage:{color:red} HAAdmin {color} [-transitionToActive serviceId
[--forceactive]]
{color:red} ./yarn HAAdmin {color}
Error: Could not find or load main class HAAdmin
Expected it should be rmadmin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3164) rmadmin command usage prints incorrect command name
Bibin A Chundatt created YARN-3164:
--
Summary: rmadmin command usage prints incorrect command name
Key: YARN-3164
URL: https://issues.apache.org/jira/browse/YARN-3164
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
/hadoop/bin{color:red} ./yarn rmadmin -transitionToActive {color}
transitionToActive: incorrect number of arguments
Usage:{color:red} HAAdmin {color} [-transitionToActive serviceId
[--forceactive]]
{color:red} ./yarn HAAdmin {color}
Error: Could not find or load main class HAAdmin
Expected it should be rmadmin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3149) Typo in message for invalid application id
Bibin A Chundatt created YARN-3149: -- Summary: Typo in message for invalid application id Key: YARN-3149 URL: https://issues.apache.org/jira/browse/YARN-3149 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Bibin A Chundatt Priority: Trivial Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3149) Typo in message for invalid application id
[
https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14307733#comment-14307733
]
Bibin A Chundatt commented on YARN-3149:
The message should be application id could any one assign to me. I would like
to submit the patch
{quote}
try {
return toApplicationId(it);
} catch (NumberFormatException n) {
throw new IllegalArgumentException(Invalid AppAttemptId:
+ appIdStr, n);
}
{quote}
Typo in message for invalid application id
--
Key: YARN-3149
URL: https://issues.apache.org/jira/browse/YARN-3149
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Bibin A Chundatt
Priority: Trivial
Attachments: screenshot-1.png
Message in console wrong when application id format wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3149) Typo in message for invalid application id
[ https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3149: --- Attachment: YARN-3149.patch Typo in message for invalid application id -- Key: YARN-3149 URL: https://issues.apache.org/jira/browse/YARN-3149 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Reporter: Bibin A Chundatt Priority: Trivial Attachments: YARN-3149.patch, screenshot-1.png Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3149) Typo in message for invalid application id
[ https://issues.apache.org/jira/browse/YARN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3149: --- Affects Version/s: 2.6.0 Typo in message for invalid application id -- Key: YARN-3149 URL: https://issues.apache.org/jira/browse/YARN-3149 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Reporter: Bibin A Chundatt Priority: Trivial Attachments: YARN-3149.patch, screenshot-1.png Message in console wrong when application id format wrong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3425) NPE during resourcemanager shutdown with node label with out datanode started
Bibin A Chundatt created YARN-3425:
--
Summary: NPE during resourcemanager shutdown with node label with
out datanode started
Key: YARN-3425
URL: https://issues.apache.org/jira/browse/YARN-3425
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Environment: 1 RM, 1 NM , 1 NN , I DN
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Configure yarn.node-labels.enabled to true
and yarn.node-labels.fs-store.root-dir /node-labels
Start resource manager without starting DN/NM
{quote}
2015-03-31 16:44:13,782 WARN org.apache.hadoop.service.AbstractService: When
stopping the service org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager
: java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.stopDispatcher(CommonNodeLabelsManager.java:261)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStop(CommonNodeLabelsManager.java:267)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:556)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:984)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:251)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1207)
{quote}
{code}
protected void stopDispatcher() {
AsyncDispatcher asyncDispatcher = (AsyncDispatcher) dispatcher;
asyncDispatcher.stop();
}
{code}
Null check missing during stop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3168) Convert site documentation from apt to markdown
[ https://issues.apache.org/jira/browse/YARN-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3168: --- Attachment: YARN-3168.20150224.1.patch Convert site documentation from apt to markdown --- Key: YARN-3168 URL: https://issues.apache.org/jira/browse/YARN-3168 Project: Hadoop YARN Issue Type: Improvement Components: documentation Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Gururaj Shetty Attachments: YARN-3168-00.patch, YARN-3168.20150224.1.patch YARN analog to HADOOP-11495 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3394) WebApplication proxy documentation is incomplete
Bibin A Chundatt created YARN-3394: -- Summary: WebApplication proxy documentation is incomplete Key: YARN-3394 URL: https://issues.apache.org/jira/browse/YARN-3394 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Reporter: Bibin A Chundatt Assignee: Naganarasimha G R Priority: Minor Webproxy documentation is incomplete hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html 1.Configuration of service start/stop as separate server 2.Steps to start as daemon service 3.Secure mode for Web proxy -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3102) Decommisioned Nodes not listed in Web UI
[ https://issues.apache.org/jira/browse/YARN-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298259#comment-14298259 ] Bibin A Chundatt commented on YARN-3102: Thanks Devaraj K for checking the issue All the time when i start the server first 2 nodemanagers and then Resource manager. As mentioned in defect describtion when i start Resourcemanager exclude file was configured with NM1 host name and both Node Manager 1 and 2 were up . Then resource mangers starts up is does send signal also 2015-01-30 17:15:11,549 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Disallowed NodeManager from HOST-10-19-92-128, Sending SHUTDOWN signal to the NodeManager. But table is shown as empty Decommisioned Nodes not listed in Web UI Key: YARN-3102 URL: https://issues.apache.org/jira/browse/YARN-3102 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Environment: 2 Node Manager and 1 Resource Manager Reporter: Bibin A Chundatt Assignee: Naganarasimha G R Priority: Minor Configure yarn.resourcemanager.nodes.exclude-path in yarn-site.xml to yarn.exlude file In RM1 machine Add Yarn.exclude with NM1 Host Name Start the node as listed below NM1,NM2 Resource manager Now check Nodes decommisioned in /cluster/nodes Number of decommisioned node is listed as 1 but Table is empty in /cluster/nodes/decommissioned (detail of Decommision node not shown) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3436) Doc WebServicesIntro.html Example Rest API url wrong
[
https://issues.apache.org/jira/browse/YARN-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493676#comment-14493676
]
Bibin A Chundatt commented on YARN-3436:
[~ozawa] Could you please review the same
Doc WebServicesIntro.html Example Rest API url wrong
Key: YARN-3436
URL: https://issues.apache.org/jira/browse/YARN-3436
Project: Hadoop YARN
Issue Type: Bug
Components: documentation, resourcemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3436.001.patch
/docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html
{quote}
Response Examples
JSON response with single resource
HTTP Request: GET
http://rmhost.domain:8088/ws/v1/cluster/{color:red}app{color}/application_1324057493980_0001
Response Status Line: HTTP/1.1 200 OK
{quote}
Url should be ws/v1/cluster/{color:red}apps{color} .
2 examples on same page are wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
[
https://issues.apache.org/jira/browse/YARN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507449#comment-14507449
]
Bibin A Chundatt commented on YARN-3525:
Hi [~ywskycn] , Thank you for looking in to the issue.
All the fair scheduler properties we mention in yarn-site only except queue
level configuration. (About 13 properties)
Reference
:http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html
Except the mentioned properties all of them follow the same prefix pattern
specific to fair. So i feel we should sync these two properties also.
Rename fair scheduler properties increment-allocation-mb and
increment-allocation-vcores
Key: YARN-3525
URL: https://issues.apache.org/jira/browse/YARN-3525
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Rename below two properties since only used by fair scheduler
{color:blue}yarn.scheduler.increment-allocation-mb{color} to
{color:red}yarn.scheduler.fair.increment-allocation-mb{color}
{color:blue}yarn.scheduler.increment-allocation-vcores{color} to
{color:red}yarn.scheduler.fair.increment-allocation-vcores{color}
All other properties only for fair scheduler are using {color:red}
yarn.scheduler.fair{color} prefix .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3525) Rename fair scheduler properties increment-allocation-mb and increment-allocation-vcores
Bibin A Chundatt created YARN-3525:
--
Summary: Rename fair scheduler properties increment-allocation-mb
and increment-allocation-vcores
Key: YARN-3525
URL: https://issues.apache.org/jira/browse/YARN-3525
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Rename below two properties since only used by fair scheduler
{color:blue}yarn.scheduler.increment-allocation-mb{color} to
{color:red}yarn.scheduler.fair.increment-allocation-mb{color}
{color:blue}yarn.scheduler.increment-allocation-vcores{color} to
{color:red}yarn.scheduler.fair.increment-allocation-vcores{color}
All other properties only for fair scheduler are using {color:red}
yarn.scheduler.fair{color} prefix .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3558) Additional containers getting reserved from RM in case of Fair scheduler
Bibin A Chundatt created YARN-3558:
--
Summary: Additional containers getting reserved from RM in case of
Fair scheduler
Key: YARN-3558
URL: https://issues.apache.org/jira/browse/YARN-3558
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, resourcemanager
Affects Versions: 2.7.0
Environment: OS :Suse 11 Sp3
Setup : 2 RM 2 NM
Scheduler : Fair scheduler
Reporter: Bibin A Chundatt
Submit PI job with 16 maps
Total container expected : 16 MAPS + 1 Reduce + 1 AM
Total containers reserved by RM is 21
Below set of containers are not being used for execution
container_1430213948957_0001_01_20
container_1430213948957_0001_01_19
RM Containers reservation and states
{code}
Processing container_1430213948957_0001_01_01 of type START
Processing container_1430213948957_0001_01_01 of type ACQUIRED
Processing container_1430213948957_0001_01_01 of type LAUNCHED
Processing container_1430213948957_0001_01_02 of type START
Processing container_1430213948957_0001_01_03 of type START
Processing container_1430213948957_0001_01_02 of type ACQUIRED
Processing container_1430213948957_0001_01_03 of type ACQUIRED
Processing container_1430213948957_0001_01_04 of type START
Processing container_1430213948957_0001_01_05 of type START
Processing container_1430213948957_0001_01_04 of type ACQUIRED
Processing container_1430213948957_0001_01_05 of type ACQUIRED
Processing container_1430213948957_0001_01_02 of type LAUNCHED
Processing container_1430213948957_0001_01_04 of type LAUNCHED
Processing container_1430213948957_0001_01_06 of type RESERVED
Processing container_1430213948957_0001_01_03 of type LAUNCHED
Processing container_1430213948957_0001_01_05 of type LAUNCHED
Processing container_1430213948957_0001_01_07 of type START
Processing container_1430213948957_0001_01_07 of type ACQUIRED
Processing container_1430213948957_0001_01_07 of type LAUNCHED
Processing container_1430213948957_0001_01_08 of type RESERVED
Processing container_1430213948957_0001_01_02 of type FINISHED
Processing container_1430213948957_0001_01_06 of type START
Processing container_1430213948957_0001_01_06 of type ACQUIRED
Processing container_1430213948957_0001_01_06 of type LAUNCHED
Processing container_1430213948957_0001_01_04 of type FINISHED
Processing container_1430213948957_0001_01_09 of type START
Processing container_1430213948957_0001_01_09 of type ACQUIRED
Processing container_1430213948957_0001_01_09 of type LAUNCHED
Processing container_1430213948957_0001_01_10 of type RESERVED
Processing container_1430213948957_0001_01_03 of type FINISHED
Processing container_1430213948957_0001_01_08 of type START
Processing container_1430213948957_0001_01_08 of type ACQUIRED
Processing container_1430213948957_0001_01_08 of type LAUNCHED
Processing container_1430213948957_0001_01_05 of type FINISHED
Processing container_1430213948957_0001_01_11 of type START
Processing container_1430213948957_0001_01_11 of type ACQUIRED
Processing container_1430213948957_0001_01_11 of type LAUNCHED
Processing container_1430213948957_0001_01_07 of type FINISHED
Processing container_1430213948957_0001_01_12 of type START
Processing container_1430213948957_0001_01_12 of type ACQUIRED
Processing container_1430213948957_0001_01_12 of type LAUNCHED
Processing container_1430213948957_0001_01_13 of type RESERVED
Processing container_1430213948957_0001_01_06 of type FINISHED
Processing container_1430213948957_0001_01_10 of type START
Processing container_1430213948957_0001_01_10 of type ACQUIRED
Processing container_1430213948957_0001_01_10 of type LAUNCHED
Processing container_1430213948957_0001_01_09 of type FINISHED
Processing container_1430213948957_0001_01_14 of type START
Processing container_1430213948957_0001_01_14 of type ACQUIRED
Processing container_1430213948957_0001_01_14 of type LAUNCHED
Processing container_1430213948957_0001_01_15 of type RESERVED
Processing container_1430213948957_0001_01_08 of type FINISHED
Processing container_1430213948957_0001_01_13 of type START
Processing container_1430213948957_0001_01_16 of type RESERVED
Processing container_1430213948957_0001_01_13 of type ACQUIRED
Processing container_1430213948957_0001_01_13 of type LAUNCHED
Processing container_1430213948957_0001_01_11 of type FINISHED
Processing container_1430213948957_0001_01_16 of type START
Processing container_1430213948957_0001_01_10 of type FINISHED
Processing container_1430213948957_0001_01_15 of type START
Processing container_1430213948957_0001_01_16 of type ACQUIRED
Processing container_1430213948957_0001_01_15 of
[jira] [Created] (YARN-3548) Missing doc for security configuration for timeline service feature
Bibin A Chundatt created YARN-3548: -- Summary: Missing doc for security configuration for timeline service feature Key: YARN-3548 URL: https://issues.apache.org/jira/browse/YARN-3548 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Reporter: Bibin A Chundatt Assignee: Gururaj Shetty Priority: Minor Documentation for the below need to be added in Timeline server documentation yarn.timeline-service.http-authentication.kerberos.principal yarn.timeline-service.http-authentication.kerberos.keytab url: /hadoop-yarn/hadoop-yarn-site/TimelineServer.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3615) Yarn and Mapred queue CLI command support for Fairscheduler
Bibin A Chundatt created YARN-3615:
--
Summary: Yarn and Mapred queue CLI command support for
Fairscheduler
Key: YARN-3615
URL: https://issues.apache.org/jira/browse/YARN-3615
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, scheduler
Reporter: Bibin A Chundatt
Assignee: Naganarasimha G R
Add support for CLI command when Fair scheduler is configured
Listed few command which needs updation
./yarn queue -status job-queue-name
*Current output*
{code}
Queue Name : root.sls_queue_2
State : RUNNING
Capacity : 100.0%
Current Capacity : 100.0%
Maximum Capacity : -100.0%
Default Node Label expression :
Accessible Node Labels :
{code}
./mapred queue -info job-queue-name
./mapred queue -list
All the below commands currently displaying based on Capacity
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3627) Preemption not triggered in Fair scheduler when maxResources is set on parent queue
Bibin A Chundatt created YARN-3627:
--
Summary: Preemption not triggered in Fair scheduler when
maxResources is set on parent queue
Key: YARN-3627
URL: https://issues.apache.org/jira/browse/YARN-3627
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, scheduler
Environment: Suse 11 SP3, 2 NM
Reporter: Bibin A Chundatt
Consider the below scenario of fair configuration
Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
Q1.1 (maxResources 4gb)
Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)
No applications are running in Q2
Submit one application with to Q1.1 with 50 maps 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.
Preemption will never get triggered since
yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the
cluster utilization is below .8.
*Fairscheduler.java*
{code}
private boolean shouldAttemptPreemption() {
if (preemptionEnabled) {
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
(float) rootMetrics.getAllocatedVirtualCores() /
clusterResource.getVirtualCores()));
}
return false;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3627) Preemption not triggered in Fair scheduler when maxResources is set on parent queue
[
https://issues.apache.org/jira/browse/YARN-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3627:
---
[~ywskycn] Thank you for discussion.
Adding his point also in comments
Set MaxResource equal to cluster capacity (instead of 0,0) is not practical
in real clusters. For example, if we only have two queues, one (Q1) for ad-hoc
jobs, and another one (Q2) for production jobs. If we let Q1 and Q2 take over
the entire cluster, one possible here: at a given time, Q1 takes 100%, but we
have some critical production jobs coming in to Q2. In that case, Q2 needs to
wait a while for preempting some resources from Q1. It takes some times. So
mostly, we don't set the maximum resource for each queue to 100%, but 70-80%
instead, to leave some space.
Preemption not triggered in Fair scheduler when maxResources is set on parent
queue
---
Key: YARN-3627
URL: https://issues.apache.org/jira/browse/YARN-3627
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, scheduler
Environment: Suse 11 SP3, 2 NM
Reporter: Bibin A Chundatt
Consider the below scenario of fair configuration
Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
Q1.1 (maxResources 4gb)
Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)
No applications are running in Q2
Submit one application with to Q1.1 with 50 maps 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.
Preemption will never get triggered since
yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the
cluster utilization is below .8.
*Fairscheduler.java*
{code}
private boolean shouldAttemptPreemption() {
if (preemptionEnabled) {
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
(float) rootMetrics.getAllocatedVirtualCores() /
clusterResource.getVirtualCores()));
}
return false;
}
{code}
Are we supposed to configure in running cluster maxResources 0mb and 0
cores so that all queues can take full cluster resources always if
available??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3627) Preemption not triggered in Fair scheduler when maxResources is set on parent queue
[
https://issues.apache.org/jira/browse/YARN-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3627:
---
Description:
Consider the below scenario of fair configuration
Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
Q1.1 (maxResources 4gb)
Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)
No applications are running in Q2
Submit one application with to Q1.1 with 50 maps 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.
Preemption will never get triggered since
yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the
cluster utilization is below .8.
*Fairscheduler.java*
{code}
private boolean shouldAttemptPreemption() {
if (preemptionEnabled) {
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
(float) rootMetrics.getAllocatedVirtualCores() /
clusterResource.getVirtualCores()));
}
return false;
}
{code}
Are we supposed to configure in running cluster maxResources 0mb and 0 cores
so that all queues can take full cluster resources always if available??
was:
Consider the below scenario of fair configuration
Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
Q1.1 (maxResources 4gb)
Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)
No applications are running in Q2
Submit one application with to Q1.1 with 50 maps 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.
Preemption will never get triggered since
yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the
cluster utilization is below .8.
*Fairscheduler.java*
{code}
private boolean shouldAttemptPreemption() {
if (preemptionEnabled) {
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
(float) rootMetrics.getAllocatedVirtualCores() /
clusterResource.getVirtualCores()));
}
return false;
}
{code}
Preemption not triggered in Fair scheduler when maxResources is set on parent
queue
---
Key: YARN-3627
URL: https://issues.apache.org/jira/browse/YARN-3627
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, scheduler
Environment: Suse 11 SP3, 2 NM
Reporter: Bibin A Chundatt
Consider the below scenario of fair configuration
Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
Q1.1 (maxResources 4gb)
Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)
No applications are running in Q2
Submit one application with to Q1.1 with 50 maps 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.
Preemption will never get triggered since
yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the
cluster utilization is below .8.
*Fairscheduler.java*
{code}
private boolean shouldAttemptPreemption() {
if (preemptionEnabled) {
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
(float) rootMetrics.getAllocatedVirtualCores() /
clusterResource.getVirtualCores()));
}
return false;
}
{code}
Are we supposed to configure in running cluster maxResources 0mb and 0
cores so that all queues can take full cluster resources always if
available??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3548) Missing doc for security configuration for timeline service feature
[ https://issues.apache.org/jira/browse/YARN-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539313#comment-14539313 ] Bibin A Chundatt commented on YARN-3548: [~ste...@apache.org] in YARN-3539 i couldn't find in TimelineServer.html the above 2 configuration . Could you please add these also to same . As i remember did face some issue during startup of server in secure mode with out those. Please do correct me if i am wrong. Missing doc for security configuration for timeline service feature --- Key: YARN-3548 URL: https://issues.apache.org/jira/browse/YARN-3548 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Reporter: Bibin A Chundatt Assignee: Gururaj Shetty Priority: Minor Documentation for the below need to be added in Timeline server documentation yarn.timeline-service.http-authentication.kerberos.principal yarn.timeline-service.http-authentication.kerberos.keytab url: /hadoop-yarn/hadoop-yarn-site/TimelineServer.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3627) Preemption not triggered in Fair scheduler when maxResources is set on parent queue
[
https://issues.apache.org/jira/browse/YARN-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541782#comment-14541782
]
Bibin A Chundatt commented on YARN-3627:
Hi [~kasha] Thank you for looking into the same . But YARN-3405 also doesn't
change *shouldAttemptPreemption()* So the primary check of threshold will
happen . And subqueue Q1-1 is not preempted since its below threshold and Q1-2
will starve for resource.
Preemption not triggered in Fair scheduler when maxResources is set on parent
queue
---
Key: YARN-3627
URL: https://issues.apache.org/jira/browse/YARN-3627
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, scheduler
Environment: Suse 11 SP3, 2 NM
Reporter: Bibin A Chundatt
Consider the below scenario of fair configuration
Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
Q1.1 (maxResources 4gb)
Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)
No applications are running in Q2
Submit one application with to Q1.1 with 50 maps 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.
Preemption will never get triggered since
yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the
cluster utilization is below .8.
*Fairscheduler.java*
{code}
private boolean shouldAttemptPreemption() {
if (preemptionEnabled) {
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
(float) rootMetrics.getAllocatedVirtualCores() /
clusterResource.getVirtualCores()));
}
return false;
}
{code}
Are we supposed to configure in running cluster maxResources 0mb and 0
cores so that all queues can take full cluster resources always if
available??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3627) Preemption not triggered in Fair scheduler when maxResources is set on parent queue
[
https://issues.apache.org/jira/browse/YARN-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541201#comment-14541201
]
Bibin A Chundatt commented on YARN-3627:
[~kasha] seems related to YARN-3405 .Will try the patch soon. Would be great if
YARN-3405 gets resolved.
Preemption not triggered in Fair scheduler when maxResources is set on parent
queue
---
Key: YARN-3627
URL: https://issues.apache.org/jira/browse/YARN-3627
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, scheduler
Environment: Suse 11 SP3, 2 NM
Reporter: Bibin A Chundatt
Consider the below scenario of fair configuration
Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
Q1.1 (maxResources 4gb)
Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)
No applications are running in Q2
Submit one application with to Q1.1 with 50 maps 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.
Preemption will never get triggered since
yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the
cluster utilization is below .8.
*Fairscheduler.java*
{code}
private boolean shouldAttemptPreemption() {
if (preemptionEnabled) {
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
(float) rootMetrics.getAllocatedVirtualCores() /
clusterResource.getVirtualCores()));
}
return false;
}
{code}
Are we supposed to configure in running cluster maxResources 0mb and 0
cores so that all queues can take full cluster resources always if
available??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3127) Apphistory url crashes when RM switches with ATS enabled
[
https://issues.apache.org/jira/browse/YARN-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3127:
---
Priority: Critical (was: Major)
Apphistory url crashes when RM switches with ATS enabled
Key: YARN-3127
URL: https://issues.apache.org/jira/browse/YARN-3127
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager, timelineserver
Affects Versions: 2.6.0
Environment: RM HA with ATS
Reporter: Bibin A Chundatt
Assignee: Naganarasimha G R
Priority: Critical
Attachments: YARN-3127.20150213-1.patch, YARN-3127.20150329-1.patch
1.Start RM with HA and ATS configured and run some yarn applications
2.Once applications are finished sucessfully start timeline server
3.Now failover HA form active to standby
4.Access timeline server URL IP:PORT/applicationhistory
Result: Application history URL fails with below info
{quote}
2015-02-03 20:28:09,511 ERROR org.apache.hadoop.yarn.webapp.View: Failed to
read the applications.
java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1643)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock.render(AppsBlock.java:80)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:67)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
at
org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
...
Caused by:
org.apache.hadoop.yarn.exceptions.ApplicationAttemptNotFoundException: The
entity for application attempt appattempt_1422972608379_0001_01 doesn't
exist in the timeline store
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplicationAttempt(ApplicationHistoryManagerOnTimelineStore.java:151)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.generateApplicationReport(ApplicationHistoryManagerOnTimelineStore.java:499)
at
org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getAllApplications(ApplicationHistoryManagerOnTimelineStore.java:108)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:84)
at
org.apache.hadoop.yarn.server.webapp.AppsBlock$1.run(AppsBlock.java:81)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
... 51 more
2015-02-03 20:28:09,512 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
handling URI: /applicationhistory
org.apache.hadoop.yarn.webapp.WebAppException: Error rendering block:
nestLevel=6 expected 5
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
at
org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:77)
{quote}
Behaviour with AHS with file based history store
-Apphistory url is working
-No attempt entries are shown for each application.
Based on inital analysis when RM switches ,application attempts from state
store are not replayed but only applications are.
So when /applicaitonhistory url is accessed it tries for all attempt id and
fails
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3651) Tracking url in ApplicaitonCLI wrong for running application
Bibin A Chundatt created YARN-3651:
--
Summary: Tracking url in ApplicaitonCLI wrong for running
application
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup
2.Submit application to cluster
3.Execute command ./yarn application -list
4.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
http://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3651) Tracking url in ApplicationCLI wrong for running application
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3651:
---
Summary: Tracking url in ApplicationCLI wrong for running application
(was: Tracking url in ApplicaitonCLI wrong for running application)
Tracking url in ApplicationCLI wrong for running application
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup
2.Submit application to cluster
3.Execute command ./yarn application -list
4.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
http://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3651) Tracking url in ApplicationCLI wrong for running application
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3651:
---
Description:
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
was:
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup
2.Submit application to cluster
3.Execute command ./yarn application -list
4.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
http://IP:64323/proxy/application_1431672734347_0003 /
Priority: Minor (was: Major)
Tracking url in ApplicationCLI wrong for running application
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Priority: Minor
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3651) Tracking url in ApplicationCLI wrong for running application
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14545353#comment-14545353
]
Bibin A Chundatt commented on YARN-3651:
Also when configured as HTTPS_ONLY why http port is opened .
HI All any comments on the same??
Tracking url in ApplicationCLI wrong for running application
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Priority: Minor
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3627) Preemption not triggered in Fair scheduler when maxResources is set on parent queue
[
https://issues.apache.org/jira/browse/YARN-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14540074#comment-14540074
]
Bibin A Chundatt commented on YARN-3627:
Hi [~kasha] any comments in the same.
Seems in this case the pre-emption logic will not work at all. Only when
preemption threshold to really low value the same will work .
Preemption not triggered in Fair scheduler when maxResources is set on parent
queue
---
Key: YARN-3627
URL: https://issues.apache.org/jira/browse/YARN-3627
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, scheduler
Environment: Suse 11 SP3, 2 NM
Reporter: Bibin A Chundatt
Consider the below scenario of fair configuration
Root (10Gb cluster resource)
--Q1 (maxResources 4gb)
Q1.1 (maxResources 4gb)
Q1.2 (maxResources 4gb)
--Q2 (maxResources 6GB)
No applications are running in Q2
Submit one application with to Q1.1 with 50 maps 4Gb gets allocated to Q1.1
Now submit application to Q1.2 the same will be starving for memory always.
Preemption will never get triggered since
yarn.scheduler.fair.preemption.cluster-utilization-threshold =.8 and the
cluster utilization is below .8.
*Fairscheduler.java*
{code}
private boolean shouldAttemptPreemption() {
if (preemptionEnabled) {
return (preemptionUtilizationThreshold Math.max(
(float) rootMetrics.getAllocatedMB() / clusterResource.getMemory(),
(float) rootMetrics.getAllocatedVirtualCores() /
clusterResource.getVirtualCores()));
}
return false;
}
{code}
Are we supposed to configure in running cluster maxResources 0mb and 0
cores so that all queues can take full cluster resources always if
available??
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-3651) Tracking url in ApplicationCLI wrong for running application
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt resolved YARN-3651.
Resolution: Won't Fix
[~devraj.jaiman] . Thank you for looking into the same.
Closing the issue as won't fix since its done intentionally.
Tracking url in ApplicationCLI wrong for running application
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Priority: Minor
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3651) Tracking url in ApplicationCLI wrong for running application
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14545979#comment-14545979
]
Bibin A Chundatt commented on YARN-3651:
Hi [~jianhe] thnnk you for looking in to the issue .
In secure mode also the url remains http . When https_only shouldnt be all
communication be https.
Shouldnt this be considered as a security issue??
Please do correct me if i am wrong.
Tracking url in ApplicationCLI wrong for running application
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Priority: Minor
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3475) Invalid Resource Exception could be handled properly when cores not available
[
https://issues.apache.org/jira/browse/YARN-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3475:
---
Summary: Invalid Resource Exception could be handled properly when cores
not available (was: Invalid Resource Exception could be handled immediately
when cores not available)
Invalid Resource Exception could be handled properly when cores not available
-
Key: YARN-3475
URL: https://issues.apache.org/jira/browse/YARN-3475
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager, resourcemanager
Environment: 1 RM , 1 DN
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Configure yarn.nodemanager.resource.cpu-vcores=2 for NM
Set mapreduce.map.cpu.vcores=5 while running sleep job n client
{code}
2015-04-10 20:37:26,111 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM.
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid
resource request, requested virtual cores 0, or requested virtual cores
max configured, requestedVirtualCores=5, maxVirtualCores=2
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:213)
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:97)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:502)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2142)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2138)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2136)
at sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:101)
at com.sun.proxy.$Proxy34.allocate(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:199)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:686)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:281)
at java.lang.Thread.run(Thread.java:745)
Caused by: 2015-04-10 20:37:27,117 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Could not contact
RM after 36 milliseconds.
2015-04-10 20:37:27,173 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Error communicating
with RM: Could not contact RM after 36 milliseconds.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Could not contact RM
after 36 milliseconds.
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:712)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:257)
at
[jira] [Created] (YARN-3479) RM switch sanity check for Resource not happening
Bibin A Chundatt created YARN-3479:
--
Summary: RM switch sanity check for Resource not happening
Key: YARN-3479
URL: https://issues.apache.org/jira/browse/YARN-3479
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler, nodemanager, resourcemanager
Reporter: Bibin A Chundatt
Assignee: Naganarasimha G R
Priority: Minor
Configure RM with CS and default calculator
{code}
nameyarn.scheduler.capacity.resource-calculator/name
valueorg.apache.hadoop.yarn.util.resource.DefaultResourceCalculator/value
{code}
2 Nm with 2 vcores configured.
Submit application with
{code}namemapreduce.map.cpu.vcores/name
value6/value
{code}
Only AM gets submitted none of the container is launching
Switch RM now and check the application status
containers starts running which is not accepted
{code}
15/04/13 15:26:19 INFO mapreduce.Job: map 38% reduce 0%
15/04/13 15:26:20 INFO mapreduce.Job: map 76% reduce 0%
15/04/13 15:26:22 INFO mapreduce.Job: map 88% reduce 0%
15/04/13 15:26:23 INFO mapreduce.Job: map 100% reduce 0%
15/04/13 15:32:01 INFO mapreduce.Job: map 0% reduce 0%
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3457) NPE when NodeManager.serviceInit fails and stopRecoveryStore called
Bibin A Chundatt created YARN-3457:
--
Summary: NPE when NodeManager.serviceInit fails and
stopRecoveryStore called
Key: YARN-3457
URL: https://issues.apache.org/jira/browse/YARN-3457
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
When NodeManager service init fails during stopRecoveryStore null pointer
exception is thrown
{code}
@Override
protected void serviceInit(Configuration conf) throws Exception {
..
try {
exec.init();
} catch (IOException e) {
throw new YarnRuntimeException(Failed to initialize container executor,
e);
}
this.context = createNMContext(containerTokenSecretManager,
nmTokenSecretManager, nmStore);
{code}
context is null when service init fails
{code}
private void stopRecoveryStore() throws IOException {
nmStore.stop();
if (context.getDecommissioned() nmStore.canRecover()) {
..
}
}
{code}
Null pointer exception thrown
{quote}
015-04-07 17:31:45,807 WARN org.apache.hadoop.service.AbstractService: When
stopping the service NodeManager : java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.stopRecoveryStore(NodeManager.java:168)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:280)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:484)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:534)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2801) Documentation development for Node labels requirment
[ https://issues.apache.org/jira/browse/YARN-2801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483093#comment-14483093 ] Bibin A Chundatt commented on YARN-2801: [~leftnoteasy] Any update on documentation for Node Labels . Its difficult to completely evaluate this feature with out documentation Documentation development for Node labels requirment Key: YARN-2801 URL: https://issues.apache.org/jira/browse/YARN-2801 Project: Hadoop YARN Issue Type: Sub-task Components: documentation Reporter: Gururaj Shetty Assignee: Wangda Tan Documentation needs to be developed for the node label requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3450) Application Master killed RPC Port AM Host not shown in CLI
Bibin A Chundatt created YARN-3450:
--
Summary: Application Master killed RPC Port AM Host not shown in
CLI
Key: YARN-3450
URL: https://issues.apache.org/jira/browse/YARN-3450
Project: Hadoop YARN
Issue Type: Bug
Components: client, resourcemanager
Reporter: Bibin A Chundatt
Assignee: Naganarasimha G R
Priority: Minor
Start Sleep job
Kill application Master process
Check status of application Attempt
When Application Master killed RPC Port , AM Host not shown in CLI
{quote}
dsperf@host-10-19-92-117:~/HADOPV1R2/install/hadoop/nodemanager/bin
{color:red} ./yarn applicationattempt -status
appattempt_1428321793042_0005_01 {color}
15/04/06 13:40:52 INFO impl.TimelineClientImpl: Timeline service address:
http://10.19.92.127:8188/ws/v1/timeline/
15/04/06 13:40:52 INFO client.AHSProxy: Connecting to Application History
server at /10.19.92.127:45034
15/04/06 13:40:53 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
Application Attempt Report :
ApplicationAttempt-Id : appattempt_1428321793042_0005_01
State : FAILED
AMContainer : container_1428321793042_0005_01_01
Tracking-URL :
http://host-10-19-92-127:45020/cluster/app/application_1428321793042_0005
{color:red}
RPC Port : -1
AM Host : N/A
{color}
Diagnostics : AM Container for appattempt_1428321793042_0005_01
exited with exitCode: 137
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3450) Application Master killed RPC Port AM Host not shown in CLI
[
https://issues.apache.org/jira/browse/YARN-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14481040#comment-14481040
]
Bibin A Chundatt commented on YARN-3450:
/applicationhistory/appattempt/appattempt_1428321793042_0005_01
In Web UI the same is shown properly
Application Attempt Overview
State FAILED
Master Containercontainer_1428321793042_0005_01_01
Node: host-10-19-92-143:49820
Tracking URL: History
Application Master killed RPC Port AM Host not shown in CLI
---
Key: YARN-3450
URL: https://issues.apache.org/jira/browse/YARN-3450
Project: Hadoop YARN
Issue Type: Bug
Components: client, resourcemanager
Reporter: Bibin A Chundatt
Assignee: Naganarasimha G R
Priority: Minor
Start Sleep job
Kill application Master process
Check status of application Attempt
When Application Master killed RPC Port , AM Host not shown in CLI
{quote}
dsperf@host-10-19-92-117:~/HADOPV1R2/install/hadoop/nodemanager/bin
{color:red} ./yarn applicationattempt -status
appattempt_1428321793042_0005_01 {color}
15/04/06 13:40:52 INFO impl.TimelineClientImpl: Timeline service address:
http://10.19.92.127:8188/ws/v1/timeline/
15/04/06 13:40:52 INFO client.AHSProxy: Connecting to Application History
server at /10.19.92.127:45034
15/04/06 13:40:53 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
Application Attempt Report :
ApplicationAttempt-Id : appattempt_1428321793042_0005_01
State : FAILED
AMContainer : container_1428321793042_0005_01_01
Tracking-URL :
http://host-10-19-92-127:45020/cluster/app/application_1428321793042_0005
{color:red}
RPC Port : -1
AM Host : N/A
{color}
Diagnostics : AM Container for appattempt_1428321793042_0005_01
exited with exitCode: 137
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3457) NPE when NodeManager.serviceInit fails and stopRecoveryStore called
[
https://issues.apache.org/jira/browse/YARN-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3457:
---
Attachment: YARN-3457.001.patch
NPE when NodeManager.serviceInit fails and stopRecoveryStore called
---
Key: YARN-3457
URL: https://issues.apache.org/jira/browse/YARN-3457
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3457.001.patch
When NodeManager service init fails during stopRecoveryStore null pointer
exception is thrown
{code}
@Override
protected void serviceInit(Configuration conf) throws Exception {
..
try {
exec.init();
} catch (IOException e) {
throw new YarnRuntimeException(Failed to initialize container
executor, e);
}
this.context = createNMContext(containerTokenSecretManager,
nmTokenSecretManager, nmStore);
{code}
context is null when service init fails
{code}
private void stopRecoveryStore() throws IOException {
nmStore.stop();
if (context.getDecommissioned() nmStore.canRecover()) {
..
}
}
{code}
Null pointer exception thrown
{quote}
015-04-07 17:31:45,807 WARN org.apache.hadoop.service.AbstractService: When
stopping the service NodeManager : java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.stopRecoveryStore(NodeManager.java:168)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:280)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:484)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:534)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3435: --- Component/s: resourcemanager AM container to be allocated Appattempt AM container shown as null -- Key: YARN-3435 URL: https://issues.apache.org/jira/browse/YARN-3435 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Environment: 1RM,1DN Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Priority: Trivial Attachments: Screenshot.png, YARN-3435.001.patch Submit yarn application Open http://rm:8088/cluster/appattempt/appattempt_1427984982805_0003_01 Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3436) Doc WebServicesIntro.html Example Rest API url wrong
[
https://issues.apache.org/jira/browse/YARN-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3436:
---
Component/s: resourcemanager
documentation
Doc WebServicesIntro.html Example Rest API url wrong
Key: YARN-3436
URL: https://issues.apache.org/jira/browse/YARN-3436
Project: Hadoop YARN
Issue Type: Bug
Components: documentation, resourcemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3436.001.patch
/docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html
{quote}
Response Examples
JSON response with single resource
HTTP Request: GET
http://rmhost.domain:8088/ws/v1/cluster/{color:red}app{color}/application_1324057493980_0001
Response Status Line: HTTP/1.1 200 OK
{quote}
Url should be ws/v1/cluster/{color:red}apps{color} .
2 examples on same page are wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3436) Doc WebServicesIntro.html Example Rest API url wrong
[
https://issues.apache.org/jira/browse/YARN-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14396314#comment-14396314
]
Bibin A Chundatt commented on YARN-3436:
Please review the same .
Doc WebServicesIntro.html Example Rest API url wrong
Key: YARN-3436
URL: https://issues.apache.org/jira/browse/YARN-3436
Project: Hadoop YARN
Issue Type: Bug
Components: documentation, resourcemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3436.001.patch
/docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html
{quote}
Response Examples
JSON response with single resource
HTTP Request: GET
http://rmhost.domain:8088/ws/v1/cluster/{color:red}app{color}/application_1324057493980_0001
Response Status Line: HTTP/1.1 200 OK
{quote}
Url should be ws/v1/cluster/{color:red}apps{color} .
2 examples on same page are wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14396312#comment-14396312 ] Bibin A Chundatt commented on YARN-3435: Thank you [~xgong] for committing the same AM container to be allocated Appattempt AM container shown as null -- Key: YARN-3435 URL: https://issues.apache.org/jira/browse/YARN-3435 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Environment: 1RM,1DN Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Priority: Trivial Fix For: 2.8.0 Attachments: Screenshot.png, YARN-3435.001.patch Submit yarn application Open http://rm:8088/cluster/appattempt/appattempt_1427984982805_0003_01 Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3457) NPE when NodeManager.serviceInit fails and stopRecoveryStore called
[
https://issues.apache.org/jira/browse/YARN-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484931#comment-14484931
]
Bibin A Chundatt commented on YARN-3457:
Thank you [~ozawa] for checking and committing patch.
NPE when NodeManager.serviceInit fails and stopRecoveryStore called
---
Key: YARN-3457
URL: https://issues.apache.org/jira/browse/YARN-3457
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Fix For: 2.8.0
Attachments: YARN-3457.001.patch
When NodeManager service init fails during stopRecoveryStore null pointer
exception is thrown
{code}
@Override
protected void serviceInit(Configuration conf) throws Exception {
..
try {
exec.init();
} catch (IOException e) {
throw new YarnRuntimeException(Failed to initialize container
executor, e);
}
this.context = createNMContext(containerTokenSecretManager,
nmTokenSecretManager, nmStore);
{code}
context is null when service init fails
{code}
private void stopRecoveryStore() throws IOException {
nmStore.stop();
if (context.getDecommissioned() nmStore.canRecover()) {
..
}
}
{code}
Null pointer exception thrown
{quote}
015-04-07 17:31:45,807 WARN org.apache.hadoop.service.AbstractService: When
stopping the service NodeManager : java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.stopRecoveryStore(NodeManager.java:168)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:280)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:484)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:534)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3425) NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed
[
https://issues.apache.org/jira/browse/YARN-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14390996#comment-14390996
]
Bibin A Chundatt commented on YARN-3425:
[~leftnoteasy] Thank you for looking into the patch and issue.
NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit
failed
--
Key: YARN-3425
URL: https://issues.apache.org/jira/browse/YARN-3425
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Environment: 1 RM, 1 NM , 1 NN , I DN
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3425.001.patch
Configure yarn.node-labels.enabled to true
and yarn.node-labels.fs-store.root-dir /node-labels
Start resource manager without starting DN/NM
{quote}
2015-03-31 16:44:13,782 WARN org.apache.hadoop.service.AbstractService: When
stopping the service
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager :
java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.stopDispatcher(CommonNodeLabelsManager.java:261)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStop(CommonNodeLabelsManager.java:267)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:556)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:984)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:251)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1207)
{quote}
{code}
protected void stopDispatcher() {
AsyncDispatcher asyncDispatcher = (AsyncDispatcher) dispatcher;
asyncDispatcher.stop();
}
{code}
Null check missing during stop
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
Bibin A Chundatt created YARN-3435: -- Summary: AM container to be allocated Appattempt AM container shown as null Key: YARN-3435 URL: https://issues.apache.org/jira/browse/YARN-3435 Project: Hadoop YARN Issue Type: Bug Environment: 1RM,1DN Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Priority: Trivial Submit yarn application Open http://rm:8088/cluster/appattempt/appattempt_1427984982805_0003_01 Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3435: --- Attachment: Screenshot.png Attaching Screen shot for bug AM container to be allocated Appattempt AM container shown as null -- Key: YARN-3435 URL: https://issues.apache.org/jira/browse/YARN-3435 Project: Hadoop YARN Issue Type: Bug Environment: 1RM,1DN Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Priority: Trivial Attachments: Screenshot.png Submit yarn application Open http://rm:8088/cluster/appattempt/appattempt_1427984982805_0003_01 Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3436) Doc WebServicesIntro.html Example Rest API url wrong
Bibin A Chundatt created YARN-3436:
--
Summary: Doc WebServicesIntro.html Example Rest API url wrong
Key: YARN-3436
URL: https://issues.apache.org/jira/browse/YARN-3436
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
/docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html
{quote}
Response Examples
JSON response with single resource
HTTP Request: GET
http://rmhost.domain:8088/ws/v1/cluster/{color:red}app{color}/application_1324057493980_0001
Response Status Line: HTTP/1.1 200 OK
{quote}
Url should be ws/v1/cluster/{color:red}apps{color} .
2 examples on same page are wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3435) AM container to be allocated Appattempt AM container shown as null
[ https://issues.apache.org/jira/browse/YARN-3435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-3435: --- Attachment: YARN-3435.001.patch AM container to be allocated Appattempt AM container shown as null -- Key: YARN-3435 URL: https://issues.apache.org/jira/browse/YARN-3435 Project: Hadoop YARN Issue Type: Bug Environment: 1RM,1DN Reporter: Bibin A Chundatt Assignee: Bibin A Chundatt Priority: Trivial Attachments: Screenshot.png, YARN-3435.001.patch Submit yarn application Open http://rm:8088/cluster/appattempt/appattempt_1427984982805_0003_01 Before the AM container is allocated -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3436) Doc WebServicesIntro.html Example Rest API url wrong
[
https://issues.apache.org/jira/browse/YARN-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3436:
---
Attachment: YARN-3436.001.patch
Attaching patch for the same. please check the same
Doc WebServicesIntro.html Example Rest API url wrong
Key: YARN-3436
URL: https://issues.apache.org/jira/browse/YARN-3436
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: YARN-3436.001.patch
/docs/current/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html
{quote}
Response Examples
JSON response with single resource
HTTP Request: GET
http://rmhost.domain:8088/ws/v1/cluster/{color:red}app{color}/application_1324057493980_0001
Response Status Line: HTTP/1.1 200 OK
{quote}
Url should be ws/v1/cluster/{color:red}apps{color} .
2 examples on same page are wrong
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3651) Tracking url in ApplicationCLI wrong for running application
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14551793#comment-14551793
]
Bibin A Chundatt commented on YARN-3651:
Hi [~devaraj.k] and [~jianhe] i agree with comments , but i feel still
improvement can be done on the same .
as per the security. After discussion with some of members *MR can define a
separate config and have NM localize the key files for AM*
So i am reopening this bug as improvement. We can further discuss on this.
Tracking url in ApplicationCLI wrong for running application
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Bug
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Priority: Minor
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3651) Support SSL for AM webapp
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3651:
---
Summary: Support SSL for AM webapp (was: Tracking url in ApplicationCLI
wrong for running application)
Support SSL for AM webapp
-
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Improvement
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Priority: Minor
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3651) Tracking url in ApplicationCLI wrong for running application
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3651:
---
Issue Type: Improvement (was: Bug)
Tracking url in ApplicationCLI wrong for running application
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Improvement
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Priority: Minor
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (YARN-3651) Support SSL for AM webapp
[
https://issues.apache.org/jira/browse/YARN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt reopened YARN-3651:
Support SSL for AM webapp
-
Key: YARN-3651
URL: https://issues.apache.org/jira/browse/YARN-3651
Project: Hadoop YARN
Issue Type: Improvement
Components: applications, resourcemanager
Affects Versions: 2.7.0
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Priority: Minor
Application URL in Application CLI wrong
Steps to reproduce
==
1. Start HA setup insecure mode
2.Configure HTTPS_ONLY
3.Submit application to cluster
4.Execute command ./yarn application -list
5.Observer tracking URL shown
{code}
15/05/15 13:34:38 INFO client.AHSProxy: Connecting to Application History
server at /IP:45034
Total number of applications (application-types: [] and states: [SUBMITTED,
ACCEPTED, RUNNING]):1
Application-Id --- Tracking-URL
application_1431672734347_0003 *http://host-10-19-92-117:13013*
{code}
*Expected*
https://IP:64323/proxy/application_1431672734347_0003 /
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3693) Duplicate parameters on service start for NM and RM
Bibin A Chundatt created YARN-3693:
--
Summary: Duplicate parameters on service start for NM and RM
Key: YARN-3693
URL: https://issues.apache.org/jira/browse/YARN-3693
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
Priority: Minor
Steps to reproduce
=
1.Install HA cluster with NM RM
2.Check process id for the same
3.ps -ef | grep pid nm
Actual result
=
Multiple parameters are duplicate like log file name, Logger type , log
directory etc.
Same is observed in RM process also
*Please find the logs below*
{quote}
dsperf 26076 1 0 12:43 ?00:00:26
/opt/dsperf/jdk1.8.0_40//bin/java -Dproc_nodemanager -Xmx1000m
-Dhadoop.log.dir=install/nodemanager/logs
-Dyarn.log.dir=install/nodemanager/logs -Dhadoop.log.file=yarn.log
-Dyarn.log.file=yarn.log -Dyarn.home.dir= -Dyarn.id.str=
-Dhadoop.root.logger=INFO,console -Dyarn.root.logger=INFO,console
-Dyarn.policy.file=hadoop-policy.xml -Dlog4j.configuration.watch=true
-Dhadoop.log.dir=install/nodemanager/logs
-Dyarn.log.dir=install/nodemanager/logs
-Dhadoop.log.file=yarn-dsperf-nodemanager-host-name.log
-Dyarn.log.file=yarn-dsperf-nodemanager-host.log -Dyarn.home.dir=
-Dyarn.id.str=dsperf {color:red}-Dhadoop.root.logger=INFO,RFA
{color}-Dyarn.root.logger=INFO,RFA
-Djava.library.path=install/nodemanager/lib/native
-Dyarn.policy.file=hadoop-policy.xml -server
-Dhadoop.log.dir=install/nodemanager/logs
-Dyarn.log.dir=install/nodemanager/logs
-Dhadoop.log.file=yarn-dsperf-nodemanager-host-name.log
-Dyarn.log.file=yarn-dsperf-nodemanager-host-name.log
-Dyarn.home.dir=install/nodemanager -Dhadoop.home.dir=install/nodemanager
{color:red}-Dhadoop.root.logger=INFO,RFA {color}-Dyarn.root.logger=INFO,RFA
-Dlog4j.configuration=log4j.properties
-Djava.library.path=install/nodemanager/lib/native -classpath XXX
org.apache.hadoop.yarn.server.nodemanager.NodeManager
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3754) Race condition when the NodeManager is shutting down and container is launched
[
https://issues.apache.org/jira/browse/YARN-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574305#comment-14574305
]
Bibin A Chundatt commented on YARN-3754:
[~rohithsharma] and [~sunilg] Have tried with build containing YARN-3585 and
YARN-3641.
org.iq80.leveldb.DBException: Closed. exception i am not able to reproduce .
Race condition when the NodeManager is shutting down and container is launched
--
Key: YARN-3754
URL: https://issues.apache.org/jira/browse/YARN-3754
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Environment: Suse 11 Sp3
Reporter: Bibin A Chundatt
Assignee: Sunil G
Priority: Critical
Attachments: NM.log
Container is launched and returned to ContainerImpl
NodeManager closed the DB connection which resulting in
{{org.iq80.leveldb.DBException: Closed}}.
*Attaching the exception trace*
{code}
2015-05-30 02:11:49,122 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Unable to update state store diagnostics for
container_e310_1432817693365_3338_01_02
java.io.IOException: org.iq80.leveldb.DBException: Closed
at
org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.storeContainerDiagnostics(NMLeveldbStateStoreService.java:261)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ContainerDiagnosticsUpdateTransition.transition(ContainerImpl.java:1109)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ContainerDiagnosticsUpdateTransition.transition(ContainerImpl.java:1101)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1129)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:83)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:246)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.iq80.leveldb.DBException: Closed
at org.fusesource.leveldbjni.internal.JniDB.put(JniDB.java:123)
at org.fusesource.leveldbjni.internal.JniDB.put(JniDB.java:106)
at
org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.storeContainerDiagnostics(NMLeveldbStateStoreService.java:259)
... 15 more
{code}
we can add a check whether DB is closed while we move container from ACQUIRED
state.
As per the discussion in YARN-3585 have add the same
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3789) Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
[
https://issues.apache.org/jira/browse/YARN-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3789:
---
Attachment: 0001-YARN-3789.patch
Attaching patch for review
Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
--
Key: YARN-3789
URL: https://issues.apache.org/jira/browse/YARN-3789
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: 0001-YARN-3789.patch
Duplicate logging from resource manager
during am limit check for each application
{code}
015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (YARN-3789) Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
Bibin A Chundatt created YARN-3789:
--
Summary: Refactor logs for LeafQueue#activateApplications() to
remove duplicate logging
Key: YARN-3789
URL: https://issues.apache.org/jira/browse/YARN-3789
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Duplicate logging from resource manager
during am limit check for each application
{code}
015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not
starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not
starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not
starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not
starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not
starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: not
starting application as amIfStarted exceeds amLimit
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3789) Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
[
https://issues.apache.org/jira/browse/YARN-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3789:
---
Attachment: 0003-YARN-3789.patch
Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
--
Key: YARN-3789
URL: https://issues.apache.org/jira/browse/YARN-3789
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: 0001-YARN-3789.patch, 0002-YARN-3789.patch,
0003-YARN-3789.patch
Duplicate logging from resource manager
during am limit check for each application
{code}
015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3789) Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
[
https://issues.apache.org/jira/browse/YARN-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3789:
---
Attachment: 0002-YARN-3789.patch
Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
--
Key: YARN-3789
URL: https://issues.apache.org/jira/browse/YARN-3789
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: 0001-YARN-3789.patch, 0002-YARN-3789.patch
Duplicate logging from resource manager
during am limit check for each application
{code}
015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3789) Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
[
https://issues.apache.org/jira/browse/YARN-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14579867#comment-14579867
]
Bibin A Chundatt commented on YARN-3789:
[~rohithsharma] have handled the review comments.
Test cases not added since patch updates only logs
Refactor logs for LeafQueue#activateApplications() to remove duplicate logging
--
Key: YARN-3789
URL: https://issues.apache.org/jira/browse/YARN-3789
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: 0001-YARN-3789.patch, 0002-YARN-3789.patch,
0003-YARN-3789.patch
Duplicate logging from resource manager
during am limit check for each application
{code}
015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
2015-06-09 17:32:40,019 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
not starting application as amIfStarted exceeds amLimit
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3838) Rest API failing when ip configured in RM address in secure https mode
[
https://issues.apache.org/jira/browse/YARN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3838:
---
Attachment: 0001-YARN-3838.patch
Rest API failing when ip configured in RM address in secure https mode
--
Key: YARN-3838
URL: https://issues.apache.org/jira/browse/YARN-3838
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Critical
Attachments: 0001-HADOOP-12096.patch, 0001-YARN-3810.patch,
0001-YARN-3838.patch, 0002-YARN-3810.patch
Steps to reproduce
===
1.Configure hadoop.http.authentication.kerberos.principal as below
{code:xml}
property
namehadoop.http.authentication.kerberos.principal/name
valueHTTP/_h...@hadoop.com/value
/property
{code}
2. In RM web address also configure IP
3. Startup RM
Call Rest API for RM {{ curl -i -k --insecure --negotiate -u : https IP
/ws/v1/cluster/info}}
*Actual*
Rest API failing
{code}
2015-06-16 19:03:49,845 DEBUG
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos credentails)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3838) Rest API failing when ip configured in RM address in secure https mode
[
https://issues.apache.org/jira/browse/YARN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14595122#comment-14595122
]
Bibin A Chundatt commented on YARN-3838:
Added patch for review 0001-YARN-3838. Please do review
Rest API failing when ip configured in RM address in secure https mode
--
Key: YARN-3838
URL: https://issues.apache.org/jira/browse/YARN-3838
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Critical
Attachments: 0001-HADOOP-12096.patch, 0001-YARN-3810.patch,
0001-YARN-3838.patch, 0002-YARN-3810.patch
Steps to reproduce
===
1.Configure hadoop.http.authentication.kerberos.principal as below
{code:xml}
property
namehadoop.http.authentication.kerberos.principal/name
valueHTTP/_h...@hadoop.com/value
/property
{code}
2. In RM web address also configure IP
3. Startup RM
Call Rest API for RM {{ curl -i -k --insecure --negotiate -u : https IP
/ws/v1/cluster/info}}
*Actual*
Rest API failing
{code}
2015-06-16 19:03:49,845 DEBUG
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos credentails)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3837) javadocs of TimelineAuthenticationFilterInitializer give wrong prefix for auth options
[
https://issues.apache.org/jira/browse/YARN-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3837:
---
Attachment: 0002-YARN-3837.patch
No testcases required since its javadoc update
javadocs of TimelineAuthenticationFilterInitializer give wrong prefix for
auth options
--
Key: YARN-3837
URL: https://issues.apache.org/jira/browse/YARN-3837
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: 0001-YARN-3837.patch, 0002-YARN-3837.patch
Original Estimate: 0.5h
Remaining Estimate: 0.5h
The javadocs for {{TimelineAuthenticationFilterInitializer}} talk about the
prefix {{yarn.timeline-service.authentication.}}, but the code uses {{
yarn.timeline-service.http-authentication.}} as the prefix.
best to use {{@value}} and let the javadocs sort it out for themselves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3838) Rest API failing when ip configured in RM address in secure https mode
[
https://issues.apache.org/jira/browse/YARN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3838:
---
Attachment: 0002-YARN-3838.patch
Updated patch since new util is not required to be added.
Rest API failing when ip configured in RM address in secure https mode
--
Key: YARN-3838
URL: https://issues.apache.org/jira/browse/YARN-3838
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Critical
Attachments: 0001-HADOOP-12096.patch, 0001-YARN-3810.patch,
0001-YARN-3838.patch, 0002-YARN-3810.patch, 0002-YARN-3838.patch
Steps to reproduce
===
1.Configure hadoop.http.authentication.kerberos.principal as below
{code:xml}
property
namehadoop.http.authentication.kerberos.principal/name
valueHTTP/_h...@hadoop.com/value
/property
{code}
2. In RM web address also configure IP
3. Startup RM
Call Rest API for RM {{ curl -i -k --insecure --negotiate -u : https IP
/ws/v1/cluster/info}}
*Actual*
Rest API failing
{code}
2015-06-16 19:03:49,845 DEBUG
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos credentails)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (YARN-3838) Rest API failing when ip configured in RM address in secure https mode
[
https://issues.apache.org/jira/browse/YARN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt moved HADOOP-12096 to YARN-3838:
-
Component/s: (was: net)
(was: security)
security
Key: YARN-3838 (was: HADOOP-12096)
Project: Hadoop YARN (was: Hadoop Common)
Rest API failing when ip configured in RM address in secure https mode
--
Key: YARN-3838
URL: https://issues.apache.org/jira/browse/YARN-3838
Project: Hadoop YARN
Issue Type: Bug
Components: security
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Critical
Attachments: 0001-HADOOP-12096.patch, 0001-YARN-3810.patch,
0002-YARN-3810.patch
Steps to reproduce
===
1.Configure hadoop.http.authentication.kerberos.principal as below
{code:xml}
property
namehadoop.http.authentication.kerberos.principal/name
valueHTTP/_h...@hadoop.com/value
/property
{code}
2. In RM web address also configure IP
3. Startup RM
Call Rest API for RM {{ curl -i -k --insecure --negotiate -u : https IP
/ws/v1/cluster/info}}
*Actual*
Rest API failing
{code}
2015-06-16 19:03:49,845 DEBUG
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos credentails)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3838) Rest API failing when ip configured in RM address in secure https mode
[
https://issues.apache.org/jira/browse/YARN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3838:
---
Component/s: (was: security)
webapp
Rest API failing when ip configured in RM address in secure https mode
--
Key: YARN-3838
URL: https://issues.apache.org/jira/browse/YARN-3838
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Critical
Attachments: 0001-HADOOP-12096.patch, 0001-YARN-3810.patch,
0002-YARN-3810.patch
Steps to reproduce
===
1.Configure hadoop.http.authentication.kerberos.principal as below
{code:xml}
property
namehadoop.http.authentication.kerberos.principal/name
valueHTTP/_h...@hadoop.com/value
/property
{code}
2. In RM web address also configure IP
3. Startup RM
Call Rest API for RM {{ curl -i -k --insecure --negotiate -u : https IP
/ws/v1/cluster/info}}
*Actual*
Rest API failing
{code}
2015-06-16 19:03:49,845 DEBUG
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos credentails)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3837) javadocs of TimelineAuthenticationFilterInitializer give wrong prefix for auth options
[
https://issues.apache.org/jira/browse/YARN-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-3837:
---
Attachment: 0001-YARN-3837.patch
Attaching patch for the same. Please assign to me if its fine
javadocs of TimelineAuthenticationFilterInitializer give wrong prefix for
auth options
--
Key: YARN-3837
URL: https://issues.apache.org/jira/browse/YARN-3837
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Affects Versions: 2.8.0
Reporter: Steve Loughran
Priority: Minor
Attachments: 0001-YARN-3837.patch
Original Estimate: 0.5h
Remaining Estimate: 0.5h
The javadocs for {{TimelineAuthenticationFilterInitializer}} talk about the
prefix {{yarn.timeline-service.authentication.}}, but the code uses {{
yarn.timeline-service.http-authentication.}} as the prefix.
best to use {{@value}} and let the javadocs sort it out for themselves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3838) Rest API failing when ip configured in RM address in secure https mode
[
https://issues.apache.org/jira/browse/YARN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594801#comment-14594801
]
Bibin A Chundatt commented on YARN-3838:
Typo in earlier comments
{quote}
As per the discussion till now we should handle in HttpServer2
{quote}
As per the discussion till now we should handle in HttpServer2.builder
Rest API failing when ip configured in RM address in secure https mode
--
Key: YARN-3838
URL: https://issues.apache.org/jira/browse/YARN-3838
Project: Hadoop YARN
Issue Type: Bug
Components: webapp
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Critical
Attachments: 0001-HADOOP-12096.patch, 0001-YARN-3810.patch,
0002-YARN-3810.patch
Steps to reproduce
===
1.Configure hadoop.http.authentication.kerberos.principal as below
{code:xml}
property
namehadoop.http.authentication.kerberos.principal/name
valueHTTP/_h...@hadoop.com/value
/property
{code}
2. In RM web address also configure IP
3. Startup RM
Call Rest API for RM {{ curl -i -k --insecure --negotiate -u : https IP
/ws/v1/cluster/info}}
*Actual*
Rest API failing
{code}
2015-06-16 19:03:49,845 DEBUG
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos credentails)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3838) Rest API failing when ip configured in RM address in secure https mode
[
https://issues.apache.org/jira/browse/YARN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594737#comment-14594737
]
Bibin A Chundatt commented on YARN-3838:
In case of resourcemanager the httpserver is started as below and the url used
is just the ip address
{{WebApps#start}}
{code}
HttpServer2.Builder builder = new HttpServer2.Builder()
.setName(name)
.addEndpoint(
URI.create(httpScheme + bindAddress
+ : + port)).setConf(conf).setFindPort(findPort)
.setACL(new AccessControlList(conf.get(
YarnConfiguration.YARN_ADMIN_ACL,
YarnConfiguration.DEFAULT_YARN_ADMIN_ACL)))
.setPathSpec(pathList.toArray(new String[0]));
{code}
Comparing the same to hdfs side for NameNode the URL is formed as below
{{DFSUtil#httpServerTemplateForNNAndJN}}
{code}
URI uri = URI.create(http://; + NetUtils.getHostPortString(httpAddr));
{code}
Seems like this is reason why there is a difference in both hdfs and yarn for
*REST api functionality when IP is configured in kerberos mode*. In case of
hdfs it works but yarn its doesnt.
Can we hange RM HTTPServer2.builder as velow
{code}
HttpServer2.Builder builder =
new HttpServer2.Builder()
.setName(name)
.addEndpoint(
URI.create(httpScheme
+ NetUtils.getHostPortString(new InetSocketAddress(
bindAddress, port
.setConf(conf)
.setFindPort(findPort)
.setACL(
new AccessControlList(conf.get(
YarnConfiguration.YARN_ADMIN_ACL,
YarnConfiguration.DEFAULT_YARN_ADMIN_ACL)))
.setPathSpec(pathList.toArray(new String[0]));
{code}
Please do correct me if i am wrong .
Rest API failing when ip configured in RM address in secure https mode
--
Key: YARN-3838
URL: https://issues.apache.org/jira/browse/YARN-3838
Project: Hadoop YARN
Issue Type: Bug
Components: security
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Critical
Attachments: 0001-HADOOP-12096.patch, 0001-YARN-3810.patch,
0002-YARN-3810.patch
Steps to reproduce
===
1.Configure hadoop.http.authentication.kerberos.principal as below
{code:xml}
property
namehadoop.http.authentication.kerberos.principal/name
valueHTTP/_h...@hadoop.com/value
/property
{code}
2. In RM web address also configure IP
3. Startup RM
Call Rest API for RM {{ curl -i -k --insecure --negotiate -u : https IP
/ws/v1/cluster/info}}
*Actual*
Rest API failing
{code}
2015-06-16 19:03:49,845 DEBUG
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
Authentication exception: GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos credentails)
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos credentails)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:399)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:519)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
