[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-713: - Attachment: YARN-713.6.patch New patch with minor more fix > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Jian He >Priority: Critical > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.1.patch, YARN-713.2.patch, YARN-713.20130910.1.patch, > YARN-713.3.patch, YARN-713.4.patch, YARN-713.5.patch, YARN-713.6.patch, > YARN-713.patch, YARN-713.patch, YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-713: - Attachment: YARN-713.5.patch > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Jian He >Priority: Critical > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.1.patch, YARN-713.2.patch, YARN-713.20130910.1.patch, > YARN-713.3.patch, YARN-713.4.patch, YARN-713.5.patch, YARN-713.patch, > YARN-713.patch, YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-713: - Attachment: YARN-713.4.patch New patch fixed the comments and added a new unit test for testing AM container retry. > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Jian He >Priority: Critical > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.1.patch, YARN-713.2.patch, YARN-713.20130910.1.patch, > YARN-713.3.patch, YARN-713.4.patch, YARN-713.patch, YARN-713.patch, > YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-713: - Attachment: YARN-713.3.patch New patch: - Move the creation of NMToken to the same place where container token is created. - Catch the exception and return empty allocation for that container if either container token or NMToken creation fails because of DNS unavailable. - Add a new field nmTokens in Allocation.java - Change AMContainerAllocatedTransition to retry if the am container is not fetchable because token creation fails. > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Jian He >Priority: Critical > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.1.patch, YARN-713.2.patch, YARN-713.20130910.1.patch, > YARN-713.3.patch, YARN-713.patch, YARN-713.patch, YARN-713.patch, > YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-713: Target Version/s: 2.3.0 > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi >Priority: Critical > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.1.patch, YARN-713.2.patch, YARN-713.20130910.1.patch, > YARN-713.patch, YARN-713.patch, YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-713: --- Attachment: YARN-713.2.patch > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi >Priority: Critical > Fix For: 2.3.0 > > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.1.patch, YARN-713.2.patch, YARN-713.20130910.1.patch, > YARN-713.patch, YARN-713.patch, YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-713: --- Attachment: YARN-713.1.patch > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi >Priority: Critical > Fix For: 2.3.0 > > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.1.patch, YARN-713.20130910.1.patch, YARN-713.patch, YARN-713.patch, > YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-713: --- Attachment: YARN-713.20130910.1.patch > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi >Priority: Critical > Fix For: 2.3.0 > > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.20130910.1.patch, YARN-713.patch, YARN-713.patch, YARN-713.patch, > YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-713: --- Attachment: YARN-713.09062013.1.patch > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi >Priority: Critical > Fix For: 2.3.0 > > Attachments: YARN-713.09052013.1.patch, YARN-713.09062013.1.patch, > YARN-713.patch, YARN-713.patch, YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-713: --- Attachment: YARN-713.09052013.1.patch > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi >Priority: Critical > Fix For: 2.3.0 > > Attachments: YARN-713.09052013.1.patch, YARN-713.patch, > YARN-713.patch, YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-713: --- Fix Version/s: (was: 2.1.0-beta) 2.3.0 > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi >Priority: Critical > Fix For: 2.3.0 > > Attachments: YARN-713.patch, YARN-713.patch, YARN-713.patch, > YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maysam Yabandeh updated YARN-713: - Attachment: YARN-713.patch The attached patch is updated with latest trunk changes. > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Maysam Yabandeh >Priority: Critical > Fix For: 2.1.0-beta > > Attachments: YARN-713.patch, YARN-713.patch, YARN-713.patch, > YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maysam Yabandeh updated YARN-713: - Attachment: YARN-713.patch The attached patch applied the comments of [~vinodkv]. The unit test is also moved to resourcemanager package close to the modules that are modified. Reviews are appreciated. > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Maysam Yabandeh >Priority: Critical > Fix For: 2.1.0-beta > > Attachments: YARN-713.patch, YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated YARN-713: --- Assignee: Maysam Yabandeh > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Assignee: Maysam Yabandeh >Priority: Critical > Attachments: YARN-713.patch, YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[
https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maysam Yabandeh updated YARN-713:
-
Attachment: YARN-713.patch
In the attached patch, the exception is handled in
RMContainerTokenSecretManager#createContainerToken by returning null. The null
values are supposed to trigger a try, as in FifoScheduler#assignContainer:
{code:java}
if (containerToken == null) {
return i; // Try again later.
}
{code}
Regarding the sweep of RM to find other places that a DNS failure should be
handled properly, I guess a cleaner approach is to directly throw
UnknownHostException instead of hiding it in a InvalidArgumentException, which
is also semantically confusing. This however would result in widespread changes
allover the project, as each user of SecurityUtil must either handle the
exception or declare it to be caught by its callers. If this approach is fine
with you guys, I can give it a go.
> ResourceManager can exit unexpectedly if DNS is unavailable
> ---
>
> Key: YARN-713
> URL: https://issues.apache.org/jira/browse/YARN-713
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Jason Lowe
>Priority: Critical
> Attachments: YARN-713.patch, YARN-713.patch
>
>
> As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could
> lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and
> that ultimately would cause the RM to exit. The RM should not exit during
> DNS hiccups.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-713) ResourceManager can exit unexpectedly if DNS is unavailable
[ https://issues.apache.org/jira/browse/YARN-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maysam Yabandeh updated YARN-713: - Attachment: YARN-713.patch A preliminary patch is attached. Three points: # With the new changes of YARN-571, I did not find a way to write a unit test that simulates a DNS error. I am temporarily changing the visibility of setTokenServiceUserIp to allow the test go thorough. # The patch catches the IllegalArgumentException and verifies whether it is related to IP resolving. I guess a cleaner way is to directly throw the UnknownHostException at SecurityUtil.java # The current patch simply logs the exception. I am wondering whether a more complicated reaction is desired, such as recirculating the event after a timeout. In general, we should determine which component is responsible of retrying a failed event. > ResourceManager can exit unexpectedly if DNS is unavailable > --- > > Key: YARN-713 > URL: https://issues.apache.org/jira/browse/YARN-713 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Critical > Attachments: YARN-713.patch > > > As discussed in MAPREDUCE-5261, there's a possibility that a DNS outage could > lead to an unhandled exception in the ResourceManager's AsyncDispatcher, and > that ultimately would cause the RM to exit. The RM should not exit during > DNS hiccups. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
