[
https://issues.apache.org/jira/browse/HADOOP-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252063#comment-13252063
]
Sanjay Radia commented on HADOOP-8198:
--------------------------------------
Eli, below is some feedback to improve the document/proposal and some
questions. At some point, the answers should move into the document.
* Intro
Not true that "Hadoop does not currently utilize multiple network
interfaces". - Hadoop relies on host-level bonding for multiple nics and this
does not allow use case 1.2 but addresses use case 1.1. Worth clarifying this
in the intro.
* Use case 1.1 Layer 7 bonding
** Some folks get thrown off by the "Layer 7 bonding" thinking that the
proposal is to configure switches, nic, etc to get layer 7 bonding. Add a
sentence to clarify.
** bandwidth and failover is the main motivation for this use case.
*** Q What are the limitations of host level bonding for workers and masters
for this use case (as opposed to use case 1.2)?
** Add text that worker is DN or TT and Master is NN or JT,
* Use case 1.2 - i understand the text but have hard time buying this use case
as a high priority. Can you please motivate this better. Can we separate this
one for phase 2 (or were you planning that anyway)?
* Use case 1.3 - you have text and filed jiras for a client config to use a
specific nic - is this a separate use case?
* Insert a section "Scope"
** Your description is general (Masters and Workers) but you limit the detailed
sections to HDFS and MR. How will things work if don't address this problem at
all layers? (see my comment below on issues/risks)
* Insert a section called "Issues/Risks"
** What if other services up the stack have not been modified and depend only
on host-level-bonding?
** Minimize changes to DfsCLient so it can remain as thin as possible for
future client side porting to other languages. (There are solutions that do not
require changes to the dfsClient that we should consider in the short term).
* 2 Requirements
** some of the requirements seem to prescribe the solution. Separate out the
solutions and insert in a section between 2 and 3 describing the solution.
** 2.1 - you are not stating a requirement here -- reword - nics can be bonded
or multi-homed. Current wildcard IP config should be used for multi-homing.
State that the current multi-homing config does not help with use case
1.1 for worker nodes.
** 2.2 Multiple interfaces on master
*** I assume this is motived completely by use case 1.2 and not use case 1.1
since we don't have bandwidth issues for masters - please add this
clarification.
*** What will this do to tokens if tokens are obtained from one of the several
interfaces on a Master.
** 2.5 Enable clients to use multiple local interfaces - what is the motivation
for this? Is this for tasks running inside Hadoop or for clients outside the
hadoop cluster. (or is this a yet another use case? If so add to use cases
section.)
** Add requirements:
*** old configs should run unchanged
*** host level bonding should continue to work
*** security should work for the proposed enhancements
*** solution should work for all protocols - hadoop native, http rpc, etc. (if
you are excluding any please clarify).
*** should work with HA
* 3.1 Example Config ...
** This section could be significantly improved. You switch between "current
config" and "how the current config's semantics are extended" and "new
configuration"; this is sometimes confusing.
* 4.3 Security
** looks like the paragraph is not complete - last sentence is cut off.
** Also added some details on what change are needed for this to work with
security. Part of the paragraph reads like requirements statements and should
move to the requirements section.
* 4.3 HA
** Add to requirements as noted above.
** Are there new issues here? Will have to failover multiple IPs of the NN etc.
> Support multiple network interfaces
> -----------------------------------
>
> Key: HADOOP-8198
> URL: https://issues.apache.org/jira/browse/HADOOP-8198
> Project: Hadoop Common
> Issue Type: New Feature
> Components: io, performance
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: MultipleNifsv1.pdf, MultipleNifsv2.pdf,
> MultipleNifsv3.pdf
>
>
> Hadoop does not currently utilize multiple network interfaces, which is a
> common user request, and important in enterprise environments. This jira
> covers a proposal for enhancements to Hadoop so it better utilizes multiple
> network interfaces. The primary motivation being improved performance,
> performance isolation, resource utilization and fault tolerance. The attached
> design doc covers the high-level use cases, requirements, a proposal for
> trunk/0.23, discussion on related features, and a proposal for Hadoop 1.x
> that covers a subset of the functionality of the trunk/0.23 proposal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
