[
https://issues.apache.org/jira/browse/YARN-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14030643#comment-14030643
]
Robert Joseph Evans commented on YARN-2140:
-------------------------------------------
We are working on similar things for storm. I am very interested in your
design, because for any streaming system to truly have a chance on YARN soft
guarantees on network I/O are critical. There are several big problems with
network I/O even if the user can effectively estimate what they will need. The
first is that the resource is not limited to a single node in the cluster. The
network has a topology and a bottlekneck can show up at any point in that
topology. So you may think you are fine because each node in a rack is not
scheduled to be using the full bandwidth that the network card(s) can support.
But you can easily have saturated the top of rack switch without knowing it.
To solve this problem you effectively have to know the topology of the
application itself. So that you can schedule the node to node network
connections within that application. if users don't know how much network they
are going to use at a high level, they will never have any idea at a low level.
But then you also have the big problem of batch being very bursty in its
network usage. The only way to solve this is going to require network hardware
support for prioritizing packets.
But I'll wait for your design before writing too much more.
> Add support for network IO isolation/scheduling for containers
> --------------------------------------------------------------
>
> Key: YARN-2140
> URL: https://issues.apache.org/jira/browse/YARN-2140
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Wei Yan
> Assignee: Wei Yan
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
