[
https://issues.apache.org/jira/browse/YARN-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250526#comment-16250526
]
Vinod Kumar Vavilapalli commented on YARN-7468:
-----------------------------------------------
bq. This is not a request for full-scale software-defined-networking
integration into YARN.
Glad you pointed this out! Though it will be interesting to see how such an
integration will look like and what fundamental building blocks will be needed
in YARN.
bq. 1A. We would setup iptables rules statically beforehand to ensure traffic
for the various YARN agreed upon cgroup contexts, bridge devices or network
namespaces could only flow where we want; we'd do this via out-of-band
configuration management – no need for YARN to do this setup.
If these rules have to be static, they cannot be tied to specific apps, but
only to more static concepts like user-name / group-name or queue name. The NM
doesn't know the queue information, so may be we should stick to user
information.
Of course, this means user information is the same on all the machines in the
YARN cluster. This is already be a requirement in secure clusters.
bq. 2. Then, when a user submit's a job, YARN would setup the OS control
(cgroup, network namespace or the bridge interface) for those processes to
match the user's name, a queue or some other deterministic handle. (We would
use that handle for our configuration-managed matching iptables rules which
would be pre-configured.)
I think we could use the same underlying linux functionality as that of traffic
shaping to tag the traffic from containers depending on the admin specific
rules. To reuse YARN-2140, we could split the underlying related
container-executor functionality into some sort of a networking module similar
to what YARN-6852 did with GPU module (but not cgroups - that part still
remains to be cleaned up).
> Provide means for container network policy control
> --------------------------------------------------
>
> Key: YARN-7468
> URL: https://issues.apache.org/jira/browse/YARN-7468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Reporter: Clay B.
> Priority: Minor
>
> To prevent data exfiltration from a YARN cluster, it would be very helpful to
> have "firewall" rules able to map to a user/queue's containers.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
