[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108978#comment-16108978
]
Eric Badger commented on YARN-6494:
-----------------------------------
bq. At a minimum, we'd need the ability to turn this off for containers that
don't need the HDFS socket, but I feel that it would be better to have a more
holistic approach, which is what I hope YARN-5534 can become. The reason I
believe we shouldn't hard code mounts is that not every container will require
that mount.
I generally agree with this approach, but adding the volume whitelist doesn't
fix the potential security issue. It only sort of mitigates it. If the
administrator allows for the socket to be in the whitelist, then any container
can ask for it. So yes, existing containers and/or containers that the attacker
does not control upon startup will not have this bind-mounted in. However, if
the attacker is the one submitting the job, they'll just ask for the socket to
be bind-mounted and will be granted that request. Basically what I'm trying to
get at is that if the administrator allows short-circuit reads, they are taking
the potential security risk. At that point, I'm not sure if it matters whether
all containers have the socket or just the ones that asked for it, especially
when the attacker can explicitly ask for it.
I think both points can be resolved by letting the administrator decide their
destiny here. We can use YARN-5534 to create a whitelist of volumes that the
jobs can specify. Then, as we touched upon in [this
comment|https://issues.apache.org/jira/browse/YARN-5534?focusedCommentId=16093026&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16093026]
in YARN-5534, we can create a default bind list, which is empty by default. If
the administrator wants the short-circuit socket for all containers, they can
add it to the default list. If they only want it for certain containers, they
can add it to the whitelist and let users ask for it.
> add mounting of HDFS Short-Circuit path for docker containers
> -------------------------------------------------------------
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Reporter: Jaeboo Jeong
> Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxxxxxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
