[
https://issues.apache.org/jira/browse/YARN-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107631#comment-16107631
]
Shane Kumpf commented on YARN-6494:
-----------------------------------
{quote}
However I thought this case is different from ordinary file mounts. I think
that the path for the short circuit is the configuration of the cluster, and
the cluster user doesn't need to consider about that path. I just think it's
just part of the cluster setup.
{quote}
Thanks for the explanation [~Jaeboo]. I do understand your point, however I'm
still not sure I agree with the idea of hard coded/automatic bind mounts in
code. IMO we need to be very, very careful about what we mount from the host
and an opt-in model puts the responsibility on administrators to at least
consider the impact of adding the mount to the whitelist. At a minimum, we'd
need the ability to turn this off for containers that don't need the HDFS
socket, but I feel that it would be better to have a more holistic approach,
which is what I hope YARN-5534 can become. The reason I believe we shouldn't
hard code mounts is that not every container will require that mount. Let's say
I'm running web servers in containers on YARN that don't use or need HDFS,
having this socket mounted exposes HDFS when it is not needed. It's also
unclear to me if a malicious user could leverage this socket to exploit the
system. I hope that helps explain my hesitation here.
> add mounting of HDFS Short-Circuit path for docker containers
> -------------------------------------------------------------
>
> Key: YARN-6494
> URL: https://issues.apache.org/jira/browse/YARN-6494
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Reporter: Jaeboo Jeong
> Assignee: Jaeboo Jeong
> Attachments: YARN-6494.001.patch, YARN-6494.002.patch
>
>
> Currently there is a error message about HDFS short-circuit when docker
> container start.
> {code}
> WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: error
> creating DomainSocket
> java.net.ConnectException: connect(2) error: No such file or directory when
> trying to connect to ‘xxxxxxx’
> at org.apache.hadoop.net.unix.DomainSocket.connect0(Native Method)
> at org.apache.hadoop.net.unix.DomainSocket.connect(DomainSocket.java:250)
> at
> org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory.createSocket(DomainSocketFactory.java:164)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.nextDomainPeer(BlockReaderFactory.java:752)
> ...
> {code}
> if dfs.client.read.shortcircuit is true and dfs.domain.socket.path isn't
> equal “”, we need to mount volume for short-circuit path.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
