Yes Jeff Thanks again.
I could successfully run standalone TF training application with
Tensorboard on docker container. Will definitely take care of silent ssh
once I start with Distributed TF..
On Tue, Feb 19, 2019 at 9:44 PM Jeff Hubbs wrote:
> Great, Vinay - I'm glad that made a
Great, Vinay - I'm glad that made a difference. When you get to the
point where you are running a cluster, the same sort of thing will have
to carry over to all nodes, with the added issue that ssh and keys must
be configured such that each of those users can shell to other nodes
without
Perfect Jeff, I clearly understand.
After changing the setup to the appropriate users and folder permissions, I
can see some progress..
Cheers..
On Fri, Feb 15, 2019 at 10:05 AM Jeff Hubbs wrote:
> On 2/14/19 11:09 PM, Vinay Kashyap wrote:
>
> I am running hadoop on my mac and all the folders
On 2/14/19 11:09 PM, Vinay Kashyap wrote:
I am running hadoop on my mac and all the folders have *myuser:staff*
as the owner. I have verified the permissions for the local dirs to be
755.
This doesn't sound right. By-the-book, there are supposed to be separate
"users" for hdfs, yarn, and
I am running hadoop on my mac and all the folders have *myuser:staff* as
the owner. I have verified the permissions for the local dirs to be 755.
I run all hadoop services with myuser and I have configured
*yarn.nodemanager.linux-container-executor.group**=staff *accordingly both
in
In case of Distributed Shell Job - ApplicationMaster runs in normal linux
container and the subsequent shell command runs inside Docker
container. The job fails even before launching AM, that is before starting
Docker Container. I think the Distributed Shell job will fail even
without Docker
Hi Prabhu,
Thanks for your reply.
I tried the configurations as per your suggestion. But I get the same error.
Is this related to container localization by any chance?.
Also, is there any log or out information which says that the docker
container runtime has been picked up.?
On Thu, Feb 14,
Hi Vinay,
Can you try specifying below configs under Docker section in
container-executor.cfg which will allow Docker Containers to use the NM
Local Dirs.
docker.allowed.ro-mounts=/data/yarn/local,,/usr/jdk64/jdk1.8.0_112/bin
I am using Hadoop 3.2.0 and trying to run a simple application in a docker
container and I have made the required configuration changes both in
*yarn-site.xml* and *container-executor.cfg* to choose
LinuxContainerExecutor and docker runtime.
I use the example of distributed shell in one of the
