Thank you very much Prabhu, Deleting the /d0 folder fixed the issue
Manuel From: Prabhu Josephraj [mailto:pjos...@cloudera.com] Sent: Thursday, October 10, 2019 6:17 PM To: Manuel Sopena Ballesteros Cc: user@hadoop.apache.org Subject: Re: can't start spark thrift after Configuring YARN container executor As per the error, spark user does not have permission to create directory under NodeManager Local Directory or the existing spark user directory is with stale uid or gid. Permission denied Can't create directory /d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 1. Check if spark user is able to create directory under NM local dir. 2. Remove /d1/hadoop/yarn/local/usercache/spark from all NMs and rerun the job. On Thu, Oct 10, 2019 at 12:13 PM Manuel Sopena Ballesteros <manuel...@garvan.org.au<mailto:manuel...@garvan.org.au>> wrote: Dear Hadoop community, I am trying to configure yarn container executor following this document https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.5/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/inst_adv_yarn_config.html I follow all the steps but after restart YARN I can’t start spark thrift server. This is the error I can see in yarn Application application_1570681803028_0018 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1570681803028_0018_000001 exited with exitCode: -1000 Failing this attempt.Diagnostics: [2019-10-10 16:49:35.322]Application application_1570681803028_0018 initialization failed (exitCode=255) with output: main : command provided 0 main : run as user is spark main : requested yarn user is spark Can't create directory /d0/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 - Permission denied Can't create directory /d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 - Permission denied Did not create any app directories For more detailed output, check the application tracking page: http://gl-hdp-ctrl03-mlx.mlx:8088/cluster/app/application_1570681803028_0018 Then click on links to logs of each attempt. . Failing the application. This is the content of container-executor.cfg [luffy@gl-hdp-ctrl01-mlx ~]$ cat /etc/hadoop/3.1.0.0-78/0/container-executor.cfg #/* # * Licensed to the Apache Software Foundation (ASF) under one # * or more contributor license agreements. See the NOTICE file # * distributed with this work for additional information # * regarding copyright ownership. The ASF licenses this file # * to you under the Apache License, Version 2.0 (the # * "License"); you may not use this file except in compliance # * with the License. You may obtain a copy of the License at # * # * http://www.apache.org/licenses/LICENSE-2.0 # * # * Unless required by applicable law or agreed to in writing, software # * distributed under the License is distributed on an "AS IS" BASIS, # * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # * See the License for the specific language governing permissions and # * limitations under the License. # */ yarn.nodemanager.local-dirs=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local yarn.nodemanager.log-dirs=/d0/hadoop/yarn/log,/d1/hadoop/yarn/log yarn.nodemanager.linux-container-executor.group=hadoop banned.users=hdfs,yarn,mapred,bin # min.user.id<http://min.user.id>=1000 min.user.id<http://min.user.id>=80 [docker] module.enabled=false docker.binary=/usr/bin/docker docker.allowed.capabilities=CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE docker.allowed.devices= docker.allowed.networks=host,none,bridge docker.allowed.ro-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local, docker.allowed.rw-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,/d0/hadoop/yarn/log,/d1/hadoop/yarn/log, docker.privileged-containers.enabled=false docker.trusted.registries= docker.allowed.volume-drivers= [gpu] module.enabled=false [cgroups] root= yarn-hierarchy= I was hopping if someone could help me troubleshooting about what YARN is trying to do and how to fix this configuration issue? Thank you very much Manuel NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed. NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
