let me input some hints information: for hadoop, you can see here about proxy_user concepts: https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/Superusers.html
for lunch spark job in mesos cluster, see this case: https://marc.info/?l=mesos-user&m=144228174927503&w=3 2016-11-13 0:17 GMT+08:00 Yu Wei <[email protected]>: > A little interesting. > > > ________________________________ > From: tommy xiao <[email protected]> > Sent: Friday, November 11, 2016 2:19 PM > To: dev > Subject: Re: Failed to launch spark jobs on mesos due to "hadoop" not found > > this is pain. if the mesos support proxy user feature like hadoop,then > every thing can easy handle. > > 2016-11-11 14:06 GMT+08:00 Yu Wei <[email protected]>: > > > I fixed the problem by adding environment variables HADOOP_HOME and > > JAVA_HOME to launch mesos-agent as below, > > > > sudo HADOOP_HOME=/home/dcos/repo/bigdata/hadoop-2.7.3/ > > JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15. > el7_2.x86_64/jre/ > > /usr/local/sbin/mesos-agent --work_dir=/var/lib/mesos/agent ... > > > > It seemed that this is caused by sudo secure_path. > > > > > > Is there any better solution to fix this problem? > > > > > > > > Thanks, > > > > Jared, (??) > > Software developer > > Interested in open source software, big data, Linux > > > > ________________________________ > > From: tommy xiao <[email protected]> > > Sent: Friday, November 11, 2016 1:52:54 PM > > To: dev > > Subject: Re: Failed to launch spark jobs on mesos due to "hadoop" not > found > > > > it seem like in same user mode. > > > > 2016-11-10 22:37 GMT+08:00 Joris Van Remoortere <[email protected]>: > > > > > Switch to the user that the agent is running as, and the directory from > > > which it is executing and ensure that you can find hadoop. For example > by > > > running > > > `which hadoop`. This may be a PATH or JAVA_HOME issue? > > > > > > - > > > *Joris Van Remoortere* > > > Mesosphere > > > > > > On Thu, Nov 10, 2016 at 6:29 AM, Yu Wei <[email protected]> wrote: > > > > > > > Hi Guys, > > > > > > > > I failed to launch spark jobs on mesos. Actually I submitted the job > to > > > > cluster successfully. > > > > > > > > But the job failed to run. > > > > > > > > I1110 18:25:11.095507 301 fetcher.cpp:498] Fetcher Info: > > > > {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/ > > > 1f8e621b-3cbf-4b86-a1c1- > > > > 9e2cf77265ee-S7\/root","items":[{"action":"BYPASS_CACHE"," > > > > uri":{"extract":true,"value":"hdfs:\/\/192.168.111.74:9090\/ > > > > bigdata\/package\/spark-examples_2.11-2.0.1.jar"}}]," > > > > sandbox_directory":"\/var\/lib\/mesos\/agent\/slaves\/ > > > > 1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-S7\/frameworks\/ > > > > 1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-0002\/executors\/ > > > > driver-20161110182510-0001\/runs\/b561328e-9110-4583-b740- > > > > 98f9653e7fc2","user":"root"} > > > > I1110 18:25:11.099799 301 fetcher.cpp:409] Fetching URI 'hdfs:// > > > > 192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar' > > > > I1110 18:25:11.099820 301 fetcher.cpp:250] Fetching directly into > the > > > > sandbox directory > > > > I1110 18:25:11.099862 301 fetcher.cpp:187] Fetching URI 'hdfs:// > > > > 192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar' > > > > E1110 18:25:11.101842 301 shell.hpp:106] Command 'hadoop version > > 2>&1' > > > > failed; this is the output: > > > > sh: hadoop: command not found > > > > Failed to fetch 'hdfs://192.168.111.74:9090/bigdata/package/spark- > > > > examples_2.11-2.0.1.jar': Failed to create HDFS client: Failed to > > execute > > > > 'hadoop version 2>&1'; the command was either not found or exited > with > > a > > > > non-zero exit status: 127 > > > > Failed to synchronize with agent (it's probably exited > > > > > > > > > > > > Actually I installed hadoop on each agent node. > > > > > > > > > > > > Any advice? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Jared, (??) > > > > Software developer > > > > Interested in open source software, big data, Linux > > > > > > > > > > > > > > > -- > > Deshi Xiao > > Twitter: xds2000 > > E-mail: xiaods(AT)gmail.com > > > > > > -- > Deshi Xiao > Twitter: xds2000 > E-mail: xiaods(AT)gmail.com > -- Deshi Xiao Twitter: xds2000 E-mail: xiaods(AT)gmail.com
