So the master/slave attach the log file when initializing. Therefore, unless you're faster than the master, it will try to attach the log file before you can create your symlink and will fail.
Can you please provide me the contents of the mesos-master.sh / mesos-slave.sh that you're running? Where did you get these, directly from trunk? Other than that, just watch for https://reviews.apache.org/r/7061/ getting submitted. On Tue, Sep 25, 2012 at 3:14 AM, praveenesh kumar <[email protected]>wrote: > Hi Benjamin, > > I changed the file permissions to 777 for my work directories and log > files. Still I don't see any server logs in webui. > Also I am using mesos-slaves.sh and mesos-master.sh command to start the > master and slave ( so I am not running binaries directly). > > Only way I can see my slave logs is my creating the links ( the thing that > I was doing before). > But I can't see master logs by anyway. I am getting the same error with and > without links. > > Regards, > Praveenesh > > > > On Tue, Sep 25, 2012 at 12:05 AM, Benjamin Mahler <[email protected] > >wrote: > > > Ah, I was confused by the asterisks in your commands. > > > > So you're running: > > $ ln -s lt-mesos-slave.INFO mesos-slave.INFO > > because your log file is named lt-mesos-slave.INFO? > > > > This suggests you're running the lt-mesos-slave binary directly, instead > of > > mesos-slave.sh. Google-glog will use basename(argv0) as the logging > > destination, so running mesos-slave.sh will produce a log file named > > mesos-slave.INFO. > > > > Regardless, I have a change that should fix this issue for you: > > https://reviews.apache.org/r/7061/ > > It's not currently in trunk, so in the interim I would suggest looking at > > permissions of the work directory you created on the master, to ensure > it's > > readable by the master process. > > > > Hope this helps! > > > > On Mon, Sep 24, 2012 at 3:32 AM, praveenesh kumar <[email protected] > > >wrote: > > > > > Hi Benjamin, > > > > > > So I did, what you suggest: > > > > > > 1. -wipe your existing symbolic links, etc > > > -remove the ln commands from your workflow > > > -run again, and see if the webui master LOG works, if not please > > attach > > > the master logs and the commands you ran > > > > > > 2. I can't still see the webui master Logs...On web UI, I am seeing > > "Failed > > > to initialize .. retrying" > > > > > > Error : > > > E0924 05:10:30.077483 10025 files.cpp:92] Error attaching path > > > '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to > > > canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO into > an > > > absolute path > > > > > > > > > Its the same error as I was getting before. > > > > > > 3. I deleted the links I created for slave nodes also. When I start > slave > > > nodes, now I am not able to see the slave node logs also. > > > > > > * tail: cannot open > > > `/usr/local/mesos-spark/mesos/work/mesos-slave.INFO' for reading: No > such > > > file or directory* > > > > > > Slave error logs -- > > > > > > E0924 05:21:00.418911 11332 files.cpp:92] Error attaching path > > > '/usr/local/mesos-spark/mesos/work/mesos-slave.INFO': Failed to > > > canonicalize /usr/local/mesos-spark/mesos/work/mesos-slave.INFO into an > > > absolute path > > > > > > E0924 05:21:00.525163 11332 slave.cpp:362] Failed to attach log file: > > > Failed to canonicalize > /usr/local/mesos-spark/mesos/work/mesos-slave.INFO > > > into an absolute path > > > > > > > > > After reading the above errors only, I created those symlinks and it > > > allowed me to see slave logs, so I thought of doing the same for master > > > also, but couldn't succeed. > > > > > > Thanks for any help. > > > > > > Regards, > > > Praveenesh > > > > > > > > > On Fri, Sep 21, 2012 at 10:40 PM, Benjamin Mahler <[email protected] > > > >wrote: > > > > > > > 1. I would suggest looking into the 'daemon' linux command man page. > > > > > > > > 3. What are you trying to accomplish with the symbolic links? They > > appear > > > > to conflict with the logging files google-glog will create. The error > > > from > > > > the master log indicates that the log files cannot be attached (and > > hence > > > > viewing logs from the webui won't work). > > > > > > > > I would suggest: > > > > -wipe your existing symbolic links, etc > > > > -remove the ln commands from your workflow > > > > -run again, and see if the webui master LOG works, if not please > > attach > > > > the master logs and the commands you ran > > > > > > > > Hope this helps! > > > > > > > > On Thu, Sep 20, 2012 at 11:42 PM, praveenesh kumar < > > [email protected] > > > > >wrote: > > > > > > > > > Thanks Benjamin. > > > > > > > > > > 1. I was wondering how can I run mesos-master and slaves as daemon > > > > process. > > > > > Nohup is giving me issues to launch them from remotely. > > > > > > > > > > 3. I have build mesos from trunk and starting meso master and slave > > by > > > > --- > > > > > > > > > > Mesos-master > > > > > > > > > > *nohup /usr/local/mesos-spark/mesos/bin/mesos-master.sh > > > > > --log_dir=/usr/local/mesos-spark/mesos/work >> > /tmp/mesos-master.txt > > &* > > > > > * > > > > > * > > > > > *ln -s lt-mesos-master.INFO mesos-master.INFO* > > > > > *ln -s lt-mesos-master.ERROR mesos-master.ERROR* > > > > > *ln -s lt-mesos-master.WARNING mesos-master.WARNING* > > > > > > > > > > > > > > > > > > > > Mesos-slave --- (Note I need to create links to all log files to > > > enable > > > > > logging from mesos web url) > > > > > > > > > > *nohup /usr/local/mesos-spark/mesos/bin/mesos-slave.sh > > > --master=br9:5050 > > > > > --log_dir=/usr/local/mesos-spark/mesos/work >> /tmp/mesos-slave.txt > > &* > > > > > > > > > > *cd /usr/local/mesos-spark/mesos/* > > > > > *chmod 777 -R work* > > > > > *cd work* > > > > > *ln -s lt-mesos-slave.ERROR mesos-slave.ERROR* > > > > > *ln -s lt-mesos-slave.WARNING mesos-slave.WARNING* > > > > > *ln -s lt-mesos-slave.INFO mesos-slave.INFO* > > > > > > > > > > > > > > > I see only these suspicious lines in lt-mesos-master.ERROR > > > > > > > > > > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg > > > > > E0918 04:28:37.111768 28729 files.cpp:92] Error attaching path > > > > > '/usr/local/mesos-spark/mesos/work/mesos-master.INFO': Failed to > > > > > canonicalize /usr/local/mesos-spark/mesos/work/mesos-master.INFO > into > > > an > > > > > absolute path > > > > > E0918 04:28:37.112239 28731 master.cpp:535] Failed to attach log > > file: > > > > > Failed to canonicalize > > > > /usr/local/mesos-spark/mesos/work/mesos-master.INFO > > > > > into an absolute path > > > > > > > > > > Regards, > > > > > Praveenesh > > > > > > > > > > > > > > > On Wed, Sep 19, 2012 at 12:26 PM, Benjamin Mahler < > > [email protected] > > > > > >wrote: > > > > > > > > > > > 1. You could run it as a daemon, what you're doing with nohup is > > fine > > > > as > > > > > > well. > > > > > > > > > > > > 2. The webui shows various resource statistics: [image: Inline > > image > > > 2] > > > > > > > > > > > > *Total memory* indicates the total amount of memory available on > > all > > > > the > > > > > > slaves. > > > > > > Total Memory should equal Used + Idle + Offered > > > > > > * > > > > > > * > > > > > > *Used:* Memory used by the frameworks. > > > > > > *Offered:* This memory has been offered but the framework hasn't > > > > > > responded yet. > > > > > > *Idle:* Any remaining memory in the system. > > > > > > > > > > > > 3. Are you running off trunk or a release? What is the command > line > > > > > you're > > > > > > running to start the master? Can you provide the master's log > > output? > > > > > > > > > > > > On Tue, Sep 18, 2012 at 11:57 PM, praveenesh kumar < > > > > [email protected] > > > > > >wrote: > > > > > > > > > > > >> Hi all, > > > > > >> > > > > > >> I have been able to successfully build and install mesos on my > > > > physical > > > > > >> nodes. I can see the mesos web-url with list of mesos slaves. > > > However > > > > I > > > > > am > > > > > >> finding few difficulties w.r.t to following things: > > > > > >> > > > > > >> *1. How to start mesos-master.sh and mesos-slave.sh as a > > background > > > > > >> process > > > > > >> > > > > > >> ? * > > > > > >> I am seeing all my output on the console. Since I want to > remotely > > > > start > > > > > >> the service on the cluster, I tried using nohup. Its working for > > > now, > > > > > but > > > > > >> is it the right way? I am hoping there should be better way to > do > > > > that. > > > > > >> > > > > > >> *2.* *Whatever memory I am seeing on the mesos-web url, is it > the > > > > total > > > > > >> memory or the free memory available on my system ?* > > > > > >> > > > > > >> I am seeing very low memory (0.001 GB) -- Does it indicate the > > > current > > > > > >> available memory ? > > > > > >> > > > > > >> 3. I have enable the "log_dir" directory, while starting the > > > > > mesos-master > > > > > >> and slaves. I can see the logs for my slave machines from mesos > > web > > > > url, > > > > > >> but not able to see logs for my master. Its saying *"Failed to > > > > > Initialize > > > > > >> .... Retrying"*. I can see work directory is there as well as > the > > > logs > > > > > are > > > > > >> > > > > > >> also there. Set the permission to 777 also. Still no success. > > > > > >> > > > > > >> > > > > > >> Any help on the above to understand mesos, would be very > helpful. > > > > > >> > > > > > >> Regards, > > > > > >> Praveenesh > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
