Hey Mike, So I’m not seeing any worker logs on my machines. The search
find / | grep worker yielded nothing. Any ideas as to why these don't exist? I'm not sure that my fluo.log.dir system property is successfully being set by the LogbackUtil class. Maybe this has something to do with it? -----Original Message----- From: Mike Walch [mailto:[email protected]] Sent: Tuesday, November 01, 2016 4:03 PM To: [email protected] Subject: Re: debugging fluo Were you able to find a worker_*.log file for each of your workers? Below are some tips for debugging: - Each YARN container should have a 'stdout' and 'stderr' file. These files may have helpful error messages. Especially if a worker failed to start. Also, any calls to System.out and System.err in your observer will be printed to these files. - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard requirment by Twill). Logback is configured using /path/to/fluo/conf/logback.xml. You should review this configuration but the root logger is configured by default to print any message that is the debug level or higher. - If you configured multiple workers, each worker will run in a different container and have a different worker_*.log file. - When a worker starts up, it prints its configuration to worker_*.log. Make sure that you configured your observers using the property 'fluo.observer.*' -Mike On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb <[email protected]> wrote: > Do you have any tips for how to make Observers log to the log files > found in the directory specified by 'yarn.nodemanager.log-dirs'? > > -----Original Message----- > From: Mike Walch [mailto:[email protected]] > Sent: Tuesday, November 01, 2016 2:36 PM > To: [email protected] > Subject: Re: debugging fluo > > Hi Caleb, > > The logs for a Fluo application can be found in YARN but they are > tricky to find. Fluo should have better documentation on this which I will > add now. > > The easiest way to view the logs for a Fluo application is to use the > web interface for the YARN resource manager ( > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_c > luster&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzY > C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6 > mq5Tns5Aewd2s&s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY&e= > ). > First, click on the application ID (i.e application_*) of your Fluo > application and then click on the latest attempt ID (appattempt_*). > You should see a list of containers. There should be a container for > the application master (typically container 1), a Fluo oracle > (typically container 2), and Fluo workers (containers 3+). You can > view the log files produced by a container by clicking on its 'logs' > link. Logs from Fluo observers will be in the worker_*.log file for > each of your worker containers. > > If you don't want to use the YARN resource manager web interface, you > can also view these logs in the directory specified by > 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config. This > method works well on one machine but on cluster your containers will > probably be on different machines. See the YARN documentation below > for more info about this property: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org > _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml&d > =CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR > 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5A > ewd2s&s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI&e= > > Best, > Mike > > On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb <[email protected]> > wrote: > > Hello, > > I'm attempting to debug a Fluo application and am having difficulty > locating the logs for my observers. I've looked within the logs for > hadoop-yarn, but am not seeing any logging statements for my observers. > Where do observers log out of the box in a normal cloudera distribution? > Do I need to do something else in addition to logging to get my > observers to generate logs? > > Thanks, > Caleb >
