Were you able to find a worker_*.log file for each of your workers? Below are some tips for debugging:
- Each YARN container should have a 'stdout' and 'stderr' file. These files may have helpful error messages. Especially if a worker failed to start. Also, any calls to System.out and System.err in your observer will be printed to these files. - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard requirment by Twill). Logback is configured using /path/to/fluo/conf/logback.xml. You should review this configuration but the root logger is configured by default to print any message that is the debug level or higher. - If you configured multiple workers, each worker will run in a different container and have a different worker_*.log file. - When a worker starts up, it prints its configuration to worker_*.log. Make sure that you configured your observers using the property 'fluo.observer.*' -Mike On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb <[email protected]> wrote: > Do you have any tips for how to make Observers log to the log files found > in the directory specified by 'yarn.nodemanager.log-dirs'? > > -----Original Message----- > From: Mike Walch [mailto:[email protected]] > Sent: Tuesday, November 01, 2016 2:36 PM > To: [email protected] > Subject: Re: debugging fluo > > Hi Caleb, > > The logs for a Fluo application can be found in YARN but they are tricky > to find. Fluo should have better documentation on this which I will add now. > > The easiest way to view the logs for a Fluo application is to use the web > interface for the YARN resource manager ( > https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_cluster&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s&s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY&e= > ). > First, click on the application ID (i.e application_*) of your Fluo > application and then click on the latest attempt ID (appattempt_*). You > should see a list of containers. There should be a container for the > application master (typically container 1), a Fluo oracle (typically > container 2), and Fluo workers (containers 3+). You can view the log files > produced by a container by clicking on its 'logs' link. Logs from Fluo > observers will be in the worker_*.log file for each of your worker > containers. > > If you don't want to use the YARN resource manager web interface, you can > also view these logs in the directory specified by > 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config. This method > works well on one machine but on cluster your containers will probably be > on different machines. See the YARN documentation below for more info about > this property: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org_docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s&s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI&e= > > Best, > Mike > > On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb <[email protected]> > wrote: > > Hello, > > I'm attempting to debug a Fluo application and am having difficulty > locating the logs for my observers. I've looked within the logs for > hadoop-yarn, but am not seeing any logging statements for my observers. > Where do observers log out of the box in a normal cloudera distribution? > Do I need to do something else in addition to logging to get my observers > to generate logs? > > Thanks, > Caleb >
