Hey Keith, Not seeing the worker logs in my container. Think my logback is configured incorrectly...
[root@c190sv193 container_1476563020088_0069_01_000002]# grep Rolling stdout 18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender] 18:36:49,215 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No compression will be used 18:36:49,230 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log 18:36:49,230 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log] -----Original Message----- From: Keith Turner [mailto:ke...@deenlo.com] Sent: Tuesday, November 01, 2016 4:33 PM To: dev@fluo.incubator.apache.org Subject: Re: debugging fluo Caleb, I just ran Fluo locally with Uno. Below is some info I am seeing. Do you see anything about RollingFileAppender in the stdout file? $ pwd /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_000004 $ ls stderr stdout worker_1_host1.log $ grep Rolling stdout 16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender] 16:17:31,574 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No compression will be used 16:17:31,583 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_000004/worker_1_host1 16:17:31,583 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_000004/worker_1_host1] Keith On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb <caleb.me...@parsons.com> wrote: > Hey Mike, > > So I’m not seeing any worker logs on my machines. The search > > find / | grep worker > > yielded nothing. Any ideas as to why these don't exist? I'm not sure > that my fluo.log.dir system property is successfully being set by the > LogbackUtil class. > Maybe this has something to do with it? > > > > > > -----Original Message----- > From: Mike Walch [mailto:mwa...@apache.org] > Sent: Tuesday, November 01, 2016 4:03 PM > To: dev@fluo.incubator.apache.org > Subject: Re: debugging fluo > > Were you able to find a worker_*.log file for each of your workers? > > Below are some tips for debugging: > > - Each YARN container should have a 'stdout' and 'stderr' file. These files > may have helpful error messages. Especially if a worker failed to start. > Also, any calls to System.out and System.err in your observer will be printed > to these files. > - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard > requirment by Twill). Logback is configured using > /path/to/fluo/conf/logback.xml. You should review this configuration but the > root logger is configured by default to print any message that is the debug > level or higher. > - If you configured multiple workers, each worker will run in a different > container and have a different worker_*.log file. > - When a worker starts up, it prints its configuration to worker_*.log. > Make sure that you configured your observers using the property > 'fluo.observer.*' > > -Mike > > On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb <caleb.me...@parsons.com> wrote: > >> Do you have any tips for how to make Observers log to the log files >> found in the directory specified by 'yarn.nodemanager.log-dirs'? >> >> -----Original Message----- >> From: Mike Walch [mailto:mwa...@apache.org] >> Sent: Tuesday, November 01, 2016 2:36 PM >> To: dev@fluo.incubator.apache.org >> Subject: Re: debugging fluo >> >> Hi Caleb, >> >> The logs for a Fluo application can be found in YARN but they are >> tricky to find. Fluo should have better documentation on this which I will >> add now. >> >> The easiest way to view the logs for a Fluo application is to use the >> web interface for the YARN resource manager ( >> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_ >> c >> luster&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdz >> Y >> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk >> 6 mq5Tns5Aewd2s&s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY&e= >> ). >> First, click on the application ID (i.e application_*) of your Fluo >> application and then click on the latest attempt ID (appattempt_*). >> You should see a list of containers. There should be a container for >> the application master (typically container 1), a Fluo oracle >> (typically container 2), and Fluo workers (containers 3+). You can >> view the log files produced by a container by clicking on its 'logs' >> link. Logs from Fluo observers will be in the worker_*.log file for >> each of your worker containers. >> >> If you don't want to use the YARN resource manager web interface, you >> can also view these logs in the directory specified by >> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config. This >> method works well on one machine but on cluster your containers will >> probably be on different machines. See the YARN documentation below >> for more info about this property: >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.or >> g >> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml& >> d >> =CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kksVZ >> R >> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5 >> A ewd2s&s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI&e= >> >> Best, >> Mike >> >> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb <caleb.me...@parsons.com> >> wrote: >> >> Hello, >> >> I'm attempting to debug a Fluo application and am having difficulty >> locating the logs for my observers. I've looked within the logs for >> hadoop-yarn, but am not seeing any logging statements for my observers. >> Where do observers log out of the box in a normal cloudera distribution? >> Do I need to do something else in addition to logging to get my >> observers to generate logs? >> >> Thanks, >> Caleb >>
