On Tue, Nov 1, 2016 at 5:01 PM, Meier, Caleb <[email protected]> wrote: > I'm using fluo-incubating. I switched "io.fluo.log" to "fluo.log" and my > worker and oracle logs appearing in my container directories. Still working > on getting my observers to log to my worker logs though.
So your observers are using slf4j? I will be at #fluo in IRC on freenode for a bit, if you want to chat there. > > -----Original Message----- > From: Mike Walch [mailto:[email protected]] > Sent: Tuesday, November 01, 2016 5:00 PM > To: [email protected] > Subject: Re: debugging fluo > > Which version of Fluo are you using? The properties 'io.fluo.log.*' were > used before Apache incubation in 1.0.0-beta-1 or 1.0.0-beta-2. These > properties were renamed in Fluo 1.0.0 to 'fluo.log.*'. You could be having > problems if you built your application using Fluo 1.0.0-beta-2 and ran it > using a Fluo 1.0.0 install. > > On Tue, Nov 1, 2016 at 4:45 PM Meier, Caleb <[email protected]> wrote: > >> Hey Keith, >> >> Not seeing the worker logs in my container. Think my logback is >> configured incorrectly... >> >> [root@c190sv193 container_1476563020088_0069_01_000002]# grep Rolling >> stdout >> 18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction >> - About to instantiate appender of type >> [ch.qos.logback.core.rolling.RollingFileAppender] >> 18:36:49,215 |-INFO in >> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No >> compression will be used >> 18:36:49,230 |-INFO in >> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log >> file >> name: >> io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log. >> host_IS_UNDEFINED.log >> 18:36:49,230 |-INFO in >> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property >> is set to >> [io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log >> .host_IS_UNDEFINED.log] >> >> -----Original Message----- >> From: Keith Turner [mailto:[email protected]] >> Sent: Tuesday, November 01, 2016 4:33 PM >> To: [email protected] >> Subject: Re: debugging fluo >> >> Caleb, >> >> I just ran Fluo locally with Uno. Below is some info I am seeing. >> Do you see anything about RollingFileAppender in the stdout file? >> >> $ pwd >> >> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/con >> tainer_1478030941494_0001_01_000004 >> $ ls >> stderr stdout worker_1_host1.log >> $ grep Rolling stdout >> 16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction >> - About to instantiate appender of type >> [ch.qos.logback.core.rolling.RollingFileAppender] >> 16:17:31,574 |-INFO in >> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No >> compression will be used >> 16:17:31,583 |-INFO in >> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log >> file >> name: >> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/con >> tainer_1478030941494_0001_01_000004/worker_1_host1 >> 16:17:31,583 |-INFO in >> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property >> is set to >> [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/co >> ntainer_1478030941494_0001_01_000004/worker_1_host1] >> >> Keith >> >> On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb <[email protected]> >> wrote: >> > Hey Mike, >> > >> > So I’m not seeing any worker logs on my machines. The search >> > >> > find / | grep worker >> > >> > yielded nothing. Any ideas as to why these don't exist? I'm not >> > sure that my fluo.log.dir system property is successfully being set >> > by the >> LogbackUtil class. >> > Maybe this has something to do with it? >> > >> > >> > >> > >> > >> > -----Original Message----- >> > From: Mike Walch [mailto:[email protected]] >> > Sent: Tuesday, November 01, 2016 4:03 PM >> > To: [email protected] >> > Subject: Re: debugging fluo >> > >> > Were you able to find a worker_*.log file for each of your workers? >> > >> > Below are some tips for debugging: >> > >> > - Each YARN container should have a 'stdout' and 'stderr' file. >> > These >> files may have helpful error messages. Especially if a worker failed >> to start. Also, any calls to System.out and System.err in your >> observer will be printed to these files. >> > - When running Fluo in YARN, Fluo must use Logback for logging (due >> > to a >> hard requirment by Twill). Logback is configured using >> /path/to/fluo/conf/logback.xml. You should review this configuration >> but the root logger is configured by default to print any message that >> is the debug level or higher. >> > - If you configured multiple workers, each worker will run in a >> different container and have a different worker_*.log file. >> > - When a worker starts up, it prints its configuration to worker_*.log. >> > Make sure that you configured your observers using the property >> 'fluo.observer.*' >> > >> > -Mike >> > >> > On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb >> > <[email protected]> >> wrote: >> > >> >> Do you have any tips for how to make Observers log to the log files >> >> found in the directory specified by 'yarn.nodemanager.log-dirs'? >> >> >> >> -----Original Message----- >> >> From: Mike Walch [mailto:[email protected]] >> >> Sent: Tuesday, November 01, 2016 2:36 PM >> >> To: [email protected] >> >> Subject: Re: debugging fluo >> >> >> >> Hi Caleb, >> >> >> >> The logs for a Fluo application can be found in YARN but they are >> >> tricky to find. Fluo should have better documentation on this which >> >> I >> will add now. >> >> >> >> The easiest way to view the logs for a Fluo application is to use >> >> the web interface for the YARN resource manager ( >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A808 >> >> 8_ >> >> c >> >> luster&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuV >> >> dz >> >> Y >> >> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeE >> >> Kk >> >> 6 mq5Tns5Aewd2s&s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY&e= >> >> ). >> >> First, click on the application ID (i.e application_*) of your Fluo >> >> application and then click on the latest attempt ID (appattempt_*). >> >> You should see a list of containers. There should be a container >> >> for the application master (typically container 1), a Fluo oracle >> >> (typically container 2), and Fluo workers (containers 3+). You can >> >> view the log files produced by a container by clicking on its 'logs' >> >> link. Logs from Fluo observers will be in the worker_*.log file >> >> for each of your worker containers. >> >> >> >> If you don't want to use the YARN resource manager web interface, >> >> you can also view these logs in the directory specified by >> >> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config. This >> >> method works well on one machine but on cluster your containers >> >> will probably be on different machines. See the YARN documentation >> >> below for more info about this property: >> >> >> >> >> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache. >> >> or >> >> g >> >> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xm >> >> l& >> >> d >> >> =CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kks >> >> VZ >> >> R >> >> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tn >> >> s5 A ewd2s&s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI&e= >> >> >> >> Best, >> >> Mike >> >> >> >> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb >> >> <[email protected]> >> >> wrote: >> >> >> >> Hello, >> >> >> >> I'm attempting to debug a Fluo application and am having difficulty >> >> locating the logs for my observers. I've looked within the logs >> >> for hadoop-yarn, but am not seeing any logging statements for my >> >> observers. >> >> Where do observers log out of the box in a normal cloudera distribution? >> >> Do I need to do something else in addition to logging to get my >> >> observers to generate logs? >> >> >> >> Thanks, >> >> Caleb >> >> >>
