On Tue, Nov 1, 2016 at 5:04 PM, Keith Turner <[email protected]> wrote: > On Tue, Nov 1, 2016 at 5:01 PM, Meier, Caleb <[email protected]> wrote: >> I'm using fluo-incubating. I switched "io.fluo.log" to "fluo.log" and my >> worker and oracle logs appearing in my container directories. Still working >> on getting my observers to log to my worker logs though. > > So your observers are using slf4j? > > I will be at #fluo in IRC on freenode for a bit, if you want to chat there.
Just to follow up on this. We did resolve part of the issue in IRC. Mike Walch figured out that old configs with io.fluo were being used. Switching that fixed some of the problems. Caleb is still not seeing output from a logger created by an observer, we have not figured this one out yet. > >> >> -----Original Message----- >> From: Mike Walch [mailto:[email protected]] >> Sent: Tuesday, November 01, 2016 5:00 PM >> To: [email protected] >> Subject: Re: debugging fluo >> >> Which version of Fluo are you using? The properties 'io.fluo.log.*' were >> used before Apache incubation in 1.0.0-beta-1 or 1.0.0-beta-2. These >> properties were renamed in Fluo 1.0.0 to 'fluo.log.*'. You could be having >> problems if you built your application using Fluo 1.0.0-beta-2 and ran it >> using a Fluo 1.0.0 install. >> >> On Tue, Nov 1, 2016 at 4:45 PM Meier, Caleb <[email protected]> wrote: >> >>> Hey Keith, >>> >>> Not seeing the worker logs in my container. Think my logback is >>> configured incorrectly... >>> >>> [root@c190sv193 container_1476563020088_0069_01_000002]# grep Rolling >>> stdout >>> 18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction >>> - About to instantiate appender of type >>> [ch.qos.logback.core.rolling.RollingFileAppender] >>> 18:36:49,215 |-INFO in >>> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No >>> compression will be used >>> 18:36:49,230 |-INFO in >>> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log >>> file >>> name: >>> io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log. >>> host_IS_UNDEFINED.log >>> 18:36:49,230 |-INFO in >>> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property >>> is set to >>> [io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log >>> .host_IS_UNDEFINED.log] >>> >>> -----Original Message----- >>> From: Keith Turner [mailto:[email protected]] >>> Sent: Tuesday, November 01, 2016 4:33 PM >>> To: [email protected] >>> Subject: Re: debugging fluo >>> >>> Caleb, >>> >>> I just ran Fluo locally with Uno. Below is some info I am seeing. >>> Do you see anything about RollingFileAppender in the stdout file? >>> >>> $ pwd >>> >>> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/con >>> tainer_1478030941494_0001_01_000004 >>> $ ls >>> stderr stdout worker_1_host1.log >>> $ grep Rolling stdout >>> 16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction >>> - About to instantiate appender of type >>> [ch.qos.logback.core.rolling.RollingFileAppender] >>> 16:17:31,574 |-INFO in >>> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No >>> compression will be used >>> 16:17:31,583 |-INFO in >>> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log >>> file >>> name: >>> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/con >>> tainer_1478030941494_0001_01_000004/worker_1_host1 >>> 16:17:31,583 |-INFO in >>> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property >>> is set to >>> [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/co >>> ntainer_1478030941494_0001_01_000004/worker_1_host1] >>> >>> Keith >>> >>> On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb <[email protected]> >>> wrote: >>> > Hey Mike, >>> > >>> > So I’m not seeing any worker logs on my machines. The search >>> > >>> > find / | grep worker >>> > >>> > yielded nothing. Any ideas as to why these don't exist? I'm not >>> > sure that my fluo.log.dir system property is successfully being set >>> > by the >>> LogbackUtil class. >>> > Maybe this has something to do with it? >>> > >>> > >>> > >>> > >>> > >>> > -----Original Message----- >>> > From: Mike Walch [mailto:[email protected]] >>> > Sent: Tuesday, November 01, 2016 4:03 PM >>> > To: [email protected] >>> > Subject: Re: debugging fluo >>> > >>> > Were you able to find a worker_*.log file for each of your workers? >>> > >>> > Below are some tips for debugging: >>> > >>> > - Each YARN container should have a 'stdout' and 'stderr' file. >>> > These >>> files may have helpful error messages. Especially if a worker failed >>> to start. Also, any calls to System.out and System.err in your >>> observer will be printed to these files. >>> > - When running Fluo in YARN, Fluo must use Logback for logging (due >>> > to a >>> hard requirment by Twill). Logback is configured using >>> /path/to/fluo/conf/logback.xml. You should review this configuration >>> but the root logger is configured by default to print any message that >>> is the debug level or higher. >>> > - If you configured multiple workers, each worker will run in a >>> different container and have a different worker_*.log file. >>> > - When a worker starts up, it prints its configuration to worker_*.log. >>> > Make sure that you configured your observers using the property >>> 'fluo.observer.*' >>> > >>> > -Mike >>> > >>> > On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb >>> > <[email protected]> >>> wrote: >>> > >>> >> Do you have any tips for how to make Observers log to the log files >>> >> found in the directory specified by 'yarn.nodemanager.log-dirs'? >>> >> >>> >> -----Original Message----- >>> >> From: Mike Walch [mailto:[email protected]] >>> >> Sent: Tuesday, November 01, 2016 2:36 PM >>> >> To: [email protected] >>> >> Subject: Re: debugging fluo >>> >> >>> >> Hi Caleb, >>> >> >>> >> The logs for a Fluo application can be found in YARN but they are >>> >> tricky to find. Fluo should have better documentation on this which >>> >> I >>> will add now. >>> >> >>> >> The easiest way to view the logs for a Fluo application is to use >>> >> the web interface for the YARN resource manager ( >>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A808 >>> >> 8_ >>> >> c >>> >> luster&d=CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuV >>> >> dz >>> >> Y >>> >> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeE >>> >> Kk >>> >> 6 mq5Tns5Aewd2s&s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY&e= >>> >> ). >>> >> First, click on the application ID (i.e application_*) of your Fluo >>> >> application and then click on the latest attempt ID (appattempt_*). >>> >> You should see a list of containers. There should be a container >>> >> for the application master (typically container 1), a Fluo oracle >>> >> (typically container 2), and Fluo workers (containers 3+). You can >>> >> view the log files produced by a container by clicking on its 'logs' >>> >> link. Logs from Fluo observers will be in the worker_*.log file >>> >> for each of your worker containers. >>> >> >>> >> If you don't want to use the YARN resource manager web interface, >>> >> you can also view these logs in the directory specified by >>> >> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config. This >>> >> method works well on one machine but on cluster your containers >>> >> will probably be on different machines. See the YARN documentation >>> >> below for more info about this property: >>> >> >>> >> >>> >> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache. >>> >> or >>> >> g >>> >> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xm >>> >> l& >>> >> d >>> >> =CwIBaQ&c=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10&r=vuVdzYC2kks >>> >> VZ >>> >> R >>> >> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8&m=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tn >>> >> s5 A ewd2s&s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI&e= >>> >> >>> >> Best, >>> >> Mike >>> >> >>> >> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb >>> >> <[email protected]> >>> >> wrote: >>> >> >>> >> Hello, >>> >> >>> >> I'm attempting to debug a Fluo application and am having difficulty >>> >> locating the logs for my observers. I've looked within the logs >>> >> for hadoop-yarn, but am not seeing any logging statements for my >>> >> observers. >>> >> Where do observers log out of the box in a normal cloudera distribution? >>> >> Do I need to do something else in addition to logging to get my >>> >> observers to generate logs? >>> >> >>> >> Thanks, >>> >> Caleb >>> >> >>>
