Re: Podling Report Reminder - November 2016

[Christopher]

I do think it would be helpful to label our issues with tags, like
"newbie". That's still a limitation imposed by INFRA with our preferred
choice of issue tracker.

On Tue, Nov 1, 2016 at 10:57 PM Christopher  wrote:

> I think growing our community is really the only outstanding area of
> concern (with respect to graduation) that we need to work on. However, I'm
> not sure how we can do that. One thing is that we could reach out to Rya,
> and any other developers who are using Fluo that we know about, to
> explicitly invite them (rather than wait for them to come to us).
>
> Aside from that, I'm not sure what else we can do. Maybe we could spend
> some dev cycles on packaging and blogging about how to use the software
> more (with examples) to help lower the barrier to entry and to highlight
> use cases?
>
> On Tue, Nov 1, 2016 at 10:16 PM Josh Elser  wrote:
>
> Thanks for posting, Keith. A couple of thoughts:
>
> * Growing the community is probably worth a few spots on the top-3
> things to do
> * Lots of good things are written up that Fluo has done, but nothing on
> how you intend to grow Fluo beyond the current size.
>
> Good enough to sign off, but this is the area of most importance to
> focus on now that you have a cadence for releases.
>
> - Josh
>
> Keith Turner wrote:
> > I just udpated the wiki if anyone wants to review the changes.
> >
> > On Mon, Oct 31, 2016 at 3:56 PM, Josh Elser
> wrote:
> >> I know you're already in a writing "mood", Keith.
> >>
> >> Don't forget about this one, please. Needs to be done by Wednesday this
> week
> >> (48hrs).
> >>
> >>
> >> johndam...@apache.org wrote:
> >>> Dear podling,
> >>>
> >>> This email was sent by an automated system on behalf of the Apache
> >>> Incubator PMC. It is an initial reminder to give you plenty of time to
> >>> prepare your quarterly board report.
> >>>
> >>> The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
> >>> The report for your podling will form a part of the Incubator PMC
> >>> report. The Incubator PMC requires your report to be submitted 2 weeks
> >>> before the board meeting, to allow sufficient time for review and
> >>> submission (Wed, November 02).
> >>>
> >>> Please submit your report with sufficient time to allow the Incubator
> >>> PMC, and subsequently board members to review and digest. Again, the
> >>> very latest you should submit your report is 2 weeks prior to the board
> >>> meeting.
> >>>
> >>> Thanks,
> >>>
> >>> The Apache Incubator PMC
> >>>
> >>> Submitting your Report
> >>>
> >>> --
> >>>
> >>> Your report should contain the following:
> >>>
> >>> *   Your project name
> >>> *   A brief description of your project, which assumes no knowledge of
> >>>   the project or necessarily of its field
> >>> *   A list of the three most important issues to address in the move
> >>>   towards graduation.
> >>> *   Any issues that the Incubator PMC or ASF Board might wish/need to
> be
> >>>   aware of
> >>> *   How has the community developed since the last report
> >>> *   How has the project developed since the last report.
> >>>
> >>> This should be appended to the Incubator Wiki page at:
> >>>
> >>> http://wiki.apache.org/incubator/November2016
> >>>
> >>> Note: This is manually populated. You may need to wait a little before
> >>> this page is created from a template.
> >>>
> >>> Mentors
> >>> ---
> >>>
> >>> Mentors should review reports for their project(s) and sign them off on
> >>> the Incubator wiki page. Signing off reports shows that you are
> >>> following the project - projects that are not signed may raise alarms
> >>> for the Incubator PMC.
> >>>
> >>> Incubator PMC
>
>

Re: Podling Report Reminder - November 2016

[Christopher]

I think growing our community is really the only outstanding area of
concern (with respect to graduation) that we need to work on. However, I'm
not sure how we can do that. One thing is that we could reach out to Rya,
and any other developers who are using Fluo that we know about, to
explicitly invite them (rather than wait for them to come to us).

Aside from that, I'm not sure what else we can do. Maybe we could spend
some dev cycles on packaging and blogging about how to use the software
more (with examples) to help lower the barrier to entry and to highlight
use cases?

On Tue, Nov 1, 2016 at 10:16 PM Josh Elser  wrote:

Thanks for posting, Keith. A couple of thoughts:

* Growing the community is probably worth a few spots on the top-3
things to do
* Lots of good things are written up that Fluo has done, but nothing on
how you intend to grow Fluo beyond the current size.

Good enough to sign off, but this is the area of most importance to
focus on now that you have a cadence for releases.

- Josh

Keith Turner wrote:
> I just udpated the wiki if anyone wants to review the changes.
>
> On Mon, Oct 31, 2016 at 3:56 PM, Josh Elser  wrote:
>> I know you're already in a writing "mood", Keith.
>>
>> Don't forget about this one, please. Needs to be done by Wednesday this
week
>> (48hrs).
>>
>>
>> johndam...@apache.org wrote:
>>> Dear podling,
>>>
>>> This email was sent by an automated system on behalf of the Apache
>>> Incubator PMC. It is an initial reminder to give you plenty of time to
>>> prepare your quarterly board report.
>>>
>>> The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
>>> The report for your podling will form a part of the Incubator PMC
>>> report. The Incubator PMC requires your report to be submitted 2 weeks
>>> before the board meeting, to allow sufficient time for review and
>>> submission (Wed, November 02).
>>>
>>> Please submit your report with sufficient time to allow the Incubator
>>> PMC, and subsequently board members to review and digest. Again, the
>>> very latest you should submit your report is 2 weeks prior to the board
>>> meeting.
>>>
>>> Thanks,
>>>
>>> The Apache Incubator PMC
>>>
>>> Submitting your Report
>>>
>>> --
>>>
>>> Your report should contain the following:
>>>
>>> *   Your project name
>>> *   A brief description of your project, which assumes no knowledge of
>>>   the project or necessarily of its field
>>> *   A list of the three most important issues to address in the move
>>>   towards graduation.
>>> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
>>>   aware of
>>> *   How has the community developed since the last report
>>> *   How has the project developed since the last report.
>>>
>>> This should be appended to the Incubator Wiki page at:
>>>
>>> http://wiki.apache.org/incubator/November2016
>>>
>>> Note: This is manually populated. You may need to wait a little before
>>> this page is created from a template.
>>>
>>> Mentors
>>> ---
>>>
>>> Mentors should review reports for their project(s) and sign them off on
>>> the Incubator wiki page. Signing off reports shows that you are
>>> following the project - projects that are not signed may raise alarms
>>> for the Incubator PMC.
>>>
>>> Incubator PMC

Re: Nice Report...

[Christopher]

Hi John. Thanks for the advice.

I think we basically agreed among ourselves that editing on the Wiki, with
discussion on the list, as needed, was sufficient. But, that was really
early on, and if we want to grow, it makes sense to utilize the mailing
list for more discussions in advance.

I think we left that question about issues empty because we didn't have
any. Though, I suppose the answer could be "No", for clarity. :)

On Tue, Nov 1, 2016 at 10:21 PM John D. Ament  wrote:

> Fluo Podling,
>
> Thanks for posting your report.  Looks good.  A couple of notes from my
>  side (as shepherd).
>
> - Try wherever possible to post/discuss the report on your MLs before
> posting to the incubator wiki.  Its not a rule per se, but it helps ensure
> that you have community involvement (clicking in the wiki and finding the
> report makes it harder to review)
>
> - This question is empty if you can provide an answer "Any issues that the
> Incubator PMC (IPMC) or ASF Board wish/need to be aware of?"
>
> Please don't hesitate if you have any questions, I monitor podling mailing
> lists.
>
> John
>

Nice Report...

[John D. Ament]

Fluo Podling,

Thanks for posting your report.  Looks good.  A couple of notes from my
 side (as shepherd).

- Try wherever possible to post/discuss the report on your MLs before
posting to the incubator wiki.  Its not a rule per se, but it helps ensure
that you have community involvement (clicking in the wiki and finding the
report makes it harder to review)

- This question is empty if you can provide an answer "Any issues that the
Incubator PMC (IPMC) or ASF Board wish/need to be aware of?"

Please don't hesitate if you have any questions, I monitor podling mailing
lists.

John

Re: Podling Report Reminder - November 2016

[Josh Elser]

Thanks for posting, Keith. A couple of thoughts:

* Growing the community is probably worth a few spots on the top-3 
things to do
* Lots of good things are written up that Fluo has done, but nothing on 
how you intend to grow Fluo beyond the current size.


Good enough to sign off, but this is the area of most importance to 
focus on now that you have a cadence for releases.


- Josh

Keith Turner wrote:

I just udpated the wiki if anyone wants to review the changes.

On Mon, Oct 31, 2016 at 3:56 PM, Josh Elser  wrote:

I know you're already in a writing "mood", Keith.

Don't forget about this one, please. Needs to be done by Wednesday this week
(48hrs).


johndam...@apache.org wrote:

Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, November 02).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
  the project or necessarily of its field
*   A list of the three most important issues to address in the move
  towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
  aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.

This should be appended to the Incubator Wiki page at:

http://wiki.apache.org/incubator/November2016

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

Podling Report Reminder - November 2016

[johndament]

Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, November 02).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.

This should be appended to the Incubator Wiki page at:

http://wiki.apache.org/incubator/November2016

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

Re: debugging fluo

[Keith Turner]

I suppose you don't see oracle files either?

$ pwd
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001
$ find .
.
./container_1478030941494_0001_01_01
./container_1478030941494_0001_01_01/stdout
./container_1478030941494_0001_01_01/stderr
./container_1478030941494_0001_01_02
./container_1478030941494_0001_01_02/stdout
./container_1478030941494_0001_01_02/stderr
./container_1478030941494_0001_01_02/oracle_0_host1
./container_1478030941494_0001_01_03
./container_1478030941494_0001_01_03/stdout
./container_1478030941494_0001_01_03/stderr
./container_1478030941494_0001_01_03/worker_0_host1
./container_1478030941494_0001_01_04
./container_1478030941494_0001_01_04/stdout
./container_1478030941494_0001_01_04/stderr
./container_1478030941494_0001_01_04/worker_1_host1

On Tue, Nov 1, 2016 at 4:40 PM, Meier, Caleb  wrote:
> Hey Keith,
>
> Not seeing the worker logs in my container.  Think my logback is configured 
> incorrectly...
>
> [root@c190sv193 container_1476563020088_0069_01_02]# grep Rolling stdout
> 18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - 
> About to instantiate appender of type 
> [ch.qos.logback.core.rolling.RollingFileAppender]
> 18:36:49,215 |-INFO in 
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No 
> compression will be used
> 18:36:49,230 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] 
> - Active log file name: 
> io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log
> 18:36:49,230 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] 
> - File property is set to 
> [io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log]
>
> -Original Message-
> From: Keith Turner [mailto:ke...@deenlo.com]
> Sent: Tuesday, November 01, 2016 4:33 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Caleb,
>
> I just ran Fluo locally with Uno.  Below is some info I am seeing.
> Do you see anything about RollingFileAppender in the stdout file?
>
> $ pwd
> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
> $ ls
> stderr  stdout  worker_1_host1.log
> $ grep Rolling stdout
> 16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
> - About to instantiate appender of type
> [ch.qos.logback.core.rolling.RollingFileAppender]
> 16:17:31,574 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No compression 
> will be used
> 16:17:31,583 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: 
> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1
> 16:17:31,583 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set 
> to 
> [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1]
>
> Keith
>
> On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb  wrote:
>> Hey Mike,
>>
>> So I’m not seeing any worker logs on my machines.  The search
>>
>> find / | grep worker
>>
>> yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
>> that my fluo.log.dir system property is successfully being set by the 
>> LogbackUtil class.
>> Maybe this has something to do with it?
>>
>>
>>
>>
>>
>> -Original Message-
>> From: Mike Walch [mailto:mwa...@apache.org]
>> Sent: Tuesday, November 01, 2016 4:03 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>>
>> Were you able to find a worker_*.log file for each of your workers?
>>
>> Below are some tips for debugging:
>>
>> - Each YARN container should have a 'stdout' and 'stderr' file.  These files 
>> may have helpful error messages.  Especially if a worker failed to start.  
>> Also, any calls to System.out and System.err in your observer will be 
>> printed to these files.
>> - When running Fluo in YARN, Fluo must use Logback for logging (due to a 
>> hard requirment by Twill). Logback is configured using 
>> /path/to/fluo/conf/logback.xml.  You should review this configuration but 
>> the root logger is configured by default to print any message that is the 
>> debug level or higher.
>> - If you configured multiple workers, each worker will run in a different 
>> container and have a different worker_*.log file.
>> - When a worker starts up, it prints its configuration to worker_*.log.
>> Make sure that you configured your observers using the property 
>> 'fluo.observer.*'
>>
>> -Mike
>>
>> On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb  wrote:
>>
>>> Do you have any tips for how to make Observers log to the log files
>>> found in the directory specified by 'yarn.nodemanager.log-dirs'?
>>>
>>> -Original Message-
>>> From: Mike Walch 

Re: debugging fluo

[Mike Walch]

Which version of Fluo are you using?  The properties 'io.fluo.log.*' were
used before Apache incubation in 1.0.0-beta-1 or 1.0.0-beta-2.  These
properties were renamed in Fluo 1.0.0 to 'fluo.log.*'.  You could be having
problems if you built your application using Fluo 1.0.0-beta-2 and ran it
using a Fluo 1.0.0 install.

On Tue, Nov 1, 2016 at 4:45 PM Meier, Caleb  wrote:

> Hey Keith,
>
> Not seeing the worker logs in my container.  Think my logback is
> configured incorrectly...
>
> [root@c190sv193 container_1476563020088_0069_01_02]# grep Rolling
> stdout
> 18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction -
> About to instantiate appender of type
> [ch.qos.logback.core.rolling.RollingFileAppender]
> 18:36:49,215 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No
> compression will be used
> 18:36:49,230 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file
> name:
> io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log
> 18:36:49,230 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is
> set to
> [io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log]
>
> -Original Message-
> From: Keith Turner [mailto:ke...@deenlo.com]
> Sent: Tuesday, November 01, 2016 4:33 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Caleb,
>
> I just ran Fluo locally with Uno.  Below is some info I am seeing.
> Do you see anything about RollingFileAppender in the stdout file?
>
> $ pwd
>
> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
> $ ls
> stderr  stdout  worker_1_host1.log
> $ grep Rolling stdout
> 16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
> - About to instantiate appender of type
> [ch.qos.logback.core.rolling.RollingFileAppender]
> 16:17:31,574 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No
> compression will be used
> 16:17:31,583 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file
> name:
> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1
> 16:17:31,583 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is
> set to
> [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1]
>
> Keith
>
> On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb 
> wrote:
> > Hey Mike,
> >
> > So I’m not seeing any worker logs on my machines.  The search
> >
> > find / | grep worker
> >
> > yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
> > that my fluo.log.dir system property is successfully being set by the
> LogbackUtil class.
> > Maybe this has something to do with it?
> >
> >
> >
> >
> >
> > -Original Message-
> > From: Mike Walch [mailto:mwa...@apache.org]
> > Sent: Tuesday, November 01, 2016 4:03 PM
> > To: dev@fluo.incubator.apache.org
> > Subject: Re: debugging fluo
> >
> > Were you able to find a worker_*.log file for each of your workers?
> >
> > Below are some tips for debugging:
> >
> > - Each YARN container should have a 'stdout' and 'stderr' file.  These
> files may have helpful error messages.  Especially if a worker failed to
> start.  Also, any calls to System.out and System.err in your observer will
> be printed to these files.
> > - When running Fluo in YARN, Fluo must use Logback for logging (due to a
> hard requirment by Twill). Logback is configured using
> /path/to/fluo/conf/logback.xml.  You should review this configuration but
> the root logger is configured by default to print any message that is the
> debug level or higher.
> > - If you configured multiple workers, each worker will run in a
> different container and have a different worker_*.log file.
> > - When a worker starts up, it prints its configuration to worker_*.log.
> > Make sure that you configured your observers using the property
> 'fluo.observer.*'
> >
> > -Mike
> >
> > On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb 
> wrote:
> >
> >> Do you have any tips for how to make Observers log to the log files
> >> found in the directory specified by 'yarn.nodemanager.log-dirs'?
> >>
> >> -Original Message-
> >> From: Mike Walch [mailto:mwa...@apache.org]
> >> Sent: Tuesday, November 01, 2016 2:36 PM
> >> To: dev@fluo.incubator.apache.org
> >> Subject: Re: debugging fluo
> >>
> >> Hi Caleb,
> >>
> >> The logs for a Fluo application can be found in YARN but they are
> >> tricky to find. Fluo should have better documentation on this which I
> will add now.
> >>
> >> The easiest way to view the logs for a Fluo application is to use the
> >> web interface for the YARN resource manager (
> >> 

RE: debugging fluo

[Meier, Caleb]

Hey Keith,

Not seeing the worker logs in my container.  Think my logback is configured 
incorrectly...

[root@c190sv193 container_1476563020088_0069_01_02]# grep Rolling stdout
18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About 
to instantiate appender of type 
[ch.qos.logback.core.rolling.RollingFileAppender]
18:36:49,215 |-INFO in 
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No compression 
will be used
18:36:49,230 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - 
Active log file name: 
io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log
18:36:49,230 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - 
File property is set to 
[io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log]

-Original Message-
From: Keith Turner [mailto:ke...@deenlo.com] 
Sent: Tuesday, November 01, 2016 4:33 PM
To: dev@fluo.incubator.apache.org
Subject: Re: debugging fluo

Caleb,

I just ran Fluo locally with Uno.  Below is some info I am seeing.
Do you see anything about RollingFileAppender in the stdout file?

$ pwd
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
$ ls
stderr  stdout  worker_1_host1.log
$ grep Rolling stdout
16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
- About to instantiate appender of type
[ch.qos.logback.core.rolling.RollingFileAppender]
16:17:31,574 |-INFO in
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No compression 
will be used
16:17:31,583 |-INFO in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: 
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1
16:17:31,583 |-INFO in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to 
[/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1]

Keith

On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb  wrote:
> Hey Mike,
>
> So I’m not seeing any worker logs on my machines.  The search
>
> find / | grep worker
>
> yielded nothing.  Any ideas as to why these don't exist?  I'm not sure 
> that my fluo.log.dir system property is successfully being set by the 
> LogbackUtil class.
> Maybe this has something to do with it?
>
>
>
>
>
> -Original Message-
> From: Mike Walch [mailto:mwa...@apache.org]
> Sent: Tuesday, November 01, 2016 4:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Were you able to find a worker_*.log file for each of your workers?
>
> Below are some tips for debugging:
>
> - Each YARN container should have a 'stdout' and 'stderr' file.  These files 
> may have helpful error messages.  Especially if a worker failed to start.  
> Also, any calls to System.out and System.err in your observer will be printed 
> to these files.
> - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard 
> requirment by Twill). Logback is configured using 
> /path/to/fluo/conf/logback.xml.  You should review this configuration but the 
> root logger is configured by default to print any message that is the debug 
> level or higher.
> - If you configured multiple workers, each worker will run in a different 
> container and have a different worker_*.log file.
> - When a worker starts up, it prints its configuration to worker_*.log.
> Make sure that you configured your observers using the property 
> 'fluo.observer.*'
>
> -Mike
>
> On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb  wrote:
>
>> Do you have any tips for how to make Observers log to the log files 
>> found in the directory specified by 'yarn.nodemanager.log-dirs'?
>>
>> -Original Message-
>> From: Mike Walch [mailto:mwa...@apache.org]
>> Sent: Tuesday, November 01, 2016 2:36 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>>
>> Hi Caleb,
>>
>> The logs for a Fluo application can be found in YARN but they are 
>> tricky to find. Fluo should have better documentation on this which I will 
>> add now.
>>
>> The easiest way to view the logs for a Fluo application is to use the 
>> web interface for the YARN resource manager ( 
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_
>> c 
>> luster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdz
>> Y
>> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk
>> 6 mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
>> ).
>> First, click on the application ID (i.e application_*) of your Fluo 
>> application and then click on the latest attempt ID (appattempt_*).
>> You should see a list of containers.  There should be a container for 
>> the application master (typically container 1), a Fluo oracle 
>> (typically container 2), and Fluo workers 

Re: debugging fluo

[Keith Turner]

I also saw the following line printed in stdout.  Maybe this line was
printed by LogbackUtil

Logging to 
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
using config ./conf/logback.xml

On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb  wrote:
> Hey Mike,
>
> So I’m not seeing any worker logs on my machines.  The search
>
> find / | grep worker
>
> yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
> that my fluo.log.dir system property is successfully being set by the 
> LogbackUtil class.
> Maybe this has something to do with it?
>
>
>
>
>
> -Original Message-
> From: Mike Walch [mailto:mwa...@apache.org]
> Sent: Tuesday, November 01, 2016 4:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Were you able to find a worker_*.log file for each of your workers?
>
> Below are some tips for debugging:
>
> - Each YARN container should have a 'stdout' and 'stderr' file.  These files 
> may have helpful error messages.  Especially if a worker failed to start.  
> Also, any calls to System.out and System.err in your observer will be printed 
> to these files.
> - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard 
> requirment by Twill). Logback is configured using 
> /path/to/fluo/conf/logback.xml.  You should review this configuration but the 
> root logger is configured by default to print any message that is the debug 
> level or higher.
> - If you configured multiple workers, each worker will run in a different 
> container and have a different worker_*.log file.
> - When a worker starts up, it prints its configuration to worker_*.log.
> Make sure that you configured your observers using the property 
> 'fluo.observer.*'
>
> -Mike
>
> On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb  wrote:
>
>> Do you have any tips for how to make Observers log to the log files
>> found in the directory specified by 'yarn.nodemanager.log-dirs'?
>>
>> -Original Message-
>> From: Mike Walch [mailto:mwa...@apache.org]
>> Sent: Tuesday, November 01, 2016 2:36 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>>
>> Hi Caleb,
>>
>> The logs for a Fluo application can be found in YARN but they are
>> tricky to find. Fluo should have better documentation on this which I will 
>> add now.
>>
>> The easiest way to view the logs for a Fluo application is to use the
>> web interface for the YARN resource manager (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_c
>> luster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzY
>> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6
>> mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
>> ).
>> First, click on the application ID (i.e application_*) of your Fluo
>> application and then click on the latest attempt ID (appattempt_*).
>> You should see a list of containers.  There should be a container for
>> the application master (typically container 1), a Fluo oracle
>> (typically container 2), and Fluo workers (containers 3+).  You can
>> view the log files produced by a container by clicking on its 'logs'
>> link.  Logs from Fluo observers will be in the worker_*.log file for
>> each of your worker containers.
>>
>> If you don't want to use the YARN resource manager web interface, you
>> can also view these logs in the directory specified by
>> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This
>> method works well on one machine but on cluster your containers will
>> probably be on different machines. See the YARN documentation below
>> for more info about this property:
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org
>> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml
>> =CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR
>> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5A
>> ewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
>>
>> Best,
>> Mike
>>
>> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb 
>> wrote:
>>
>> Hello,
>>
>> I'm attempting to debug a Fluo application and am having difficulty
>> locating the logs for my observers.  I've looked within the logs for
>> hadoop-yarn, but am not seeing any logging statements for my observers.
>> Where do observers log out of the box in a normal cloudera distribution?
>> Do I need to do something else in addition to logging to get my
>> observers to generate logs?
>>
>> Thanks,
>> Caleb
>>

Re: debugging fluo

[Mike Walch]

Yes.  Keith, you are right. Logging config changes need to be made to
/path/to/fluo/apps/$APPNAME/conf/logback.xml

On Tue, Nov 1, 2016 at 4:21 PM Keith Turner  wrote:

> On Tue, Nov 1, 2016 at 4:02 PM, Mike Walch  wrote:
> > Were you able to find a worker_*.log file for each of your workers?
> >
> > Below are some tips for debugging:
> >
> > - Each YARN container should have a 'stdout' and 'stderr' file.  These
> > files may have helpful error messages.  Especially if a worker failed to
> > start.  Also, any calls to System.out and System.err in your observer
> will
> > be printed to these files.
> > - When running Fluo in YARN, Fluo must use Logback for logging (due to a
> > hard requirment by Twill). Logback is configured using
> > /path/to/fluo/conf/logback.xml.  You should review this configuration but
> > the root logger is configured by default to print any message that is the
> > debug level or higher.
>
> Each application has its own config.  That config gets copied from
> /path/to/fluo/conf/ to /path/to/fluo/apps/$APPNAME/conf when the Fluo
> app is created.  So wouldn't logging config changes need to be made to
> /path/to/fluo/apps/$APPNAME/conf/logback.xml?
>
>
> > - If you configured multiple workers, each worker will run in a different
> > container and have a different worker_*.log file.
> > - When a worker starts up, it prints its configuration to worker_*.log.
> > Make sure that you configured your observers using the property
> > 'fluo.observer.*'
> >
> > -Mike
> >
> > On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb 
> wrote:
> >
> >> Do you have any tips for how to make Observers log to the log files
> found
> >> in the directory specified by 'yarn.nodemanager.log-dirs'?
> >>
> >> -Original Message-
> >> From: Mike Walch [mailto:mwa...@apache.org]
> >> Sent: Tuesday, November 01, 2016 2:36 PM
> >> To: dev@fluo.incubator.apache.org
> >> Subject: Re: debugging fluo
> >>
> >> Hi Caleb,
> >>
> >> The logs for a Fluo application can be found in YARN but they are tricky
> >> to find. Fluo should have better documentation on this which I will add
> now.
> >>
> >> The easiest way to view the logs for a Fluo application is to use the
> web
> >> interface for the YARN resource manager (
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_cluster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
> >> ).
> >> First, click on the application ID (i.e application_*) of your Fluo
> >> application and then click on the latest attempt ID (appattempt_*). You
> >> should see a list of containers.  There should be a container for the
> >> application master (typically container 1), a Fluo oracle (typically
> >> container 2), and Fluo workers (containers 3+).  You can view the log
> files
> >> produced by a container by clicking on its 'logs' link.  Logs from Fluo
> >> observers will be in the worker_*.log file for each of your worker
> >> containers.
> >>
> >> If you don't want to use the YARN resource manager web interface, you
> can
> >> also view these logs in the directory specified by
> >> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This method
> >> works well on one machine but on cluster your containers will probably
> be
> >> on different machines. See the YARN documentation below for more info
> about
> >> this property:
> >>
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org_docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
> >>
> >> Best,
> >> Mike
> >>
> >> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb 
> >> wrote:
> >>
> >> Hello,
> >>
> >> I'm attempting to debug a Fluo application and am having difficulty
> >> locating the logs for my observers.  I've looked within the logs for
> >> hadoop-yarn, but am not seeing any logging statements for my observers.
> >> Where do observers log out of the box in a normal cloudera distribution?
> >> Do I need to do something else in addition to logging to get my
> observers
> >> to generate logs?
> >>
> >> Thanks,
> >> Caleb
> >>
>

Re: debugging fluo

[Keith Turner]

Caleb,

I just ran Fluo locally with Uno.  Below is some info I am seeing.
Do you see anything about RollingFileAppender in the stdout file?

$ pwd
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
$ ls
stderr  stdout  worker_1_host1.log
$ grep Rolling stdout
16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
- About to instantiate appender of type
[ch.qos.logback.core.rolling.RollingFileAppender]
16:17:31,574 |-INFO in
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No
compression will be used
16:17:31,583 |-INFO in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log
file name: 
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1
16:17:31,583 |-INFO in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property
is set to 
[/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1]

Keith

On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb  wrote:
> Hey Mike,
>
> So I’m not seeing any worker logs on my machines.  The search
>
> find / | grep worker
>
> yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
> that my fluo.log.dir system property is successfully being set by the 
> LogbackUtil class.
> Maybe this has something to do with it?
>
>
>
>
>
> -Original Message-
> From: Mike Walch [mailto:mwa...@apache.org]
> Sent: Tuesday, November 01, 2016 4:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Were you able to find a worker_*.log file for each of your workers?
>
> Below are some tips for debugging:
>
> - Each YARN container should have a 'stdout' and 'stderr' file.  These files 
> may have helpful error messages.  Especially if a worker failed to start.  
> Also, any calls to System.out and System.err in your observer will be printed 
> to these files.
> - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard 
> requirment by Twill). Logback is configured using 
> /path/to/fluo/conf/logback.xml.  You should review this configuration but the 
> root logger is configured by default to print any message that is the debug 
> level or higher.
> - If you configured multiple workers, each worker will run in a different 
> container and have a different worker_*.log file.
> - When a worker starts up, it prints its configuration to worker_*.log.
> Make sure that you configured your observers using the property 
> 'fluo.observer.*'
>
> -Mike
>
> On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb  wrote:
>
>> Do you have any tips for how to make Observers log to the log files
>> found in the directory specified by 'yarn.nodemanager.log-dirs'?
>>
>> -Original Message-
>> From: Mike Walch [mailto:mwa...@apache.org]
>> Sent: Tuesday, November 01, 2016 2:36 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>>
>> Hi Caleb,
>>
>> The logs for a Fluo application can be found in YARN but they are
>> tricky to find. Fluo should have better documentation on this which I will 
>> add now.
>>
>> The easiest way to view the logs for a Fluo application is to use the
>> web interface for the YARN resource manager (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_c
>> luster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzY
>> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6
>> mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
>> ).
>> First, click on the application ID (i.e application_*) of your Fluo
>> application and then click on the latest attempt ID (appattempt_*).
>> You should see a list of containers.  There should be a container for
>> the application master (typically container 1), a Fluo oracle
>> (typically container 2), and Fluo workers (containers 3+).  You can
>> view the log files produced by a container by clicking on its 'logs'
>> link.  Logs from Fluo observers will be in the worker_*.log file for
>> each of your worker containers.
>>
>> If you don't want to use the YARN resource manager web interface, you
>> can also view these logs in the directory specified by
>> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This
>> method works well on one machine but on cluster your containers will
>> probably be on different machines. See the YARN documentation below
>> for more info about this property:
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org
>> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml
>> =CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR
>> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5A
>> ewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
>>
>> Best,
>> Mike
>>
>> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb 
>> wrote:
>>
>> Hello,

RE: debugging fluo

[Meier, Caleb]

Hey Mike,

So I’m not seeing any worker logs on my machines.  The search

find / | grep worker

yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
that my fluo.log.dir system property is successfully being set by the 
LogbackUtil class.
Maybe this has something to do with it?


  


-Original Message-
From: Mike Walch [mailto:mwa...@apache.org] 
Sent: Tuesday, November 01, 2016 4:03 PM
To: dev@fluo.incubator.apache.org
Subject: Re: debugging fluo

Were you able to find a worker_*.log file for each of your workers?

Below are some tips for debugging:

- Each YARN container should have a 'stdout' and 'stderr' file.  These files 
may have helpful error messages.  Especially if a worker failed to start.  
Also, any calls to System.out and System.err in your observer will be printed 
to these files.
- When running Fluo in YARN, Fluo must use Logback for logging (due to a hard 
requirment by Twill). Logback is configured using 
/path/to/fluo/conf/logback.xml.  You should review this configuration but the 
root logger is configured by default to print any message that is the debug 
level or higher.
- If you configured multiple workers, each worker will run in a different 
container and have a different worker_*.log file.
- When a worker starts up, it prints its configuration to worker_*.log.
Make sure that you configured your observers using the property 
'fluo.observer.*'

-Mike

On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb  wrote:

> Do you have any tips for how to make Observers log to the log files 
> found in the directory specified by 'yarn.nodemanager.log-dirs'?
>
> -Original Message-
> From: Mike Walch [mailto:mwa...@apache.org]
> Sent: Tuesday, November 01, 2016 2:36 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Hi Caleb,
>
> The logs for a Fluo application can be found in YARN but they are 
> tricky to find. Fluo should have better documentation on this which I will 
> add now.
>
> The easiest way to view the logs for a Fluo application is to use the 
> web interface for the YARN resource manager ( 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_c
> luster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzY
> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6
> mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
> ).
> First, click on the application ID (i.e application_*) of your Fluo 
> application and then click on the latest attempt ID (appattempt_*). 
> You should see a list of containers.  There should be a container for 
> the application master (typically container 1), a Fluo oracle 
> (typically container 2), and Fluo workers (containers 3+).  You can 
> view the log files produced by a container by clicking on its 'logs' 
> link.  Logs from Fluo observers will be in the worker_*.log file for 
> each of your worker containers.
>
> If you don't want to use the YARN resource manager web interface, you 
> can also view these logs in the directory specified by 
> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This 
> method works well on one machine but on cluster your containers will 
> probably be on different machines. See the YARN documentation below 
> for more info about this property:
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org
> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml
> =CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR
> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5A
> ewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
>
> Best,
> Mike
>
> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb 
> wrote:
>
> Hello,
>
> I'm attempting to debug a Fluo application and am having difficulty 
> locating the logs for my observers.  I've looked within the logs for 
> hadoop-yarn, but am not seeing any logging statements for my observers.
> Where do observers log out of the box in a normal cloudera distribution?
> Do I need to do something else in addition to logging to get my 
> observers to generate logs?
>
> Thanks,
> Caleb
>

Re: debugging fluo

[Mike Walch]

Were you able to find a worker_*.log file for each of your workers?

Below are some tips for debugging:

- Each YARN container should have a 'stdout' and 'stderr' file.  These
files may have helpful error messages.  Especially if a worker failed to
start.  Also, any calls to System.out and System.err in your observer will
be printed to these files.
- When running Fluo in YARN, Fluo must use Logback for logging (due to a
hard requirment by Twill). Logback is configured using
/path/to/fluo/conf/logback.xml.  You should review this configuration but
the root logger is configured by default to print any message that is the
debug level or higher.
- If you configured multiple workers, each worker will run in a different
container and have a different worker_*.log file.
- When a worker starts up, it prints its configuration to worker_*.log.
Make sure that you configured your observers using the property
'fluo.observer.*'

-Mike

On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb  wrote:

> Do you have any tips for how to make Observers log to the log files found
> in the directory specified by 'yarn.nodemanager.log-dirs'?
>
> -Original Message-
> From: Mike Walch [mailto:mwa...@apache.org]
> Sent: Tuesday, November 01, 2016 2:36 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Hi Caleb,
>
> The logs for a Fluo application can be found in YARN but they are tricky
> to find. Fluo should have better documentation on this which I will add now.
>
> The easiest way to view the logs for a Fluo application is to use the web
> interface for the YARN resource manager (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_cluster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
> ).
> First, click on the application ID (i.e application_*) of your Fluo
> application and then click on the latest attempt ID (appattempt_*). You
> should see a list of containers.  There should be a container for the
> application master (typically container 1), a Fluo oracle (typically
> container 2), and Fluo workers (containers 3+).  You can view the log files
> produced by a container by clicking on its 'logs' link.  Logs from Fluo
> observers will be in the worker_*.log file for each of your worker
> containers.
>
> If you don't want to use the YARN resource manager web interface, you can
> also view these logs in the directory specified by
> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This method
> works well on one machine but on cluster your containers will probably be
> on different machines. See the YARN documentation below for more info about
> this property:
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org_docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
>
> Best,
> Mike
>
> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb 
> wrote:
>
> Hello,
>
> I'm attempting to debug a Fluo application and am having difficulty
> locating the logs for my observers.  I've looked within the logs for
> hadoop-yarn, but am not seeing any logging statements for my observers.
> Where do observers log out of the box in a normal cloudera distribution?
> Do I need to do something else in addition to logging to get my observers
> to generate logs?
>
> Thanks,
> Caleb
>

Re: Podling Report Reminder - November 2016

[Keith Turner]

I just udpated the wiki if anyone wants to review the changes.

On Mon, Oct 31, 2016 at 3:56 PM, Josh Elser  wrote:
> I know you're already in a writing "mood", Keith.
>
> Don't forget about this one, please. Needs to be done by Wednesday this week
> (48hrs).
>
>
> johndam...@apache.org wrote:
>>
>> Dear podling,
>>
>> This email was sent by an automated system on behalf of the Apache
>> Incubator PMC. It is an initial reminder to give you plenty of time to
>> prepare your quarterly board report.
>>
>> The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
>> The report for your podling will form a part of the Incubator PMC
>> report. The Incubator PMC requires your report to be submitted 2 weeks
>> before the board meeting, to allow sufficient time for review and
>> submission (Wed, November 02).
>>
>> Please submit your report with sufficient time to allow the Incubator
>> PMC, and subsequently board members to review and digest. Again, the
>> very latest you should submit your report is 2 weeks prior to the board
>> meeting.
>>
>> Thanks,
>>
>> The Apache Incubator PMC
>>
>> Submitting your Report
>>
>> --
>>
>> Your report should contain the following:
>>
>> *   Your project name
>> *   A brief description of your project, which assumes no knowledge of
>>  the project or necessarily of its field
>> *   A list of the three most important issues to address in the move
>>  towards graduation.
>> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
>>  aware of
>> *   How has the community developed since the last report
>> *   How has the project developed since the last report.
>>
>> This should be appended to the Incubator Wiki page at:
>>
>> http://wiki.apache.org/incubator/November2016
>>
>> Note: This is manually populated. You may need to wait a little before
>> this page is created from a template.
>>
>> Mentors
>> ---
>>
>> Mentors should review reports for their project(s) and sign them off on
>> the Incubator wiki page. Signing off reports shows that you are
>> following the project - projects that are not signed may raise alarms
>> for the Incubator PMC.
>>
>> Incubator PMC

Re: debugging fluo

[Mike Walch]

Hi Caleb,

The logs for a Fluo application can be found in YARN but they are tricky to
find. Fluo should have better documentation on this which I will add now.

The easiest way to view the logs for a Fluo application is to use the web
interface for the YARN resource manager (http://localhost:8088/cluster).
First, click on the application ID (i.e application_*) of your Fluo
application and then click on the latest attempt ID (appattempt_*). You
should see a list of containers.  There should be a container for the
application master (typically container 1), a Fluo oracle (typically
container 2), and Fluo workers (containers 3+).  You can view the log files
produced by a container by clicking on its 'logs' link.  Logs from Fluo
observers will be in the worker_*.log file for each of your worker
containers.

If you don't want to use the YARN resource manager web interface, you can
also view these logs in the directory specified by
'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This method
works well on one machine but on cluster your containers will probably be
on different machines. See the YARN documentation below for more info about
this property:

https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

Best,
Mike

On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb  wrote:

Hello,

I'm attempting to debug a Fluo application and am having difficulty
locating the logs for my observers.  I've looked within the logs for
hadoop-yarn, but am not seeing any logging statements for my observers.
Where do observers log out of the box in a normal cloudera distribution?
Do I need to do something else in addition to logging to get my observers
to generate logs?

Thanks,
Caleb

debugging fluo

[Meier, Caleb]

Hello,

I'm attempting to debug a Fluo application and am having difficulty locating 
the logs for my observers.  I've looked within the logs for hadoop-yarn, but am 
not seeing any logging statements for my observers.  Where do observers log out 
of the box in a normal cloudera distribution?  Do I need to do something else 
in addition to logging to get my observers to generate logs?

Thanks,
Caleb