Re: New 'fluo-yarn' repo

[Mike Walch]

Yes.  Fluo-yarn will be on a different release schedule.  A new release of
Fluo should in most cases not require a new release of fluo-yarn.  New
releases of fluo-yarn will most likely be necessary to upgrade Apache Twill
to support newer releases of Hadoop YARN.

On Mon, Jul 17, 2017 at 4:26 PM Christopher <ctubb...@apache.org> wrote:

> Is the idea to have fluo-yarn on a different release cycle?
>
>
> On Mon, Jul 17, 2017 at 3:52 PM Mike Walch <mwa...@apache.org> wrote:
>
> > I would like to create new repo (called 'fluo-yarn') using Gitbox to hold
> > code for launching Fluo applications in YARN.  This is related to my
> recent
> > pull request [1]. Does anyone have any objections to this?
> >
> > [1]: https://github.com/apache/incubator-fluo/pull/883
> >
>

New 'fluo-yarn' repo

[Mike Walch]

I would like to create new repo (called 'fluo-yarn') using Gitbox to hold
code for launching Fluo applications in YARN.  This is related to my recent
pull request [1]. Does anyone have any objections to this?

[1]: https://github.com/apache/incubator-fluo/pull/883

Re: [VOTE] Apache Fluo Recipes 1.1.0-incubating-rc1

[Mike Walch]

+1

* Verified sha1 & md5 hashes matched
* Verified signatures
* Built and ran ITs for Webindex & Stresso using fluo-recipes jars in
staging repo

On Mon, Jun 12, 2017 at 7:16 PM Keith Turner  wrote:

> Fluo Developers,
>
> Please consider the following candidate for Fluo Recipes 1.1.0-incubating.
>
> Git Commit:
> 4769fa9a3a43af030de318c0423a7cae5c5eb4fe
> Branch:
> 1.1.0-incubating-rc1
>
> If this vote passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Fluo Recipes 1.1.0-incubating' -s
> rel/fluo-recipes-1.1.0-incubating \
> 4769fa9a3a43af030de318c0423a7cae5c5eb4fe
> Staging repo:
> https://repository.apache.org/content/repositories/orgapachefluo-1018
> Source (official release artifact):
>
> https://repository.apache.org/content/repositories/orgapachefluo-1018/org/apache/fluo/fluo-recipes/1.1.0-incubating/fluo-recipes-1.1.0-incubating-source-release.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for
> a given artifact.)
>
> All artifacts were built and staged with:
> mvn release:prepare && mvn release:perform
>
> Signing keys are available at
> https://www.apache.org/dist/incubator/fluo/KEYS
> (Expected fingerprint: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)
>
> Release notes will be created during the vote.
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1.1.0-incubating release of Apache Fluo Recipes.
>
> Voting is open until at least Thu Jun 15 23:30:00 UTC 2017
> (Thu Jun 15 19:30:00 EDT 2017 / Thu Jun 15 16:30:00 PDT 2017)
>
> After this, the release manager can call the vote at any time.
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
> https://repository.apache.org/content/repositories/orgapachefluo-1018/
> # note the trailing slash is needed
>

Re: [VOTE] Fluo 1.1.0-incubating-rc1

[Mike Walch]

+1

* Verified sha1 & md5 hashes matched
* Verified signatures
* Was able to build and run integration tests for Fluo-recipes & Webindex
using staging repo


On Fri, Jun 2, 2017 at 12:54 PM Keith Turner  wrote:

> Fluo Developers,
>
> Please consider the following candidate for Fluo 1.1.0-incubating.
>
> Git Commit:
> ad8ee492e2f435405f98d825781098c55186f4fb
> Branch:
> 1.1.0-incubating-rc1
>
> If this vote passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Fluo 1.1.0-incubating' -s
> rel/fluo-1.1.0-incubating \
> ad8ee492e2f435405f98d825781098c55186f4fb
>
> Staging repo:
> https://repository.apache.org/content/repositories/orgapachefluo-1017
> Source (official release artifact):
>
> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz
> Binary:
> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for
> a given artifact.)
>
> All artifacts were built and staged with:
> mvn release:prepare && mvn release:perform
>
> Signing keys are available at
> https://www.apache.org/dist/incubator/fluo/KEYS
> (Expected fingerprint: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)
>
> Release notes (in progress) can be found at:
> https://fluo.apache.org/.../1.1.0-incubating
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1.1.0-incubating release of Apache Fluo.
>
> This vote will end on Mon Jun  5 17:00:00 UTC 2017
> (Mon Jun  5 13:00:00 EDT 2017 / Mon Jun  5 10:00:00 PDT 2017)
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
> https://repository.apache.org/content/repositories/orgapachefluo-1017/
> # note the trailing slash is needed
>

Re: [DISCUSS] Improve how Fluo applications are run

[Mike Walch]

> From a user standpoint, what are the main differences from the current
way of running Fluo apps?

>From a user perspective, the main differences are:

* running Fluo applications is handled by downstream projects and upstream
tarball is deprecated.  These downstream projects share upstream code to
keep them simple.
* the fluo.properties file is split up into two files
(connection.properties & application.properties).  The
application.properties file is only required for initialization.  After
initialization, these properties will be retrieved from Zookeeper. This
allows a lot of commands like scanning, starting oracles & workers, etc to
only require connection.properties and the application name.

> Are your fluo-local and fluo-yarn repos functional?

No.. they have scripts but need a lot of Java code written.

> I noticed in some of the issues that the command had a package of
core.command.   I was thinking it may be better to put commands in
their own module, so the commands can have a dependency like
jcommander w/o adding that to core.

Putting commands in their own module sounds good to me.

On Mon, May 22, 2017 at 10:59 AM Keith Turner <ke...@deenlo.com> wrote:

> From a user standpoint, what are the main differences from the current
> way of running Fluo apps?
>
> Are your fluo-local and fluo-yarn repos functional?
>
> I noticed in some of the issues that the command had a package of
> core.command.   I was thinking it may be better to put commands in
> their own module, so the commands can have a dependency like
> jcommander w/o adding that to core.
>
> On Fri, May 19, 2017 at 1:18 PM, Mike Walch <mwa...@apache.org> wrote:
> > For Fluo 1.2, I would like to improve how Fluo applications are
> > launched/run. This involves deprecating the current tarball distribution
> > and creating downstream repos that create distributions for running Fluo
> > applications locally and cluster managements tools suchs as YARN, Mesos,
> > Kubernetes, etc.
> >
> > I have created repos for running Fluo locally and YARN and have started
> to
> > design their functionality by creating scripts and documentation. No Java
> > code has been written yet. The repos will eventually be moved to Apache
> > infrastructure.
> >
> > https://github.com/mikewalch/fluo-local
> > https://github.com/mikewalch/fluo-yarn
> >
> > In order to complete this task, a lot of new functionality/commands need
> to
> > created upstream in Fluo. To coordinate this work, I created several
> issues
> > under the 1.2 milestone.
> >
> > https://github.com/apache/incubator-fluo/milestone/6
> >
> > If you are interested in this work, review the documentation/scripts in
> > fluo-local & fluo-yarn as well as the issues linked to above and let me
> > know if you have any ideas or suggestions.
>

[DISCUSS] Improve how Fluo applications are run

[Mike Walch]

For Fluo 1.2, I would like to improve how Fluo applications are
launched/run. This involves deprecating the current tarball distribution
and creating downstream repos that create distributions for running Fluo
applications locally and cluster managements tools suchs as YARN, Mesos,
Kubernetes, etc.

I have created repos for running Fluo locally and YARN and have started to
design their functionality by creating scripts and documentation. No Java
code has been written yet. The repos will eventually be moved to Apache
infrastructure.

https://github.com/mikewalch/fluo-local
https://github.com/mikewalch/fluo-yarn

In order to complete this task, a lot of new functionality/commands need to
created upstream in Fluo. To coordinate this work, I created several issues
under the 1.2 milestone.

https://github.com/apache/incubator-fluo/milestone/6

If you are interested in this work, review the documentation/scripts in
fluo-local & fluo-yarn as well as the issues linked to above and let me
know if you have any ideas or suggestions.

Re: [DISCUSS] Releasing Fluo 1.1.0

[Mike Walch]

I am working on changes to how Fluo launches applications in YARN.
However, my changes are not close to being ready for a PR.  I am OK with
1.1.0 being released and this getting into 1.2.0.

On Fri, Apr 21, 2017 at 12:46 PM Keith Turner  wrote:

> I am thinking of initiating a release of 1.1.0.  I would like to get
> the changes out that make writing spark jobs[1] and observers
> easier[2][3].  Would also like to release the changes to release that
> make notification scanning more efficient[4][5].
>
> Does anyone else have other changes they would like to see in 1.1.0?
>
> [1]: https://github.com/apache/incubator-fluo/issues/813
> [2]: https://github.com/apache/incubator-fluo/issues/816
> [3]: https://github.com/apache/incubator-fluo/pull/817
> [4]: https://github.com/apache/incubator-fluo/issues/500
> [5]: https://github.com/apache/incubator-fluo/pull/822
>

Re: [DISCUSS] New API for setting up observers.

[Mike Walch]

This is a great idea.  How would users configure their UserObserversFactory
in fluo.properties?  Would there still be observer properties in
fluo.properties or just application properties?

On Thu, Feb 23, 2017 at 5:36 PM Keith Turner  wrote:

> The current way observers are set up in Fluo requires specifying a
> class for each observers in configuration.  This is cumbersome and
> prevents using lambdas and anonymous inner classes.  It also makes it
> hard to follow what a Fluo Application is doing.  This cumbersome way
> of setting things up propagates forward into higher level constructs
> in Fluo Recipes making them also cumbersome to use.
>
> I think it would be much simpler if the user only had to specify one
> factory class in configuration that created all observers.  This
> factory would be free to pair a lambda with a column for an observer.
> Observed columns would no longer be tightly coupled with a specific
> class.  I am thinking the factory would look something like the
> following.
>
> interface ObserversFactory {
>   Map getObservers(SimpleConfiguration
> applicationConfig);
> }
>
> A user would implement this interface with a class that creates all of
> their observers.
>
> class UserObserversFactory implements ObserversFactory {
>   Map getObservers(SimpleConfiguration
> appConfig) {
> //all of users observers are setup here in code, which is much
> easier to follow than current way of configuring each observer class
> individually.
>  HashMap observers = ...;
>  observers.put(col1, tx,row,col -> ); //an observer that's a lambda
>  ExportQueue.addObserver(observers, appConfig, "queueId", exports
> -> );  //using a lambda to handle exports...
>   }
> }
>
> The user would configure Fluo to use the above observer factory.
>
> I am thinking of trying to implement this for 1.1, but wanted to see
> if anyone had any input before I started.  If it seems to work well, I
> was thinking of deprecating the current way of configuring observers.
> I would update webindex and fluo recipes in parallel to evaluate the
> changes.
>

Re: third party service to poll Fluo for absence of event

[Mike Walch]

Hi Caleb,

I wouldn't try to follow our Twill code too closely.  We have a lot of
abstraction in the code that I don't think you need.  We do use TwillRunner
and TwillController.  One major tip is that you should consider whether or
not you want to pull in all of Twill dependencies (which may cause
dependency conflicts). You can avoid pulling in Twill's dependencies by
creating a bundled jar and using Twill's BundleJarRunnable.  We didn't do
this in Fluo right now but I would like to in the future.  If you want to
see an example, take a look at the new accumulo-testing repo[1] which uses
Twill to launch tests using bundled jars. In particular, look at the class
YarnAccumuloTestRunner[2].

Best,
Mike

[1]: https://github.com/apache/accumulo-testing
[2]:
https://github.com/apache/accumulo-testing/blob/master/yarn/src/main/java/org/apache/accumulo/testing/yarn/YarnAccumuloTestRunner.java


On Thu, Feb 2, 2017 at 5:15 PM Meier, Caleb  wrote:

> Thanks for the input.  I'm currently looking at creating some sort of
> coordinator (which wraps a ScheduledExecutorService to generate periodic
> notifications) and a collection of workers (to process the periodic queries
> as they are issued).  Most of the interaction between the workers and
> coordinator will be via Kafka (develop some sort of protocol to ensure that
> more than one worker isn't getting assigned the same query).  At any rate,
> I was thinking of implementing these components as TwillRunnables.
> However, it seems like the Twill documentation is a bit sparse.  Given that
> you guys implemented Fluo as a TwillApplication, do you have any
> insight/advice for writing TwillApplications?  In particular, how is your
> FluoTwillApp being run?  All of the examples I've seen create a client with
> a TwillRunner and TwillController.  It seems like you 've created your own
> version of a YarnAppRunner -- what role is that playing in running the
> FluoTwillApp?  Moreover, it is also unclear to me whether the
> TwillRunnables are bound to the client -- if the client terminates do the
> runnables terminate as well?  So essentially, it is unclear to me how
> create a long running application in Twill that is not bound to a
> particular client.  Sorry that this is a little off topic, but any help,
> references to documentation/examples would be very appreciated.
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office:  (703)797-3066 <(703)%20797-3066>
> caleb.me...@parsons.com ♦ www.parsons.com
>
> -Original Message-
> From: Keith Turner [mailto:ke...@deenlo.com]
> Sent: Wednesday, February 01, 2017 11:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Wed, Feb 1, 2017 at 9:54 PM, Christopher  wrote:
> > On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb 
> > wrote:
> >
> >> Yeah, this seems pretty reasonable to me.  I guess it then boils down
> >> to the nitty gritty of do I store results in Fluo and have my service
> >> query Fluo (I think you guys actually advise against that in your
> >> documentation), or export results and then have the service query
> >> some external index that I am exporting to.
> >>
> >>
> > I'm not sure we advise against it, so much as recognize that it may
> > not be suitable for certain use cases and may not meet query
> > performance expectations (
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_=CwIFaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs=
> ).
> >
>
> I would advise against querying Fluo for low latency queries.
> However, this external service thats checking a few stats within Fluo and
> injecting new notifications probably does not care about latency.
>
> The reason Fluo is not geared towards low latency is that it does lazy
> recovery of failed transactions.   Failed transactions are not cleaned
> up until something tries to read the data, which could significantly delay
> reads.
>
> > In any case, your observer need not write the final "last occurrence"
> > entries into a Fluo table. It could write them anywhere.
> >
> >
> >> Regarding timestamps, does the oracle server provide actual
> >> timestamps or just logical timestamps?  That is, could I use the
> >> timestamps that the server provides to define some sort of now()
> >> function to obtain the current time to compare with the times of
> incoming events?
> >>
> >
> > Just logical time, and it delivers batches to limit locking, so it can
> > appear to jump ahead spontaneously. I'm not sure the OracleServer is
> > suitable for this purpose. What level of precision are you going for?
> > It might be enough to just run NTP, if you 

Re: Podling Report Reminder - February 2017

[Mike Walch]

Looks great.  Thanks Keith!

On Wed, Feb 1, 2017 at 10:48 AM Keith Turner  wrote:

> I updated the Fluo report on the wiki.  Does anyone have any suggested
> changes?
>
> https://wiki.apache.org/incubator/February2017
>
> On Tue, Jan 31, 2017 at 10:02 PM,   wrote:
> > Dear podling,
> >
> > This email was sent by an automated system on behalf of the Apache
> > Incubator PMC. It is an initial reminder to give you plenty of time to
> > prepare your quarterly board report.
> >
> > The board meeting is scheduled for Wed, 15 February 2017, 10:30 am PDT.
> > The report for your podling will form a part of the Incubator PMC
> > report. The Incubator PMC requires your report to be submitted 2 weeks
> > before the board meeting, to allow sufficient time for review and
> > submission (Wed, February 01).
> >
> > Please submit your report with sufficient time to allow the Incubator
> > PMC, and subsequently board members to review and digest. Again, the
> > very latest you should submit your report is 2 weeks prior to the board
> > meeting.
> >
> > Thanks,
> >
> > The Apache Incubator PMC
> >
> > Submitting your Report
> >
> > --
> >
> > Your report should contain the following:
> >
> > *   Your project name
> > *   A brief description of your project, which assumes no knowledge of
> > the project or necessarily of its field
> > *   A list of the three most important issues to address in the move
> > towards graduation.
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> > aware of
> > *   How has the community developed since the last report
> > *   How has the project developed since the last report.
> >
> > This should be appended to the Incubator Wiki page at:
> >
> > https://wiki.apache.org/incubator/February2017
> >
> > Note: This is manually populated. You may need to wait a little before
> > this page is created from a template.
> >
> > Mentors
> > ---
> >
> > Mentors should review reports for their project(s) and sign them off on
> > the Incubator wiki page. Signing off reports shows that you are
> > following the project - projects that are not signed may raise alarms
> > for the Incubator PMC.
> >
> > Incubator PMC
>

Re: debugging fluo

[Mike Walch]

Which version of Fluo are you using?  The properties 'io.fluo.log.*' were
used before Apache incubation in 1.0.0-beta-1 or 1.0.0-beta-2.  These
properties were renamed in Fluo 1.0.0 to 'fluo.log.*'.  You could be having
problems if you built your application using Fluo 1.0.0-beta-2 and ran it
using a Fluo 1.0.0 install.

On Tue, Nov 1, 2016 at 4:45 PM Meier, Caleb <caleb.me...@parsons.com> wrote:

> Hey Keith,
>
> Not seeing the worker logs in my container.  Think my logback is
> configured incorrectly...
>
> [root@c190sv193 container_1476563020088_0069_01_02]# grep Rolling
> stdout
> 18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction -
> About to instantiate appender of type
> [ch.qos.logback.core.rolling.RollingFileAppender]
> 18:36:49,215 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No
> compression will be used
> 18:36:49,230 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file
> name:
> io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log
> 18:36:49,230 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is
> set to
> [io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log]
>
> -Original Message-
> From: Keith Turner [mailto:ke...@deenlo.com]
> Sent: Tuesday, November 01, 2016 4:33 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Caleb,
>
> I just ran Fluo locally with Uno.  Below is some info I am seeing.
> Do you see anything about RollingFileAppender in the stdout file?
>
> $ pwd
>
> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
> $ ls
> stderr  stdout  worker_1_host1.log
> $ grep Rolling stdout
> 16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
> - About to instantiate appender of type
> [ch.qos.logback.core.rolling.RollingFileAppender]
> 16:17:31,574 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No
> compression will be used
> 16:17:31,583 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file
> name:
> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1
> 16:17:31,583 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is
> set to
> [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1]
>
> Keith
>
> On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb <caleb.me...@parsons.com>
> wrote:
> > Hey Mike,
> >
> > So I’m not seeing any worker logs on my machines.  The search
> >
> > find / | grep worker
> >
> > yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
> > that my fluo.log.dir system property is successfully being set by the
> LogbackUtil class.
> > Maybe this has something to do with it?
> >
> >
> >
> >
> >
> > -Original Message-
> > From: Mike Walch [mailto:mwa...@apache.org]
> > Sent: Tuesday, November 01, 2016 4:03 PM
> > To: dev@fluo.incubator.apache.org
> > Subject: Re: debugging fluo
> >
> > Were you able to find a worker_*.log file for each of your workers?
> >
> > Below are some tips for debugging:
> >
> > - Each YARN container should have a 'stdout' and 'stderr' file.  These
> files may have helpful error messages.  Especially if a worker failed to
> start.  Also, any calls to System.out and System.err in your observer will
> be printed to these files.
> > - When running Fluo in YARN, Fluo must use Logback for logging (due to a
> hard requirment by Twill). Logback is configured using
> /path/to/fluo/conf/logback.xml.  You should review this configuration but
> the root logger is configured by default to print any message that is the
> debug level or higher.
> > - If you configured multiple workers, each worker will run in a
> different container and have a different worker_*.log file.
> > - When a worker starts up, it prints its configuration to worker_*.log.
> > Make sure that you configured your observers using the property
> 'fluo.observer.*'
> >
> > -Mike
> >
> > On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb <caleb.me...@parsons.com>
> wrote:
> >
> >> Do you have any tips for how to make Observers log to the log files
> >> found in the directory specified by 'yarn.nodemanager.log-dirs'?
> >>
> >> -Original Message-
> >> From: Mike Walch [mailto:mwa...@apache.org]
>

Re: debugging fluo

[Mike Walch]

Yes.  Keith, you are right. Logging config changes need to be made to
/path/to/fluo/apps/$APPNAME/conf/logback.xml

On Tue, Nov 1, 2016 at 4:21 PM Keith Turner <ke...@deenlo.com> wrote:

> On Tue, Nov 1, 2016 at 4:02 PM, Mike Walch <mwa...@apache.org> wrote:
> > Were you able to find a worker_*.log file for each of your workers?
> >
> > Below are some tips for debugging:
> >
> > - Each YARN container should have a 'stdout' and 'stderr' file.  These
> > files may have helpful error messages.  Especially if a worker failed to
> > start.  Also, any calls to System.out and System.err in your observer
> will
> > be printed to these files.
> > - When running Fluo in YARN, Fluo must use Logback for logging (due to a
> > hard requirment by Twill). Logback is configured using
> > /path/to/fluo/conf/logback.xml.  You should review this configuration but
> > the root logger is configured by default to print any message that is the
> > debug level or higher.
>
> Each application has its own config.  That config gets copied from
> /path/to/fluo/conf/ to /path/to/fluo/apps/$APPNAME/conf when the Fluo
> app is created.  So wouldn't logging config changes need to be made to
> /path/to/fluo/apps/$APPNAME/conf/logback.xml?
>
>
> > - If you configured multiple workers, each worker will run in a different
> > container and have a different worker_*.log file.
> > - When a worker starts up, it prints its configuration to worker_*.log.
> > Make sure that you configured your observers using the property
> > 'fluo.observer.*'
> >
> > -Mike
> >
> > On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb <caleb.me...@parsons.com>
> wrote:
> >
> >> Do you have any tips for how to make Observers log to the log files
> found
> >> in the directory specified by 'yarn.nodemanager.log-dirs'?
> >>
> >> -Original Message-
> >> From: Mike Walch [mailto:mwa...@apache.org]
> >> Sent: Tuesday, November 01, 2016 2:36 PM
> >> To: dev@fluo.incubator.apache.org
> >> Subject: Re: debugging fluo
> >>
> >> Hi Caleb,
> >>
> >> The logs for a Fluo application can be found in YARN but they are tricky
> >> to find. Fluo should have better documentation on this which I will add
> now.
> >>
> >> The easiest way to view the logs for a Fluo application is to use the
> web
> >> interface for the YARN resource manager (
> >>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_cluster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
> >> ).
> >> First, click on the application ID (i.e application_*) of your Fluo
> >> application and then click on the latest attempt ID (appattempt_*). You
> >> should see a list of containers.  There should be a container for the
> >> application master (typically container 1), a Fluo oracle (typically
> >> container 2), and Fluo workers (containers 3+).  You can view the log
> files
> >> produced by a container by clicking on its 'logs' link.  Logs from Fluo
> >> observers will be in the worker_*.log file for each of your worker
> >> containers.
> >>
> >> If you don't want to use the YARN resource manager web interface, you
> can
> >> also view these logs in the directory specified by
> >> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This method
> >> works well on one machine but on cluster your containers will probably
> be
> >> on different machines. See the YARN documentation below for more info
> about
> >> this property:
> >>
> >>
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org_docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
> >>
> >> Best,
> >> Mike
> >>
> >> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb <caleb.me...@parsons.com>
> >> wrote:
> >>
> >> Hello,
> >>
> >> I'm attempting to debug a Fluo application and am having difficulty
> >> locating the logs for my observers.  I've looked within the logs for
> >> hadoop-yarn, but am not seeing any logging statements for my observers.
> >> Where do observers log out of the box in a normal cloudera distribution?
> >> Do I need to do something else in addition to logging to get my
> observers
> >> to generate logs?
> >>
> >> Thanks,
> >> Caleb
> >>
>

Re: debugging fluo

[Mike Walch]

Were you able to find a worker_*.log file for each of your workers?

Below are some tips for debugging:

- Each YARN container should have a 'stdout' and 'stderr' file.  These
files may have helpful error messages.  Especially if a worker failed to
start.  Also, any calls to System.out and System.err in your observer will
be printed to these files.
- When running Fluo in YARN, Fluo must use Logback for logging (due to a
hard requirment by Twill). Logback is configured using
/path/to/fluo/conf/logback.xml.  You should review this configuration but
the root logger is configured by default to print any message that is the
debug level or higher.
- If you configured multiple workers, each worker will run in a different
container and have a different worker_*.log file.
- When a worker starts up, it prints its configuration to worker_*.log.
Make sure that you configured your observers using the property
'fluo.observer.*'

-Mike

On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb <caleb.me...@parsons.com> wrote:

> Do you have any tips for how to make Observers log to the log files found
> in the directory specified by 'yarn.nodemanager.log-dirs'?
>
> -Original Message-
> From: Mike Walch [mailto:mwa...@apache.org]
> Sent: Tuesday, November 01, 2016 2:36 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Hi Caleb,
>
> The logs for a Fluo application can be found in YARN but they are tricky
> to find. Fluo should have better documentation on this which I will add now.
>
> The easiest way to view the logs for a Fluo application is to use the web
> interface for the YARN resource manager (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_cluster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
> ).
> First, click on the application ID (i.e application_*) of your Fluo
> application and then click on the latest attempt ID (appattempt_*). You
> should see a list of containers.  There should be a container for the
> application master (typically container 1), a Fluo oracle (typically
> container 2), and Fluo workers (containers 3+).  You can view the log files
> produced by a container by clicking on its 'logs' link.  Logs from Fluo
> observers will be in the worker_*.log file for each of your worker
> containers.
>
> If you don't want to use the YARN resource manager web interface, you can
> also view these logs in the directory specified by
> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This method
> works well on one machine but on cluster your containers will probably be
> on different machines. See the YARN documentation below for more info about
> this property:
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org_docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5Aewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
>
> Best,
> Mike
>
> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb <caleb.me...@parsons.com>
> wrote:
>
> Hello,
>
> I'm attempting to debug a Fluo application and am having difficulty
> locating the logs for my observers.  I've looked within the logs for
> hadoop-yarn, but am not seeing any logging statements for my observers.
> Where do observers log out of the box in a normal cloudera distribution?
> Do I need to do something else in addition to logging to get my observers
> to generate logs?
>
> Thanks,
> Caleb
>

Re: debugging fluo

[Mike Walch]

Hi Caleb,

The logs for a Fluo application can be found in YARN but they are tricky to
find. Fluo should have better documentation on this which I will add now.

The easiest way to view the logs for a Fluo application is to use the web
interface for the YARN resource manager (http://localhost:8088/cluster).
First, click on the application ID (i.e application_*) of your Fluo
application and then click on the latest attempt ID (appattempt_*). You
should see a list of containers.  There should be a container for the
application master (typically container 1), a Fluo oracle (typically
container 2), and Fluo workers (containers 3+).  You can view the log files
produced by a container by clicking on its 'logs' link.  Logs from Fluo
observers will be in the worker_*.log file for each of your worker
containers.

If you don't want to use the YARN resource manager web interface, you can
also view these logs in the directory specified by
'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This method
works well on one machine but on cluster your containers will probably be
on different machines. See the YARN documentation below for more info about
this property:

https://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

Best,
Mike

On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb  wrote:

Hello,

I'm attempting to debug a Fluo application and am having difficulty
locating the logs for my observers.  I've looked within the logs for
hadoop-yarn, but am not seeing any logging statements for my observers.
Where do observers log out of the box in a normal cloudera distribution?
Do I need to do something else in addition to logging to get my observers
to generate logs?

Thanks,
Caleb

Re: [TEST] Fluo 1.0.0-incubating-rc0

[Mike Walch]

I checked over rc0 and didn't find any issues.  While rc1 needs to be
latest commit, the release artifacts look good.

On Mon, Sep 19, 2016 at 5:32 PM Christopher  wrote:

> My plan is to submit an RC1 for a vote tomorrow.
>
> On Fri, Sep 16, 2016 at 4:39 PM Christopher  wrote:
>
> > This is just a test. I'll keep the staging repo around until we get to a
> > final approved release. Mainly right now what I'm looking for is a sanity
> > check to make sure that the build artifacts pushed to the staging repo
> are
> > what we expect things like jars, tarballs, etc., all named
> correctly...
> > no unexpected shaded or test jars, or anything else we didn't expect to
> get
> > pushed.
> >
> > This test staging repo is mostly about getting the build system correct
> > and the wording of the email template right, and less about actually
> > testing the contents (though, that's welcome too).
> >
> > On Fri, Sep 16, 2016 at 4:27 PM Keith Turner  wrote:
> >
> >> -1  because would like to get a few API changes in that were just made
> >> in before releasing 1.0.0.
> >>
> >> I have not had a chance to review this RC.  I would still like to
> >> review this RC to look for any issues witht he release process.  I
> >> plan to look at it Mon morning, can we keep the staging repo around
> >> till then?
> >>
> >> On Wed, Sep 14, 2016 at 10:31 PM, Christopher 
> >> wrote:
> >> > The following is a test of my script from
> >> > https://github.com/apache/incubator-fluo/pull/769
> >> > Please begin reviewing what it has produced so I can iron out the
> >> release
> >> > process a bit, and so you can find any build-bugs prior to a release
> >> > candidate vote with RC1.
> >> >
> >> > =
> >> > Fluo Developers,
> >> >
> >> > Please consider the following candidate for Fluo 1.0.0-incubating.
> >> >
> >> > Git Commit:
> >> > f7a7aaf194ec2c0bef85d97e7ce939232d9a1491
> >> > Branch:
> >> > 1.0.0-incubating-rc0
> >> >
> >> > If this vote passes, a gpg-signed tag will be created using:
> >> > git tag -f -m 'Apache Fluo 1.0.0-incubating' -s
> >> > rel/fluo-1.0.0-incubating \
> >> > f7a7aaf194ec2c0bef85d97e7ce939232d9a1491
> >> >
> >> > Staging repo:
> >> > https://repository.apache.org/content/repositories/orgapachefluo-1011
> >> > Source (official release artifact):
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachefluo-1011/org/apache/fluo/fluo/1.0.0-incubating/fluo-1.0.0-incubating-source-release.tar.gz
> >> > Binary:
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachefluo-1011/org/apache/fluo/fluo/1.0.0-incubating/fluo-1.0.0-incubating-bin.tar.gz
> >> > (Append ".sha1", ".md5", or ".asc" to download the signature/hash for
> a
> >> > given artifact.)
> >> >
> >> > All artifacts were built and staged with:
> >> > mvn release:prepare && mvn release:perform
> >> >
> >> > Signing keys are available at
> >> > https://www.apache.org/dist/incubator/fluo/KEYS
> >> > (Expected fingerprint: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D)
> >> >
> >> > Release notes (in progress) can be found at:
> >> > https://fluo.apache.org/.../1.0.0-incubating
> >> >
> >> > Please vote one of:
> >> > [ ] +1 - I have verified and accept...
> >> > [ ] +0 - I have reservations, but not strong enough to vote against...
> >> > [ ] -1 - Because..., I do not accept...
> >> > ... these artifacts as the 1.0.0-incubating release of Apache Fluo.
> >> >
> >> > This vote will end on Sun Sep 18 02:30:00 UTC 2016
> >> > (Sat Sep 17 22:30:00 EDT 2016 / Sat Sep 17 19:30:00 PDT 2016)
> >> >
> >> > Thanks!
> >> >
> >> > P.S. Hint: download the whole staging repo with
> >> > wget -erobots=off -r -l inf -np -nH \
> >> >
> >> https://repository.apache.org/content/repositories/orgapachefluo-1011/
> >> > # note the trailing slash is needed
> >>
> >
>

Re: [VOTE] Apache Fluo Parent POM 1 (rc3) and Build Resources 1.0.0 (rc1)

[Mike Walch]

+1

* Verified sha1 & md5 hashes for fluo-parent & build-resoures
* Verified signatures for fluo-parent & build-resources
* Built Fluo and Fluo recipes using staging repo and new fluo-parent

On Fri, Aug 5, 2016 at 11:11 AM Keith Turner  wrote:

> +1
>
>  * sigs and hashes are good on parent pom and resources jar
>  * able to build Fluo and Fluo Recipes using staging repo and parent
> pom version 1-incubating
>  * Resources eclipse-formatter.xml and java-checkstyle.xml in jar are
> same as files in rc1 branch
>  * Parent pom in staging repo is same as parent pom in rc3 branch
>
> On Thu, Aug 4, 2016 at 4:48 PM, Christopher  wrote:
> > Fluo Developers,
> >
> > I'm combining these two votes, because they are a bit interdependent, and
> > both are small.
> >
> > Please consider the following candidates for Fluo Parent POM 1-incubating
> > and Fluo Build Resources 1.0.0-incubating.
> >
> > Git Commits/branches:
> > 02d4ea2332598a94285985ee8a1c8e92a42b4770 (build-resources-1.0.0-rc1)
> > 95c48e3f14faf5cdca259d8ec60ec68b640fce1e (fluo-parent-1-rc3)
> >
> > If this vote passes, a vote will be brought to the IPMC, and if that
> > passes, gpg-signed tags will be created using:
> > git tag -f -m 'Apache Fluo Build Resources 1.0.0-incubating' -s
> > rel/build-resources-1.0.0-incubating
> > 02d4ea2332598a94285985ee8a1c8e92a42b4770
> > git tag -f -m 'Apache Fluo Parent POM 1-incubating' -s
> > rel/fluo-parent-1-incubating 95c48e3f14faf5cdca259d8ec60ec68b640fce1e
> >
> > Staging repo:
> > https://repository.apache.org/content/repositories/orgapachefluo-1004
> >
> > Source artifacts:
> >
> https://repository.apache.org/content/repositories/orgapachefluo-1004/org/apache/fluo/build-resources/1.0.0-incubating/build-resources-1.0.0-incubating-source-release.tar.gz
> >
> https://repository.apache.org/content/repositories/orgapachefluo-1004/org/apache/fluo/fluo-parent/1-incubating/fluo-parent-1-incubating-source-release.tar.gz
> > (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
> > given artifact.)
> >
> > Signing key fingerprint is: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D
> >
> > Please vote one of:
> > [ ] +1 - I have verified and accept...
> > [ ] +0 - I have reservations, but not strong enough to vote against...
> > [ ] -1 - Because..., I do not accept...
> > ... these artifacts as the 1-incubating release of Apache Fluo Parent
> POM.
> >
> > This vote will end on Sun Aug 07 21:00:00 UTC 2016
> > (Sun Aug 07 17:00:00 EDT 2016 / Sun Aug 07 14:00:00 PDT 2016)
> >
> > Thanks!
> >
> > P.S. Hint: download the whole staging repo with
> > wget -erobots=off -r -l inf -np -nH \
> >
> https://repository.apache.org/content/repositories/orgapachefluo-1004/
> > # note the trailing slash is needed
>

Re: Github pull request and mailing list

[Mike Walch]

The dev list would be nice if only one email was sent for a new pull
request.  However, I think emails are sent every time someone comments on
the PR.  Therefore, I prefer them being sent to the notifications list.

On Wed, Aug 3, 2016 at 11:00 AM Christopher  wrote:

> For completeness, I'll respond here saying the same: I prefer notifications
> list. I can watch the GitHub repo if i want more prominent notifications.
>
> On Wed, Aug 3, 2016, 10:58 Keith Turner  wrote:
>
> > Currently github pull request are not generating emails to any list.
> > I want to file an issue w/ infra to fix this, but I was wondering if
> > anyone had a preference on which list to send to?  I am slightly
> > leaning towards sending the emails to the notfications list, but I can
> > see value in having those going to the dev list.
> >
> > I accidentally sent this to the wrong list.   Christopher had
> > responded the he preferred notifications.
> >
> > https://groups.google.com/forum/#!msg/fluo-dev/Nxz6ndYVRxM/2amblvRwBAAJ
> >
>

Re: Test staging release

[Mike Walch]

I was able to build fluo-recipes using fluo-parent 1-incubating & fluo
1.0.0-incubating from the staging repo.  I am ready to vote.

On Tue, Jul 26, 2016 at 11:19 AM Keith Turner  wrote:

> I was able to build fluo-quickstart, fluo-recipes, phrasecount,
> fluo-stress, and webindex against fluo 1.0.0-incubating from this staging
> repo.  I ran mvn verify on each of those projects.
>
> Are we ready to vote on the parent pom?
>
> On Mon, Jul 25, 2016 at 6:50 PM, Christopher  wrote:
>
> > I tested staging a release today for fluo-parent and fluo:
> > https://repository.apache.org/content/repositories/orgapachefluo-1001/
> >
> > I created a few follow-on issues to address, and opened one PR to fix
> some
> > immediate issues.
> >
>

Re: Projects using GitHub issues

[Mike Walch]

We do have a 'how to contribute' page (
https://fluo.apache.org/how-to-contribute/) but it needs to be updated with
how we are working around problems with GH issues.

On Tue, Jun 28, 2016 at 3:44 PM Keith Turner  wrote:

> On Tue, Jun 28, 2016 at 3:35 PM, Josh Elser  wrote:
>
> > Ya I groked what you guys were doing this morning, but didn't realize
> that
> > was why.
> >
> > Is there any process written down on the website as it evolves?
> >
>
> No, we can do that.
>
>
> > On Jun 28, 2016 10:06 AM, "Keith Turner"  wrote:
> >
> > > Thanks Josh.  We have realized how much of an impediment not being able
> > to
> > > assign and label issues is.  Feel free to point people to our wiki page
> > for
> > > the 1.0.0 release to show what we are doing since we can not label and
> > > assign.  Normally we would have tagged and assigned the issues instead
> of
> > > creating that page.
> > >
> > > On Tue, Jun 28, 2016 at 12:35 PM, Josh Elser 
> > wrote:
> > >
> > > > FYI, I'll be trying to keep an eye on this as well..
> > > > -- Forwarded message --
> > > > From: "John D. Ament" 
> > > > Date: Jun 28, 2016 7:17 AM
> > > > Subject: Projects using GitHub issues
> > > > To: "gene...@incubator.apache.org" 
> > > > Cc:
> > > >
> > > > All,
> > > > >
> > > > > I started a discussion on legal discuss, as I was told VP Legal
> > > approved
> > > > > using GitHub issues.  I have no specific concerns, other than
> > ensuring
> > > > that
> > > > > permissions are propagated.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/d2cb0eed30d72976a2f54893f132c1fe300a86d2fdbce1422763c9f0@%3Clegal-discuss.apache.org%3E
> > > > >
> > > > > Its clear to me that legal-discuss isn't where this should start,
> so
> > > I'm
> > > > > not sure why VP Legal approved.  Anyways, I wanted to get opinions
> > from
> > > > the
> > > > > incubator on who should be discussing this issue.
> > > > >
> > > > > John
> > > > >
> > > >
> > >
> >
>

Re: Contribute in Fluo

[Mike Walch]

Hi Garvit,

We have some documentation on how to contribute (
http://fluo.apache.org/how-to-contribute/) on our website.  Also, if you
don't find an issue that you want work on, one way to contribute is to
build a example Fluo application (see
http://fluo.apache.org/related-projects/ for examples) and provide feedback
through new issues or pull requests.

-Mike

On Tue, Jun 21, 2016 at 2:42 PM Christopher  wrote:

> Hi Garvit,
>
> Sure. We're just getting set up transitioning to the Apache Incubator.
> We're using GitHub issues to track bugs/features:
> https://github.com/apache/incubator-fluo/issues and the current workflow
> is
> to submit pull requests for feature changes.
>
> We're still working on setting up the website (https://fluo.apache.org/),
> so if you have any suggestions there, that would be helpful, too.
>
> On Tue, Jun 21, 2016 at 2:20 PM Garvit Bansal 
> wrote:
>
> > Hi All,
> >
> > My name is Garvit and I like to contribute in Fluo. I found this project
> > interesting and want to make contribution to it. Can somebody help me how
> > to start with, bug fixes or something like that which will help me to get
> > in flow.
> >
> > Garvit
> > --
> > *Garvit Bansal,*
> > Software Developer at Flipkart
> > B.Tech in Computer Science and Engineering
> > The LNM Institute of Information Technology
> > Jaipur, Rajasthan-302001
> > *Mobile:* 9610728532, 9886384276
> > *Website:* garvitbansal244.appspot.com
> >
> > 
> >
>