Re: About fluo-bytes

[Keith Turner]

On Wed, Aug 9, 2017 at 11:26 PM, Dylan Hutchison
<dhutc...@cs.washington.edu> wrote:
> It could be useful to cite some related work.  There is a project called
> Weld
> <https://blog.acolyer.org/2017/01/16/weld-a-common-runtime-for-high-performance-data-analytics/>
>  (source code <https://github.com/weld-project/weld>) in development at
> Databricks.  They aim to create a common runtime underneath a bunch of
> python libraries (NumPy, Tensorflow, Spark, Pandas, etc).  When these
> different libraries use a common runtime (e.g., common data structures),
> there is no need to copy data between them.  Weld goes a step further in
> the ability to optimize expressions across these libraries, rather than
> just data transfer.
>
> I think there are many projects that aim to provide "common ground" between
> a set of related libraries.  Intel's Nervana Graph
> <https://www.intelnervana.com/intel-nervana-graph-preview-release/> (source
> code <https://github.com/NervanaSystems/ngraph>) is another that comes to
> mind for ML frameworks.  The difference of FluoBytes is that it is
> low-level and well-scoped, making it modular and easy to use in any Java

It's currently well-scoped, I have been wondering how can it be kept
that way going forward...

> project.  FluoBytes does not aim to boil the ocean by optimizing all kinds
> of expressions on bytes; it only relates to the storage and transfer of
> bytes.
>
> On Wed, Aug 9, 2017 at 11:54 AM, Christopher <ctubb...@apache.org> wrote:
>
>> Apparently, you can't fork and create pull requests until the repo is
>> non-empty. So, I pushed a commit with just the LICENSE and NOTICE files.
>> Feel free to sanity check those. Everything else I'll put in a proper PR
>> for review.
>>
>> On Wed, Aug 9, 2017 at 10:12 AM Keith Turner <ke...@deenlo.com> wrote:
>>
>> > On Tue, Aug 8, 2017 at 7:50 PM, Christopher <ctubb...@apache.org> wrote:
>> > > On Tue, Aug 8, 2017 at 1:33 PM Keith Turner <ke...@deenlo.com> wrote:
>> > >
>> > >> On Fri, Aug 4, 2017 at 4:42 PM, Christopher <ctubb...@apache.org>
>> > wrote:
>> > >> > Fluoers,
>> > >> >
>> > >> > I created a fluo-bytes repository in GitBox[1], so we can try to
>> > create a
>> > >>
>> > >> This is great. I will take a stab at putting together the initial PR
>> > >> for the repo unless someone else was interested.
>> > >>
>> > >>
>> > > It was my intention to put some effort into this this week, but I don't
>> > > mind collaborating. I just don't want to be stuck doing only reviews.
>> :)
>> >
>> > I'll wait for your initial PR to the repo.  Let me know if you want
>> > help with anything before then.
>> >
>> > >
>> > >
>> > >
>> > >> > dependency-free, standalone implementation of the basic Bytes
>> > features we
>> > >> > need, based on Keith's observations in his blog post[2].
>> > >>
>> > >> At some point we need to circle back to the openjdk discuss list and
>> > >> let them know we are working on this.  There were a few people there
>> > >> who expressed interest in a project like this.  Maybe we can do that
>> > >> after we get the basic readme and initial import up. Reading this post
>> > >> made me think of the readme a lot.
>> > >>
>> > >>
>> > > +1; do you have a link to that discuss thread? I'm not familiar with
>> > this,
>> > > and was not a participant.
>> >
>> > http://mail.openjdk.java.net/pipermail/discuss/2016-November/004062.html
>> >
>> > >
>> > >
>> > >> >
>> > >> > Over the next few weeks, I'd like to try to start using it to
>> create a
>> > >> > small library of the following:
>> > >> >
>> > >> > * A ByteSequence interface (analogous to CharSequence)
>> > >> > * BytesBuilder (analogous to StringBuilder)
>> > >> > * an immutable Bytes implementation of ByteSequence (analogous to
>> > String)
>> > >> >
>> > >> > Maybe later, we can add useful InputStream and OutputStream
>> > >> implementations
>> > >> > and other useful tools, but it should always be a small library
>> with a
>> > >> > narrow focus on manipulating byte sequences.
>> > &

Re: ASF Board Report for Fluo - Initial Reminder for August 2017

[Keith Turner]

On Wed, Aug 9, 2017 at 1:34 PM, Christopher <ctubb...@apache.org> wrote:
> On Wed, Aug 9, 2017 at 10:41 AM Keith Turner <ke...@deenlo.com> wrote:
>
>> On Tue, Aug 8, 2017 at 7:36 PM, Christopher <ctubb...@apache.org> wrote:
>> > On Tue, Aug 8, 2017 at 5:47 PM Keith Turner <ke...@deenlo.com> wrote:
>> >>
>> >> For reference I used the following to collect stats
>> >>
>> >>
>> >>
>> >>
>> https://github.com/issues?utf8=%E2%9C%93=is%3Aissue+repo%3Aapache%2Ffluo+repo%3Aapache%2Ffluo-recipes+repo%3Aapache%2Ffluo-website+closed%3A%3E%3D2017-05-01+
>> >>
>> >>
>> https://github.com/issues?utf8=%E2%9C%93=is%3Aissue+repo%3Aapache%2Ffluo+repo%3Aapache%2Ffluo-recipes+repo%3Aapache%2Ffluo-website+created%3A%3E%3D2017-05-01+
>> >>
>> >>
>> https://github.com/issues?utf8=%E2%9C%93=is%3Apr+repo%3Aapache%2Ffluo+repo%3Aapache%2Ffluo-recipes+repo%3Aapache%2Ffluo-website+closed%3A%3E%3D2017-05-01+
>> >>
>> >>
>> https://github.com/issues?utf8=%E2%9C%93=is%3Apr+repo%3Aapache%2Ffluo+repo%3Aapache%2Ffluo-recipes+repo%3Aapache%2Ffluo-website+created%3A%3E%3D2017-05-01+
>> >> git log --since 2017-05-01 | grep Author | sort -u
>> >>
>> >> On Tue, Aug 8, 2017 at 5:46 PM, Keith Turner <ke...@deenlo.com> wrote:
>> >> > Here is a draft of the report
>> >> >
>> >> > ## Description:
>> >> >
>> >> >  - Apache Fluo is an open source implementation of Percolator (which
>> >> > populates
>> >> >Google's search index) for Apache Accumulo. With Fluo, users can
>> >> > continuously
>> >> >join new data into large existing data sets without reprocessing
>> all
>> >> > data.
>> >> >Unlike batch and streaming frameworks, Fluo offers much lower
>> latency
>> >> > and can
>> >> >operate on extremely large data sets.
>> >
>> >
>> > I'm not a big fan of this description. There's some elements of debatable
>> > accuracy and relevancy. I suggest dropping the history/background and
>>
>> What do you think is inaccurate?
>>
>>

Good feedback and discussion, definitely influenced what I put in the
report.  I never responded because I was in a time crunch trying to
get the report out the door.  I am going to open an issue to improve
the intro on the website based in this discussion.

> It's not that it's inaccurate... it's that it's temporal. Saying that it
> offers "much lower latency" compared to others is not likely to be a
> resilient statement as advances in those other frameworks progress. It may

I think its reasonable to write things that one believes are presently
true.  If there is a concern can always explicitly state its subject
to change.

> be true... but it could not be true at any given moment. Same with "can
> operate on extremely large data sets"... it's not necessarily accurate to
> say this is "unlike" others, because others may be able to do that, too, at
> any given moment.
>
> More importantly, I think, is whether or not these are relevant. Our docs
> can explain more details about history, background, prior technologies, and
> comparisons with other things. Those things should not distract from
> answering "What is *this* in front of me?" question. I especially think

My goal for contextualizing was to explain "what is this in front of
me" in relative terms.  Sometimes its easier to understand a thing
relative to other things that you know.

> this or the board report... the board isn't going to want to see an
> explanation of what Percolator is, or an explanation of limitations in
> other software... they just want a reminder of what this particular Apache
> project is about.
>
>
>>
>> > comparison stuff, and using a more "get to the point" description which
>> lets
>> > users know why they should care about Fluo, like:
>> >
>> > Apache Fluo is a distributed processing system which allows users to
>> > continuously and incrementally join new data with extremely large
>> existing
>> > data sets without reprocessing all the data. It provides low-latency,
>> > high-throughput data processing into Apache Accumulo, with cross-node
>> > transaction support and a notification system to automatically process
>> new
>> > data with a user-defined workflow.
>>
>>
>> Thinking about this some more base on your feedback, I think there are
>> three different aspects that a

Re: About fluo-bytes

[Keith Turner]

On Tue, Aug 8, 2017 at 7:50 PM, Christopher <ctubb...@apache.org> wrote:
> On Tue, Aug 8, 2017 at 1:33 PM Keith Turner <ke...@deenlo.com> wrote:
>
>> On Fri, Aug 4, 2017 at 4:42 PM, Christopher <ctubb...@apache.org> wrote:
>> > Fluoers,
>> >
>> > I created a fluo-bytes repository in GitBox[1], so we can try to create a
>>
>> This is great. I will take a stab at putting together the initial PR
>> for the repo unless someone else was interested.
>>
>>
> It was my intention to put some effort into this this week, but I don't
> mind collaborating. I just don't want to be stuck doing only reviews. :)

I'll wait for your initial PR to the repo.  Let me know if you want
help with anything before then.

>
>
>
>> > dependency-free, standalone implementation of the basic Bytes features we
>> > need, based on Keith's observations in his blog post[2].
>>
>> At some point we need to circle back to the openjdk discuss list and
>> let them know we are working on this.  There were a few people there
>> who expressed interest in a project like this.  Maybe we can do that
>> after we get the basic readme and initial import up. Reading this post
>> made me think of the readme a lot.
>>
>>
> +1; do you have a link to that discuss thread? I'm not familiar with this,
> and was not a participant.

http://mail.openjdk.java.net/pipermail/discuss/2016-November/004062.html

>
>
>> >
>> > Over the next few weeks, I'd like to try to start using it to create a
>> > small library of the following:
>> >
>> > * A ByteSequence interface (analogous to CharSequence)
>> > * BytesBuilder (analogous to StringBuilder)
>> > * an immutable Bytes implementation of ByteSequence (analogous to String)
>> >
>> > Maybe later, we can add useful InputStream and OutputStream
>> implementations
>> > and other useful tools, but it should always be a small library with a
>> > narrow focus on manipulating byte sequences.
>> >
>> > The idea is that this will be semver, but will very strong prefer to
>> avoid
>> > ever going to a breaking 2.0 change, instead insuring it will be
>> backwards
>> > compatible for a *LONG* time, making it safe for use in other projects'
>> > APIs.
>>
>> When creating the readme, it would be good to try to explain the
>> rational for avoiding dropping methods.  I attempted that in my blog
>> post, but not sure how well I did at getting the point across.  I
>> think it would be best shown with an example that shows how it can be
>> hard to use two projects where one uses newer methods and another uses
>> older dropped methods.  Couple that with both projects having the
>> library in their API and its a huge headache for any users of both
>> libraries.
>>
>>
> +1
>
>
>> >
>> > I think this library would be useful not only for Fluo's API, but as a
>> > separate dependency-free library, it could be easily reused by many other
>>
>> We will also need to explain why dependencies are so important.  If
>> their were dependencies they would also need to follow very strict API
>> guarantees.  Having Java standard libs as the only dep is good because
>> Java itself is very rigorous about its API.
>>
>> Another thing that will need to be explained well in the readme is the
>> benefit of multiple APIs using the same immutable type, it avoids
>> copies when going between APIs.
>>
>>
> +1; it sounds like you've already got the README half written ;)

An extremely rough outline is half written.

>
>
>> > projects, such as Accumulo (and anybody else).
>> >
>> > [1]: https://github.com/apache/fluo-bytes
>> > [2]: https://fluo.apache.org/blog/2016/11/10/immutable-bytes/
>>

Re: New 'fluo-yarn' repo

[Keith Turner]

On Mon, Jul 17, 2017 at 4:38 PM, Mike Walch  wrote:
> Yes.  Fluo-yarn will be on a different release schedule.  A new release of

This is off topic, but reading this made me think : Moving the
documentation from the Fluo repo to the website will make it easier to
document using Fluo+YARN, Fluo+Mesos, etc.

> Fluo should in most cases not require a new release of fluo-yarn.  New
> releases of fluo-yarn will most likely be necessary to upgrade Apache Twill
> to support newer releases of Hadoop YARN.
>
> On Mon, Jul 17, 2017 at 4:26 PM Christopher  wrote:
>
>> Is the idea to have fluo-yarn on a different release cycle?
>>
>>
>> On Mon, Jul 17, 2017 at 3:52 PM Mike Walch  wrote:
>>
>> > I would like to create new repo (called 'fluo-yarn') using Gitbox to hold
>> > code for launching Fluo applications in YARN.  This is related to my
>> recent
>> > pull request [1]. Does anyone have any objections to this?
>> >
>> > [1]: https://github.com/apache/incubator-fluo/pull/883
>> >
>>

Re: [ANNOUNCE] Apache Fluo Recipes 1.1.0-incubating released

[Keith Turner]

 Thanks for the tip!  That would make the announcements much better.
We will do that in the future.

On Thu, Jun 29, 2017 at 2:52 PM, sebb <seb...@gmail.com> wrote:
> What is the project about? Why should I be interested in it?
> [rhetorical questions]
>
> The Announce emails are sent to people not on the developer or user lists.
> Most will have no idea what the project is about.
>
> So the e-mails should contain at least brief details of what the
> product does, and some info on why the new release might be of
> interest to them.
>
> Readers should not have to click the link to find out the basic information
> (although of course it is useful to have such links for further detail).
>
> Please can you add that information to future announce mails?
>
> Thanks.
>
>
> On 29 June 2017 at 19:06, Keith Turner <ke...@deenlo.com> wrote:
>> The Apache Fluo (incubating) team is happy to announce the release of
>> Fluo Recipes 1.1.0-incubating:
>>
>> https://fluo.apache.org/release/fluo-recipes-1.1.0-incubating/
>>
>> This release includes significant API improvements that make writing
>> applications faster and easier.
>>
>> Thanks,
>> The Apache Fluo team
>>
>> =
>>
>> *Disclaimer*
>>
>> Apache Fluo is an effort undergoing incubation at The Apache Software
>> Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
>> required of all newly accepted projects until a further review indicates
>> that the infrastructure, communications, and decision making process have
>> stabilized in a manner consistent with other successful ASF projects. While
>> incubation status is not necessarily a reflection of the completeness or
>> stability of the code, it does indicate that the project has yet to be
>> fully endorsed by the ASF.
>>
>> -
>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
>> For additional commands, e-mail: general-h...@incubator.apache.org
>>

Re: [VOTE] Apache Fluo Recipes 1.1.0-incubating-rc1

[Keith Turner]

+1

 * Sigs and hashes are good.
 * Was able to run webindex using jars from staging repo

On Mon, Jun 12, 2017 at 7:16 PM, Keith Turner <ke...@deenlo.com> wrote:
> Fluo Developers,
>
> Please consider the following candidate for Fluo Recipes 1.1.0-incubating.
>
> Git Commit:
> 4769fa9a3a43af030de318c0423a7cae5c5eb4fe
> Branch:
> 1.1.0-incubating-rc1
>
> If this vote passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Fluo Recipes 1.1.0-incubating' -s
> rel/fluo-recipes-1.1.0-incubating \
> 4769fa9a3a43af030de318c0423a7cae5c5eb4fe
> Staging repo: 
> https://repository.apache.org/content/repositories/orgapachefluo-1018
> Source (official release artifact):
> https://repository.apache.org/content/repositories/orgapachefluo-1018/org/apache/fluo/fluo-recipes/1.1.0-incubating/fluo-recipes-1.1.0-incubating-source-release.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for
> a given artifact.)
>
> All artifacts were built and staged with:
> mvn release:prepare && mvn release:perform
>
> Signing keys are available at https://www.apache.org/dist/incubator/fluo/KEYS
> (Expected fingerprint: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)
>
> Release notes will be created during the vote.
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1.1.0-incubating release of Apache Fluo Recipes.
>
> Voting is open until at least Thu Jun 15 23:30:00 UTC 2017
> (Thu Jun 15 19:30:00 EDT 2017 / Thu Jun 15 16:30:00 PDT 2017)
>
> After this, the release manager can call the vote at any time.
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
> https://repository.apache.org/content/repositories/orgapachefluo-1018/
> # note the trailing slash is needed

[RESULT] [VOTE] Apache Fluo 1.1.0-incubating (rc1)

[Keith Turner]

The vote passed with 3 +1 and no -1. Thanks Josh, Justin, John, and
sebb for looking at this release.

Kicking of a concurrent recipes release

[Keith Turner]

I am thinking of kicking off a concurrent recipes release that depends
on the Fluo 1.1.0-incubating release that currently being voted on.
Before I go through the trouble wanted to see if there was any
objections.

[VOTE] Apache Fluo 1.1.0-incubating (rc1)

[Keith Turner]

Dear IPMC,

Please vote for the following release candidate of Apache Fluo
1.1.0-incubating.

PPMC vote thread:
https://lists.apache.org/thread.html/c815b7a22fab10281ab2d4e88418a47a5903ba85327058bb727c7ccd@%3Cdev.fluo.apache.org%3E
https://lists.apache.org/thread.html/99508ef4a2b782b82494e4250713c1d0917b75cf7149bec64455bea2@%3Cdev.fluo.apache.org%3E

Staged dist artifacts:
https://dist.apache.org/repos/dist/dev/incubator/fluo/fluo/1.1.0-incubating-rc1/

Staged Maven repository:
https://repository.apache.org/content/repositories/orgapachefluo-1017/

Signing KEYS:
https://www.apache.org/dist/incubator/fluo/KEYS
(fingerprint for this release: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)

Git repo:
https://gitbox.apache.org/repos/asf/incubator-fluo
(branch: 1.1.0-incubating-rc1,
commit: ad8ee492e2f435405f98d825781098c55186f4fb)

This vote will end on Fri Jun  9 20:30:00 UTC 2017
(Fri Jun  9 16:30:00 EDT 2017 / Fri Jun  9 13:30:00 PDT 2017)

Re: [VOTE] Fluo 1.1.0-incubating-rc1

[Keith Turner]

On Tue, Jun 6, 2017 at 1:10 PM, Christopher <ctubb...@apache.org> wrote:
> No more comments from me. We can address the issues I saw later.

I opened two issues.  Also, I updated the release documentation PR I
am working on to mention checking the date in the NOTICE file.

https://github.com/apache/incubator-fluo/issues/871
https://github.com/apache/incubator-fluo/issues/872

>
> On Tue, Jun 6, 2017, 13:01 Keith Turner <ke...@deenlo.com> wrote:
>
>> On Tue, Jun 6, 2017 at 12:40 PM, Josh Elser <els...@apache.org> wrote:
>> > On 6/6/17 12:29 PM, Mike Walch wrote:
>> >>>
>> >>> No problem, thanks for taking the time to look over it.   Does it
>> >>
>> >> matter that some votes were after the window?
>> >>
>> >> The release manager (at their discretion) should be able to give voters
>> >> more time (especially when the vote opens late in the week like this
>> did).
>> >> Voting should just continue after the deadline until the release manager
>> >> ends it (they can still end it at the deadline).  We should change the
>> >> form
>> >> email that opens the vote to have the following text:
>> >>
>> >> This vote will remain open until at least Mon Jun  5 17:00:00 UTC 2017
>> >> (Mon Jun  5 13:00:00 EDT 2017 / Mon Jun  5 10:00:00 PDT 2017). Voting
>> can
>> >> continue after this deadline until the release manager sends an email
>> >> ending the vote.
>> >
>> >
>> > Exactly. The 72-hour window is meant to be a minimum, not a maximum. The
>> > amended text is nice.
>>
>> Ok that makes sense to me.   I will wait to see if Christopher has any
>> more comments before closing the vote.
>>

Re: [VOTE] Fluo 1.1.0-incubating-rc1

[Keith Turner]

On Mon, Jun 5, 2017 at 7:03 PM, Christopher <ctubb...@apache.org> wrote:
> Oh, interesting. That seems wrong. The Maven convention is that the test
> directory is supposed to contain code for testing that particular module.
> If there's nothing to test, it shouldn't exist. A separate module with
> integration test code should exist in src/main, like in Accumulo's
> accumulo-test module. Breaking that Maven convention seems to be resulting
> in confusing behavior of the maven-source-plugin and maven-javadoc-plugin.

Is this something you want to fix before releasing?

>
> On Mon, Jun 5, 2017 at 6:13 PM Keith Turner <ke...@deenlo.com> wrote:
>
>> That module only has a src/test dir.  There is no src/main dir.  Maybe
>> that is the cause.
>>
>> On Mon, Jun 5, 2017 at 5:55 PM, Christopher <ctubb...@apache.org> wrote:
>> > Neither fluo-integration jar which is missing javadocs are included in
>> the
>> > lib directory either, so I'm guessing they're just for testing. I'd still
>> > expect a javadoc and source jar for them, though.
>> >
>> > On Mon, Jun 5, 2017 at 5:43 PM Christopher <ctubb...@apache.org> wrote:
>> >
>> >> The good:
>> >>
>> >> GPG signatures, hashes, SHA1 in Jar metadata all look good.
>> >> Source tarball matches SHA1 (except for generated DEPENDENCIES file from
>> >> Apache parent POM)
>> >> Tests with ITs pass
>> >> DISCLAIMER and LICENSE files are present and good
>> >>
>> >> The bad:
>> >>
>> >> The NOTICE file has the wrong copyright date (says 2016; should be 2017,
>> >> or 2016-2017 to be consistent with the generated NOTICE files in each
>> jar's
>> >> META-INF).
>> >>
>> >> The confusing/questionable:
>> >>
>> >> There's a few missing source and javadoc jars. I'm not sure if that's
>> >> intended, or how it happened:
>> >> fluo-integration-1.1.0-incubating-tests.jar is missing both
>> corresponding
>> >> source and javadoc jar
>> >> fluo-integration-1.1.0-incubating.jar is missing a corresponding javadoc
>> >> jar
>> >>
>> >> Several ERROR logs in the console during the build; probably expected,
>> >> from tests, but it's confusing, and could possibly be tests which aren't
>> >> checking for valid return codes or something. Could also just be a bad
>> log
>> >> configuration for the ITs:
>> >>
>> >> 2017-06-05 17:28:22,502 [config.FluoConfiguration] ERROR: Client
>> >> properties should not be set in your configuration if MiniFluo is
>> >> configured to start its own accumulo (indicated by
>> fluo.mini.start.accumulo
>> >> being set to true)
>> >> 2017-06-05 17:29:48,251 [server.AbstractNonblockingServer$FrameBuffer]
>> >> ERROR: Unexpected throwable while invoking!
>> >> 2017-06-05 17:29:55,779 [server.AbstractNonblockingServer$FrameBuffer]
>> >> ERROR: Read a frame size of 1633837924, which is bigger than the maximum
>> >> allowable buffer size for ALL connections.
>> >>
>> >>
>> >> On Mon, Jun 5, 2017 at 4:21 PM Keith Turner <ke...@deenlo.com> wrote:
>> >>
>> >>> +1
>> >>>
>> >>>  * sig and hashes all checked out
>> >>>  * Was able to build and test Fluo Recipes, Stress test, and Webindex
>> >>> against staging repo
>> >>>  * Was able to run web index against tar ball from staging repo
>> >>>  * source tar matches commit
>> >>>
>> >>>
>> >>> On Fri, Jun 2, 2017 at 12:53 PM, Keith Turner <ke...@deenlo.com>
>> wrote:
>> >>> > Fluo Developers,
>> >>> >
>> >>> > Please consider the following candidate for Fluo 1.1.0-incubating.
>> >>> >
>> >>> > Git Commit:
>> >>> > ad8ee492e2f435405f98d825781098c55186f4fb
>> >>> > Branch:
>> >>> > 1.1.0-incubating-rc1
>> >>> >
>> >>> > If this vote passes, a gpg-signed tag will be created using:
>> >>> > git tag -f -m 'Apache Fluo 1.1.0-incubating' -s
>> >>> rel/fluo-1.1.0-incubating \
>> >>> > ad8ee492e2f435405f98d825781098c55186f4fb
>> >>> >
>> >>> > Staging repo:
>> >>> https://repository.apache.org/content/repositories/orgapachefluo-1017
>

Re: [VOTE] Fluo 1.1.0-incubating-rc1

[Keith Turner]

On Tue, Jun 6, 2017 at 11:50 AM, Josh Elser <els...@apache.org> wrote:
> +1 (binding)
>
> * xsums/sigs OK
> * NOTICE has wrong copyright year (as ctubbsii already pointed out -- fix
> for release.next IMO)
> * No bin files in source release
> * apache-rat:check passes
> * DISCLAIMER is present
> * Can build from source
> * L look good (tricky as always, folks). Glanced through the list of
> things pulled down by fetch.sh and that appears fine (compatible licenses)
>
> Sorry for missing the stated window.

No problem, thanks for taking the time to look over it.   Does it
matter that some votes were after the window?


>
> - Josh
>
>
> On 6/2/17 12:53 PM, Keith Turner wrote:
>>
>> Fluo Developers,
>>
>> Please consider the following candidate for Fluo 1.1.0-incubating.
>>
>> Git Commit:
>>  ad8ee492e2f435405f98d825781098c55186f4fb
>> Branch:
>>  1.1.0-incubating-rc1
>>
>> If this vote passes, a gpg-signed tag will be created using:
>>  git tag -f -m 'Apache Fluo 1.1.0-incubating' -s
>> rel/fluo-1.1.0-incubating \
>>  ad8ee492e2f435405f98d825781098c55186f4fb
>>
>> Staging repo:
>> https://repository.apache.org/content/repositories/orgapachefluo-1017
>> Source (official release artifact):
>>
>> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz
>> Binary:
>> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz
>> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for
>> a given artifact.)
>>
>> All artifacts were built and staged with:
>>  mvn release:prepare && mvn release:perform
>>
>> Signing keys are available at
>> https://www.apache.org/dist/incubator/fluo/KEYS
>> (Expected fingerprint: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)
>>
>> Release notes (in progress) can be found at:
>> https://fluo.apache.org/.../1.1.0-incubating
>>
>> Please vote one of:
>> [ ] +1 - I have verified and accept...
>> [ ] +0 - I have reservations, but not strong enough to vote against...
>> [ ] -1 - Because..., I do not accept...
>> ... these artifacts as the 1.1.0-incubating release of Apache Fluo.
>>
>> This vote will end on Mon Jun  5 17:00:00 UTC 2017
>> (Mon Jun  5 13:00:00 EDT 2017 / Mon Jun  5 10:00:00 PDT 2017)
>>
>> Thanks!
>>
>> P.S. Hint: download the whole staging repo with
>>  wget -erobots=off -r -l inf -np -nH \
>>
>> https://repository.apache.org/content/repositories/orgapachefluo-1017/
>>  # note the trailing slash is needed
>>
>

Re: [VOTE] Fluo 1.1.0-incubating-rc1

[Keith Turner]

+1

 * sig and hashes all checked out
 * Was able to build and test Fluo Recipes, Stress test, and Webindex
against staging repo
 * Was able to run web index against tar ball from staging repo
 * source tar matches commit


On Fri, Jun 2, 2017 at 12:53 PM, Keith Turner <ke...@deenlo.com> wrote:
> Fluo Developers,
>
> Please consider the following candidate for Fluo 1.1.0-incubating.
>
> Git Commit:
> ad8ee492e2f435405f98d825781098c55186f4fb
> Branch:
> 1.1.0-incubating-rc1
>
> If this vote passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Fluo 1.1.0-incubating' -s rel/fluo-1.1.0-incubating 
> \
> ad8ee492e2f435405f98d825781098c55186f4fb
>
> Staging repo: 
> https://repository.apache.org/content/repositories/orgapachefluo-1017
> Source (official release artifact):
> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz
> Binary: 
> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for
> a given artifact.)
>
> All artifacts were built and staged with:
> mvn release:prepare && mvn release:perform
>
> Signing keys are available at https://www.apache.org/dist/incubator/fluo/KEYS
> (Expected fingerprint: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)
>
> Release notes (in progress) can be found at:
> https://fluo.apache.org/.../1.1.0-incubating
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1.1.0-incubating release of Apache Fluo.
>
> This vote will end on Mon Jun  5 17:00:00 UTC 2017
> (Mon Jun  5 13:00:00 EDT 2017 / Mon Jun  5 10:00:00 PDT 2017)
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
> https://repository.apache.org/content/repositories/orgapachefluo-1017/
> # note the trailing slash is needed

Re: [VOTE] Fluo 1.1.0-incubating-rc1

[Keith Turner]

Its very much a work in progress, but I have started working on the
release notes.

https://github.com/keith-turner/incubator-fluo-website/blob/1.1.0-release-notes/_posts/release/2017-06-02-fluo-1.1.0-incubating.md

On Fri, Jun 2, 2017 at 12:53 PM, Keith Turner <ke...@deenlo.com> wrote:
> Fluo Developers,
>
> Please consider the following candidate for Fluo 1.1.0-incubating.
>
> Git Commit:
> ad8ee492e2f435405f98d825781098c55186f4fb
> Branch:
> 1.1.0-incubating-rc1
>
> If this vote passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Fluo 1.1.0-incubating' -s rel/fluo-1.1.0-incubating 
> \
> ad8ee492e2f435405f98d825781098c55186f4fb
>
> Staging repo: 
> https://repository.apache.org/content/repositories/orgapachefluo-1017
> Source (official release artifact):
> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz
> Binary: 
> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for
> a given artifact.)
>
> All artifacts were built and staged with:
> mvn release:prepare && mvn release:perform
>
> Signing keys are available at https://www.apache.org/dist/incubator/fluo/KEYS
> (Expected fingerprint: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)
>
> Release notes (in progress) can be found at:
> https://fluo.apache.org/.../1.1.0-incubating
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1.1.0-incubating release of Apache Fluo.
>
> This vote will end on Mon Jun  5 17:00:00 UTC 2017
> (Mon Jun  5 13:00:00 EDT 2017 / Mon Jun  5 10:00:00 PDT 2017)
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
> https://repository.apache.org/content/repositories/orgapachefluo-1017/
> # note the trailing slash is needed

Re: [VOTE] Fluo 1.1.0-incubating-rc1

[Keith Turner]

None of this changed the commit hash for the release.  It just put the
commit for the release at the top of the branch.

On Fri, Jun 2, 2017 at 1:16 PM, Keith Turner <ke...@deenlo.com> wrote:
> I made a slight mistake I pushed the 1.1.0-incubating-rc1 branch with
> the commit that transition form 1.1.0 to 1.2.0-SNAP.  I force pushed
> and corrected the branch.  I also pushed a branch
> 1.1.0-incubating-rc1-next with that transition commit.
>
> On Fri, Jun 2, 2017 at 12:53 PM, Keith Turner <ke...@deenlo.com> wrote:
>> Fluo Developers,
>>
>> Please consider the following candidate for Fluo 1.1.0-incubating.
>>
>> Git Commit:
>> ad8ee492e2f435405f98d825781098c55186f4fb
>> Branch:
>> 1.1.0-incubating-rc1
>>
>> If this vote passes, a gpg-signed tag will be created using:
>> git tag -f -m 'Apache Fluo 1.1.0-incubating' -s 
>> rel/fluo-1.1.0-incubating \
>> ad8ee492e2f435405f98d825781098c55186f4fb
>>
>> Staging repo: 
>> https://repository.apache.org/content/repositories/orgapachefluo-1017
>> Source (official release artifact):
>> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz
>> Binary: 
>> https://repository.apache.org/content/repositories/orgapachefluo-1017/org/apache/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz
>> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for
>> a given artifact.)
>>
>> All artifacts were built and staged with:
>> mvn release:prepare && mvn release:perform
>>
>> Signing keys are available at https://www.apache.org/dist/incubator/fluo/KEYS
>> (Expected fingerprint: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)
>>
>> Release notes (in progress) can be found at:
>> https://fluo.apache.org/.../1.1.0-incubating
>>
>> Please vote one of:
>> [ ] +1 - I have verified and accept...
>> [ ] +0 - I have reservations, but not strong enough to vote against...
>> [ ] -1 - Because..., I do not accept...
>> ... these artifacts as the 1.1.0-incubating release of Apache Fluo.
>>
>> This vote will end on Mon Jun  5 17:00:00 UTC 2017
>> (Mon Jun  5 13:00:00 EDT 2017 / Mon Jun  5 10:00:00 PDT 2017)
>>
>> Thanks!
>>
>> P.S. Hint: download the whole staging repo with
>> wget -erobots=off -r -l inf -np -nH \
>> https://repository.apache.org/content/repositories/orgapachefluo-1017/
>> # note the trailing slash is needed

Welcome new committer/IPMC member Chris McTague

[Keith Turner]

The Incubator Project Management Committee (IPMC) for Apache Fluo
has invited Chris McTague to become a committer/IPMC member and we
are pleased to announce that he has accepted.

Chris, feel free to say a few words about your development interests.

Completed Stress Test

[Keith Turner]

I just completed a successful stress test[3] run in prep for 1.1.0
release. Commit f0403a1 was used for the test. I sent a tweet[1] out
about it and I am providing more details here.

For the test I loaded an initial 1 billion random numbers via map
reduce.  Then I incrementally loaded 100 sets of 1 million random
numbers.  All random numbers were between 0 and 1 trillion.

Twelve m3.xlarge EC2 nodes were used.  Ten worker nodes.  One leader
node running ZK, NN, Accumulo master, and RM.  One node ran Grafana
and InfluxDB.  Muchos[2] was used to set it all up.

At the end of the test I looked at the next Oracle timestamp and
divided this by two to get an estimate of the number of transactions.
Based on this I think ~409 million transactions were executed during
the test run.  There were constant transaction collisions during the
test between 50 and 150 per second, the test is designed to produce
these.

Below is the final output from the test.

*Verifying Fluo & MapReduce results match*
Success! Fluo & MapReduce both calculated 1099396113 unique integers

Below is a summary of settings that were changed for performance.

I set the following Accumulo table settings

table.durability=flush
table.compaction.major.ratio=1.5
table.file.compress.blocksize=8K
table.file.compress.blocksize.index=32K
table.file.compress.type=snappy
table.file.replication=2

I set the following Accumulo system settings

tserver.cache.data.size=4536M
tserver.cache.index.size=512M
tserver.memory.maps.max=512M

I set the following Fluo props

fluo.worker.num.threads=128
fluo.loader.num.threads=32
fluo.loader.queue.size=64
fluo.yarn.worker.max.memory.mb=3584
fluo.yarn.worker.instances=10
io.fluo.impl.tx.commit.memory=41943040

For HDFS, local reads were enabled.  This makes a noticeable
difference in transactions per second.  I also ran a compaction after
the initial bulk import of a billion numbers to get ensure data was
local.

Watching Accumulo's monitor page, index cache hits were always at
100%.  Data cache oscillated  between 90% and 100% hit rate.  I think
as transactions worked on data higher up the tree, the hit rate went
up.

Accumulo 1.7.3, Zookeeper 3.4.9, Hadoop 2.7.2, Centos 7, and OpenJDK 8
(from Centos) were used to run the test.

[1]: https://twitter.com/ApacheFluo/status/867425323742351360
[2]: https://github.com/astralway/muchos/
[3]: https://github.com/astralway/stresso

Re: [DISCUSS] Improve how Fluo applications are run

[Keith Turner]

There are few possible goals with these changes :

  * support multiple launchers
  * stop using the local app dirs and store all app info in zookeeper and hdfs

Instead of creating a new fluo-local, maybe we could make the
following changes to the current fluo tarball and command.  Maybe
still create a fluo-yarn tarball for launching yarn.

Deprecate the following commands (these commands are yarn and/or app
dir related) :
  fluo list
  fluo new 
  fluo init 
  fluo start 
  fluo stop 
  fluo kill 
  fluo status 
  fluo info 

Add the following commands :

  # sets up a new app, putting its jars in hdfs and config in zookeeper
  fluo setup  {list of app jars}

  # updates a app, putting its jars in hdfs and config in zookeeper
  fluo update   {list of app jars}

  # retrieves an apps jars from hdfs
  fluo jars  

  # retrieves an apps config from zookeeper and prints it stdout
  fluo config 

  # deletes an apps data in zookeeper and hdfs
  fluo delete 

Following commands stay the same :

  # list all apps... or if given an app name lists info about that app
  fluo list [optional app name]
  fluo classpath
  fluo wait 
  fluo version
  fluo exec{}
  fluo scan 

On Fri, May 19, 2017 at 1:18 PM, Mike Walch  wrote:
> For Fluo 1.2, I would like to improve how Fluo applications are
> launched/run. This involves deprecating the current tarball distribution
> and creating downstream repos that create distributions for running Fluo
> applications locally and cluster managements tools suchs as YARN, Mesos,
> Kubernetes, etc.
>
> I have created repos for running Fluo locally and YARN and have started to
> design their functionality by creating scripts and documentation. No Java
> code has been written yet. The repos will eventually be moved to Apache
> infrastructure.
>
> https://github.com/mikewalch/fluo-local
> https://github.com/mikewalch/fluo-yarn
>
> In order to complete this task, a lot of new functionality/commands need to
> created upstream in Fluo. To coordinate this work, I created several issues
> under the 1.2 milestone.
>
> https://github.com/apache/incubator-fluo/milestone/6
>
> If you are interested in this work, review the documentation/scripts in
> fluo-local & fluo-yarn as well as the issues linked to above and let me
> know if you have any ideas or suggestions.

Re: [DISCUSS] Improve how Fluo applications are run

[Keith Turner]

On Fri, May 19, 2017 at 1:18 PM, Mike Walch  wrote:
> For Fluo 1.2, I would like to improve how Fluo applications are
> launched/run. This involves deprecating the current tarball distribution
> and creating downstream repos that create distributions for running Fluo
> applications locally and cluster managements tools suchs as YARN, Mesos,
> Kubernetes, etc.
>
> I have created repos for running Fluo locally and YARN and have started to
> design their functionality by creating scripts and documentation. No Java
> code has been written yet. The repos will eventually be moved to Apache
> infrastructure.
>
> https://github.com/mikewalch/fluo-local
> https://github.com/mikewalch/fluo-yarn
>
> In order to complete this task, a lot of new functionality/commands need to
> created upstream in Fluo. To coordinate this work, I created several issues
> under the 1.2 milestone.
>
> https://github.com/apache/incubator-fluo/milestone/6
>
> If you are interested in this work, review the documentation/scripts in
> fluo-local & fluo-yarn as well as the issues linked to above and let me
> know if you have any ideas or suggestions.


I want to help with this and work on parts of it, not sure which parts yet.

Re: [DISCUSS] Improve how Fluo applications are run

[Keith Turner]

>From a user standpoint, what are the main differences from the current
way of running Fluo apps?

Are your fluo-local and fluo-yarn repos functional?

I noticed in some of the issues that the command had a package of
core.command.   I was thinking it may be better to put commands in
their own module, so the commands can have a dependency like
jcommander w/o adding that to core.

On Fri, May 19, 2017 at 1:18 PM, Mike Walch  wrote:
> For Fluo 1.2, I would like to improve how Fluo applications are
> launched/run. This involves deprecating the current tarball distribution
> and creating downstream repos that create distributions for running Fluo
> applications locally and cluster managements tools suchs as YARN, Mesos,
> Kubernetes, etc.
>
> I have created repos for running Fluo locally and YARN and have started to
> design their functionality by creating scripts and documentation. No Java
> code has been written yet. The repos will eventually be moved to Apache
> infrastructure.
>
> https://github.com/mikewalch/fluo-local
> https://github.com/mikewalch/fluo-yarn
>
> In order to complete this task, a lot of new functionality/commands need to
> created upstream in Fluo. To coordinate this work, I created several issues
> under the 1.2 milestone.
>
> https://github.com/apache/incubator-fluo/milestone/6
>
> If you are interested in this work, review the documentation/scripts in
> fluo-local & fluo-yarn as well as the issues linked to above and let me
> know if you have any ideas or suggestions.

Re: Testing for 1.1.0 release

[Keith Turner]

With #827[1] and #839[2] fixed, I plan to start running test next week.

[1]: https://github.com/apache/incubator-fluo/issues/827
[2]: https://github.com/apache/incubator-fluo/issues/839

On Wed, May 17, 2017 at 3:30 PM, Michael Wall <mjw...@gmail.com> wrote:
> I am going to work https://github.com/apache/incubator-fluo/issues/827
> (Bytes.copyTo)
> tomorrow, but you shouldn't hold up testing.
>
> On Wed, May 17, 2017 at 3:24 PM Keith Turner <ke...@deenlo.com> wrote:
>
>> I have been running some quick test of Webindex[1] and Stresso[2]
>> against 1.1.0-SNAPSHOT and have found a few bugs.  I am going to run
>> some longer test on EC2 in preparation for a 1.1.0 release.  Before I
>> do this, I wanted to see if anyone had an changes they would like to
>> make before  this 1.1.0 testing is done.  Let me know of anything and
>> I can hold off on testing.
>>
>> [1]: https://github.com/astralway/webindex
>> [2]: https://github.com/astralway/stresso
>>

Re: Testing for 1.1.0 release

[Keith Turner]

On Wed, May 17, 2017 at 3:39 PM, Christopher <ctubb...@apache.org> wrote:
> Can only assign to committers.

bummer

>
> On Wed, May 17, 2017, 15:37 Keith Turner <ke...@deenlo.com> wrote:
>
>> I tried assigning that issue to you (we now have gitbox), but was
>> unable too.  Seems there is only a set list that we can assign to.
>>
>> On Wed, May 17, 2017 at 3:30 PM, Michael Wall <mjw...@gmail.com> wrote:
>> > I am going to work https://github.com/apache/incubator-fluo/issues/827
>> > (Bytes.copyTo)
>> > tomorrow, but you shouldn't hold up testing.
>> >
>> > On Wed, May 17, 2017 at 3:24 PM Keith Turner <ke...@deenlo.com> wrote:
>> >
>> >> I have been running some quick test of Webindex[1] and Stresso[2]
>> >> against 1.1.0-SNAPSHOT and have found a few bugs.  I am going to run
>> >> some longer test on EC2 in preparation for a 1.1.0 release.  Before I
>> >> do this, I wanted to see if anyone had an changes they would like to
>> >> make before  this 1.1.0 testing is done.  Let me know of anything and
>> >> I can hold off on testing.
>> >>
>> >> [1]: https://github.com/astralway/webindex
>> >> [2]: https://github.com/astralway/stresso
>> >>
>>

Re: Testing for 1.1.0 release

[Keith Turner]

I tried assigning that issue to you (we now have gitbox), but was
unable too.  Seems there is only a set list that we can assign to.

On Wed, May 17, 2017 at 3:30 PM, Michael Wall <mjw...@gmail.com> wrote:
> I am going to work https://github.com/apache/incubator-fluo/issues/827
> (Bytes.copyTo)
> tomorrow, but you shouldn't hold up testing.
>
> On Wed, May 17, 2017 at 3:24 PM Keith Turner <ke...@deenlo.com> wrote:
>
>> I have been running some quick test of Webindex[1] and Stresso[2]
>> against 1.1.0-SNAPSHOT and have found a few bugs.  I am going to run
>> some longer test on EC2 in preparation for a 1.1.0 release.  Before I
>> do this, I wanted to see if anyone had an changes they would like to
>> make before  this 1.1.0 testing is done.  Let me know of anything and
>> I can hold off on testing.
>>
>> [1]: https://github.com/astralway/webindex
>> [2]: https://github.com/astralway/stresso
>>

Testing for 1.1.0 release

[Keith Turner]

I have been running some quick test of Webindex[1] and Stresso[2]
against 1.1.0-SNAPSHOT and have found a few bugs.  I am going to run
some longer test on EC2 in preparation for a 1.1.0 release.  Before I
do this, I wanted to see if anyone had an changes they would like to
make before  this 1.1.0 testing is done.  Let me know of anything and
I can hold off on testing.

[1]: https://github.com/astralway/webindex
[2]: https://github.com/astralway/stresso

Re: Repository Moves (updated git URLs)

[Keith Turner]

ok, I closed the issue I opened as a duplicate

On Tue, May 16, 2017 at 2:49 PM, Christopher <ctubb...@apache.org> wrote:
> Yep, was discussing this with Chris on the original jira issue.
>
> On Tue, May 16, 2017, 14:45 Keith Turner <ke...@deenlo.com> wrote:
>
>> Notifications seem to be going to the wrong list.  I opened :
>>
>> https://issues.apache.org/jira/servicedesk/agent/INFRA/issue/INFRA-14169
>>
>> On Fri, May 12, 2017 at 4:52 PM, Christopher <ctubb...@apache.org> wrote:
>> > Hi all,
>> >
>> > Fluo's repositories have moved to GitBox[1], a newer Git service provided
>> > by Apache which has a few new features than the old one. As a result, the
>> > URLs have changed. You should update your local clones, Jenkins builds,
>> CI
>> > stuffs, etc. to point to the new location. We will be updating our
>> website
>> > docs, pom.xml files, and other references to our git repos soon with the
>> > new URLs.
>> >
>> > The new URLs can be updated (assuming your git remote is called "origin")
>> > similar to:
>> >
>> > (cd incubator-fluo && git remote set-url origin
>> > https://gitbox.apache.org/repos/asf/incubator-fluo)
>> > (cd incubator-fluo-recipes && git remote set-url origin
>> > https://gitbox.apache.org/repos/asf/incubator-fluo-recipes)
>> > (cd incubator-fluo-website && git remote set-url origin
>> > https://gitbox.apache.org/repos/asf/incubator-fluo-website)
>> >
>> > [1]: https://issues.apache.org/jira/browse/INFRA-14121
>>

Re: Repository Moves (updated git URLs)

[Keith Turner]

Notifications seem to be going to the wrong list.  I opened :

https://issues.apache.org/jira/servicedesk/agent/INFRA/issue/INFRA-14169

On Fri, May 12, 2017 at 4:52 PM, Christopher  wrote:
> Hi all,
>
> Fluo's repositories have moved to GitBox[1], a newer Git service provided
> by Apache which has a few new features than the old one. As a result, the
> URLs have changed. You should update your local clones, Jenkins builds, CI
> stuffs, etc. to point to the new location. We will be updating our website
> docs, pom.xml files, and other references to our git repos soon with the
> new URLs.
>
> The new URLs can be updated (assuming your git remote is called "origin")
> similar to:
>
> (cd incubator-fluo && git remote set-url origin
> https://gitbox.apache.org/repos/asf/incubator-fluo)
> (cd incubator-fluo-recipes && git remote set-url origin
> https://gitbox.apache.org/repos/asf/incubator-fluo-recipes)
> (cd incubator-fluo-website && git remote set-url origin
> https://gitbox.apache.org/repos/asf/incubator-fluo-website)
>
> [1]: https://issues.apache.org/jira/browse/INFRA-14121

Re: Podling Report Reminder - May 2017

[Keith Turner]

I updated the Wiki.  Let me know of any desired mutations.

https://wiki.apache.org/incubator/May2017

On Mon, May 1, 2017 at 10:22 AM, Josh Elser  wrote:
> *ping*
>
> Don't forget, please! Due this Wednesday.
>
>
> johndam...@apache.org wrote:
>>
>> Dear podling,
>>
>> This email was sent by an automated system on behalf of the Apache
>> Incubator PMC. It is an initial reminder to give you plenty of time to
>> prepare your quarterly board report.
>>
>> The board meeting is scheduled for Wed, 17 May 2017, 10:30 am PDT.
>> The report for your podling will form a part of the Incubator PMC
>> report. The Incubator PMC requires your report to be submitted 2 weeks
>> before the board meeting, to allow sufficient time for review and
>> submission (Wed, May 03).
>>
>> Please submit your report with sufficient time to allow the Incubator
>> PMC, and subsequently board members to review and digest. Again, the
>> very latest you should submit your report is 2 weeks prior to the board
>> meeting.
>>
>> Thanks,
>>
>> The Apache Incubator PMC
>>
>> Submitting your Report
>>
>> --
>>
>> Your report should contain the following:
>>
>> *   Your project name
>> *   A brief description of your project, which assumes no knowledge of
>>  the project or necessarily of its field
>> *   A list of the three most important issues to address in the move
>>  towards graduation.
>> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
>>  aware of
>> *   How has the community developed since the last report
>> *   How has the project developed since the last report.
>> *   How does the podling rate their own maturity.
>>
>> This should be appended to the Incubator Wiki page at:
>>
>> https://wiki.apache.org/incubator/May2017
>>
>> Note: This is manually populated. You may need to wait a little before
>> this page is created from a template.
>>
>> Mentors
>> ---
>>
>> Mentors should review reports for their project(s) and sign them off on
>> the Incubator wiki page. Signing off reports shows that you are
>> following the project - projects that are not signed may raise alarms
>> for the Incubator PMC.
>>
>> Incubator PMC

Re: Podling Report Reminder - May 2017

[Keith Turner]

I'll work on it this afternoon or tomorrow morning.

On Mon, May 1, 2017 at 10:22 AM, Josh Elser  wrote:
> *ping*
>
> Don't forget, please! Due this Wednesday.
>
>
> johndam...@apache.org wrote:
>>
>> Dear podling,
>>
>> This email was sent by an automated system on behalf of the Apache
>> Incubator PMC. It is an initial reminder to give you plenty of time to
>> prepare your quarterly board report.
>>
>> The board meeting is scheduled for Wed, 17 May 2017, 10:30 am PDT.
>> The report for your podling will form a part of the Incubator PMC
>> report. The Incubator PMC requires your report to be submitted 2 weeks
>> before the board meeting, to allow sufficient time for review and
>> submission (Wed, May 03).
>>
>> Please submit your report with sufficient time to allow the Incubator
>> PMC, and subsequently board members to review and digest. Again, the
>> very latest you should submit your report is 2 weeks prior to the board
>> meeting.
>>
>> Thanks,
>>
>> The Apache Incubator PMC
>>
>> Submitting your Report
>>
>> --
>>
>> Your report should contain the following:
>>
>> *   Your project name
>> *   A brief description of your project, which assumes no knowledge of
>>  the project or necessarily of its field
>> *   A list of the three most important issues to address in the move
>>  towards graduation.
>> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
>>  aware of
>> *   How has the community developed since the last report
>> *   How has the project developed since the last report.
>> *   How does the podling rate their own maturity.
>>
>> This should be appended to the Incubator Wiki page at:
>>
>> https://wiki.apache.org/incubator/May2017
>>
>> Note: This is manually populated. You may need to wait a little before
>> this page is created from a template.
>>
>> Mentors
>> ---
>>
>> Mentors should review reports for their project(s) and sign them off on
>> the Incubator wiki page. Signing off reports shows that you are
>> following the project - projects that are not signed may raise alarms
>> for the Incubator PMC.
>>
>> Incubator PMC

New name for CollisionFreeMap

[Keith Turner]

https://github.com/apache/incubator-fluo-recipes/issues/127

If anyone has any suggestions I would love to hear them.  I think it
would be nice to rename this before a 1.1.0 release.

Reducing scanning needed to find notifications

[Keith Turner]

I am currently experimenting with the concept in the following GH comment.

https://github.com/apache/incubator-fluo/issues/500#issuecomment-291199063

If anyone has any feedback/questions on this proposed change, let me know.

Keith

Re: [DISCUSS] New API for setting up observers.

[Keith Turner]

On Thu, Feb 23, 2017 at 7:03 PM, Christopher <ctubb...@apache.org> wrote:
> I like the idea, but I am curious how Fluo should handle the case where the
> Observer implementation is different for different workers (either the
> factory provides different Observer implementations on different workers,
> or the workers are running different versions).

This can happen currently, with different implementations of the same
class.  Using YARN to launch workers makes this situation less likely
because YARN will launch all processes with the same code.  Also
commit e211990[1] added code to deal with YARN leaving stragglers
behind as outlined in #482[2].  If not using YARN to launch workers,
then the issue of workers using different versions of code is more
likely to occur.

[1]: 
https://github.com/apache/incubator-fluo/commit/e211990f0e96b88485cd2e522cbf6032524cd1a3
[2]: https://github.com/apache/incubator-fluo/issues/482


>
> On Thu, Feb 23, 2017 at 5:36 PM Keith Turner <ke...@deenlo.com> wrote:
>
>> The current way observers are set up in Fluo requires specifying a
>> class for each observers in configuration.  This is cumbersome and
>> prevents using lambdas and anonymous inner classes.  It also makes it
>> hard to follow what a Fluo Application is doing.  This cumbersome way
>> of setting things up propagates forward into higher level constructs
>> in Fluo Recipes making them also cumbersome to use.
>>
>> I think it would be much simpler if the user only had to specify one
>> factory class in configuration that created all observers.  This
>> factory would be free to pair a lambda with a column for an observer.
>> Observed columns would no longer be tightly coupled with a specific
>> class.  I am thinking the factory would look something like the
>> following.
>>
>> interface ObserversFactory {
>>   Map<ObservedColumn, Observer> getObservers(SimpleConfiguration
>> applicationConfig);
>> }
>>
>> A user would implement this interface with a class that creates all of
>> their observers.
>>
>> class UserObserversFactory implements ObserversFactory {
>>   Map<ObservedColumn, Observer> getObservers(SimpleConfiguration
>> appConfig) {
>> //all of users observers are setup here in code, which is much
>> easier to follow than current way of configuring each observer class
>> individually.
>>  HashMap observers = ...;
>>  observers.put(col1, tx,row,col -> ); //an observer that's a lambda
>>  ExportQueue.addObserver(observers, appConfig, "queueId", exports
>> -> );  //using a lambda to handle exports...
>>   }
>> }
>>
>> The user would configure Fluo to use the above observer factory.
>>
>> I am thinking of trying to implement this for 1.1, but wanted to see
>> if anyone had any input before I started.  If it seems to work well, I
>> was thinking of deprecating the current way of configuring observers.
>> I would update webindex and fluo recipes in parallel to evaluate the
>> changes.
>>
> --
> Christopher

Re: third party service to poll Fluo for absence of event

[Keith Turner]

If you have time to share as you move forward, I am interested in
learning from your experiences with using Kafka and Fluo together. I
have wanted to experiment with this in order to see if Fluo needed any
changes to support better interoperation.   However, I have not had
the time.  I opened #795[1], but its based on speculation not
experience.

[1]: https://github.com/apache/incubator-fluo/issues/795

On Thu, Feb 2, 2017 at 5:15 PM, Meier, Caleb <caleb.me...@parsons.com> wrote:
> Thanks for the input.  I'm currently looking at creating some sort of 
> coordinator (which wraps a ScheduledExecutorService to generate periodic 
> notifications) and a collection of workers (to process the periodic queries 
> as they are issued).  Most of the interaction between the workers and 
> coordinator will be via Kafka (develop some sort of protocol to ensure that 
> more than one worker isn't getting assigned the same query).  At any rate, I 
> was thinking of implementing these components as TwillRunnables.  However, it 
> seems like the Twill documentation is a bit sparse.  Given that you guys 
> implemented Fluo as a TwillApplication, do you have any insight/advice for 
> writing TwillApplications?  In particular, how is your FluoTwillApp being 
> run?  All of the examples I've seen create a client with a TwillRunner and 
> TwillController.  It seems like you 've created your own version of a 
> YarnAppRunner -- what role is that playing in running the FluoTwillApp?  
> Moreover, it is also unclear to me whether the TwillRunnables are bound to 
> the client -- if the client terminates do the runnables terminate as well?  
> So essentially, it is unclear to me how create a long running application in 
> Twill that is not bound to a particular client.  Sorry that this is a little 
> off topic, but any help, references to documentation/examples would be very 
> appreciated.
>
> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office:  (703)797-3066
> caleb.me...@parsons.com ♦ www.parsons.com
>
> -Original Message-
> From: Keith Turner [mailto:ke...@deenlo.com]
> Sent: Wednesday, February 01, 2017 11:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Wed, Feb 1, 2017 at 9:54 PM, Christopher <ctubb...@apache.org> wrote:
>> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb <caleb.me...@parsons.com>
>> wrote:
>>
>>> Yeah, this seems pretty reasonable to me.  I guess it then boils down
>>> to the nitty gritty of do I store results in Fluo and have my service
>>> query Fluo (I think you guys actually advise against that in your
>>> documentation), or export results and then have the service query
>>> some external index that I am exporting to.
>>>
>>>
>> I'm not sure we advise against it, so much as recognize that it may
>> not be suitable for certain use cases and may not meet query
>> performance expectations (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__fluo.apache.org_docs_fluo-2Drecipes_1.0.0-2Dincubating_export-2Dqueue_=CwIFaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=zqJSJTFo90FyUVCiF79uq3P0FHnxr0MLFKbsPsHGgyk=spmwJN_FBTO6TBBT2dne8sbE7MRMrlhz8lLPpfPZBbs=
>>  ).
>>
>
> I would advise against querying Fluo for low latency queries.
> However, this external service thats checking a few stats within Fluo and 
> injecting new notifications probably does not care about latency.
>
> The reason Fluo is not geared towards low latency is that it does lazy
> recovery of failed transactions.   Failed transactions are not cleaned
> up until something tries to read the data, which could significantly delay 
> reads.
>
>> In any case, your observer need not write the final "last occurrence"
>> entries into a Fluo table. It could write them anywhere.
>>
>>
>>> Regarding timestamps, does the oracle server provide actual
>>> timestamps or just logical timestamps?  That is, could I use the
>>> timestamps that the server provides to define some sort of now()
>>> function to obtain the current time to compare with the times of incoming 
>>> events?
>>>
>>
>> Just logical time, and it delivers batches to limit locking, so it can
>> appear to jump ahead spontaneously. I'm not sure the OracleServer is
>> suitable for this purpose. What level of precision are you going for?
>> It might be enough to just run NTP, if you don't need more precision
>> than "within seconds".
>>
>>
>>> __

Re: third party service to poll Fluo for absence of event

[Keith Turner]

On Wed, Feb 1, 2017 at 9:54 PM, Christopher  wrote:
> On Wed, Feb 1, 2017 at 10:04 AM Meier, Caleb 
> wrote:
>
>> Yeah, this seems pretty reasonable to me.  I guess it then boils down to
>> the nitty gritty of do I store results in Fluo and have my service query
>> Fluo (I think you guys actually advise against that in your documentation),
>> or export results and then have the service query some external index that
>> I am exporting to.
>>
>>
> I'm not sure we advise against it, so much as recognize that it may not be
> suitable for certain use cases and may not meet query performance
> expectations (
> http://fluo.apache.org/docs/fluo-recipes/1.0.0-incubating/export-queue/).
>

I would advise against querying Fluo for low latency queries.
However, this external service thats checking a few stats within Fluo
and injecting new notifications probably does not care about latency.

The reason Fluo is not geared towards low latency is that it does lazy
recovery of failed transactions.   Failed transactions are not cleaned
up until something tries to read the data, which could significantly
delay reads.

> In any case, your observer need not write the final "last occurrence"
> entries into a Fluo table. It could write them anywhere.
>
>
>> Regarding timestamps, does the oracle server provide actual timestamps or
>> just logical timestamps?  That is, could I use the timestamps that the
>> server provides to define some sort of now() function to obtain the current
>> time to compare with the times of incoming events?
>>
>
> Just logical time, and it delivers batches to limit locking, so it can
> appear to jump ahead spontaneously. I'm not sure the OracleServer is
> suitable for this purpose. What level of precision are you going for? It
> might be enough to just run NTP, if you don't need more precision than
> "within seconds".
>
>
>> 
>> From: Christopher 
>> Sent: Tuesday, January 31, 2017 5:08 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: third party service to poll Fluo for absence of event
>>
>> You could write an observer which rolls up timestamps from all the events
>> you are concerned about, and puts the most recent event timestamp into a
>> centralized place, which you could poll. If there is no ingest of these
>> events, then the last timestamp in this central place will exceed some
>> threshold and the poller could detect that and trigger additional actions.
>>
>> On Tue, Jan 31, 2017 at 3:51 PM Meier, Caleb 
>> wrote:
>>
>> > Hello,
>> >
>> > I’m looking into using Fluo to develop an event based notification system
>> > that incrementally generates events of increasing complexity.  The one
>> > issue that I’m running into is how to handle the non-event event.  That
>> is,
>> > Fluo (as I understand it) is not well-suited to handle the following
>> > request: “generate a notification if no events of a given type have
>> > occurred within the last 24 hours”.  This is because it is a push based
>> > notification framework that only generates notifications when things
>> > actually happen.  So the question is, has anyone looked into developing a
>> > service for generating notifications at regular intervals (even if
>> > something doesn’t happen) that works with Fluo?  I’m toying with the idea
>> > of creating some sort of Twill application that tells Fluo to wake up at
>> > regular intervals to generate a notification about the set of events
>> > falling within the given time window. Before doing this I just wanted to
>> > make sure that something like this does not already exist, and I also
>> want
>> > to get a sense of how bad an idea it is to delegate some of the logic of
>> > this periodic notification service to Fluo.   Would it be better to
>> > separate out the temporal portion of my notification request to be
>> > processed entirely outside of Fluo to avoid transactional overhead?
>> >
>> > Caleb A. Meier, Ph.D.
>> > Software Engineer II ♦ Analyst
>> > Parsons Corporation
>> > 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
>> > Office:  (703)797-3066 <(703)%20797-3066> <(703)%20797-3066>
>> > caleb.me...@parsons.com ♦
>> www.parsons.com<
>> > http://www.parsons.com/>
>> >
>> > --
>> Christopher
>>
> --
> Christopher

Re: third party service to poll Fluo for absence of event

[Keith Turner]

On Wed, Feb 1, 2017 at 9:53 AM, Meier, Caleb <caleb.me...@parsons.com> wrote:
> Hey Keith,
>
> Right now the project is in its infancy so I don't have concrete examples of 
> how events are distributed, but I think we would like everything to scale as 
> needed.  Right now, the user is in charge of how events are modeled and how 
> many different event types are included.
>
> Regarding the use cases that you mentioned, I think we would be more 
> interested in find which event types haven't been updated or haven't occurred 
> within the last 24 hours.  I'm not ruling out the possibility of tracking 
> updates to an individual event though.
>
> More generally, we're interested in issuing periodic notifications.  The 
> non-event event is kind of a canonical example.  Another use case would be 
> "generate a notification if more than 100 events of a given type have 
> occurred within the last 24 hours",  and the user specifies that they want a 
> notification to be generated every hour.  In the case that 1000 events happen 
> in the first hour and then no other events occur during that 24 hour period, 
> 1000 notifications would be generated by Fluo during the first hour, but then 
> nothing would happen the remaining 23 hours.  We would need some service 
> (that possibly utilizes an observer if that makes sense) to periodically 
> detect how many events have occurred within the specified window and generate 
> a notification.  So in this case, every hour for the next 23 hours, something 
> would check and find that 1000 events occurred within the window and generate 
> a notification.
>
> Did that clarify things?

Yeah.  I feel like this could be broken into two parts.

 * A many to few update recipe with short circuiting.
 * Something that periodically scans the "few" row/cols and possibly
injects a new action.

Below is an example to show what I mean by short circuiting.

 1. Processing event 100934 with event type X
 2. Update the last update hour for event type X only if its value is
less than current hour

The Collision Free Map (CFM) could be used for these many to few
updates.  However, it lacks the short circuit optimization.  It will
always queue the updates, even when the applied update will not change
the current value.

The external scan service periodically scan the few event type last
update times and inject new notifications.  This mechanism could use
transactions to avoid problems from accidentally running multiple
external services.

> 
> From: Keith Turner <ke...@deenlo.com>
> Sent: Tuesday, January 31, 2017 6:43 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: third party service to poll Fluo for absence of event
>
> On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb <caleb.me...@parsons.com> wrote:
>> Hello,
>>
>> I’m looking into using Fluo to develop an event based notification system 
>> that incrementally generates events of increasing complexity.  The one issue 
>> that I’m running into is how to handle the non-event event.  That is, Fluo 
>> (as I understand it) is not well-suited to handle the following request: 
>> “generate a notification if no events of a given type have occurred within 
>> the last 24 hours”.  This is because it is a push based notification 
>> framework that only generates notifications when things actually happen.  So 
>> the question is, has anyone looked into developing a service for generating 
>> notifications at regular intervals (even if something doesn’t happen) that 
>> works with Fluo?  I’m toying with the idea of creating some sort of Twill 
>> application that tells Fluo to wake up at regular intervals to generate a 
>> notification about the set of events falling within the given time window. 
>> Before doing this I just wanted to make sure that something like this does 
>> not already exist, and I also want to get a sense of how bad an idea it is 
>> to delegate some of the logic of this periodic notification service to Fluo. 
>>   Would it be better to separate out the temporal portion of my notification 
>> request to be processed entirely outside of Fluo to avoid transactional 
>> overhead?
>>
>
> At one point in time I was thinking of delayed notifications, but I
> can't remember what my specific use case was. I am not sure if this is
> good idea or not.
>
> How many event types are there in your use case?  Just curious about
> the order of magnitude.  Like is it ~100 event types and millions of
> events?
>
> Just to make sure I understand your use case, are you interested in
> finding which event types were not updated in the last 24hrs?  Or are
> you interested in finding which events were no

Re: Podling Report Reminder - February 2017

[Keith Turner]

I updated the Fluo report on the wiki.  Does anyone have any suggested changes?

https://wiki.apache.org/incubator/February2017

On Tue, Jan 31, 2017 at 10:02 PM,   wrote:
> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 15 February 2017, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, February 01).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/February2017
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC

Re: third party service to poll Fluo for absence of event

[Keith Turner]

On Tue, Jan 31, 2017 at 3:50 PM, Meier, Caleb  wrote:
> Hello,
>
> I’m looking into using Fluo to develop an event based notification system 
> that incrementally generates events of increasing complexity.  The one issue 
> that I’m running into is how to handle the non-event event.  That is, Fluo 
> (as I understand it) is not well-suited to handle the following request: 
> “generate a notification if no events of a given type have occurred within 
> the last 24 hours”.  This is because it is a push based notification 
> framework that only generates notifications when things actually happen.  So 
> the question is, has anyone looked into developing a service for generating 
> notifications at regular intervals (even if something doesn’t happen) that 
> works with Fluo?  I’m toying with the idea of creating some sort of Twill 
> application that tells Fluo to wake up at regular intervals to generate a 
> notification about the set of events falling within the given time window. 
> Before doing this I just wanted to make sure that something like this does 
> not already exist, and I also want to get a sense of how bad an idea it is to 
> delegate some of the logic of this periodic notification service to Fluo.   
> Would it be better to separate out the temporal portion of my notification 
> request to be processed entirely outside of Fluo to avoid transactional 
> overhead?
>

At one point in time I was thinking of delayed notifications, but I
can't remember what my specific use case was. I am not sure if this is
good idea or not.

How many event types are there in your use case?  Just curious about
the order of magnitude.  Like is it ~100 event types and millions of
events?

Just to make sure I understand your use case, are you interested in
finding which event types were not updated in the last 24hrs?  Or are
you interested in finding which events were not updated in the last
24hrs?  If none of these, would you be able to describe the use case
you want with an example?

> Caleb A. Meier, Ph.D.
> Software Engineer II ♦ Analyst
> Parsons Corporation
> 1911 N. Fort Myer Drive, Suite 800 ♦ Arlington, VA 22209
> Office:  (703)797-3066
> caleb.me...@parsons.com ♦ 
> www.parsons.com
>

ApacheCon Talk

[Keith Turner]

Is anyone considering giving a talk about Fluo at Apache Con[1] North
America?  I just saw that the deadline for submissions was Feb 11th.

[1]: http://www.apachecon.com/

Fluo Accumulo Summit Talks

[Keith Turner]

Since the videos are up from Accumulo Summit 2016, I created a blog
post linking to the slides and videos.

http://fluo.apache.org/blog/2017/01/10/accumulo-summit-17/

New blog post

[Keith Turner]

I just posted a new blog post about loading data into Fluo using Spark.

http://fluo.apache.org/blog/2016/12/22/spark-load/

Re: Doing twitter wrong

[Keith Turner]

One of the articles you linked mentioned that not having a bio was
suspicious to search, so I created a bio.  The article said it may
take a few days for the profiles change to propagate to search.  Also
my account is new and does not have many follower, which the article
mentioned.  As an experiment, yesterday I created a test tweet to
#apachefluotour w/o a link.  This tweet showed up in search.  I tried
creating another link with a link to the gist, this did not show up in
search.  Josh tried creating a test tweet with a link to the gist and
#apachefluotour.  His link showed up in search.  So I suppose twitter
trust Josh more than me.  We both deleted our test tweets.

On Tue, Nov 15, 2016 at 7:36 PM, Christopher <ctubb...@apache.org> wrote:
> These might be helpful:
>
> http://hubshout.com/?Why-don%27t-my-Twitter-hashtags-show-up-in-Twitter-search=732
> https://support.twitter.com/articles/66018
>
>
> On Mon, Nov 14, 2016 at 9:54 AM Keith Turner <ke...@deenlo.com> wrote:
>
>> I sent out a tweet[1] with the hashtag #apachefluotour.  The tweet
>> does not show up in search[2].  Not sure why.
>>
>> [1]: https://twitter.com/rkturn2/status/796897289566650368
>> [2]: https://twitter.com/hashtag/apachefluotour
>>

Doing twitter wrong

[Keith Turner]

I sent out a tweet[1] with the hashtag #apachefluotour.  The tweet
does not show up in search[2].  Not sure why.

[1]: https://twitter.com/rkturn2/status/796897289566650368
[2]: https://twitter.com/hashtag/apachefluotour

Fluo is on Twitter

[Keith Turner]

I updated the web site with this info when I created the Twitter
account for Fluo. However, I just realized I did not mention the new
Twitter account on the mailing list.

https://twitter.com/ApacheFluo

Re: New Blog Post

[Keith Turner]

On Thu, Nov 10, 2016 at 5:20 PM, Dylan Hutchison
<dhutc...@cs.washington.edu> wrote:
> Thanks for writing the post Keith.
>
> How would you compare Fluo's Bytes to Guava's ImmutableList?

I think the following elements are key for an immutable byte array wrapper.

 * good interoperability  with byte[], String, ByteBuffer,
InputStream, OutputStream, etc.
 * an efficient reusable builder that also interoperates with type
mentioned above
 * good hashCode, equals, and compareTo functions

A list representation might be nice, however the performance
implications you mentioned below may be severe. The list
implementation would provide some of the elements mentioned above.
Would still need to provide additional specialization for the interop.

>
> Bytes would be more efficient at face value because it used a byte[]
> internally rather than an Object[] (that is a Byte[]). Apart from that, I
> imagine that the two implementations share a lot of functionality, such as
> how to handle sub-arrays of a Bytes (= sublists of an immutable list, which
> Guava implements very nicely). Did Bytes take inspiration from Guava or was
> it built from the ground up?

Its slowly evolved over time, so its hard to remember. I think
inspiration was taken from parts of Guava, but I am not sure what
parts.  Also Java's String was an inspiration.

>
> On Nov 10, 2016 1:11 PM, "Keith Turner" <ke...@deenlo.com> wrote:
>
> I just added a new blog post to the Fluo website.
>
> http://fluo.apache.org/blog/2016/11/10/immutable-bytes/

New Blog Post

[Keith Turner]

I just added a new blog post to the Fluo website.

http://fluo.apache.org/blog/2016/11/10/immutable-bytes/

Re: debugging fluo

[Keith Turner]

On Tue, Nov 1, 2016 at 5:04 PM, Keith Turner <ke...@deenlo.com> wrote:
> On Tue, Nov 1, 2016 at 5:01 PM, Meier, Caleb <caleb.me...@parsons.com> wrote:
>> I'm using fluo-incubating.  I switched "io.fluo.log" to "fluo.log" and my 
>> worker and oracle logs appearing in my container directories.  Still working 
>> on getting my observers to log to my worker logs though.
>
> So your observers are using slf4j?
>
> I will be at #fluo in IRC on freenode for a bit, if you want to chat there.

Just to follow up on this.  We did resolve part of the issue in IRC.
Mike Walch figured out that old configs with io.fluo were being used.
Switching that fixed some of the problems.  Caleb is still not seeing
output from a logger created by an observer, we have not figured this
one out yet.


>
>>
>> -Original Message-
>> From: Mike Walch [mailto:mwa...@apache.org]
>> Sent: Tuesday, November 01, 2016 5:00 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>>
>> Which version of Fluo are you using?  The properties 'io.fluo.log.*' were 
>> used before Apache incubation in 1.0.0-beta-1 or 1.0.0-beta-2.  These 
>> properties were renamed in Fluo 1.0.0 to 'fluo.log.*'.  You could be having 
>> problems if you built your application using Fluo 1.0.0-beta-2 and ran it 
>> using a Fluo 1.0.0 install.
>>
>> On Tue, Nov 1, 2016 at 4:45 PM Meier, Caleb <caleb.me...@parsons.com> wrote:
>>
>>> Hey Keith,
>>>
>>> Not seeing the worker logs in my container.  Think my logback is
>>> configured incorrectly...
>>>
>>> [root@c190sv193 container_1476563020088_0069_01_02]# grep Rolling
>>> stdout
>>> 18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
>>> - About to instantiate appender of type
>>> [ch.qos.logback.core.rolling.RollingFileAppender]
>>> 18:36:49,215 |-INFO in
>>> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No
>>> compression will be used
>>> 18:36:49,230 |-INFO in
>>> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log
>>> file
>>> name:
>>> io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.
>>> host_IS_UNDEFINED.log
>>> 18:36:49,230 |-INFO in
>>> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property
>>> is set to
>>> [io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log
>>> .host_IS_UNDEFINED.log]
>>>
>>> -Original Message-
>>> From: Keith Turner [mailto:ke...@deenlo.com]
>>> Sent: Tuesday, November 01, 2016 4:33 PM
>>> To: dev@fluo.incubator.apache.org
>>> Subject: Re: debugging fluo
>>>
>>> Caleb,
>>>
>>> I just ran Fluo locally with Uno.  Below is some info I am seeing.
>>> Do you see anything about RollingFileAppender in the stdout file?
>>>
>>> $ pwd
>>>
>>> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/con
>>> tainer_1478030941494_0001_01_04
>>> $ ls
>>> stderr  stdout  worker_1_host1.log
>>> $ grep Rolling stdout
>>> 16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
>>> - About to instantiate appender of type
>>> [ch.qos.logback.core.rolling.RollingFileAppender]
>>> 16:17:31,574 |-INFO in
>>> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No
>>> compression will be used
>>> 16:17:31,583 |-INFO in
>>> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log
>>> file
>>> name:
>>> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/con
>>> tainer_1478030941494_0001_01_04/worker_1_host1
>>> 16:17:31,583 |-INFO in
>>> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property
>>> is set to
>>> [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/co
>>> ntainer_1478030941494_0001_01_04/worker_1_host1]
>>>
>>> Keith
>>>
>>> On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb <caleb.me...@parsons.com>
>>> wrote:
>>> > Hey Mike,
>>> >
>>> > So I’m not seeing any worker logs on my machines.  The search
>>> >
>>> > find / | grep worker
>>> >
>>> > yielded nothing.  Any ideas as to why these don't exist?  I'm not
>>> > sure that my fluo.log.dir system property is successfully being set
>>> > by the
>>> LogbackUtil class.
&

Re: Podling Report Reminder - November 2016

[Keith Turner]

On Wed, Nov 2, 2016 at 5:36 PM, Aaron D. Mihalik
<aaron.miha...@gmail.com> wrote:
> I took you all up on your offer and joined the fluo dev list :)
>
> And may I say: Excellent documentation and processes!  Spent a lot of time
> referring to your release process documentation and inspecting your git
> repo while I was cutting the first Rya release.  Thanks for the excellent
> and clean examples.

Thanks.  We have benefited from many lessons learned from working on Accumulo.

>
> --Aaron
>
> On Wed, Nov 2, 2016 at 5:30 PM Josh Elser <josh.el...@gmail.com> wrote:
>
>> Keith Turner wrote:
>> > On Tue, Nov 1, 2016 at 10:57 PM, Christopher<ctubb...@apache.org>
>> wrote:
>> >> >  I think growing our community is really the only outstanding area of
>> >> >  concern (with respect to graduation) that we need to work on.
>> However, I'm
>> >> >  not sure how we can do that. One thing is that we could reach out to
>> Rya,
>> >> >  and any other developers who are using Fluo that we know about, to
>> >> >  explicitly invite them (rather than wait for them to come to us).
>> >> >
>> >> >  Aside from that, I'm not sure what else we can do. Maybe we could
>> spend
>> >
>> > I think we can also give talks to help raise awareness.
>> > We can create integration with other communities (like Kafka and Nifi)
>> > and then blog within those communities.
>> > We can also blog about possible integration with other communites...
>> > like maybe Fluo could be used with Presto to keep a secondary index up
>> > to date.. we could blog about how we would think that would work.
>> >
>>
>> Everybody loves secondary indexing ;). In general, this kind of
>> cross-pollination is a great way to get people interested.
>>
>> Blogs, tutorials, presentations are a great way to help find these kinds
>> of people that you can get involved in Fluo.
>>

Re: Nice Report...

[Keith Turner]

I reposted the content to the wiki.

On Wed, Nov 2, 2016 at 12:54 PM, Keith Turner <ke...@deenlo.com> wrote:
> Thanks for taking a look.  Its odd, when I look at the wiki page now I
> don't see the updates I made for Fluo yesterday.
>
> On Tue, Nov 1, 2016 at 10:21 PM, John D. Ament <johndam...@apache.org> wrote:
>> Fluo Podling,
>>
>> Thanks for posting your report.  Looks good.  A couple of notes from my
>>  side (as shepherd).
>>
>> - Try wherever possible to post/discuss the report on your MLs before
>> posting to the incubator wiki.  Its not a rule per se, but it helps ensure
>> that you have community involvement (clicking in the wiki and finding the
>> report makes it harder to review)
>>
>> - This question is empty if you can provide an answer "Any issues that the
>> Incubator PMC (IPMC) or ASF Board wish/need to be aware of?"
>>
>> Please don't hesitate if you have any questions, I monitor podling mailing
>> lists.
>>
>> John

Re: Nice Report...

[Keith Turner]

Thanks for taking a look.  Its odd, when I look at the wiki page now I
don't see the updates I made for Fluo yesterday.

On Tue, Nov 1, 2016 at 10:21 PM, John D. Ament  wrote:
> Fluo Podling,
>
> Thanks for posting your report.  Looks good.  A couple of notes from my
>  side (as shepherd).
>
> - Try wherever possible to post/discuss the report on your MLs before
> posting to the incubator wiki.  Its not a rule per se, but it helps ensure
> that you have community involvement (clicking in the wiki and finding the
> report makes it harder to review)
>
> - This question is empty if you can provide an answer "Any issues that the
> Incubator PMC (IPMC) or ASF Board wish/need to be aware of?"
>
> Please don't hesitate if you have any questions, I monitor podling mailing
> lists.
>
> John

Re: debugging fluo

[Keith Turner]

I suppose you don't see oracle files either?

$ pwd
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001
$ find .
.
./container_1478030941494_0001_01_01
./container_1478030941494_0001_01_01/stdout
./container_1478030941494_0001_01_01/stderr
./container_1478030941494_0001_01_02
./container_1478030941494_0001_01_02/stdout
./container_1478030941494_0001_01_02/stderr
./container_1478030941494_0001_01_02/oracle_0_host1
./container_1478030941494_0001_01_03
./container_1478030941494_0001_01_03/stdout
./container_1478030941494_0001_01_03/stderr
./container_1478030941494_0001_01_03/worker_0_host1
./container_1478030941494_0001_01_04
./container_1478030941494_0001_01_04/stdout
./container_1478030941494_0001_01_04/stderr
./container_1478030941494_0001_01_04/worker_1_host1

On Tue, Nov 1, 2016 at 4:40 PM, Meier, Caleb <caleb.me...@parsons.com> wrote:
> Hey Keith,
>
> Not seeing the worker logs in my container.  Think my logback is configured 
> incorrectly...
>
> [root@c190sv193 container_1476563020088_0069_01_02]# grep Rolling stdout
> 18:36:49,174 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - 
> About to instantiate appender of type 
> [ch.qos.logback.core.rolling.RollingFileAppender]
> 18:36:49,215 |-INFO in 
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@5ab860f4 - No 
> compression will be used
> 18:36:49,230 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] 
> - Active log file name: 
> io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log
> 18:36:49,230 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] 
> - File property is set to 
> [io.fluo.log.dir_IS_UNDEFINED/io.fluo.log.app_IS_UNDEFINED_io.fluo.log.host_IS_UNDEFINED.log]
>
> -Original Message-
> From: Keith Turner [mailto:ke...@deenlo.com]
> Sent: Tuesday, November 01, 2016 4:33 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Caleb,
>
> I just ran Fluo locally with Uno.  Below is some info I am seeing.
> Do you see anything about RollingFileAppender in the stdout file?
>
> $ pwd
> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
> $ ls
> stderr  stdout  worker_1_host1.log
> $ grep Rolling stdout
> 16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
> - About to instantiate appender of type
> [ch.qos.logback.core.rolling.RollingFileAppender]
> 16:17:31,574 |-INFO in
> ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No compression 
> will be used
> 16:17:31,583 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: 
> /home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1
> 16:17:31,583 |-INFO in
> ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set 
> to 
> [/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1]
>
> Keith
>
> On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb <caleb.me...@parsons.com> wrote:
>> Hey Mike,
>>
>> So I’m not seeing any worker logs on my machines.  The search
>>
>> find / | grep worker
>>
>> yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
>> that my fluo.log.dir system property is successfully being set by the 
>> LogbackUtil class.
>> Maybe this has something to do with it?
>>
>>
>>
>>
>>
>> -Original Message-
>> From: Mike Walch [mailto:mwa...@apache.org]
>> Sent: Tuesday, November 01, 2016 4:03 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>>
>> Were you able to find a worker_*.log file for each of your workers?
>>
>> Below are some tips for debugging:
>>
>> - Each YARN container should have a 'stdout' and 'stderr' file.  These files 
>> may have helpful error messages.  Especially if a worker failed to start.  
>> Also, any calls to System.out and System.err in your observer will be 
>> printed to these files.
>> - When running Fluo in YARN, Fluo must use Logback for logging (due to a 
>> hard requirment by Twill). Logback is configured using 
>> /path/to/fluo/conf/logback.xml.  You should review this configuration but 
>> the root logger is configured by default to print any message that is the 
>> debug level or higher.
>> - If you configured multiple workers, each worker will run in a different 
>> container and have a different worker_*.log file.
>> - When a worker starts up, it prints its configuration to worker_

Re: debugging fluo

[Keith Turner]

I also saw the following line printed in stdout.  Maybe this line was
printed by LogbackUtil

Logging to 
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
using config ./conf/logback.xml

On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb  wrote:
> Hey Mike,
>
> So I’m not seeing any worker logs on my machines.  The search
>
> find / | grep worker
>
> yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
> that my fluo.log.dir system property is successfully being set by the 
> LogbackUtil class.
> Maybe this has something to do with it?
>
>
>
>
>
> -Original Message-
> From: Mike Walch [mailto:mwa...@apache.org]
> Sent: Tuesday, November 01, 2016 4:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Were you able to find a worker_*.log file for each of your workers?
>
> Below are some tips for debugging:
>
> - Each YARN container should have a 'stdout' and 'stderr' file.  These files 
> may have helpful error messages.  Especially if a worker failed to start.  
> Also, any calls to System.out and System.err in your observer will be printed 
> to these files.
> - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard 
> requirment by Twill). Logback is configured using 
> /path/to/fluo/conf/logback.xml.  You should review this configuration but the 
> root logger is configured by default to print any message that is the debug 
> level or higher.
> - If you configured multiple workers, each worker will run in a different 
> container and have a different worker_*.log file.
> - When a worker starts up, it prints its configuration to worker_*.log.
> Make sure that you configured your observers using the property 
> 'fluo.observer.*'
>
> -Mike
>
> On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb  wrote:
>
>> Do you have any tips for how to make Observers log to the log files
>> found in the directory specified by 'yarn.nodemanager.log-dirs'?
>>
>> -Original Message-
>> From: Mike Walch [mailto:mwa...@apache.org]
>> Sent: Tuesday, November 01, 2016 2:36 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>>
>> Hi Caleb,
>>
>> The logs for a Fluo application can be found in YARN but they are
>> tricky to find. Fluo should have better documentation on this which I will 
>> add now.
>>
>> The easiest way to view the logs for a Fluo application is to use the
>> web interface for the YARN resource manager (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_c
>> luster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzY
>> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6
>> mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
>> ).
>> First, click on the application ID (i.e application_*) of your Fluo
>> application and then click on the latest attempt ID (appattempt_*).
>> You should see a list of containers.  There should be a container for
>> the application master (typically container 1), a Fluo oracle
>> (typically container 2), and Fluo workers (containers 3+).  You can
>> view the log files produced by a container by clicking on its 'logs'
>> link.  Logs from Fluo observers will be in the worker_*.log file for
>> each of your worker containers.
>>
>> If you don't want to use the YARN resource manager web interface, you
>> can also view these logs in the directory specified by
>> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This
>> method works well on one machine but on cluster your containers will
>> probably be on different machines. See the YARN documentation below
>> for more info about this property:
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org
>> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml
>> =CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR
>> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5A
>> ewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
>>
>> Best,
>> Mike
>>
>> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb 
>> wrote:
>>
>> Hello,
>>
>> I'm attempting to debug a Fluo application and am having difficulty
>> locating the logs for my observers.  I've looked within the logs for
>> hadoop-yarn, but am not seeing any logging statements for my observers.
>> Where do observers log out of the box in a normal cloudera distribution?
>> Do I need to do something else in addition to logging to get my
>> observers to generate logs?
>>
>> Thanks,
>> Caleb
>>

Re: debugging fluo

[Keith Turner]

Caleb,

I just ran Fluo locally with Uno.  Below is some info I am seeing.
Do you see anything about RollingFileAppender in the stdout file?

$ pwd
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04
$ ls
stderr  stdout  worker_1_host1.log
$ grep Rolling stdout
16:17:31,557 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
- About to instantiate appender of type
[ch.qos.logback.core.rolling.RollingFileAppender]
16:17:31,574 |-INFO in
ch.qos.logback.core.rolling.FixedWindowRollingPolicy@bc1b008 - No
compression will be used
16:17:31,583 |-INFO in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log
file name: 
/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1
16:17:31,583 |-INFO in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property
is set to 
[/home/kturner/uno/install/logs/yarn/application_1478030941494_0001/container_1478030941494_0001_01_04/worker_1_host1]

Keith

On Tue, Nov 1, 2016 at 4:22 PM, Meier, Caleb  wrote:
> Hey Mike,
>
> So I’m not seeing any worker logs on my machines.  The search
>
> find / | grep worker
>
> yielded nothing.  Any ideas as to why these don't exist?  I'm not sure
> that my fluo.log.dir system property is successfully being set by the 
> LogbackUtil class.
> Maybe this has something to do with it?
>
>
>
>
>
> -Original Message-
> From: Mike Walch [mailto:mwa...@apache.org]
> Sent: Tuesday, November 01, 2016 4:03 PM
> To: dev@fluo.incubator.apache.org
> Subject: Re: debugging fluo
>
> Were you able to find a worker_*.log file for each of your workers?
>
> Below are some tips for debugging:
>
> - Each YARN container should have a 'stdout' and 'stderr' file.  These files 
> may have helpful error messages.  Especially if a worker failed to start.  
> Also, any calls to System.out and System.err in your observer will be printed 
> to these files.
> - When running Fluo in YARN, Fluo must use Logback for logging (due to a hard 
> requirment by Twill). Logback is configured using 
> /path/to/fluo/conf/logback.xml.  You should review this configuration but the 
> root logger is configured by default to print any message that is the debug 
> level or higher.
> - If you configured multiple workers, each worker will run in a different 
> container and have a different worker_*.log file.
> - When a worker starts up, it prints its configuration to worker_*.log.
> Make sure that you configured your observers using the property 
> 'fluo.observer.*'
>
> -Mike
>
> On Tue, Nov 1, 2016 at 3:33 PM Meier, Caleb  wrote:
>
>> Do you have any tips for how to make Observers log to the log files
>> found in the directory specified by 'yarn.nodemanager.log-dirs'?
>>
>> -Original Message-
>> From: Mike Walch [mailto:mwa...@apache.org]
>> Sent: Tuesday, November 01, 2016 2:36 PM
>> To: dev@fluo.incubator.apache.org
>> Subject: Re: debugging fluo
>>
>> Hi Caleb,
>>
>> The logs for a Fluo application can be found in YARN but they are
>> tricky to find. Fluo should have better documentation on this which I will 
>> add now.
>>
>> The easiest way to view the logs for a Fluo application is to use the
>> web interface for the YARN resource manager (
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__localhost-3A8088_c
>> luster=CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzY
>> C2kksVZR5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6
>> mq5Tns5Aewd2s=_4PS5z_vu1bkhpZBdzJXjbGsCvMBboMqoLBIOBnRAEY=
>> ).
>> First, click on the application ID (i.e application_*) of your Fluo
>> application and then click on the latest attempt ID (appattempt_*).
>> You should see a list of containers.  There should be a container for
>> the application master (typically container 1), a Fluo oracle
>> (typically container 2), and Fluo workers (containers 3+).  You can
>> view the log files produced by a container by clicking on its 'logs'
>> link.  Logs from Fluo observers will be in the worker_*.log file for
>> each of your worker containers.
>>
>> If you don't want to use the YARN resource manager web interface, you
>> can also view these logs in the directory specified by
>> 'yarn.nodemanager.log-dirs' of your 'yarn-site.xml' config.  This
>> method works well on one machine but on cluster your containers will
>> probably be on different machines. See the YARN documentation below
>> for more info about this property:
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__hadoop.apache.org
>> _docs_r2.7.0_hadoop-2Dyarn_hadoop-2Dyarn-2Dcommon_yarn-2Ddefault.xml
>> =CwIBaQ=Nwf-pp4xtYRe0sCRVM8_LWH54joYF7EKmrYIdfxIq10=vuVdzYC2kksVZR
>> 5STiFwDpzJ7CrMHCgeo_4WXTD0qo8=S88pZ1xYAkVw1LehCcB3YDzFFeEKk6mq5Tns5A
>> ewd2s=SS8YTOgIAWUmBnKkHN2Eu2-h6WyEHqlNvJO-D5EKFiI=
>>
>> Best,
>> Mike
>>
>> On Tue, Nov 1, 2016 at 1:29 PM Meier, Caleb 
>> wrote:
>>
>> Hello,

Re: Podling Report Reminder - November 2016

[Keith Turner]

I just udpated the wiki if anyone wants to review the changes.

On Mon, Oct 31, 2016 at 3:56 PM, Josh Elser  wrote:
> I know you're already in a writing "mood", Keith.
>
> Don't forget about this one, please. Needs to be done by Wednesday this week
> (48hrs).
>
>
> johndam...@apache.org wrote:
>>
>> Dear podling,
>>
>> This email was sent by an automated system on behalf of the Apache
>> Incubator PMC. It is an initial reminder to give you plenty of time to
>> prepare your quarterly board report.
>>
>> The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
>> The report for your podling will form a part of the Incubator PMC
>> report. The Incubator PMC requires your report to be submitted 2 weeks
>> before the board meeting, to allow sufficient time for review and
>> submission (Wed, November 02).
>>
>> Please submit your report with sufficient time to allow the Incubator
>> PMC, and subsequently board members to review and digest. Again, the
>> very latest you should submit your report is 2 weeks prior to the board
>> meeting.
>>
>> Thanks,
>>
>> The Apache Incubator PMC
>>
>> Submitting your Report
>>
>> --
>>
>> Your report should contain the following:
>>
>> *   Your project name
>> *   A brief description of your project, which assumes no knowledge of
>>  the project or necessarily of its field
>> *   A list of the three most important issues to address in the move
>>  towards graduation.
>> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
>>  aware of
>> *   How has the community developed since the last report
>> *   How has the project developed since the last report.
>>
>> This should be appended to the Incubator Wiki page at:
>>
>> http://wiki.apache.org/incubator/November2016
>>
>> Note: This is manually populated. You may need to wait a little before
>> this page is created from a template.
>>
>> Mentors
>> ---
>>
>> Mentors should review reports for their project(s) and sign them off on
>> the Incubator wiki page. Signing off reports shows that you are
>> following the project - projects that are not signed may raise alarms
>> for the Incubator PMC.
>>
>> Incubator PMC

Re: [VOTE] Apache Fluo Recipes 1.0.0-incubating (rc1)

[Keith Turner]

The vote passed with 3 +1 votes and 0 -1 votes.  Thanks Josh, Justin
and Drew for inspecting the RC.

On Mon, Oct 24, 2016 at 12:33 PM, Keith Turner <ke...@deenlo.com> wrote:
> Dear IPMC,
>
> Please vote for the following release candidate of Apache Fluo Recipes
> 1.0.0-incubating.
>
> PPMC vote thread:
> https://lists.apache.org/thread.html/d1e8ae5cef7c4ab1eac8192b742a3229b346335fec64afd080284dd4@%3Cdev.fluo.apache.org%3E
>
> Staged dist artifacts:
> https://dist.apache.org/repos/dist/dev/incubator/fluo/fluo-recipes/1.0.0-incubating-rc1/
>
> Staged Maven repository:
> https://repository.apache.org/content/repositories/orgapachefluo-1016/
>
> Signing KEYS:
> https://www.apache.org/dist/incubator/fluo/KEYS
> (fingerprint for this release: CF72CA07C8BC86A1C862765F9AACFB56352ACF76)
>
> Git repo:
> https://git-wip-us.apache.org/repos/asf/incubator-fluo-recipes
> (branch: 1.0.0-incubating-rc1,
> commit: 682eff983f1fe6e60b75c36d3b2f782c6a93b155)
>
> This vote will end on Thu Oct 27 17:00:00 UTC 2016
> (Thu Oct 27 13:00:00 EDT 2016 / Thu Oct 27 10:00:00 PDT 2016)

Article on community building

[Keith Turner]

I found the following article on community building interesting. Its
annoying that we can not label issues for first time contributors in GH.
It would be nice to be able to label issues and then post a link to a GH
issue search to some of the places mentioned in the article.

http://hood.ie/blog/welcoming-communities.html

Re: Hadoop Weekly #191

[Keith Turner]

That's interesting.   Seems their website has not been updated for a few
weeks.  The last issue on the website is currently 10/2.

On Sun, Oct 23, 2016 at 8:05 PM, Josh Elser  wrote:

> Congrats, the 1.0.0-incubating release was picked up by Hadoop Weekly :)
> -- Forwarded message --
> From: "Hadoop Weekly" 
> Date: Oct 23, 2016 19:21
> Subject: Hadoop Weekly #191
> To: 
> Cc:
>
> Hadoop Weekly
> > Issue #191
> > 23 October 2016
> >
> > This week's issue is short and sweet with a few technical posts, two
> > interesting news articles, and several exciting releases (including
> Apache
> > Kafka 0.10.1.0). With Spark Summit Europe this week, expect lots of great
> > content in the next issue. And if you're attending, please send
> interesting
> > slides/talks my way!
> >
> > Technical
> > ===
> >
> > Cloudera's CDH supports intra-node disk balancing since version 5.8.2
> > (it's also part of the 3.0.0 alpha Apache release). Using this feature, a
> > data node can rebalance data blocks across disks using the `hdfs
> > diskbalancer` command. This post describes how the tool works and shows
> how
> > to run it.
> >
> > http://blog.cloudera.com/blog/2016/10/how-to-use-the-new-
> > hdfs-intra-datanode-disk-balancer-in-apache-hadoop/
> >
> >
> > This post demonstrates the capabilities of the spark.ml library by
> > building a logistic regression model to predict malignancy of cases from
> > the Wisconsin Diagnostic Breast Cancer data set. The example code covers
> > parsing, exploring a dataset with built-in statistics, extracting
> features
> > from the input dataset, training the model, and evaluating the model.
> >
> > https://www.mapr.com/blog/predicting-breast-cancer-
> > using-apache-spark-machine-learning-logistic-regression
> >
> >
> > The Amazon Big Data blog has a tutorial for running RStudio with sparklyr
> > on EMR. Thanks to a bootstrap action, a cluster complete with RStudio
> > running on the master, can be launched with a single command.
> >
> > https://aws.amazon.com/blogs/big-data/running-sparklyr-
> > rstudios-r-interface-to-spark-on-amazon-emr/
> >
> >
> > The Databricks blog features a list of seven tips for debugging Apache
> > Spark code on Databricks. Most of the suggestions, like "Scale up Spark
> > jobs slowly for really large datasets" and "Examine the partitioning for
> > your dataset," are generally applicable to all Spark users.
> >
> > https://databricks.com/blog/2016/10/18/7-tips-to-debug-
> > apache-spark-code-faster-with-databricks.html
> >
> >
> > News
> > 
> >
> > InfoQ has an interview with Yahoo VP of Engineering, Peter Cnudde. Topics
> > covered include Hadoop, Spark adoption at Yahoo (mostly for in-memory
> > computing, not for ETL), and Caffe-on-Spark for deep learning.
> >
> > https://www.infoq.com/articles/peter-cnudde-yahoo-big-data
> >
> >
> > ZDNet contributor Tony Baer has read between the lines when it comes to
> > recent benchmarks by Cloudera and Hortonworks. The takeaways are as
> > follows: 1) "SQL's the gateway drug to Hadoop." 2) Cloudera is trying to
> > challenge Amazon (in this case Redshift), and 3) Hortonworks (via Hive's
> > Live Long and Prosper) has caught up on the investment Cloudera made in
> > Impala.
> >
> > http://www.zdnet.com/article/sql-on-hadoop-benchmarks-get-serious/
> >
> >
> > Releases
> > ===
> >
> > Apache Kafka 0.10.1.0 was released this week. It contains improvements
> > from over 500 pull requests and the implementation of 15 Kafka
> Improvement
> > Proposals. The Confluent blog has the highlights of
> additions/improvements
> > to Kafka Server (time-based indexes, replication quotas, and improved log
> > compaction), improvements to Kafka client APIs (interactive queries for
> > Kafak Streams, improved memory management, secure quotas, and more), and
> > bug fixes.
> >
> > http://mail-archives.apache.org/mod_mbox/kafka-users/
> > 201610.mbox/%3CCAJL4t_oz9q4T9vn6Z-EBoazWJFyqHw4Y0L-
> > PTowD%2BpFhcPv0VQ%40mail.gmail.com%3E
> > http://www.confluent.io/blog/announcing-apache-kafka-0-10-1-0/
> >
> > Apache Fluo (incubating), recently had its first release since entering
> > the incubator. Fluo is a tool for making "incremental updates to large
> data
> > sets stored in Apache Accumulo" a la Google's Perculator.
> >
> > https://fluo.apache.org/release/fluo-1.0.0-incubating/
> >
> >
> > Apache Flume 1.7.0 was released. It adds support for a `taildir` source
> > and includes a number of improvements and bug fixes. Many of these are
> > around Flume's integration with Apache Kafka.
> >
> > http://flume.apache.org/releases/1.7.0.html
> >
> >
> > Apache NiFi 0.7.1 was released as a follow-up to July's 0.7.0 release
> > (version 1.0.0 was also recently released—in August). This release adds a
> > number of improvements and bug fixes.
> >
> > https://cwiki.apache.org/confluence/display/NIFI/
> > Release+Notes#ReleaseNotes-Version0.7.1
> >
> >
> > 

Re: [VOTE] Fluo Recipes 1.0.0-incubating-rc0

[Keith Turner]

On Tue, Oct 18, 2016 at 6:29 PM, Christopher <ctubb...@apache.org> wrote:
> -1
>
> * Reviewed signatures, hashes
> * Commit matches tarball (with the exception of the generated DEPENDENCIES
> file, which is created by the maven-remote-resources-plugin)
> * Builds fine with tests, but with a compiler warning about unused
> auto-closeable in FluoSparkHelperIT.java which should be fixed
>
> Issues (minor):
> * The NOTICE and DISCLAIMER files at the top incorrectly identifies itself
> as Apache Fluo, rather than Apache Fluo Recipes. This might be okay with
> DISCLAIMER (though, some wording to explain the relationship Apache Fluo
> Recipes has to Apache Fluo would help, but it's certainly confusing for
> NOTICE.
> * The modules don't have the "Apache" branding in their  elements in
> the pom. This results in NOTICE files that don't have the Apache branding
> on the project name, and jar manifests which have the same. Also, the
> generated NOTICE files in the jars for the modules look funny "Fluo Recipes
> Accumulo". Should we name them more like "Fluo Recipes For Accumulo"?
> * We can/should also update the  fields and NOTICE files to have "
> (incubating)" suffixed at the end.

https://github.com/apache/incubator-fluo-recipes/pull/115

>
> Other:
>
> * saw weird Implementation-URLs in jar manifests. Not really sure these add
> any value... but the URLs certainly shouldn't be expected to exist (they
> don't)
>
>
> On Tue, Oct 18, 2016 at 3:33 PM Christopher <ctubb...@apache.org> wrote:
>
>> I'm reviewing this now, but wanted to mention that the branch didn't match
>> the commit (it was one commit ahead), so I just now pushed a rollback of
>> that 1 commit to the rc0 branch, and I staged that extra commit to an
>> rc0-next branch.
>>
>> On Tue, Oct 18, 2016 at 2:24 PM Josh Elser <josh.el...@gmail.com> wrote:
>>
>> Keith Turner wrote:
>> > On Tue, Oct 18, 2016 at 1:37 PM, Josh Elser<josh.el...@gmail.com>
>> wrote:
>> >> Keith Turner wrote:
>> >>>> * Saw libthrift in the parent pom.xml, but could not see usages of
>> that
>> >>>>>   anywhere in the child modules. (not a release issue, just curious)
>> >>>
>> >>> Its in the dep mgmt section.  The reason is its needed to build
>> >>> recipes against accumulo 1.8.0.  Fluo also uses thrift and depends on
>> >>> 0.9.1.  If thrift 0.9.1 ends up on path, then MiniAccumulo will hang.
>> >>> I can add a comment to the pom.
>> >>>
>> >> Ah, ok. I just did a grep and did see any usages of Thrift in the
>> recipes
>> >> which made me wonder why it was defined (even if only in depMgmt).
>> >> Overriding Fluo's thrift dependency makes sense -- you don't need to
>> >> document on my account.
>> >
>> > It took me a bit to remember why it was there. I made a PR to add a
>> > comment to pom.
>>
>> Ok, sounds great, guys. I just assumed I was being dense and did not
>> want to demand a comment to be added :)
>>
>>

Re: [VOTE] Fluo Recipes 1.0.0-incubating-rc0

[Keith Turner]

On Tue, Oct 18, 2016 at 1:37 PM, Josh Elser <josh.el...@gmail.com> wrote:
> Keith Turner wrote:
>>>
>>> * Saw libthrift in the parent pom.xml, but could not see usages of that
>>> >  anywhere in the child modules. (not a release issue, just curious)
>>
>>
>> Its in the dep mgmt section.  The reason is its needed to build
>> recipes against accumulo 1.8.0.  Fluo also uses thrift and depends on
>> 0.9.1.  If thrift 0.9.1 ends up on path, then MiniAccumulo will hang.
>> I can add a comment to the pom.
>>
>
> Ah, ok. I just did a grep and did see any usages of Thrift in the recipes
> which made me wonder why it was defined (even if only in depMgmt).
> Overriding Fluo's thrift dependency makes sense -- you don't need to
> document on my account.

It took me a bit to remember why it was there. I made a PR to add a
comment to pom.

Re: [VOTE] Fluo Recipes 1.0.0-incubating-rc0

[Keith Turner]

On Tue, Oct 18, 2016 at 12:20 PM, Josh Elser <els...@apache.org> wrote:
> +1
>
> * xsums/sigs OK
> * Can build and run tests
> * No binaries in source release
> * mvn apache-rat:check passes (no exclusions)
> * DISCLAIMER present
> * L are good
> * README looks good.
>
> Some comments:
>
> * Keith, you could consider adding your apache.org email addr to your key.
> Not required, but a nice to have.
> * Would prefer to see "apache-fluo-recipes-1.0.0-incubating..." as the file
> name.
> * Saw libthrift in the parent pom.xml, but could not see usages of that
> anywhere in the child modules. (not a release issue, just curious)

Its in the dep mgmt section.  The reason is its needed to build
recipes against accumulo 1.8.0.  Fluo also uses thrift and depends on
0.9.1.  If thrift 0.9.1 ends up on path, then MiniAccumulo will hang.
I can add a comment to the pom.

>
> - Josh
>
> Keith Turner wrote:
>>
>> Fluo Developers,
>>
>> Please consider the following candidate for Fluo Recipes 1.0.0-incubating.
>>
>> Git Commit:
>>  2eb367e173721607a2cb958927e3c753b40188ed
>> Branch:
>>  1.0.0-incubating-rc0
>>
>> If this vote passes, a gpg-signed tag will be created using:
>>  git tag -f -m 'Apache Fluo Recipes 1.0.0-incubating' \
>>  -s rel/fluo-recipes-1.0.0-incubating \
>>  2eb367e173721607a2cb958927e3c753b40188ed
>>
>> Staging repo:
>> https://repository.apache.org/content/repositories/orgapachefluo-1014
>>
>> Source (official release artifact):
>>
>> https://repository.apache.org/content/repositories/orgapachefluo-1014/org/apache/fluo/fluo-recipes/1.0.0-incubating/fluo-recipes-1.0.0-incubating-source-release.tar.gz
>> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
>> given artifact.)
>>
>> All artifacts were built and staged with:
>>  mvn release:prepare&&  mvn release:perform
>>
>> Signing keys are available at
>> https://www.apache.org/dist/incubator/fluo/KEYS
>> (Expected fingerprint: CF72 CA07 C8BC 86A1 C862  765F 9AAC FB56 352A CF76)
>>
>> Release notes (in progress) can be found at:
>> https://fluo.apache.org/.../1.0.0-incubating (not yet started)
>>
>> Please vote one of:
>> [ ] +1 - I have verified and accept...
>> [ ] +0 - I have reservations, but not strong enough to vote against...
>> [ ] -1 - Because..., I do not accept...
>> ... these artifacts as the 1.0.0-incubating release of Apache Fluo.
>>
>> This vote will end on Thur Oct 20 18:10:00 UTC 2016
>> (Fri Oct 14:10:00 EDT 2016 / Fri Oct 20 11:10:00 PDT 2016)
>>
>> Thanks!
>>
>> P.S. Hint: download the whole staging repo with
>>  wget -erobots=off -r -l inf -np -nH \
>>
>> https://repository.apache.org/content/repositories/orgapachefluo-1014/
>>  # note the trailing slash is needed

Re: [VOTE] Fluo Recipes 1.0.0-incubating-rc0

[Keith Turner]

+1

 * Sigs on all artifacts check out
 * Source tar contents match commit
 * Successfully ran mvn verify for stress, webindex, and phrasecount
use staging repo

On Mon, Oct 17, 2016 at 2:08 PM, Keith Turner <ke...@deenlo.com> wrote:
> Fluo Developers,
>
> Please consider the following candidate for Fluo Recipes 1.0.0-incubating.
>
> Git Commit:
> 2eb367e173721607a2cb958927e3c753b40188ed
> Branch:
> 1.0.0-incubating-rc0
>
> If this vote passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Fluo Recipes 1.0.0-incubating' \
> -s rel/fluo-recipes-1.0.0-incubating \
> 2eb367e173721607a2cb958927e3c753b40188ed
>
> Staging repo:
> https://repository.apache.org/content/repositories/orgapachefluo-1014
>
> Source (official release artifact):
> https://repository.apache.org/content/repositories/orgapachefluo-1014/org/apache/fluo/fluo-recipes/1.0.0-incubating/fluo-recipes-1.0.0-incubating-source-release.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
> given artifact.)
>
> All artifacts were built and staged with:
> mvn release:prepare && mvn release:perform
>
> Signing keys are available at
> https://www.apache.org/dist/incubator/fluo/KEYS
> (Expected fingerprint: CF72 CA07 C8BC 86A1 C862  765F 9AAC FB56 352A CF76)
>
> Release notes (in progress) can be found at:
> https://fluo.apache.org/.../1.0.0-incubating (not yet started)
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1.0.0-incubating release of Apache Fluo.
>
> This vote will end on Thur Oct 20 18:10:00 UTC 2016
> (Fri Oct 14:10:00 EDT 2016 / Fri Oct 20 11:10:00 PDT 2016)
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
> https://repository.apache.org/content/repositories/orgapachefluo-1014/
> # note the trailing slash is needed

Re: [VOTE] Fluo Recipes 1.0.0-incubating-rc0

[Keith Turner]

On Mon, Oct 17, 2016 at 7:46 PM, Christopher <ctubb...@apache.org> wrote:
> Were you intending this to be a vote thread or just a test? By convention,

I was intending it to be a test, but as I looked at it I could not
find any problems.  So I decided to try voting on it.


> I usually reserve rc0 for test of the release process itself, and rc1+ for
> actual release candidates... but you may have something else in mind. I
> just want to be sure before I prioritize testing this RC over other stuffs.
>
> On Mon, Oct 17, 2016, 16:42 Mike Walch <mwa...@apache.org> wrote:
>
>> +1
>>
>> * Verified signatures and hashes for source release
>> * Verified that recipes jars exist in staging repo
>> * Ran 'mvn verify' for stresso and webindex using staging repo
>> * Ran webindex dev server using staging repo
>>
>> On Mon, Oct 17, 2016 at 2:08 PM Keith Turner <ke...@deenlo.com> wrote:
>>
>> > Fluo Developers,
>> >
>> > Please consider the following candidate for Fluo Recipes
>> 1.0.0-incubating.
>> >
>> > Git Commit:
>> > 2eb367e173721607a2cb958927e3c753b40188ed
>> > Branch:
>> > 1.0.0-incubating-rc0
>> >
>> > If this vote passes, a gpg-signed tag will be created using:
>> > git tag -f -m 'Apache Fluo Recipes 1.0.0-incubating' \
>> > -s rel/fluo-recipes-1.0.0-incubating \
>> > 2eb367e173721607a2cb958927e3c753b40188ed
>> >
>> > Staging repo:
>> > https://repository.apache.org/content/repositories/orgapachefluo-1014
>> >
>> > Source (official release artifact):
>> >
>> >
>> https://repository.apache.org/content/repositories/orgapachefluo-1014/org/apache/fluo/fluo-recipes/1.0.0-incubating/fluo-recipes-1.0.0-incubating-source-release.tar.gz
>> > (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
>> > given artifact.)
>> >
>> > All artifacts were built and staged with:
>> > mvn release:prepare && mvn release:perform
>> >
>> > Signing keys are available at
>> > https://www.apache.org/dist/incubator/fluo/KEYS
>> > (Expected fingerprint: CF72 CA07 C8BC 86A1 C862  765F 9AAC FB56 352A
>> CF76)
>> >
>> > Release notes (in progress) can be found at:
>> > https://fluo.apache.org/.../1.0.0-incubating (not yet started)
>> >
>> > Please vote one of:
>> > [ ] +1 - I have verified and accept...
>> > [ ] +0 - I have reservations, but not strong enough to vote against...
>> > [ ] -1 - Because..., I do not accept...
>> > ... these artifacts as the 1.0.0-incubating release of Apache Fluo.
>> >
>> > This vote will end on Thur Oct 20 18:10:00 UTC 2016
>> > (Fri Oct 14:10:00 EDT 2016 / Fri Oct 20 11:10:00 PDT 2016)
>> >
>> > Thanks!
>> >
>> > P.S. Hint: download the whole staging repo with
>> > wget -erobots=off -r -l inf -np -nH \
>> >
>> https://repository.apache.org/content/repositories/orgapachefluo-1014/
>> > # note the trailing slash is needed
>> >
>>

[VOTE] Fluo Recipes 1.0.0-incubating-rc0

[Keith Turner]

Fluo Developers,

Please consider the following candidate for Fluo Recipes 1.0.0-incubating.

Git Commit:
2eb367e173721607a2cb958927e3c753b40188ed
Branch:
1.0.0-incubating-rc0

If this vote passes, a gpg-signed tag will be created using:
git tag -f -m 'Apache Fluo Recipes 1.0.0-incubating' \
-s rel/fluo-recipes-1.0.0-incubating \
2eb367e173721607a2cb958927e3c753b40188ed

Staging repo:
https://repository.apache.org/content/repositories/orgapachefluo-1014

Source (official release artifact):
https://repository.apache.org/content/repositories/orgapachefluo-1014/org/apache/fluo/fluo-recipes/1.0.0-incubating/fluo-recipes-1.0.0-incubating-source-release.tar.gz
(Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
given artifact.)

All artifacts were built and staged with:
mvn release:prepare && mvn release:perform

Signing keys are available at
https://www.apache.org/dist/incubator/fluo/KEYS
(Expected fingerprint: CF72 CA07 C8BC 86A1 C862  765F 9AAC FB56 352A CF76)

Release notes (in progress) can be found at:
https://fluo.apache.org/.../1.0.0-incubating (not yet started)

Please vote one of:
[ ] +1 - I have verified and accept...
[ ] +0 - I have reservations, but not strong enough to vote against...
[ ] -1 - Because..., I do not accept...
... these artifacts as the 1.0.0-incubating release of Apache Fluo.

This vote will end on Thur Oct 20 18:10:00 UTC 2016
(Fri Oct 14:10:00 EDT 2016 / Fri Oct 20 11:10:00 PDT 2016)

Thanks!

P.S. Hint: download the whole staging repo with
wget -erobots=off -r -l inf -np -nH \
https://repository.apache.org/content/repositories/orgapachefluo-1014/
# note the trailing slash is needed

Re: Updated Fluo 1.0.0 release plan

[Keith Turner]

On Wed, Aug 31, 2016 at 2:45 PM, Mike Walch  wrote:
> To prepare for the 1.0.0 release of Fluo, I updated the release plan.
>
> https://github.com/apache/incubator-fluo/wiki/1.0.0-Release-Plan
>
> I separated the completed work from possible work.  As for remaining work,
> it looks like we have 2 pending pull requests and 5 outstanding 'nice to
> have' issues.
>
> Keith & Chris, any status on the two pending PRs?

I resolved both of those.

>
> Of the 5 outstanding issues, only one issue (#714) has been claimed.
> Garvit, can you give a status on #714?  If you are busy and can't complete
> it, that's no problem. We can push it off or see if someone else wants to
> work it for 1.0.0.

I may try to take a crack at #714, if not we can just shelf it.

>
> What does everyone think about the other 4 issues? My view is that they can
> be pushed off to a later release.

I agree I also think we can push the other 4 off.

>
> Is there any other work for 1.0.0 that I am missing in the release plan?

Re: First release status

[Keith Turner]

I updated Fluo Recipes to use the newly released parent pom.

On Thu, Aug 11, 2016 at 6:29 PM, Christopher  wrote:
> So, now that the vote on the incubator list passed, I've moved the release
> artifacts to the right area in SVN, for mirroring, and released the Maven
> repository.
>
> I've also cleaned up the old RC branches and staging repos, and merged in
> the outstanding PR for updating the reference to the parent POM in the
> master branch's POM.
>
> I've created a ticket for updating the website downloads page (
> https://github.com/apache/incubator-fluo-website/issues/17)
>
> Once we update the downloads page and let the mirrors propagate, we can do
> an [ANNOUNCE] thread on this list, but I don't think we need to bother
> sending to the wider annou...@apache.org. I don't think they're significant
> enough to warrant the broad announcement... especially since they aren't
> very useful on their own.
>
> Are there any other remaining tasks which I have not thought of?

Re: [DISCUSS] Trademark discussions and plan

[Keith Turner]

On Fri, Aug 5, 2016 at 3:03 PM, Josh Elser <josh.el...@gmail.com> wrote:
> Keith Turner wrote:
>>
>> On Fri, Aug 5, 2016 at 2:48 PM, Josh Elser<josh.el...@gmail.com>  wrote:
>>>
>>> >  Thanks for all the points, Keith and Mike. This was very helpful to
>>> > me. I
>>> >  assumed formerly-known-as-Zetten was more about deploying Fluo itself,
>>> > but
>>> >  it seems it is more about the prerequisites for Fluo. I was just
>>> > assuming a
>>> >  bit too much it seems:)
>>> >
>>> >  While it might be a little premature before there is an official
>>> > release, I
>>> >  think instructions on the website for how a user installs/configures
>>> > Fluo
>>> >  would be great and help attract new users.
>>
>>
>> This will happen.  Mike has been taking markdown instructions in Fluo
>> itself and posting those on the website in the past.   This includes
>> install instructions.   I just put up a PR to remove the links to
>> fluo-dev (now called Uno) and zetten (now called Muchos) from those
>> install instructions.  I think this is a nice improvement.   The
>> install instructions should not have been pointing to those w/o the
>> caveat about not using them for production.  But I think its better
>> just not pointing to them at all.
>>
>> https://github.com/apache/incubator-fluo/pull/754
>>
>
> Thanks for the pointer!
>
> I think making sure that it is crystal clear that Unos and Muchos can be
> used to prepare an environment to run Fluo is important. Will watch that
> pull request for future updates.


I agree.  I updated both projects introduction in an attempt to better
communicate this.

https://github.com/astralway/uno
https://github.com/astralway/muchos

Re: Where is fluo-parent?

[Keith Turner]

On Sat, Aug 6, 2016 at 12:53 PM, Christopher <ctubb...@apache.org> wrote:
> On Sat, Aug 6, 2016, 09:42 Keith Turner <ke...@deenlo.com> wrote:
>
>> Its in another branch, until we can get it released you will need to
>> install it.  The parent depends on build resources which is another
>> branch, need to install that also.  I would install the ones we are
>> currently voting on for release in the branches below below.   Then
>> change the the version of the parent Fluo's pom to 1-incubating.
>>
>> https://github.com/apache/incubator-fluo/tree/fluo-parent-1-rc3
>> https://github.com/apache/incubator-fluo/tree/build-resources-1.0.0-rc1
>>
>> so do
>>   git checkout fluo-parent-1-rc3; mvn install
>>   git checkout build-resources-1.0.0-rc1; mvn install
>>   then checkout master and change the parent version
>>
>> The reason I would use the rc versions at the moment is that the
>> current versions have a small problem that the rc versions dont.
>>
>>
> Oh? What small problem?

The versions are misaligned in the branches between parent pom and
resources pom.  The parent pom points to a non snapshot version of the
resources.  The resources declares itself as a snapshot.  The RC
branches do not have this problem.


>
>>

Re: Where is fluo-parent?

[Keith Turner]

Its in another branch, until we can get it released you will need to
install it.  The parent depends on build resources which is another
branch, need to install that also.  I would install the ones we are
currently voting on for release in the branches below below.   Then
change the the version of the parent Fluo's pom to 1-incubating.

https://github.com/apache/incubator-fluo/tree/fluo-parent-1-rc3
https://github.com/apache/incubator-fluo/tree/build-resources-1.0.0-rc1

so do
  git checkout fluo-parent-1-rc3; mvn install
  git checkout build-resources-1.0.0-rc1; mvn install
  then checkout master and change the parent version

The reason I would use the rc versions at the moment is that the
current versions have a small problem that the rc versions dont.

On Sat, Aug 6, 2016 at 9:16 AM, Michael Wall  wrote:
> I checked out https://git-wip-us.apache.org/repos/asf/incubator-fluo.git,
> and tried to build.  Here is what I got
>
> mvn clean install
> [INFO] Scanning for projects...
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [FATAL] Non-resolvable parent POM for
> org.apache.fluo:fluo-project:1.0.0-incubating-SNAPSHOT: Could not find
> artifact org.apache.fluo:fluo-parent:pom:1-SNAPSHOT and
> 'parent.relativePath' points at wrong local POM @ line 18, column 11
>  @
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR]   The project
> org.apache.fluo:fluo-project:1.0.0-incubating-SNAPSHOT
> (/Users/mjwall/git/incubator-fluo/pom.xml) has 1 error
> [ERROR] Non-resolvable parent POM for
> org.apache.fluo:fluo-project:1.0.0-incubating-SNAPSHOT: Could not find
> artifact org.apache.fluo:fluo-parent:pom:1-SNAPSHOT and
> 'parent.relativePath' points at wrong local POM @ line 18, column 11 ->
> [Help 2]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> [ERROR] [Help 2]
> http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
>
> Where is the source code for fluo-parent so I can build it myself?
>
> Thanks

Re: [VOTE] Fluo Parent Pom 1-incubating (rc1)

[Keith Turner]

I did the following :

 * Checked signature, was good
 * Diffed tarball and branch, looked good (DEPENDENCIES was only in the
tarball)
 * Build Fluo master(modified commit 5ed9f2d)  using parent pom and staging
repo, build was successful.

On Tue, Jul 26, 2016 at 1:44 PM, Christopher  wrote:

> Fluo Developers,
>
> Please consider the following candidate for Fluo Parent POM 1-incubating.
>
> Git Commit:
> 4473de40fa562e6caff48b87e93aa2f23a6b885b
> Branch:
> fluo-parent-1-rc1
>
> If this vote passes, a vote will be brought to the IPMC, and if that
> passes, a gpg-signed tag will be created using:
> git tag -f -m 'Apache Fluo Parent POM 1-incubating' -s
> rel/fluo-parent-1-incubating 4473de40fa562e6caff48b87e93aa2f23a6b885b
>
> Staging repo:
> https://repository.apache.org/content/repositories/orgapachefluo-1002
> Source (official release artifact):
>
> https://repository.apache.org/content/repositories/orgapachefluo-1002/org/apache/fluo/fluo-parent/1-incubating/fluo-parent-1-incubating-source-release.tar.gz
> (Append ".sha1", ".md5", or ".asc" to download the signature/hash for a
> given artifact.)
>
> Signing key fingerprint is: 8CC4F8A2B29C2B040F2B835D6F0CDAE700B6899D
>
> Please vote one of:
> [ ] +1 - I have verified and accept...
> [ ] +0 - I have reservations, but not strong enough to vote against...
> [ ] -1 - Because..., I do not accept...
> ... these artifacts as the 1-incubating release of Apache Fluo Parent POM.
>
> This vote will end on Fri Jul 29 18:00:00 UTC 2016
> (Fri Jul 29 14:00:00 EDT 2016 / Fri Jul 29 11:00:00 PDT 2016)
>
> Thanks!
>
> P.S. Hint: download the whole staging repo with
> wget -erobots=off -r -l inf -np -nH \
> https://repository.apache.org/content/repositories/orgapachefluo-1002/
> # note the trailing slash is needed
>

Re: Test staging release

[Keith Turner]

I was able to build fluo-quickstart, fluo-recipes, phrasecount,
fluo-stress, and webindex against fluo 1.0.0-incubating from this staging
repo.  I ran mvn verify on each of those projects.

Are we ready to vote on the parent pom?

On Mon, Jul 25, 2016 at 6:50 PM, Christopher  wrote:

> I tested staging a release today for fluo-parent and fluo:
> https://repository.apache.org/content/repositories/orgapachefluo-1001/
>
> I created a few follow-on issues to address, and opened one PR to fix some
> immediate issues.
>

Notifications list

[Keith Turner]

Should Github issues and pull request be going to the notifications lists?

Re: [DISCUSS] Establishing PPMC

[Keith Turner]

+1

On Tue, Jul 5, 2016 at 4:16 PM, Christopher  wrote:

> It occurs to me that we haven't actually made it a point to clarify who the
> PPMC members are. However, we'll need a PPMC established in order to vote
> on releases, before calling a vote on the general@incubator list for IPMC
> voting.
>
> I think the PPMC should consist of those on the initial proposal. I believe
> this makes sense, since these are the folks who were bootstrapped into the
> private@fluo mailing list.
>
> Additionally, I think it makes sense (and is simpler all around, if we
> treat PMC == committers).
>

Re: Contribute in Fluo

[Keith Turner]

Yeah looking at those would be good.  We are in an odd situation since we
moved to Apache.  We can not currently assign issues on GH.  Hopefully this
will change soon.  Mike decided to update the wiki with the issues he is
working on for the time being.  If you don't find anything let me know.

On Tue, Jun 28, 2016 at 12:54 PM, Garvit Bansal <garvitbansal...@gmail.com>
wrote:

> Yes I will be happy to work on another task.
> I looked on some issue which are nice to have in 1.0.0 release plan. Can I
> pick any issue from there.
>
> On Tue, Jun 28, 2016 at 10:00 PM, Keith Turner <ke...@deenlo.com> wrote:
>
> > Garvit,
> >
> > Thanks again for the patch.  If you are interested in another starter
> > issue, let me know.
> >
> > Keith
> >
> > On Tue, Jun 28, 2016 at 9:06 AM, Garvit Bansal <
> garvitbansal...@gmail.com>
> > wrote:
> >
> > > Sure. I can work on this task to start with.
> > >
> > > On Tue, Jun 28, 2016 at 1:04 AM, Keith Turner <ke...@deenlo.com>
> wrote:
> > >
> > > >  I would suggest 662[1] as a good starter issue.  To implement this
> > add a
> > > > new task to the existing timer[2] that periodically logs
> currentTs[3].
> > > >
> > > > [1]: https://github.com/apache/incubator-fluo/issues/662
> > > > [2]:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-fluo/blob/2b2d784fc4e5f96af8346d779274e2b1a3246e0c/modules/core/src/main/java/org/apache/fluo/core/oracle/OracleServer.java#L164
> > > > [3]:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-fluo/blob/2b2d784fc4e5f96af8346d779274e2b1a3246e0c/modules/core/src/main/java/org/apache/fluo/core/oracle/OracleServer.java#L86
> > > >
> > > >
> > > > On Mon, Jun 27, 2016 at 1:38 PM, Garvit Bansal <
> > > garvitbansal...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Keith,
> > > > >
> > > > > Thanks for reply. I looked on some open issues in github but not
> able
> > > to
> > > > > identify whether it is small bug or big.
> > > > > It will help me a lot if you can tell me how to start with.
> > > > >
> > > > > Garvit
> > > > >
> > > > > On Mon, Jun 27, 2016 at 10:08 PM, Keith Turner <ke...@deenlo.com>
> > > wrote:
> > > > >
> > > > > > Garvit,
> > > > > >
> > > > > > Where you able to find anything to work on?  If not I can help
> you
> > > find
> > > > > > something, just let me know.  We can chat on IRC.
> > > > > >
> > > > > > Keith
> > > > > >
> > > > > > On Tue, Jun 21, 2016 at 3:12 PM, Garvit Bansal <
> > > > > garvitbansal...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Christopher and Mike. I will look for issues on which I
> > can
> > > > > start
> > > > > > > with in incubator-fluo repo
> > > > > > >
> > > > > > > On Wed, Jun 22, 2016 at 12:25 AM, Mike Walch <
> mwa...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Garvit,
> > > > > > > >
> > > > > > > > We have some documentation on how to contribute (
> > > > > > > > http://fluo.apache.org/how-to-contribute/) on our website.
> > > Also,
> > > > if
> > > > > > you
> > > > > > > > don't find an issue that you want work on, one way to
> > contribute
> > > is
> > > > > to
> > > > > > > > build a example Fluo application (see
> > > > > > > > http://fluo.apache.org/related-projects/ for examples) and
> > > provide
> > > > > > > > feedback
> > > > > > > > through new issues or pull requests.
> > > > > > > >
> > > > > > > > -Mike
> > > > > > > >
> > > > > > > > On Tue, Jun 21, 2016 at 2:42 PM Christopher <
> > ctubb...@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Garvit,
> > > > > > > > >
> >

Re: Contribute in Fluo

[Keith Turner]

Garvit,

Thanks again for the patch.  If you are interested in another starter
issue, let me know.

Keith

On Tue, Jun 28, 2016 at 9:06 AM, Garvit Bansal <garvitbansal...@gmail.com>
wrote:

> Sure. I can work on this task to start with.
>
> On Tue, Jun 28, 2016 at 1:04 AM, Keith Turner <ke...@deenlo.com> wrote:
>
> >  I would suggest 662[1] as a good starter issue.  To implement this add a
> > new task to the existing timer[2] that periodically logs currentTs[3].
> >
> > [1]: https://github.com/apache/incubator-fluo/issues/662
> > [2]:
> >
> >
> https://github.com/apache/incubator-fluo/blob/2b2d784fc4e5f96af8346d779274e2b1a3246e0c/modules/core/src/main/java/org/apache/fluo/core/oracle/OracleServer.java#L164
> > [3]:
> >
> >
> https://github.com/apache/incubator-fluo/blob/2b2d784fc4e5f96af8346d779274e2b1a3246e0c/modules/core/src/main/java/org/apache/fluo/core/oracle/OracleServer.java#L86
> >
> >
> > On Mon, Jun 27, 2016 at 1:38 PM, Garvit Bansal <
> garvitbansal...@gmail.com>
> > wrote:
> >
> > > Hi Keith,
> > >
> > > Thanks for reply. I looked on some open issues in github but not able
> to
> > > identify whether it is small bug or big.
> > > It will help me a lot if you can tell me how to start with.
> > >
> > > Garvit
> > >
> > > On Mon, Jun 27, 2016 at 10:08 PM, Keith Turner <ke...@deenlo.com>
> wrote:
> > >
> > > > Garvit,
> > > >
> > > > Where you able to find anything to work on?  If not I can help you
> find
> > > > something, just let me know.  We can chat on IRC.
> > > >
> > > > Keith
> > > >
> > > > On Tue, Jun 21, 2016 at 3:12 PM, Garvit Bansal <
> > > garvitbansal...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Christopher and Mike. I will look for issues on which I can
> > > start
> > > > > with in incubator-fluo repo
> > > > >
> > > > > On Wed, Jun 22, 2016 at 12:25 AM, Mike Walch <mwa...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi Garvit,
> > > > > >
> > > > > > We have some documentation on how to contribute (
> > > > > > http://fluo.apache.org/how-to-contribute/) on our website.
> Also,
> > if
> > > > you
> > > > > > don't find an issue that you want work on, one way to contribute
> is
> > > to
> > > > > > build a example Fluo application (see
> > > > > > http://fluo.apache.org/related-projects/ for examples) and
> provide
> > > > > > feedback
> > > > > > through new issues or pull requests.
> > > > > >
> > > > > > -Mike
> > > > > >
> > > > > > On Tue, Jun 21, 2016 at 2:42 PM Christopher <ctubb...@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > Hi Garvit,
> > > > > > >
> > > > > > > Sure. We're just getting set up transitioning to the Apache
> > > > Incubator.
> > > > > > > We're using GitHub issues to track bugs/features:
> > > > > > > https://github.com/apache/incubator-fluo/issues and the
> current
> > > > > workflow
> > > > > > > is
> > > > > > > to submit pull requests for feature changes.
> > > > > > >
> > > > > > > We're still working on setting up the website (
> > > > > https://fluo.apache.org/
> > > > > > ),
> > > > > > > so if you have any suggestions there, that would be helpful,
> too.
> > > > > > >
> > > > > > > On Tue, Jun 21, 2016 at 2:20 PM Garvit Bansal <
> > > > > garvitbansal...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > My name is Garvit and I like to contribute in Fluo. I found
> > this
> > > > > > project
> > > > > > > > interesting and want to make contribution to it. Can somebody
> > > help
> > > > me
> > > > > > how
> > > > > > > > to start with, bug fixes or something like that which will
> help
> > > 

1.0.0 Release Plan

[Keith Turner]

Mike and I went throught the issues and generated an initial 1.0.0 release
plan[1].

We were thinking of setting a goal date of July 18 to have a release
candidate ready for vote.  If that date is a problem for anyone, let us
know.

I think anyone should be able to edit the wiki.

[1]https://github.com/apache/incubator-fluo/wiki/1.0.0-Release-Plan

Re: Contribute in Fluo

[Keith Turner]

 I would suggest 662[1] as a good starter issue.  To implement this add a
new task to the existing timer[2] that periodically logs currentTs[3].

[1]: https://github.com/apache/incubator-fluo/issues/662
[2]:
https://github.com/apache/incubator-fluo/blob/2b2d784fc4e5f96af8346d779274e2b1a3246e0c/modules/core/src/main/java/org/apache/fluo/core/oracle/OracleServer.java#L164
[3]:
https://github.com/apache/incubator-fluo/blob/2b2d784fc4e5f96af8346d779274e2b1a3246e0c/modules/core/src/main/java/org/apache/fluo/core/oracle/OracleServer.java#L86


On Mon, Jun 27, 2016 at 1:38 PM, Garvit Bansal <garvitbansal...@gmail.com>
wrote:

> Hi Keith,
>
> Thanks for reply. I looked on some open issues in github but not able to
> identify whether it is small bug or big.
> It will help me a lot if you can tell me how to start with.
>
> Garvit
>
> On Mon, Jun 27, 2016 at 10:08 PM, Keith Turner <ke...@deenlo.com> wrote:
>
> > Garvit,
> >
> > Where you able to find anything to work on?  If not I can help you find
> > something, just let me know.  We can chat on IRC.
> >
> > Keith
> >
> > On Tue, Jun 21, 2016 at 3:12 PM, Garvit Bansal <
> garvitbansal...@gmail.com>
> > wrote:
> >
> > > Thanks Christopher and Mike. I will look for issues on which I can
> start
> > > with in incubator-fluo repo
> > >
> > > On Wed, Jun 22, 2016 at 12:25 AM, Mike Walch <mwa...@apache.org>
> wrote:
> > >
> > > > Hi Garvit,
> > > >
> > > > We have some documentation on how to contribute (
> > > > http://fluo.apache.org/how-to-contribute/) on our website.  Also, if
> > you
> > > > don't find an issue that you want work on, one way to contribute is
> to
> > > > build a example Fluo application (see
> > > > http://fluo.apache.org/related-projects/ for examples) and provide
> > > > feedback
> > > > through new issues or pull requests.
> > > >
> > > > -Mike
> > > >
> > > > On Tue, Jun 21, 2016 at 2:42 PM Christopher <ctubb...@apache.org>
> > wrote:
> > > >
> > > > > Hi Garvit,
> > > > >
> > > > > Sure. We're just getting set up transitioning to the Apache
> > Incubator.
> > > > > We're using GitHub issues to track bugs/features:
> > > > > https://github.com/apache/incubator-fluo/issues and the current
> > > workflow
> > > > > is
> > > > > to submit pull requests for feature changes.
> > > > >
> > > > > We're still working on setting up the website (
> > > https://fluo.apache.org/
> > > > ),
> > > > > so if you have any suggestions there, that would be helpful, too.
> > > > >
> > > > > On Tue, Jun 21, 2016 at 2:20 PM Garvit Bansal <
> > > garvitbansal...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > My name is Garvit and I like to contribute in Fluo. I found this
> > > > project
> > > > > > interesting and want to make contribution to it. Can somebody
> help
> > me
> > > > how
> > > > > > to start with, bug fixes or something like that which will help
> me
> > to
> > > > get
> > > > > > in flow.
> > > > > >
> > > > > > Garvit
> > > > > > --
> > > > > > *Garvit Bansal,*
> > > > > > Software Developer at Flipkart
> > > > > > B.Tech in Computer Science and Engineering
> > > > > > The LNM Institute of Information Technology
> > > > > > Jaipur, Rajasthan-302001
> > > > > > *Mobile:* 9610728532, 9886384276
> > > > > > *Website:* garvitbansal244.appspot.com
> > > > > >
> > > > > > <https://in.linkedin.com/pub/garvit-bansal/50/129/433>
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *Garvit Bansal,*
> > > Software Developer at Flipkart
> > > B.Tech in Computer Science and Engineering
> > > The LNM Institute of Information Technology
> > > Jaipur, Rajasthan-302001
> > > *Mobile:* 9610728532, 9886384276
> > > *Website:* garvitbansal244.appspot.com
> > >
> > > <https://in.linkedin.com/pub/garvit-bansal/50/129/433>
> > >
> >
>
>
>
> --
> *Garvit Bansal,*
> Software Developer at Flipkart
> B.Tech in Computer Science and Engineering
> The LNM Institute of Information Technology
> Jaipur, Rajasthan-302001
> *Mobile: *9886384276
> *Website: *www.garvitbansal.xyz
>
> <https://in.linkedin.com/pub/garvit-bansal/50/129/433>
>

Re: Incoming commits@ list

[Keith Turner]

Yeah that sounds great.  Thanks Josh.

On Thu, Jun 16, 2016 at 2:19 PM, Josh Elser  wrote:

> Infra gently suggested that we have a commits@ list for consistency with
> other projects.
>
> I just submitted the form to create it and will send a note when it's made.
>

Re: JIRA vs Github Issues

[Keith Turner]

On Tue, Jun 14, 2016 at 2:17 PM, Keith Turner <ke...@deenlo.com> wrote:

>
>
> On Tue, Jun 14, 2016 at 1:23 PM, Josh Elser <josh.el...@gmail.com> wrote:
>
>>
>> Josh Elser wrote:
>>
>>>
>>>
>>> Keith Turner wrote:
>>>
>>>> > If INFRA can do the GH transition now, let's go ahead
>>>>> and do
>>>>> > it. It seems like the consensus has already been reached that GH>>
>>>>> JIRA
>>>>> > and INFRA has not directly persuaded against GH per your last
>>>>> messages,
>>>>> > Keith. Let's take action now.
>>>>> >
>>>>>
>>>>
>>>> Ok yeah, based on Mike and Christopher's past messages its pretty clear
>>>> that they are in favor. I will update the INFRA issue and try to
>>>> figure out
>>>> the next step.
>>>>
>>>>
>>> Thanks for driving, Keith, and sorry for not taking more action myself
>>> to push this :)
>>>
>>
>> Any update? How can I help?
>>
>
> I don't know.  At this point waiting on INFRA.  I dropped into hipchat and
> stated we had finalized our decision and updated the issue. I asked if
> there was anything else we needed to do, no answer.  So I asked if we would
> need to add someone to the GH fluo-io org to do the transfer and someone
> said probably.
>

I did this sometime last week.



>
> Mike dropped into hipchat this morning and asked INFRA about the issue and
> did not get an answer.
>
>
>
>

Re: JIRA vs Github Issues

[Keith Turner]

On Tue, Jun 14, 2016 at 1:23 PM, Josh Elser <josh.el...@gmail.com> wrote:

>
> Josh Elser wrote:
>
>>
>>
>> Keith Turner wrote:
>>
>>> > If INFRA can do the GH transition now, let's go ahead
>>>> and do
>>>> > it. It seems like the consensus has already been reached that GH>>
>>>> JIRA
>>>> > and INFRA has not directly persuaded against GH per your last
>>>> messages,
>>>> > Keith. Let's take action now.
>>>> >
>>>>
>>>
>>> Ok yeah, based on Mike and Christopher's past messages its pretty clear
>>> that they are in favor. I will update the INFRA issue and try to
>>> figure out
>>> the next step.
>>>
>>>
>> Thanks for driving, Keith, and sorry for not taking more action myself
>> to push this :)
>>
>
> Any update? How can I help?
>

I don't know.  At this point waiting on INFRA.  I dropped into hipchat and
stated we had finalized our decision and updated the issue. I asked if
there was anything else we needed to do, no answer.  So I asked if we would
need to add someone to the GH fluo-io org to do the transfer and someone
said probably.

Mike dropped into hipchat this morning and asked INFRA about the issue and
did not get an answer.

Re: JIRA vs Github Issues

[Keith Turner]

On Wed, Jun 8, 2016 at 8:21 PM, Christopher <ctubb...@apache.org> wrote:

> Personally, I'd rather use GitHub issues anyway. The minor inconvenience in
> the case of occasional unresponsive reporters I think would be minimal and
> acceptable (there is the additional annoyance of not being able to manage
> labels on issues, but I think even that is minor).
>
> Using GitHub issues could even help advance the state of development on ASF
> projects, by offering an additional use case to motivate further
> convenience integration. We could even help create or test a service which
> authenticates using ASF project credentials and integrates with GitHub's
> API to manage issues. I've recently been looking at GitHub OAuth integrated
> applications, and it doesn't seem like a difficult service to create. Or,
> perhaps the M.A.T.T. stuff will be online and available to us soon enough
> and it won't matter.
>
> But, if the majority feel JIRA would be better in the meantime, then we can
> convert the existing issues to JIRA, transfer the repo, and have issues
> disabled. I strongly prefer GH issues over JIRA, but defer to the majority.
>

I don't have a strong preference.  I slightly prefer GH issues.  At this
point, my main motivation is moving forward.  Don't want to be stuck in
limbo forever.  I'll pop into hipchat and find out what the timeliness on
transferring to GH would be and what level of effort it requires on INFRA's
part.  I'll report back what I find.



>
> On Wed, Jun 8, 2016 at 2:23 PM Keith Turner <ke...@deenlo.com> wrote:
>
> > On Wed, Jun 8, 2016 at 1:14 PM, Josh Elser <josh.el...@gmail.com> wrote:
> >
> > > IMO, use JIRA to start, and if/when INFRA gives their "blessing" that
> GH
> > > issues is a supported way to go, we can consider switching then.
> >
> >
> > I am in favor of that.
> >
> > Also I have been researching the option of transferring the existing GH
> > project to Apache and not using GH issues.  I asked INFRA about this and
> if
> > we are not using GH issues, then they will turn issues off for the Repo.
> > On a personal repo (with one issue) I tried turning off issues to see
> what
> > would happen.  After doing this I could not longer access issues.  Also a
> > link to the issue started returning 404.   I turned issues back on and
> the
> > issue was still there and the link started working.   Based on this
> > behaviour, I am not in favor of transferring the repo if we use Jira.   I
> > would like links to existing issues to keep working.
> >
> >
> > >
> > >
> > > Keith Turner wrote:
> > >
> > >> I spoke w/ INFRA on Hipchat and got some more info[1].  So we will not
> > be
> > >> able to close issue on GH and there is no timeline on when we would be
> > >> able
> > >> to.  Given this we need to decide if we would like to use GH issues or
> > >> JIRA.   Also we can transfer the existing project to Apache and not
> use
> > GH
> > >> issues.
> > >>
> > >> I am leaning towards using Jira.  Not being able to close issues could
> > be
> > >> a
> > >> pain.
> > >>
> > >> [1]:
> > >>
> > >>
> >
> https://issues.apache.org/jira/browse/INFRA-11901?focusedCommentId=15320755=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15320755
> > >>
> > >>
> >
>

Re: State of things

[Keith Turner]

On Thu, May 26, 2016 at 1:43 PM, Billie Rinaldi  wrote:

> I guess the reporting schedule is flexible. We could change our reporting
> group, or leave it as is and do a trivial report this month. I'd lean
>

A trivial report seems good.


> towards just doing the report, frankly. Maybe we'll make some additional
> infra progress before the report is due next week.
>
> Billie
>
> On Wed, May 25, 2016 at 7:55 PM, Josh Elser  wrote:
>
> > Seeing the note come across for your first board report (I believe you'll
> > have 3 monthly reports before moving to quarterly), it reminded me to
> check
> > where things are currently held up.
> >
> > As I remember/see it:
> >
> > * All committers should have ASF accts (I think Mike was missing, and I
> > think I saw the acct notice go across private@incubator)
> > * No Git repos yet (source or website)
> > * No JIRA project
> > * I think Billie sent me a link to a PR which sets up all of the karma
> > when you have the repos show up
> >
> > Have I missed anything? Would you like us to ping INFRA to see what's
> > going on (I've been offline a bunch this week -- haven't been following
> the
> > MLs per normal).
> >
> > Given the above is accurate, I think we should just push your first
> report
> > off another month (your first report will be July instead of June, and
> you
> > would do monthly through September) since it would be nothing other than
> "p
> > much waiting on infra".
> >
> > - Josh
> >
>