Re: maybe a handy script for working with PRs

[Sean Busbey]

May I humbly suggest y'all check out the Apache Yetus project?

http://yetus.apache.org/

It includes a framework for running some community decided automated
tests against new contributions and works for both JIRA attachments
and PRs in github.

http://yetus.apache.org/documentation/0.4.0/precommit-basic/

It also has a stripped down tool for just doing the fetch / apply
dance called 'smart-apply-patch'.



On Thu, Mar 23, 2017 at 9:01 PM, Otto Fowler  wrote:
> Nice,
> I like this one because I can just dump a pr in ~/tmp or wherever -or- in
> my current.
>
> I’ll star this though :)
>
>
> On March 23, 2017 at 18:55:52, Andy LoPresto (alopre...@apache.org) wrote:
>
> Thanks Otto. I have a similar bash function I use:
>
> function gpr() {
> echo "Fetching and checking out new git branch pr$@..."
> git fetch --all
> git checkout master
> git pull upstream master
> git checkout "upstream/pr/$@"
> git checkout -b "pr$@“
> }
>
> Usage: For PR 1234
>
> $ gpr 1234
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 23, 2017, at 7:00 AM, Otto Fowler  wrote:
>
> On the metron project, many of our committers use a couple of scripts when
> working with PRs, either cloning and testing or preparing commits.
> I have created a version of the checkout-pr script for the nifi project,
> and though maybe someone would be interested.
>
> This work is a fork of Nick Allen’s original repo for Metron.
>
> https://github.com/ottobackwards/commit-pr-stuff
>
> If you look in the nifi folder, there is a checkout-nifi-pr script, which
> when called will checkout a pr for you to work with.
> I am not sure if you have something similar already, but I thought I would
> throw this over the wall just the same.
>
> O

Re: [REMINDER] Please signoff when committing other people's changes

[Sean Busbey]

On Thu, Mar 2, 2017 at 6:00 PM, Andre  wrote:
> James,
>
> There's no doubt the Sign-off-by is redundant (as GIT itself holds that
> information, reason why GH is still able to show the information without
> the sign-of-by stamp), however, I agree with your view around positive
> action and easy to refer as Bryan pointed.
>


One important case where the signed-off-by isn't redundant (at least
within git) is when there is more than one reviewer. This has been
useful in some other projects, especially if folks get into the habit
and there are multiple places reviews happen (e.g. both on JIRA and
GitHub PRs). It's also been super helpful when non-committers are
doing reviews.

Another is when we get to the point of maintaining enough release
lines that the person doing a backport commit might not be the person
who did the initial review. The signed-off-by still carries useful
information (albeit something we can kind-of get from JIRA) about who
all did reviews.

Anyone interested in presenting at Data Day Texas?

[Sean Busbey]

See below for info about this day long conference in Austin. Anyone got
something they can submit?

>
> -- Forwarded message --
> From: "LynnBender / GeekAustin" 
> Date: Sep 8, 2016 16:49
> Subject: More Speakers - and other Data Day Texas news (NLP, Graphs,
IoT...)
> Cc:
>
> Friends,
>
> We've been doing Data Day since 2011, but I am really excited about the
upcoming installment. I think that we're really going to blow the roof off
the AT this time around. Data Day Texas is the largest independent data
event within 1500 miles of Texas -- and it happens in Austin. Woot!
>
> We've added another batch of speakers to the website:
> http://datadaytexas.com/speakers
>
> This is just the beginning. We have about 55 more
speakers/sessions/workshops to announce.
>
> The NLP tracks (NLP Day) are shaping up. Gabor Melli has agreed to speak
at Data Day and hold a workshop.
> http://datadaytexas.com/nlp-day-texas-2017
> We have about 15 more NLP speakers to announce
>
> We are currently working on sessions and speakers for the following
featured tracks:
>
> - Graph Data, Theory and Algorithms (Graph Day!) We'll start announcing
speakers next week.
> - Data Wrangling
> - Time Series Data
> - IoT
> - Deep Learning  / TensorFlow
> - Open Data (Announcing first batch of speakers soon!)
>
> We have quite a few folks from Austin speaking this year. We're still
waiting on bios and pics (hint hint). If you are doing something cool and
want to share it at *the largest data conference within 1500 miles of
Texas*, check out our proposals page:
> http://datadaytexas.com/proposals
>
> Ticket prices go up a little each week. Save money and get your ticket
now.
>
> Cheers,
>
> Lynn Bender
> Data Day / GeekAustin
> Copyright © 2016 GeekAustin, All rights reserved.
> You are receiving this email because your are a member of GeekAustin or
you have attending one of our events. If you do not wish to receive note of
our events, please unsubscribe. Our feelings won't be hurt -- well, maybe
just a little.
>
> Our mailing address is:
> GeekAustin
> 2900 West Anderson Lane C200
> Austin, TX 78757
>
> Add us to your address book
>
>
> Want to change how you receive these emails?
> You can update your preferences or unsubscribe from this list
>

[DISCUSS] Using the new 'help wanted' tool from comdev

[Sean Busbey]

Hi folks!

ASF comdev has put up a great new tool for funneling in new folks:

https://helpwanted.apache.org/

How about we brainstorm a few things here (maybe some beginner JIRAs
we can flesh out a little?) and then file?

-Sean

[NOTICE] jira lockdown

[Sean Busbey]

Hi folks!

Just a quick heads-up that the ASF JIRA is currently locked down to
counter a spam attack. Unfortunately, this lock down prevents our
normal open-policy that allows anyone with a JIRA account to create,
assign, and comment on issues.

If you are caught up in this, please drop me a note either on or off
list with your JIRA user name and I'll get you added to a formal JIRA
role so that you can interact with the NiFi project.

-Sean

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[Sean Busbey]

No magic that I know of. Each project that I've seen do this just
pushed new tags. Some make the tag point to the prior release tag,
some have it point to the commit that the prior release tag points to.
Those who want to be fancy use git cli to make the date match.

On Mon, Apr 18, 2016 at 10:58 PM, Joe Witt <joe.w...@gmail.com> wrote:
> Is there some good git magic to follow to do this Sean?  I could push
> new signed tags?
>
> On Mon, Apr 18, 2016 at 11:54 PM, Sean Busbey <bus...@apache.org> wrote:
>> The "rel/" prefix is part of the implementation of an ASF policy that 
>> requires a
>> protected immutable tag for releases. It will be present on all
>> releases going forward and we should probably add them for our prior
>> releases.
>>
>>
>> On Mon, Apr 18, 2016 at 12:28 PM, Aldrin Piri <aldrinp...@gmail.com> wrote:
>>> The "rel" prefix is a bit of a deviation from what we have been doing.  Up
>>> to this point, it had been x.y.z-RC# up until a final release occurred
>>> resulting in just an x.y.z tag.
>>>
>>> On Mon, Apr 18, 2016 at 1:19 PM, James Wing <jvw...@gmail.com> wrote:
>>>
>>>> Found it, thank you.  Is "rel/nifi-0.6.1" the going-forward tag format?
>>>>
>>>> On Mon, Apr 18, 2016 at 9:59 AM, Joe Witt <joe.w...@gmail.com> wrote:
>>>>
>>>> > Yep!  Thanks for catching that. Had made but not pushed.  Should be there
>>>> > now.
>>>> >
>>>> > On Mon, Apr 18, 2016 at 12:10 PM, James Wing <jvw...@gmail.com> wrote:
>>>> > > Thanks, Joe.  Will the 0.6.1 git tag be published as part of the
>>>> release?
>>>> > >
>>>> > > On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt <joew...@apache.org> wrote:
>>>> > >
>>>> > >> Hello
>>>> > >>
>>>> > >> The Apache NiFi team would like to announce the release of Apache NiFi
>>>> > >> 0.6.1.
>>>> > >>
>>>> > >> Apache NiFi is an easy to use, powerful, and reliable system to
>>>> > >> process and distribute data.  Apache NiFi was made for dataflow.  It
>>>> > >> supports highly configurable directed graphs of data routing,
>>>> > >> transformation, and system mediation logic.
>>>> > >>
>>>> > >> More details on Apache NiFi can be found here:
>>>> > >>   http://nifi.apache.org/
>>>> > >>
>>>> > >> The release artifacts can be downloaded from here:
>>>> > >>   http://nifi.apache.org/download.html
>>>> > >>
>>>> > >> Maven artifacts have been made available here:
>>>> > >>
>>>> > >>
>>>> >
>>>> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
>>>> > >>
>>>> > >> Release note highlights can be found here:
>>>> > >>
>>>> > >>
>>>> >
>>>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
>>>> > >>
>>>> > >> Thank you
>>>> > >> The Apache NiFi team
>>>> > >>
>>>> >
>>>>

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[Sean Busbey]

The "rel/" prefix is part of the implementation of an ASF policy that requires a
protected immutable tag for releases. It will be present on all
releases going forward and we should probably add them for our prior
releases.


On Mon, Apr 18, 2016 at 12:28 PM, Aldrin Piri  wrote:
> The "rel" prefix is a bit of a deviation from what we have been doing.  Up
> to this point, it had been x.y.z-RC# up until a final release occurred
> resulting in just an x.y.z tag.
>
> On Mon, Apr 18, 2016 at 1:19 PM, James Wing  wrote:
>
>> Found it, thank you.  Is "rel/nifi-0.6.1" the going-forward tag format?
>>
>> On Mon, Apr 18, 2016 at 9:59 AM, Joe Witt  wrote:
>>
>> > Yep!  Thanks for catching that. Had made but not pushed.  Should be there
>> > now.
>> >
>> > On Mon, Apr 18, 2016 at 12:10 PM, James Wing  wrote:
>> > > Thanks, Joe.  Will the 0.6.1 git tag be published as part of the
>> release?
>> > >
>> > > On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:
>> > >
>> > >> Hello
>> > >>
>> > >> The Apache NiFi team would like to announce the release of Apache NiFi
>> > >> 0.6.1.
>> > >>
>> > >> Apache NiFi is an easy to use, powerful, and reliable system to
>> > >> process and distribute data.  Apache NiFi was made for dataflow.  It
>> > >> supports highly configurable directed graphs of data routing,
>> > >> transformation, and system mediation logic.
>> > >>
>> > >> More details on Apache NiFi can be found here:
>> > >>   http://nifi.apache.org/
>> > >>
>> > >> The release artifacts can be downloaded from here:
>> > >>   http://nifi.apache.org/download.html
>> > >>
>> > >> Maven artifacts have been made available here:
>> > >>
>> > >>
>> >
>> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
>> > >>
>> > >> Release note highlights can be found here:
>> > >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
>> > >>
>> > >> Thank you
>> > >> The Apache NiFi team
>> > >>
>> >
>>

Re: [DISCUSS] From Contributor to Committer

[Sean Busbey]

Thanks for starting this Tony!

As a PMC member, I really try to focus on things that help the
community where we tend to have limited bandwidth: reviews weigh
heavily, as does helping out new folks on users@, and doing public
talking/workshops.

I also am inclined to vote in favor of folks who show the kind of
project dedication that we expect from PMC members. While we still
need to do a better job of describing those things, at the moment I'm
thinking of things like voting on release candidates, watching out for
our trademarks, and investing the time needed to handle our licensing
responsibilities.

On Sun, Apr 10, 2016 at 9:38 AM, Tony Kurc  wrote:
> First off, I recommend this reading this page to understand what the Apache
> NiFi PMC draws from when making a decision
>
> http://community.apache.org/contributors/index.html
>
> I thought it would be helpful for me to walk through how I interpret that
> guidance, and what that means for NiFi. For those that didn't read, there
> are four aspects of contribution that are worth considering someone for
> committership: community, project, documentation and code. Really, the
> committer decision comes down to: has this person built up enough merit in
> the community that I have a high degree of confidence that I trust him/her
> with write access to the code and website.
>
> Given that merit and trust are subjective measures, how does the PMC make
> those decisions? We, the PMC, have attempted to make this as evidence-based
> as possible. When discussing a contributor for being considered for
> committer access, we attempt to put together a corpus of interaction in the
> community, both negative and positive, and use this as a basis for
> discussion. The interaction with the community can include:
>
> - Interaction on the mailing lists - is this person helping others? Is this
> person using the community to enhance his/her understanding of the project
> or the apache foundation?
> - Code contributions - is this person contributing code that advances the
> project? How important is the code? Is this a niche capability, a core
> capability? How challenging was the code? Was the code improving the
> quality of the project (bug fix, adding  tests, or code that comes along
> with comprehensive unit and/or integration tests). How does this person
> react to criticism of his/her contribution? Is this person reacting
> positively to patch or pull request feedback? Is the code high quality?
> - Assisting others with their contributions - is this person providing
> useful comments on pull requests or patches? Is this person testing new
> features/functionality and providing feedback on the mailing list?
> - Participating in project votes and discussions: is this person helping to
> verify releases? Providing input to the roadmap? Is this person using the
> lists to get feedback on features he/she plan to implement?
> - Documentation contributions - is this person helping the community by
> blogging? providing patches to the web page or in-app docs? contributing to
> the project wiki?
> - Other community/project activities - has this person organized or talked
> at a meetup? has this person briefed at a conference or workshop?
> - "Going over and beyond" factor - Has this person done something
> exceptional to demonstrate dedication to the project? e.g. did this person
> go to great lengths to fix or diagnose a critical issue?
>
> An underlying theme of the above: the ASF code of conduct [1] is taken
> seriously by the PMC - while interacting with the community, was this
> person adhering to the guidelines? Are we seeing a pattern of openness,
> empathy, inquisitiveness, and willingness to cooperate? Has this person
> shown remorse for interaction that may have violated the code of conduct
> and a positive trend since?
>
> It helps for a committer to have evidence supporting all four aspects of
> contribution. It also helps to have demonstrated this over an extended
> period of time. I personally like to see at least 3 months of strong
> contribution.
>
> This is a start of the discussion, I'm hoping others can weigh in.
>
> 1. http://www.apache.org/foundation/policies/conduct.html
>
> Tony

Re: [VOTE] Establish Apache MiNiFi, a subproject of Apache NIFi

[Sean Busbey]

+1

On Sat, Apr 9, 2016 at 12:13 AM, Joe Witt  wrote:
> Team,
>
> Following a solid discussion [1] in early January regarding
> establishment of MiNiFi as a subproject of NiFi I'd like to call a
> formal vote to record this important community decision and
> established consensus.
>
> Through lazy consensus [2] the repositories were setup for source,
> JIRA, and git-hub mirror [3].  There has since been solid progress and
> discussion on a number of MINIFI JIRAs.  The core goals of MiNiFi have
> been documented on Confluence/Wiki [4].  It was also reported in our
> April 2016 board report that we have established MiNiFi based on these
> things.  But, lets make sure we get this right.
>
> [1] https://s.apache.org/czuz
> [2] https://s.apache.org/Ucxs
> [3] https://cwiki.apache.org/confluence/display/NIFI/Supporting+Infrastructure
> [4] https://cwiki.apache.org/confluence/display/NIFI/MiNiFi
>
> I am a +1 and am very excited by the progress made thus far!
>
> The vote will be open for 72 hours and be a majority rule vote.
>
> [ ] +1 Establish Apache MiNiFi, a subproject of Apache NiFi
> [ ]   0 Do not care
> [ ]  -1 Do not establish Apache MiNiFi
>
> Thanks
> Joe

Re: NiFi Installation : Subscribtion required

[Sean Busbey]

For help with vendor specific questions, please use the appropriate
vendor support channel.

FYI, the Hortonworks community support track that covers NiFi is:

https://community.hortonworks.com/spaces/63/data-flow-track.html

On Tue, Apr 5, 2016 at 4:25 AM, Aankita Kaur  wrote:
>  I am unable to install NiFi in Hortonworks 2.3.2 environment.Currently, I
> tried https://github.com/abajwa-hw/ambari-nifi-service tutorial but have
> not yet started with its services.Also since its half way installed so I
> dont know how to remove its services first & start with new installationin
> hortonworks environment.Please help
> --
> *Warm Regards*
>
> *Dr. AANKITA KAUR*



-- 
busbey

Re: [DISCUSS] git branching model

[Sean Busbey]

On Mon, Apr 4, 2016 at 4:40 PM, Adam Lamar <adamond...@gmail.com> wrote:
> On Mon, Apr 4, 2016 at 11:14 AM, Sean Busbey <bus...@apache.org> wrote:
>> You're correct, a github PR only targets a single branch and Travis-CI
>> only checks how the PR does at its own commit hash (that is, it
>> doesn't even check what the target branch would look like post-merge).
>>
>
> Hey Sean, it looks like Travis-CI does merge into the specified merge
> branch before running. For example, my ListS3 PR [1] has commit
> 2f7e89e, but the Travis-CI status page [1] shows commit e0868c2. If
> you find that in github [2] you'll see it is the merge of 2f7e89e and
> 6f5fb59, where 6f5fb59 is the location of master when I submitted the
> PR.
>

Excellent. I am very happy to find that I'm incorrect. I must just be
used to using Travis-CI with very old PRs. :)

Re: [DISCUSS] git branching model

[Sean Busbey]

On Mon, Apr 4, 2016 at 1:39 PM, Tony Kurc  wrote:
> Sean, how about yetus? Is this something yetus can do with the patch
> testing in an automated way?

Right now Yetus only checks a single target branch.

For commits that can apply/merge cleanly we could add something to
check multiple branches. I'd expect this would be a non-trivial
feature addition, esp since personalities would have to pick
strategies (cherry pick to list of branches, vs attempt
merge-forward). We could start with the former since it's probably
simpler.

Mind filing a JIRA so we don't lose track of the feature request?

If anyone in the NiFi community would like to grow how much they
understand bash and implementer would be a nice bonus. :)

Re: [DISCUSS] git branching model

[Sean Busbey]

You're correct, a github PR only targets a single branch and Travis-CI
only checks how the PR does at its own commit hash (that is, it
doesn't even check what the target branch would look like post-merge).

On Mon, Apr 4, 2016 at 12:05 PM, Matt Gilman  wrote:
> To my knowledge, when you open a PR in GitHub it only allows the
> contributor to select a single branch. The reviewer would need to check the
> compatibility if that contribution needs to be applied to both. Depending
> on the contribution, a separate PR may be required as the codebases
> diverge. In practice though, most of the work on master (1.x) will focus on
> the framework which is where diverging is most likely to occur.
> Consequently, framework bug fixes are the most likely place where we may
> need separate PRs.
>
> On Mon, Apr 4, 2016 at 12:46 PM, Tony Kurc  wrote:
>
>> Is travis ci / githib able to give immediate feedback if a PR doesn't merge
>> into both?
>>
>> On Mon, Apr 4, 2016 at 12:33 PM, Matt Gilman 
>> wrote:
>>
>> > All,
>> >
>> > I have completed the branching discussed last week.
>> >
>> > - master contains the current 1.x baseline - Future 1.x releases will
>> start
>> > from here
>> > - 0.x contains the 0.x baseline - Future 0.x releases will start from
>> here
>> >
>> > Going forward all PRs will need to be merged the either or both branches
>> as
>> > appropriate to ensure it's included in subsequent releases. I will be
>> > updating the quick start and contributor guide to describe the
>> distinction
>> > between the two branches.
>> >
>> > Thanks!
>> >
>> > Matt
>> >
>> > On Thu, Mar 31, 2016 at 4:35 PM, Matt Gilman 
>> > wrote:
>> >
>> > > The majority consensus is to have master point to our 1.x baseline
>> going
>> > > forward. Unless there are any strong objections I will set everything
>> up
>> > on
>> > > Monday (4/4) morning.
>> > >
>> > > - Create a 0.x branch for all future 0.x releases based on the current
>> > > state of master.
>> > > - Apply all 1.x commits from the temporary 1.x branch to master.
>> > > - Delete the temporary 1.x branch.
>> > > - Update the quickstart page and contribution guide to detail the
>> > > distinction between the 0.x and master branches.
>> > > - Send another email to @dev once this has been completed.
>> > >
>> > > Reminder: Going forward once this has been completed all commits will
>> > need
>> > > to be made to both branches as appropriate.
>> > >
>> > > Thanks!
>> > >
>> > > Matt
>> > >
>> > > On Tue, Mar 29, 2016 at 3:05 PM, Matt Gilman 
>> > > wrote:
>> > >
>> > >> Matt,
>> > >>
>> > >> I agree that the PRs would need to be merged to both baselines at
>> > >> contribution time. If the contribution applies cleanly the reviewer
>> > could
>> > >> certainly handle the commit themselves. However, if additional code
>> > changes
>> > >> are required because the baselines have diverged, the contributor
>> would
>> > >> probably need to submit another PR. This additional effort should only
>> > be
>> > >> necessary until we're able to perform the first 1.x release.
>> > >>
>> > >> Aldrin,
>> > >>
>> > >> I definitely understand your thoughts regarding (1) and (2). This is
>> why
>> > >> I wanted to pose the options before just jumping into one approach vs
>> > the
>> > >> other. I personally prefer the GitHub style PR process. I realize this
>> > is
>> > >> more cumbersome but hopefully the number of conflicts should be small
>> as
>> > >> folks are already starting to focus their efforts on the framework for
>> > 1.x.
>> > >>
>> > >> Matt
>> > >>
>> > >> On Tue, Mar 29, 2016 at 10:55 AM, Aldrin Piri 
>> > >> wrote:
>> > >>
>> > >>> I think I prefer option 2 considering, what may be the incorrect
>> > >>> assumption, that rebasing 1.x on 0.x / pushing into 1.x would be
>> > easier.
>> > >>> Based on outstanding PRs/Patches in conjunction with release cadence
>> > >>> there
>> > >>> will be more 0.x releases planned. Until we reached the point where
>> the
>> > >>> first 1.x release is in sight, I think (2) makes sense just from
>> > >>> minimizing
>> > >>> impedance where the majority of effort will occur (new/updated
>> > >>> extensions)
>> > >>> and then switching to (1) when we are scheduling 1.x as next
>> (exclusive
>> > >>> of
>> > >>> any patch builds).  This seems to work out when I try to reason about
>> > it,
>> > >>> but admittedly, am coming at this heavily from my own anecdotal
>> > >>> perspective
>> > >>> given my flow of reviewing.
>> > >>>
>> > >>> Matt, excellent points to consider.
>> > >>>
>> > >>> Do not want to go too much on a tangent from the current
>> conversation,
>> > >>> but
>> > >>> I think we need to harness automation as much as possible.  Not sure
>> > >>> Travis
>> > >>> can do this or do so easily (short of two PRs) and this may arguably
>> > >>> shift
>> > >>> things in favor of patches and the model that 

Re: Zookeeper issues mentioned in a talk about storm / heron

[Sean Busbey]

HBase has also had issues with ZK at modest (~couple hundred node)
scale when using it to act as an intermediary for heartbeats and to do
work assignment.

On Thu, Mar 31, 2016 at 4:33 PM, Tony Kurc  wrote:
> My commentary that didn't accompany this - it appears Storm was using
> zookeeper in a similar way as the road we're heading down, and it was such
> a major bottleneck that they moved key value storage and heartbeating out
> into separate services, and then re-engineering (i.e. built Heron). Before
> we get too dependent on zookeeper, may be worth learning some lessons from
> the crew that built Heron or from a team that learned zookeeper lessons
> scale like accumulo.
>
> On Thu, Mar 24, 2016 at 6:22 PM, Tony Kurc  wrote:
>
>> I mentioned slides I saw at the meetup about zookeeper perils at scale in
>> storm, here are slides, i couldn't find a video after some limited
>> searching.
>> https://qconsf.com/system/files/presentation-slides/heron-qcon-2015.pdf
>>

Re: EncryptContent issues after NIFI-1257 and NIFI-1259

[Sean Busbey]

In the mean time can we call this out in the release notes as a known
issue so that folks using things as Alan was know about it before htey
upgrade?

On Tue, Mar 29, 2016 at 12:58 PM, Andy LoPresto
 wrote:
> Thanks Alan. I don’t anticipate it being a large effort. I have it marked as
> minor and will bump it if resources are strained.
>
> Andy LoPresto
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 29, 2016, at 10:32 AM, Alan Jackoway  wrote:
>
> Honestly, it's not clear to me that we should handle this scenario. The
> only reason I would propose fixing it is to handle people (like me) who did
> it wrong and then upgraded. Requiring a keyring isn't that unusual, and the
> docs are pretty specific. I just didn't read them.
>
> Alan
>
> On Tue, Mar 29, 2016 at 1:28 PM, Andy LoPresto 
> wrote:
>
> Alan,
>
> The processor properties for public keyring file and secret keyring file
> are fairly explicit in their names, so when I upgraded the BouncyCastle
> dependencies, I wrote logic that performs strict validation on the file
> format because the underlying library code changed substantially. I was
> unaware anyone was using the individual key file there.
>
> I have created a Jira [1] for 0.7.0 to add custom logic to handle this
> scenario.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1694
>
> Andy LoPresto
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 29, 2016, at 8:03 AM, Alan Jackoway  wrote:
>
> I don't get a stacktrace. Probably because it is a validation failure and
> the error is caught at
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/EncryptContent.java#L288
>
> I couldn't get your template to work without the gpgkeyring file. However,
> that clued me into what I believe is the problem.
>
> I have not been using a public keyring file, but rather the public key
> itself. Somehow that used to work, but the parameter has always been called
> Public Keyring File so I was using it wrong the whole time.
>
> I attached the encrypt template that is working for me back in 0.3.0 (and
> should work in 0.4.1 but not 0.5.1)
>
> To fix it for 0.5.1, I had to make a real keyring file AND change the user
> id to be the right thing.
>
> This feels like a regression to me, but one where I was not following the
> instructions all along.
>
> Thanks,
> Alan
>
> On Tue, Mar 29, 2016 at 1:15 AM, Andy LoPresto 
> wrote:
>
>
> The only other thing I can think of off the top of my head is that the
> userID specification may have changed with the BouncyCastle upgrade and the
> provided userID of just an email may be incomplete? In my testing, I had to
> specify the "name", "description", and "email" fields from the key in the
> format below in order to match the exact format that the library reads from
> the keyring.
>
> userID = "Name (Description) "
>
> You can test this and evaluate what the library sees as the key userID by
> attaching a remote debugger to your running instance and evaluating inside
> the iterator loop here [1].
>
> I'm not sure what version of GPG you're running, but it is worth
> investigating if the format of the stored key no longer matches how NiFi
> was reading it.
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/crypto/OpenPGPKeyBasedEncryptor.java#L200
>
>
>
> Andy LoPresto
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 28, 2016, at 18:24, Andy LoPresto 
>
> wrote:
>
>
> Forgot to mention you’ll want to change the input/output directories in
>
> the GetFile and PutFile processors, as well as the paths to the public and
> secret keyring, the user ID, and the password for the EncryptContent
> processors.
>
>
> Andy LoPresto
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 28, 2016, at 4:04 PM, Andy LoPresto 
>
> wrote:
>
>
> Hi Alan,
>
> I am investigating this issue (spinning up an instance, setting up a
>
> flow that involves PGP encryption and decryption, etc.) to verify.
>
>
> As an aside, the setting for “Key Derivation Function” is irrelevant
>
> if “Encryption Algorithm” is set to “PGP” or “PGP_ASCII_ARMOR”. The KDF is
> required for symmetric encryption (deriving a key from the provided
> password), but not used for PGP encryption/decryption at all.
> Unfortunately, we cannot currently display/hide or change the required-ness
> of processor properties based on the value of other properties. There is an
> existing Jira open [1] to 

Re: quick reminder on new nars

[Sean Busbey]

we could make a parent pom for all the nar modules.

wanna see what that looks like?

On Tue, Mar 15, 2016 at 8:46 PM, Joe Witt  wrote:
> Team,
>
> During the previous build/release cycle it was found that
> javadocs/sources were being made for the Nar bundles themselves and
> was causing invalid licensing/notice information to be present.  All
> the existing bundles and the archetypes were fixed for this.  Just be
> sure on new nars to include these as well if you aren't copying from
> something existing or using the archetype.  I just fixed a couple of
> them for new things in the 0.6.0 release.
>
> The nar pom itself should have a properties section such as
>
> 
> true
> true
> 
>
> Perhaps there is a nicer maven way of ensuring this doesn't happen for Nars.
>
> Thanks
> Joe

Re: [VOTE] Release Apache NiFi 0.5.0 (RC3)

[Sean Busbey]

-1 (binding)

bad:

* several -source and -javadoc jars contain improper LICENSE/NOTICE files
(they cover things "in the binary artifact" that are not in the artifact
containing the LICENSE/NOTICE file)

ex:

* -source and -javadoc for war packaging
* -source for nar packaging

questionable:

* Our LICENSE/NOTICE for war packaging is in "WEB-INF/classes/META-INF/".
it's not clear to me if this is the correct location, but I can't find
anything one way or the other. My intuition says it should be META-INF at
the top level because the wars have that folder for other things.

good:

* sigs + checksums match
* source tag matches source tarball
* source builds binaries

On Fri, Feb 12, 2016 at 5:27 PM, Tony Kurc  wrote:

> Hello
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-0.5.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1073
>
>
> The Git tag is nifi-0.5.0-RC3
> The Git commit ID is 8309dba80bbc0e6c0dfe5b8f93750d424f5b9a90
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=8309dba80bbc0e6c0dfe5b8f93750d424f5b9a90
>
> This release candidate is a branch off of master at
> cae5b109c09be80df77f2d767220b82850178ed5.
>
> Checksums of nifi-0.5.0-source-release.zip:
> MD5: 5d20a5bb2fcdb3d96b2c62949db1c780
> SHA1: e1c541d27373b94a58c8ede4eae7f6a7eb51c5a0
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/tkurc.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/dev/nifi/KEYS
>
> 106 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12334158
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.0
>
> Migration guidance can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>
> The vote will be open for 96 hours (24 extra hours due to late Friday
> release candidate)
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.  The
> please vote:
>
> [ ] +1 Release this package as nifi-0.5.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...
>



-- 
Sean

ApacheCon presence?

[Sean Busbey]

Hi folks!

CFPs for ApacheCon NA Core and ApacheCon NA Big Data close in a few
days, on 12 Feb.

Anyone already planning to attend? Anyone already submit a proposal?

I'd really like to see NiFi have a good showing.

-Sean

Re: Javadoc

[Sean Busbey]

Hi Devin!

Sorry for the gap. We're tracking the need for a web accessible copy
of our API javadocs in NIFI-943. I don't believe anyone has taken up
the task yet, so if you're looking to get more involved in the project
we'd love the help!

On Sun, Jan 17, 2016 at 1:38 PM, Devin Fisher
 wrote:
> Might be a silly question but I can't find a link the Javadocs anywhere on
> your website. I've googled for it and only find the reference to the
> Javadocs in the NiFi Developer’s Guide
> . I'm hopeful that I'm just
> not seeing it. If not, what is the recommended way to get a hold of it?
>
> Also, thanks in advanced.
>
> Devin

Re: [DISCUSS] roadmap for the next 6-12 months

[Sean Busbey]

On Fri, Jan 8, 2016 at 8:29 AM, Joe Witt  wrote:
> We should also have a discussion on how long we should be committed to
> supporting the 0.x line and what that means.  We need to document a
> commitment for the community.


Worth a dedicated thread?

Presuming that we're going to use "1.0.0" as the start of a stricter
versioning policy, I think the first issue is deciding how long we'll
support major versions once that happens. If the expectation is that
"pre-1.0" versions involve more latitude as we work out what NiFi
needs to better find a fit for the public good, then I think it's
reasonable for our continuing support period to be less than a
'normal' major version. But we have to know what that looks like
first.

-- 
Sean

Re: Syslog Classes

[Sean Busbey]

The interface audience and stability annotations are a great idea to
use. Please use the publicly-consumable ones in Apache Yetus instead
of those in Hadoop. :)

http://yetus.apache.org/documentation/0.1.0/audience-annotations-apidocs/

On Tue, Jan 5, 2016 at 7:34 PM, Matthew Burgess  wrote:
> That’s my bad, and I can’t blame bourbon this time :)
>
> Hadoop has an annotation class for InterfaceStability [1]. It is used to 
> annotate interfaces with a “contract” about whether they are likely to change 
> or not (example [2]). They use values like Stable, Unstable, and Evolving, 
> explained in javadoc [3].  I thought maybe this was the kind of thing you 
> were referring to when you mentioned annotating NiFi classes with a sort of 
> contract about their potential volatility?
>
> Regards,
> Matt
>
> [1] 
> https://hadoop.apache.org/docs/r2.7.0/api/org/apache/hadoop/classification/InterfaceStability.html
> [2] 
> https://hadoop.apache.org/docs/r2.7.0/api/org/apache/hadoop/yarn/api/records/ContainerReport.html
> [3] 
> https://hadoop.apache.org/docs/r2.7.0/api/src-html/org/apache/hadoop/classification/InterfaceStability.html#line.42
>
>
>
>
>
> On 1/5/16, 7:43 PM, "Tony Kurc"  wrote:
>
>>I think I parsed your sentence differently than you intended. Was your
>>"this" in your opening sentence "what Tony described" or "what Matt is
>>going describe"?
>>On Jan 5, 2016 7:35 PM, "Matt Burgess"  wrote:
>>
>>> Roger that. This is what Hadoop does, for an API method (class, etc.) in
>>> Java it is annotated as @Stable or @Unstable. I was just referring to the
>>> semantics of when you might expect an @Unstable method to change, for
>>> example. Or am I still misunderstanding what you mean?
>>>
>>> Regards,
>>> Matt
>>>
>>> Sent from my iPhone
>>>
>>> > On Jan 5, 2016, at 7:29 PM, Tony Kurc  wrote:
>>> >
>>> > Matt,
>>> > What I'm talking about is annotating individual fields,  methods, and
>>> > classes, giving some contract other than the access modifiers of java.
>>>
>



-- 
Sean

Re: Redesign User Interface (UI)

[Sean Busbey]

+1 to "UI redesign warrants a major version increment"

I know that we're "pre 1.0", but this sounds like it's time to figure out
what we need to include to go over that cliff.

On Wed, Jan 6, 2016 at 12:22 PM, Tony Kurc  wrote:

> Looks great so far!
>
> I saw the target release of 0.6, which surprised me a bit. I would have
> expected this significant of a change would warrant a major release
> increment.
>
> On Wed, Jan 6, 2016 at 1:08 PM, Rob Moran  wrote:
>
> > Greetings ​​NiFi community,
> >
> > NIFI-1323 [1] has been mentioned in (at least one) previous thread. In
> case
> > you are unaware, it involves beginning work on a series of UI
> improvements.
> >
> > I'd like to point your attention to the wiki page [2] where you can read
> up
> > on factors that are driving this effort.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-1323
> > [2]
> > https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface
> >
> > Thanks,
> > Rob
> >
>



-- 
Sean

Re: [DRAFT][REPORT] Apache NiFi - January 2016

[Sean Busbey]

On Sun, Jan 3, 2016 at 12:04 PM, Joe Witt <joew...@apache.org> wrote:
>  - New commmitters:
> - Ricky Saltzer was added as a committer on Wed Oct 21 2015
> - Toivo Adams was added as a committer on Tue Nov 03 2015
> - Sean Busbey was added as a committer on Wed Nov 25 2015
> - Michael W Moser was added as a committer on Sun Oct 18 2015
> - Joe Percivall was added as a committer on Mon Dec 14 2015


could we sort these by descending date (as we sort the releases)

Re: Testing handling of static class methods

[Sean Busbey]

You could use PowerMock to either call private static methods directly
or to mock them out.

https://github.com/jayway/powermock/wiki/MockStatic

https://github.com/jayway/powermock/wiki/MockPrivate

https://github.com/jayway/powermock/wiki/BypassEncapsulation#invoking-a-private-method

On Fri, Dec 18, 2015 at 4:42 PM, Joe Skora  wrote:
> Wrapping createMessageProducer() in an instance method is a good
> suggestion, but it seems overkill just to enable testing.  Prompted by
> Oleg's suggestion, I got around instance variable visibility with
> Reflection, which is nice because it doesn't require "private" be changed
> to "protected" in the class under test and doesn't require an inner test
> class.  But, that doesn't appear to be possible for static methods, so
> wrapping with class methods may be the only choice.  Hopefully, I've missed
> something.
>
>
> On Fri, Dec 18, 2015 at 3:58 PM, Bryan Bende  wrote:
>
>> If you get it into a protected instance method, you can also make an inner
>> class in your test, something like TestablePutJMS extends PutJMS, and
>> overrides that method to return a mock or whatever you want. That is a
>> common pattern in a lot of the processor tests.
>>
>> On Fri, Dec 18, 2015 at 3:44 PM, Matt Burgess  wrote:
>>
>> > You could move the one static call into an instance method of PutJMS, and
>> > use Mockito.spy() to get a partial mock of the processor, then use when()
>> > to override the instance method in the test. Not sure if that's how it's
>> > done in other places but it's worked for me in the past.
>> >
>> > Regards,
>> > Matt
>> >
>> > Sent from my iPhone
>> >
>> > > On Dec 18, 2015, at 3:20 PM, Joe Skora  wrote:
>> > >
>> > > For unit testing, one problem I've run into is overriding the returns
>> > from
>> > > static class methods.
>> > >
>> > > For instance, PutJMS contains this code:
>> > >
>> > > try {
>> > >>wrappedProducer = JmsFactory.createMessageProducer(context, true);
>> > >>logger.info("Connected to JMS server {}",
>> > >>new Object[]{context.getProperty(URL).getValue()});
>> > >> } catch (final JMSException e) {
>> > >>logger.error("Failed to connect to JMS Server due to {}", new
>> > >> Object[]{e});
>> > >>session.transfer(flowFiles, REL_FAILURE);
>> > >>context.yield();
>> > >>return;
>> > >> }
>> > >
>> > > where JmsFactory.createmessageProducer call being defined as
>> > >
>> > > public static WrappedMessageProducer createMessageProducer(...
>> > >
>> > > which presents a problem since it can't be easily overridden for a unit
>> > > test.  Exercising the
>> > >
>> > > How you handle this problem?
>> > >
>> > > Regards,
>> > > Joe
>> >
>>

Re: discuss nifi 0.4.1

[Sean Busbey]

with the Kafka client change backed out for 0.5.0, I'm good to go with
0.4.1 on the rest of the changes.

On Fri, Dec 18, 2015 at 10:11 PM, Joe Witt <joe.w...@gmail.com> wrote:
> Sean,
>
> Yeah i don't disagree with that point.  The caveat being it was only a
> change to that client not a change to support the new client API and
> the behavior with existing clients old and new verified.
>
> I'd prefer to stick with 0.4.1 and if you still think it is best to
> actually just revert that commit and apply it toward 0.5.0.
>
> What do you think?
>
> Thanks
> Joe
>
> On Fri, Dec 18, 2015 at 11:08 PM, Sean Busbey <bus...@apache.org> wrote:
>> Can we update to 0.5.0 instead? The kafka client change isn't
>> something I'd expect in a patch release.
>>
>> On Fri, Dec 18, 2015 at 9:54 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>> ok - so master is presently on 041 and it does indeed appear to be all
>>> incremental friendly fixes.  So looks like we can just use the normal
>>> process.  As excited as I was to use cherry-pick doesn't look like it
>>> is needed.
>>>
>>> The bugs fixed on 041 so far are all nice cleanup items and things
>>> which have been problematic for quite a while.  However, there are a
>>> few site-to-site issues that would create some pretty annoying issues
>>> for users so best to eliminate them.  And big thanks to Matt Clarke
>>> for finding and reporting them!
>>>
>>> Gonna prep an RC.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 17, 2015 at 7:53 PM, Tony Kurc <trk...@gmail.com> wrote:
>>>> I have no objection to "because we should be able to do this well!" as a
>>>> reason.
>>>>
>>>> On Thu, Dec 17, 2015 at 7:45 PM, Oleg Zhurakousky <
>>>> ozhurakou...@hortonworks.com> wrote:
>>>>
>>>>> Generally RCs are used to address that level of validation, so in the end
>>>>> I still think it's a more of a culture one chooses. One common  example;
>>>>> x.x.1+ = maintenance, x.1+.0 = minor features + bugs and 1+.0.0 = major
>>>>> features.
>>>>>
>>>>> In any event IMHO the ability to quickly release maintenance releases is
>>>>> very important  as it showcases our attention to quality
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> > On Dec 17, 2015, at 19:32, Tony Kurc <trk...@gmail.com> wrote:
>>>>> >
>>>>> > I'm not sure I understand "more validation" reasoning - won't features 
>>>>> > at
>>>>> > the end have very little validation?
>>>>> >
>>>>> >> On Thu, Dec 17, 2015 at 7:26 PM, Ryan Blue <b...@cloudera.com> wrote:
>>>>> >>
>>>>> >> Another reason to release 0.4.1 is to allow the additions that warrant
>>>>> >> 0.5.0 to have more validation before release. With a regular release
>>>>> cycle,
>>>>> >> features can go in at the beginning to have more time for catching bugs
>>>>> in
>>>>> >> them. I also agree with what Sean said below.
>>>>> >>
>>>>> >> rb
>>>>> >>
>>>>> >>> On 12/17/2015 04:00 PM, Sean Busbey wrote:
>>>>> >>>
>>>>> >>> On Thu, Dec 17, 2015 at 5:50 PM, Tony Kurc <trk...@gmail.com> wrote:
>>>>> >>>
>>>>> >>> s/features/buxfixes/
>>>>> >>>>
>>>>> >>>> On Thu, Dec 17, 2015 at 6:50 PM, Tony Kurc <trk...@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> Is there a reason to not just cut a 0.5.0 instead of grafting 0.5.0
>>>>> >>>>> features onto 0.4.1?
>>>>> >>> This is a good question.
>>>>> >>>
>>>>> >>> Some downstream users might have different change processes for a
>>>>> patch vs
>>>>> >>> minor release, so making a 0.4.1 that fixes what we determine to be a
>>>>> >>> substantial gap in the 0.4 line would be nice for them.
>>>>> >>>
>>>>> >>> While we might be a young project now, it would be good to already 
>>>>> >>> have
>>>>> >>> the
>>>>> >>> habit practiced for when we have more users in enterprise settings.
>>>>> >>>
>>>>> >>> On the other hand, 0.4.0 just happened, so a release in 3 days would
>>>>> >>> minimize the number of "stuck on 0.4.0" folks.
>>>>> >>
>>>>> >> --
>>>>> >> Ryan Blue
>>>>> >> Software Engineer
>>>>> >> Cloudera, Inc.
>>>>> >>
>>>>>



-- 
Sean

Re: discuss nifi 0.4.1

[Sean Busbey]

Can we update to 0.5.0 instead? The kafka client change isn't
something I'd expect in a patch release.

On Fri, Dec 18, 2015 at 9:54 PM, Joe Witt <joe.w...@gmail.com> wrote:
> ok - so master is presently on 041 and it does indeed appear to be all
> incremental friendly fixes.  So looks like we can just use the normal
> process.  As excited as I was to use cherry-pick doesn't look like it
> is needed.
>
> The bugs fixed on 041 so far are all nice cleanup items and things
> which have been problematic for quite a while.  However, there are a
> few site-to-site issues that would create some pretty annoying issues
> for users so best to eliminate them.  And big thanks to Matt Clarke
> for finding and reporting them!
>
> Gonna prep an RC.
>
> Thanks
> Joe
>
> On Thu, Dec 17, 2015 at 7:53 PM, Tony Kurc <trk...@gmail.com> wrote:
>> I have no objection to "because we should be able to do this well!" as a
>> reason.
>>
>> On Thu, Dec 17, 2015 at 7:45 PM, Oleg Zhurakousky <
>> ozhurakou...@hortonworks.com> wrote:
>>
>>> Generally RCs are used to address that level of validation, so in the end
>>> I still think it's a more of a culture one chooses. One common  example;
>>> x.x.1+ = maintenance, x.1+.0 = minor features + bugs and 1+.0.0 = major
>>> features.
>>>
>>> In any event IMHO the ability to quickly release maintenance releases is
>>> very important  as it showcases our attention to quality
>>>
>>> Sent from my iPhone
>>>
>>> > On Dec 17, 2015, at 19:32, Tony Kurc <trk...@gmail.com> wrote:
>>> >
>>> > I'm not sure I understand "more validation" reasoning - won't features at
>>> > the end have very little validation?
>>> >
>>> >> On Thu, Dec 17, 2015 at 7:26 PM, Ryan Blue <b...@cloudera.com> wrote:
>>> >>
>>> >> Another reason to release 0.4.1 is to allow the additions that warrant
>>> >> 0.5.0 to have more validation before release. With a regular release
>>> cycle,
>>> >> features can go in at the beginning to have more time for catching bugs
>>> in
>>> >> them. I also agree with what Sean said below.
>>> >>
>>> >> rb
>>> >>
>>> >>> On 12/17/2015 04:00 PM, Sean Busbey wrote:
>>> >>>
>>> >>> On Thu, Dec 17, 2015 at 5:50 PM, Tony Kurc <trk...@gmail.com> wrote:
>>> >>>
>>> >>> s/features/buxfixes/
>>> >>>>
>>> >>>> On Thu, Dec 17, 2015 at 6:50 PM, Tony Kurc <trk...@gmail.com> wrote:
>>> >>>>
>>> >>>> Is there a reason to not just cut a 0.5.0 instead of grafting 0.5.0
>>> >>>>> features onto 0.4.1?
>>> >>> This is a good question.
>>> >>>
>>> >>> Some downstream users might have different change processes for a
>>> patch vs
>>> >>> minor release, so making a 0.4.1 that fixes what we determine to be a
>>> >>> substantial gap in the 0.4 line would be nice for them.
>>> >>>
>>> >>> While we might be a young project now, it would be good to already have
>>> >>> the
>>> >>> habit practiced for when we have more users in enterprise settings.
>>> >>>
>>> >>> On the other hand, 0.4.0 just happened, so a release in 3 days would
>>> >>> minimize the number of "stuck on 0.4.0" folks.
>>> >>
>>> >> --
>>> >> Ryan Blue
>>> >> Software Engineer
>>> >> Cloudera, Inc.
>>> >>
>>>

Re: discuss nifi 0.4.1

[Sean Busbey]

Sounds reasonable to me. branching off of the last release tag and then
cherry picking a conservative subset of fixes for a patch release has
worked well for me on another project.

It's implied in your email, but just to confirm, you're only suggesting
grabbing *some* of the currently-in-0.5.0 issues right? specifically just
those that you added a 0.4.1 fixversion to?


On Thu, Dec 17, 2015 at 2:29 PM, Joe Witt  wrote:

> team,
>
> matt clarke just discovered an interesting case that appears to expose
> a defect in site-to-site.  The details of it are still being worked
> out as you can see in NIFI-1301.  And this issue has been around for a
> very long time but it still feels like something worth addressing in
> an incremental/bug release (0.4.1).
>
> I looked at already addressed bugs on 050 and added the to fix
> versions of 041 as well.  What I am wondering here is a bit of a
> proper usage and thinking with Git.  Would it make sense to branch off
> master right where 0.4.1-SNAPSHOT started, then cherry pick the
> commits into this new branch, and just release that branch never
> needing then to merge that back to master since these fixes are all
> already on master anyway?
>
>
> https://issues.apache.org/jira/browse/NIFI-1301?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.4.1%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> Thanks
> Joe
>



-- 
Sean

Re: discuss nifi 0.4.1

[Sean Busbey]

I wouldn't want to see the kafka client upgrade in a patch release.

On Thu, Dec 17, 2015 at 5:37 PM, Joe Witt  wrote:

> OK thanks for confirming proper git fu.
>
> Yeah I was meaning to just grab bugs.  The master branch already has stuff
> that seems to warrant a minor bump (maybe) so wanted to understand a bug
> only route.
>
> Matt understand your point on dependent commits.  Will check that out.
>
> Thanks
> Joe
> On Dec 17, 2015 6:32 PM, "Matt Gilman"  wrote:
>
> > I see, that does appear to be the case. What your suggesting sounds good.
> > Though we should review the tickets that addressed UI bugs/improvements.
> I
> > realize you probably specifically just chose the JIRAs that addressed
> bugs.
> > I'd want to make sure that the tickets included don't have a dependency
> on
> > the tickets excluded. For instance, merge conflicts because a commit in
> an
> > included ticket is based off the code base after a commit in an excluded
> > ticket.
> >
> > Matt
> >
> > On Thu, Dec 17, 2015 at 6:17 PM, Matt Gilman 
> > wrote:
> >
> > > Are there some commits on master that we don't want included in 0.4.1?
> If
> > > not, wouldn't we just follow our normal release process?
> > >
> > > Matt
> > >
> > >
> > > On Thu, Dec 17, 2015 at 6:15 PM, Ryan Blue  wrote:
> > >
> > >> Branching from master at the start of 0.4.1-SNAPSHOT and
> cherry-picking
> > >> makes sense to me.
> > >>
> > >> rb
> > >>
> > >>
> > >> On 12/17/2015 12:29 PM, Joe Witt wrote:
> > >>
> > >>> team,
> > >>>
> > >>> matt clarke just discovered an interesting case that appears to
> expose
> > >>> a defect in site-to-site.  The details of it are still being worked
> > >>> out as you can see in NIFI-1301.  And this issue has been around for
> a
> > >>> very long time but it still feels like something worth addressing in
> > >>> an incremental/bug release (0.4.1).
> > >>>
> > >>> I looked at already addressed bugs on 050 and added the to fix
> > >>> versions of 041 as well.  What I am wondering here is a bit of a
> > >>> proper usage and thinking with Git.  Would it make sense to branch
> off
> > >>> master right where 0.4.1-SNAPSHOT started, then cherry pick the
> > >>> commits into this new branch, and just release that branch never
> > >>> needing then to merge that back to master since these fixes are all
> > >>> already on master anyway?
> > >>>
> > >>>
> > >>>
> >
> https://issues.apache.org/jira/browse/NIFI-1301?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.4.1%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> > >>>
> > >>> Thanks
> > >>> Joe
> > >>>
> > >>>
> > >>
> > >> --
> > >> Ryan Blue
> > >> Software Engineer
> > >> Cloudera, Inc.
> > >>
> > >
> > >
> >
>



-- 
Sean

Re: discuss nifi 0.4.1

[Sean Busbey]

On Thu, Dec 17, 2015 at 5:50 PM, Tony Kurc  wrote:

> s/features/buxfixes/
>
> On Thu, Dec 17, 2015 at 6:50 PM, Tony Kurc  wrote:
>
> > Is there a reason to not just cut a 0.5.0 instead of grafting 0.5.0
> > features onto 0.4.1?
> >
>

This is a good question.

Some downstream users might have different change processes for a patch vs
minor release, so making a 0.4.1 that fixes what we determine to be a
substantial gap in the 0.4 line would be nice for them.

While we might be a young project now, it would be good to already have the
habit practiced for when we have more users in enterprise settings.

On the other hand, 0.4.0 just happened, so a release in 3 days would
minimize the number of "stuck on 0.4.0" folks.

-- 
Sean

Re: discuss nifi 0.4.1

[Sean Busbey]

On Thu, Dec 17, 2015 at 6:32 PM, Tony Kurc  wrote:

> I'm not sure I understand "more validation" reasoning - won't features at
> the end have very little validation?
>
> On Thu, Dec 17, 2015 at 7:26 PM, Ryan Blue  wrote:
>
> > Another reason to release 0.4.1 is to allow the additions that warrant
> > 0.5.0 to have more validation before release. With a regular release
> cycle,
> > features can go in at the beginning to have more time for catching bugs
> in
> > them. I also agree with what Sean said below.
> >
>

I presume Ryan just meant that the things that have gone in now would have
more time.

This is the current history since 0.4.0 branched for RC:

* 04e9606 - (HEAD, origin/master, origin/HEAD, master) NIFI-1290 Document th
* 7c87968 - NIFI-1218 addressed PR comments (29 hours ago)
* 3763523 - NIFI-1218 upgraded Kafka to 0.9.0.0 client API Tested and valida
* b19ff7c - NIFI-1215: - Only showing the run duration setting when applicab
* 51b8ecd - NIFI-1185: - Using banners from the NCM rather than replicating
* c75b5cf - NIFI-1119: - Addressing race condition that caused the revision
* 17be1c2 - NIFI-1206: - Only enabling the enable/disable toolbar icon when
* f9f0443 - NIFI-1119: - Also refreshing flow revision when the user clicks
*   a7b09a5 - NIFI-1122 release vote passess. Merge branch 'NIFI-1122_nifi-0
|\
| * d755e43 - (origin/NIFI-1122_nifi-0.4.0-RC2) NIFI-1122_nifi-0.4.0-RC2prep
| * b66c029 - (nifi-0.4.0-RC2, nifi-0.4.0) NIFI-1122_nifi-0.4.0-RC2prepare r
* | 8070a9f - NIFI-1104: - Using the appropriate attributes based on the con
* | 854c667 - NIFI-1211 Adding a .travis.yml to provide CI and adding an exc
|/
* fb65cf1 - NIFI-1271: Yield funnels and ports for nifi.bored.yield.duration

Nothing here gives me heartburn about 0.5.0 going out on short release
cycle.


-- 
Sean

Re: [ANNOUNCE] New Apache NiFi Committer: Joseph (Joe) Percivall

[Sean Busbey]

Congrats and welcome Joe!

On Mon, Dec 14, 2015 at 2:33 PM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Thanks Tony!
>
> I'm excited to continue working on such a great and innovative project.
>
> Joe
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
>
> On Monday, December 14, 2015 2:28 PM, Tony Kurc  wrote:
>
>
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Joe
> Percivall has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Joe's hard work and
> generous contributions to the project. We look forward to his continued
> involvement in the project.
>
> Joe has been working on some awesome capabilites in NiFi such as the image
> processing bundle, improving the ExecuteStreamCommand, the long overdue
> refactoring of InvokeHTTP, and dealing with the fun of building and
> operating across platforms.
>
> Welcome and congratulations!
>
> Tony
>



-- 
Sean

Re: [VOTE] Release Apache NiFi 0.4.0 (rc2)

[Sean Busbey]

On Fri, Dec 11, 2015 at 8:06 AM, Sean Busbey <bus...@cloudera.com> wrote:

> +1 (binding)
>
> * verified signatures / checksums
> * source and binary artifacts passed check from RAT.
> * verified source artifact against tag
> * verified bin artifacts match each other and one generated from source
> artifact
> * spot checked licenses
> * build passes
> * bin artifact runs simple flow
>
> Ran into NIFI-1278, but I don't think it's sufficient to block the release.
>
>
>
Sorry, was in a hurry when I wrote this up. Should have been

* build passes (after accounting for failures/errors that I think are from
NIFI-1278



-- 
Sean

Re: [VOTE] Release Apache NiFi 0.4.0 (rc2)

[Sean Busbey]

+1 (binding)

* verified signatures / checksums
* source and binary artifacts passed check from RAT.
* verified source artifact against tag
* verified bin artifacts match each other and one generated from source
artifact
* spot checked licenses
* build passes
* bin artifact runs simple flow

Ran into NIFI-1278, but I don't think it's sufficient to block the release.



On Tue, Dec 8, 2015 at 12:29 PM, Joe Witt  wrote:

> Hello NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 0.4.0.
>
> The source zip, including signatures, digests, and associated
> convenience binaries can be found at
>   https://dist.apache.org/repos/dist/dev/nifi/nifi-0.4.0/
>
> The staged maven artifacts of the build can be found at
>   https://repository.apache.org/content/repositories/orgapachenifi-1065
>
> The Git tag is nifi-0.4.0-RC2
> The Git commit ID is b66c029090f395c0cbc001fd483e86895b133e46
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=b66c029090f395c0cbc001fd483e86895b133e46
>
> Checksums of NiFi 0.4.0 Source Release
> MD5: da733f8fdb520a0346dcda59940b2c12
> SHA1: 82fffbc5f8d7e4724bbe2f794bdde39396dae745
>
> Release artifacts are signed with the following key
>   https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here
>   https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 161 issues were closed/resolved for this release
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12333070
>
> Release note highlights
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.4.0
>
> Migration/Upgrade guidance
>   https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>   https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
>
> Then please vote:
>
> [ ] +1 Release this package as Apache NiFi 0.4.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>



-- 
Sean

Re: Release wrangling: 1 week until our hopeful 0.4.0 release

[Sean Busbey]

"Before I kick out Apache NiFi 0.4.0 RC1 I propose we spend a day or so
doing the same sort of items we would have and if it is quiet-ish then
we proceed with putting out an RC.  "

we would have if what?

On Tue, Dec 1, 2015 at 10:19 PM, Joe Witt <joe.w...@gmail.com> wrote:

> Team,
>
> As you can see here [1] we are 'done' as far as 0.4.0 known tickets
> go.  I am ready to create the RC and would love to do so.
>
> Before I kick out Apache NiFi 0.4.0 RC1 I propose we spend a day or so
> doing the same sort of items we would have and if it is quiet-ish then
> we proceed with putting out an RC.  There are a tremendous amount of
> tickets/items involved in this release [2] with a few that are rather
> substantive changes/features such as [3] for example.
>
> [1]
> https://issues.apache.org/jira/browse/NIFI-1237?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.4.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> [2]
> https://issues.apache.org/jira/browse/NIFI-1122?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.4.0%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
>
> [3] https://issues.apache.org/jira/browse/NIFI-655
>
> Thanks
> Joe
>
> On Fri, Nov 20, 2015 at 11:05 AM, Tony Kurc <trk...@gmail.com> wrote:
> > FWIW I submitted at patch for lineendings that I don't believe will
> > conflict with NIFI-655
> > On Nov 20, 2015 8:57 AM, "Joe Witt" <joe.w...@gmail.com> wrote:
> >
> >> Getting closer!
> >>
> >>
> >>
> https://issues.apache.org/jira/browse/NIFI-1122?jql=project%20%3D%20NIFI%20AND%20fixVersion%20%3D%200.4.0%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> >>
> >> Key stuff left:
> >> - Wrap up NIFI-655 uname/pword stuff
> >> - Make the kafka client config a little more flexible
> >> - Make the user experience better for processors requiring input
> >> - Fix up the tail file stuff which we all now appreciate as being
> >> extraordinarily edge case challenged
> >>
> >> Then
> >> - Tony fixes line endings (except for tests which require specific
> endings)
> >> - I will do RM to kick out RC
> >> - We all do focused testing
> >> - We do the vote
> >>
> >> We're a little behind the ideal timeline but these are definitely
> >> things worth waiting for and we're still finding bugs during testing.
> >> Once the popcorn stops popping as much (so to speak) we can zero in on
> >> a release.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Mon, Nov 16, 2015 at 11:47 AM, Joe Percivall
> >> <joeperciv...@yahoo.com.invalid> wrote:
> >> > Per NiFi-1165: a discussion is occurring on the ticket:
> >> https://issues.apache.org/jira/browse/NIFI-1165
> >> >
> >> >
> >> > Overall, most of the issues are identified and pending a fix from
> Mark,
> >> Oleg and I. The issues were encountered on two different windows 8
> machines
> >> by me and on windows 2012 R2 by Mark. My configuration is maven 3.3.3
> and
> >> Java 1.8.0_45 (on the machine I have in front of me).
> >> >
> >> > Should have a patch resolving the issues in the next couple days.
> >> >
> >> > Joe
> >> >
> >> > - - - - - - Joseph Percivall
> >> > linkedin.com/in/Percivall
> >> > e: joeperciv...@yahoo.com
> >> >
> >> >
> >> >
> >> >
> >> > On Monday, November 16, 2015 10:42 AM, Sean Busbey <
> bus...@cloudera.com>
> >> wrote:
> >> >
> >> >
> >> >
> >> > re: NIFI-1165 I also have a windows 7 laptop I can test on. (though
> it is
> >> > low power)
> >> >
> >> >
> >> > On Sun, Nov 15, 2015 at 10:48 AM, Aldrin Piri <aldrinp...@gmail.com>
> >> wrote:
> >> >
> >> >> I have another set of eyes for NIFI-748.  Will do so now.
> >> >>
> >> >> On Sun, Nov 15, 2015 at 10:39 AM, Tony Kurc <trk...@gmail.com>
> wrote:
> >> >>
> >> >> > For those not watching commits@nifi
> >> >> > I need another set of eyes on the review for NIFI-748
> >> >> >
> >> >> > On Sun, Nov 15, 2015 at 8:33 AM, Joe Witt <joe.w...@gmail.com>
> wrote:
> >> >> >
> >> >> > > NIFI-1082 (this should move to next 

Re: Nifi processor last execution time

[Sean Busbey]

Sam,

For now I would store this information in HDFS. Alternatively, if you have
it ZooKeeper would also work.

On Thu, Nov 12, 2015 at 10:45 PM, Sam Kamau  wrote:

> I am working on a processor that loads data to HDFS for an RDBMS using
> ExecuteSQL processor. The data has a last update time that I would like to
> use to load records that have changed since the last load.
>
> How best can I go about saving the last execution time so I can use it as a
> filter in the query.
>
> Thanks
>



-- 
Sean

Re: Release wrangling: 1 week until our hopeful 0.4.0 release

[Sean Busbey]

Has anyone had a chance to do a pass through Feature Proposals to move out
any that aren't going to make 0.4.0?

https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals

On Thu, Nov 12, 2015 at 8:13 AM, Tony Kurc  wrote:

> https://issues.apache.org/jira/browse/NIFI-61 - awaiting an answer before
> patch can be completed
> https://issues.apache.org/jira/browse/NIFI-655 - Based on feature branch
> activity, is close?
> https://issues.apache.org/jira/browse/NIFI-696 - awaiting a patch marking
> method as deprecated (assigned to me, but if someone else wants to take it
> and I review, thats cool too)
> https://issues.apache.org/jira/browse/NIFI-812 -  a bit confused about
> this
> one. patch in NIFI-1086 will close this?
> https://issues.apache.org/jira/browse/NIFI-973 - awaiting review?
> https://issues.apache.org/jira/browse/NIFI-980 - (see 812 confusion)
> presumably closed when NIFI-1086 is closed
> https://issues.apache.org/jira/browse/NIFI-1009 (same!)
> https://issues.apache.org/jira/browse/NIFI-1054 I'm at the ready to submit
> a patch at the 11th hour. .gitattributes may be a candidate for 0.5.0
> https://issues.apache.org/jira/browse/NIFI-1073 - some bug fixes awaiting
> review. if we don't have review bandwidth can be pushed to next release
> https://issues.apache.org/jira/browse/NIFI-1081 - reviewed, probably needs
> revision. Presuming the revisions can be done
> https://issues.apache.org/jira/browse/NIFI-1082 - Unclear as to status. Is
> this awaiting review?
> https://issues.apache.org/jira/browse/NIFI-1086 - Seems like this is the
> lynchpin for several tickets, probably should be a priority
> https://issues.apache.org/jira/browse/NIFI-1097 - Awaiting review?
> https://issues.apache.org/jira/browse/NIFI-1108 - Awaiting comment from
> Mark Payne.
> https://issues.apache.org/jira/browse/NIFI-1109 - awaiting a merge to
> master?
> https://issues.apache.org/jira/browse/NIFI-1127 - awaiting review?
> https://issues.apache.org/jira/browse/NIFI-1132 - awaiting review
> https://issues.apache.org/jira/browse/NIFI-1133 - awaiting review
> https://issues.apache.org/jira/browse/NIFI-1153 - awaiting review
> https://issues.apache.org/jira/browse/NIFI-1155 - blocker bug awaiting
> patch
>
> Seems like we're in good shape. I will have some review bandwidth this
> evening, so if you start reviewing, please note that in the jiras.
>
> Can we get answers for 1108 and 61 and contemplate a slip to 0.5.0?
>
> Tony
>



-- 
Sean

Re: Avro tests are broken on Windows?

[Sean Busbey]

On Tue, Nov 3, 2015 at 8:58 AM, Bryan Bende  wrote:
> I don't really have a way of trying this since I don't have a windows
> machine, but I noticed that the processor writes a new line like this:
>
> System.lineSeparator().getBytes(StandardCharsets.UTF_8)
>
> but in the test it looks for "}\n{" so maybe if that \n was replaced with
> System.lineSeparator() it would work??
>


Friendly reminder that all ASF committers have access to a free MSDN
subscription, which includes Windows VM access:

https://svn.apache.org/repos/private/committers/donated-licenses/msdn-subscription.html

-- 
Sean

Re: spam to the dev mailing list

[Sean Busbey]

FWIW, the mailing list FAQ[1] says that any messages moderators don't
respond to are treated as spam and silently dropped after 5 days. By
actively rejecting you send a notice message back to the sender.


[1]: http://apache.org/dev/committers.html#mail-moderate

On Fri, Oct 30, 2015 at 9:00 AM, Joe Witt <joe.w...@gmail.com> wrote:
> Tony Kurc is the other moderator at this time.
>
> On Fri, Oct 30, 2015 at 9:58 AM, Sean Busbey <bus...@cloudera.com> wrote:
>> Are the rest of the PMC already moderators?
>>
>> On Fri, Oct 30, 2015 at 8:19 AM, Joe Witt <joew...@apache.org> wrote:
>>> Team,
>>>
>>> As a moderator I'm getting tons of spam emails to dev@nifi.  I cannot
>>> keep up with rejections and am not sure if that means they'll leak
>>> through.  I'm not quite sure how to go about getting it address so if
>>> some come through my apologies.
>>>
>>> About to make a NiFi spam bot!
>>>
>>> Joe
>>
>>
>>
>> --
>> Sean



-- 
Sean

Re: spam to the dev mailing list

[Sean Busbey]

Yes, of course. I meant to address Joe's initial concern that by not
keeping up with incoming messages spam might leak through.

On Fri, Oct 30, 2015 at 9:04 AM, Tony Kurc <trk...@gmail.com> wrote:
> Sean,
> The challenging bit is not missing a legit message among the spam.
>
> On Fri, Oct 30, 2015 at 10:02 AM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> FWIW, the mailing list FAQ[1] says that any messages moderators don't
>> respond to are treated as spam and silently dropped after 5 days. By
>> actively rejecting you send a notice message back to the sender.
>>
>>
>> [1]: http://apache.org/dev/committers.html#mail-moderate
>>
>> On Fri, Oct 30, 2015 at 9:00 AM, Joe Witt <joe.w...@gmail.com> wrote:
>> > Tony Kurc is the other moderator at this time.
>> >
>> > On Fri, Oct 30, 2015 at 9:58 AM, Sean Busbey <bus...@cloudera.com>
>> wrote:
>> >> Are the rest of the PMC already moderators?
>> >>
>> >> On Fri, Oct 30, 2015 at 8:19 AM, Joe Witt <joew...@apache.org> wrote:
>> >>> Team,
>> >>>
>> >>> As a moderator I'm getting tons of spam emails to dev@nifi.  I cannot
>> >>> keep up with rejections and am not sure if that means they'll leak
>> >>> through.  I'm not quite sure how to go about getting it address so if
>> >>> some come through my apologies.
>> >>>
>> >>> About to make a NiFi spam bot!
>> >>>
>> >>> Joe
>> >>
>> >>
>> >>
>> >> --
>> >> Sean
>>
>>
>>
>> --
>> Sean
>>



-- 
Sean

Re: [ANNOUNCE] New Apache NiFi Committer Toivo Adams

[Sean Busbey]

Congrats Toivo! looking forward to more great contributions!

On Thu, Oct 29, 2015 at 9:16 AM, Mark Payne  wrote:
> All,
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Toivo
> Adams has accepted the PMC's invitation to become a committer on the Apache
> NiFi project. We greatly appreciate all of Toivo's hard work and generous
> contributions to the project. We look forward to his continued involvement
> in the project.
>
> Welcome, Toivo, and congratulations!



-- 
Sean

Re: [ANNOUNCE] New Apache NiFi Committer Ricky Saltzer

[Sean Busbey]

Congrats Ricky! very well deserved.

On Wed, Oct 21, 2015 at 2:04 PM, Tony Kurc  wrote:
> NiFi Community!
>
> Great news!
>
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Ricky
> Saltzer has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Ricky's hard work and
> generous contributions to the project. We look forward to his continued
> involvement in the project.
>
> Welcome Ricky, and congratulations!
>
> Tony



-- 
Sean

Re: Newbie Dev Contribution : Questions

[Sean Busbey]

Oleg,

Could we convince you to contribute a version of the IDE integration
stuff that works within the current codebase?

On Fri, Oct 16, 2015 at 5:39 AM, Venkatesh Sellappa
 wrote:
> Thanks a bunch.
> This is very good.
>
> I think one of the things that threw me off-course is the names of tests
> don;t really correspond with the Class Names. Learning wheels , i guess.
>
> Oleg: https://github.com/olegz/nifi-ide-integration - Very useful this.
>
> Is it a good idea to add a link to the above from the Developer's Guide ?
>
> Venky.
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Newbie-Dev-Contribution-Questions-tp3118p3123.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.



-- 
Sean

A few questions on tracking project participation

[Sean Busbey]

When a jira is completed by a non-committer, who does the jira get assigned to?

I couldn't find a relevant section in the contributor guide (maybe we
need a "for committers" section?).

If I wanted to get a sense of project activity (across committers and
non-committers alike), how would I get the best possible information?
Could I just look at git? Could I just look at jira? What if I wanted
to break things down by release? What if what I care about is review
activity?

-- 
Sean

Re: A few questions on tracking project participation

[Sean Busbey]

well that's frustrating.

It was approximately:

[1]: The use of ammending-author started as a way of handling
gnarly cherry-picks back to earlier release lines in HBase, but
I've seen it show up on a couple of multiple-author patches
generally.

The HBase reference guide is 404 at the moment, so the best
link I can find is the mailing list thread the explanation references:

http://search-hadoop.com/m/DHED4wHGYS



On Tue, Oct 20, 2015 at 3:46 PM, Tony Kurc <trk...@gmail.com> wrote:
> Sean,
> I think your footnote link got chopped.
>
> Tony
>
> On Tue, Oct 20, 2015 at 4:39 PM, Sean Busbey <bus...@cloudera.com> wrote:
>
>> Well, I admit that my questions were as much to get more questions as
>> to advance an idea of answers. :)
>>
>>
>> On Tue, Oct 20, 2015 at 3:16 PM, Tony Kurc <trk...@gmail.com> wrote:
>> > Sean,
>> > Can you better define "completed by a non-committer"?
>> >
>>
>> Where the majority of the work is done by a non-commiter. I'd even err
>> this on the side of "any substantial", but that's largely because I try to
>> err
>> on the side of encouraging new people with visible credit.
>>
>>
>> > Some cases to consider
>> > Is "a jira" complete when a patch is submitted or when it is merged into
>> > the code repository?
>>
>> definitely when all work is complete, and for things that involve patches
>> to the code base, that would mean merged (either into a release line
>> or into a feature branch if the jira is a part of a mutli-jira effort with
>> such
>> a feature branch).
>>
>>
>> > If a patch requires a bit of work to integrate (and test), where does
>> that
>> > fall?
>>
>> In other projects I usually cover this with the commit's "signed-off-by".
>> if
>> there's a substantial amount of changes involved then an additional
>> "ammending-author" tag per extra contributor (e.g. hbase does this
>> as policy[1]). Figuring out when the tag is needed is largely a judgement
>> call AFAICT.
>>
>> In both of those cases, I'd personally prefer the jira go to whomever did
>> the bulk of the work. 'did the bulk of the work' could be the original
>> author or the
>> integrator depending on jira phrasing, e.g. 'incorporate this external
>> work' is different from a patch that is 80% done by an initial contributor
>> but finished by another.
>>
>> I've definitely seen projects take the other substantially less ambiguous
>> approach. Apache Curator, for example, assigns jiras to whatever
>> committer handled the merge.
>>
>> > What if a patch provider doesn't have a jira account?
>>
>> How would we track provenance of the contribution / grant to the ASF?
>> I guess by git information? At the risk of derailing this thread, is git
>> how we're doing this now?
>>
>> > What if integrating a patch a substantial rewrite of the patch?
>>
>> I'd usually push back on the contributor to do the rewrite in most cases
>> and use "amending-author" in git as described above for either the
>> original or amending author.
>>
>>
>> >
>> > I can certainly tell you what I did based on my ability to "introspect"
>> our
>> > team conventions from jira, commit log and mailing list in lieu of a
>> > documented committer guide.
>> >
>>
>> That would be wonderful. Would you be fine if I rolled the info up into
>> a PMC guide?
>>
>> --
>> Sean
>>



-- 
Sean

Re: Newbie Dev Contribution : Questions

[Sean Busbey]

A PR would be grand. Thanks in advance!

On Tue, Oct 20, 2015 at 3:54 PM, Oleg Zhurakousky
<ozhurakou...@hortonworks.com> wrote:
> Sure, I am ok with adding to contributor guide. Let me know how should I 
> proceed with it (e.g. PR or a paragraph via this thread so someone can add it 
> to the guide)
>
> Cheers
> Oleg
>
> Sent from my iPhone
>
>> On Oct 20, 2015, at 13:07, Sean Busbey <bus...@cloudera.com> wrote:
>>
>> I was thinking the contributor guide, but anywhere we can add a URL to
>> is fine by me. :)
>>
>> On Tue, Oct 20, 2015 at 2:40 PM, Oleg Zhurakousky
>> <ozhurakou...@hortonworks.com> wrote:
>>> Sure, although I am not sure how it would manifest itself since this is 
>>> simply a set of instructions which could go to docs/wiki etc. Is that what 
>>> you meant?
>>>
>>> Cheers
>>> Oleg
>>>
>>>> On Oct 20, 2015, at 3:20 PM, Sean Busbey <bus...@cloudera.com> wrote:
>>>>
>>>> Oleg,
>>>>
>>>> Could we convince you to contribute a version of the IDE integration
>>>> stuff that works within the current codebase?
>>>>
>>>> On Fri, Oct 16, 2015 at 5:39 AM, Venkatesh Sellappa
>>>> <vs186...@outlook.com> wrote:
>>>>> Thanks a bunch.
>>>>> This is very good.
>>>>>
>>>>> I think one of the things that threw me off-course is the names of tests
>>>>> don;t really correspond with the Class Names. Learning wheels , i guess.
>>>>>
>>>>> Oleg: https://github.com/olegz/nifi-ide-integration - Very useful this.
>>>>>
>>>>> Is it a good idea to add a link to the above from the Developer's Guide ?
>>>>>
>>>>> Venky.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: 
>>>>> http://apache-nifi-developer-list.39713.n7.nabble.com/Newbie-Dev-Contribution-Questions-tp3118p3123.html
>>>>> Sent from the Apache NiFi Developer List mailing list archive at 
>>>>> Nabble.com.
>>>>
>>>>
>>>>
>>>> --
>>>> Sean
>>
>>
>>
>> --
>> Sean
>>



-- 
Sean

Re: Newbie Dev Contribution : Questions

[Sean Busbey]

I was thinking the contributor guide, but anywhere we can add a URL to
is fine by me. :)

On Tue, Oct 20, 2015 at 2:40 PM, Oleg Zhurakousky
<ozhurakou...@hortonworks.com> wrote:
> Sure, although I am not sure how it would manifest itself since this is 
> simply a set of instructions which could go to docs/wiki etc. Is that what 
> you meant?
>
> Cheers
> Oleg
>
>> On Oct 20, 2015, at 3:20 PM, Sean Busbey <bus...@cloudera.com> wrote:
>>
>> Oleg,
>>
>> Could we convince you to contribute a version of the IDE integration
>> stuff that works within the current codebase?
>>
>> On Fri, Oct 16, 2015 at 5:39 AM, Venkatesh Sellappa
>> <vs186...@outlook.com> wrote:
>>> Thanks a bunch.
>>> This is very good.
>>>
>>> I think one of the things that threw me off-course is the names of tests
>>> don;t really correspond with the Class Names. Learning wheels , i guess.
>>>
>>> Oleg: https://github.com/olegz/nifi-ide-integration - Very useful this.
>>>
>>> Is it a good idea to add a link to the above from the Developer's Guide ?
>>>
>>> Venky.
>>>
>>>
>>>
>>> --
>>> View this message in context: 
>>> http://apache-nifi-developer-list.39713.n7.nabble.com/Newbie-Dev-Contribution-Questions-tp3118p3123.html
>>> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
>>
>>
>>
>> --
>> Sean
>>
>



-- 
Sean

Re: A few questions on tracking project participation

[Sean Busbey]

Well, I admit that my questions were as much to get more questions as
to advance an idea of answers. :)


On Tue, Oct 20, 2015 at 3:16 PM, Tony Kurc  wrote:
> Sean,
> Can you better define "completed by a non-committer"?
>

Where the majority of the work is done by a non-commiter. I'd even err
this on the side of "any substantial", but that's largely because I try to err
on the side of encouraging new people with visible credit.


> Some cases to consider
> Is "a jira" complete when a patch is submitted or when it is merged into
> the code repository?

definitely when all work is complete, and for things that involve patches
to the code base, that would mean merged (either into a release line
or into a feature branch if the jira is a part of a mutli-jira effort with such
a feature branch).


> If a patch requires a bit of work to integrate (and test), where does that
> fall?

In other projects I usually cover this with the commit's "signed-off-by". if
there's a substantial amount of changes involved then an additional
"ammending-author" tag per extra contributor (e.g. hbase does this
as policy[1]). Figuring out when the tag is needed is largely a judgement
call AFAICT.

In both of those cases, I'd personally prefer the jira go to whomever did
the bulk of the work. 'did the bulk of the work' could be the original
author or the
integrator depending on jira phrasing, e.g. 'incorporate this external
work' is different from a patch that is 80% done by an initial contributor
but finished by another.

I've definitely seen projects take the other substantially less ambiguous
approach. Apache Curator, for example, assigns jiras to whatever
committer handled the merge.

> What if a patch provider doesn't have a jira account?

How would we track provenance of the contribution / grant to the ASF?
I guess by git information? At the risk of derailing this thread, is git
how we're doing this now?

> What if integrating a patch a substantial rewrite of the patch?

I'd usually push back on the contributor to do the rewrite in most cases
and use "amending-author" in git as described above for either the
original or amending author.


>
> I can certainly tell you what I did based on my ability to "introspect" our
> team conventions from jira, commit log and mailing list in lieu of a
> documented committer guide.
>

That would be wonderful. Would you be fine if I rolled the info up into
a PMC guide?

-- 
Sean

Re: 1.0.0 Branch Guidance

[Sean Busbey]

Before we make a 1.0 branch we should get consensus on what a
minimally viable 1.0 release would need to contain and set a target
release date (even if it's 6-9 months out).

Otherwise it's too easy to end up in a situation where the 1.0 branch
perpetually languishing while releases continue out of the older
release line.

Kind of related, how long do we plan to support major releases? Once
1.Y comes out, how long will we keep making 0.Xs? What about once
there's a 2.Z?

On Tue, Sep 29, 2015 at 7:58 AM, Dan Bress  wrote:
> OK, sounds good to me.  If I had to choose between keeping most of the 
> communities work 'parked' as PullRequests and patches vs. merged into a 1.X 
> branch that we keep up to date with master, I'd vote for 1.X branch.
>
> Dan Bress
> Software Engineer
> ONYX Consulting Services
>
> 
> From: Aldrin Piri 
> Sent: Tuesday, September 29, 2015 8:48 AM
> To: dev@nifi.apache.org
> Subject: Re: 1.0.0 Branch Guidance
>
> Dan,
>
> I don't think we are at the point where 1.0 is our next release.  The items
> to be included for that release as per the feature proposals (whether
> directly as a feature or as an implicit dependency for one or more of those
> features) are some considerable efforts and while there may not be many
> releases between 0.3.0 and that point, it will likely be too long to go
> without any at all.
>
> Certainly agree on the long living branch being an effort of itself, but
> may be the course we have to take to keep the project moving.
>
> On Tue, Sep 29, 2015 at 8:41 AM, Dan Bress  wrote:
>
>> Aldrin,
>>I definitely like the idea of creating separate branches for the 0.3.X
>> work and the 1.X.X work.
>>
>>My thoughts would be to make 'master' the 1.X version, and make a
>> branch for the 0.3.X work.  The reason being, I would imagine most of the
>> work the community is doing should be slated for 1.X, where as less
>> work(e.g. bug fixes) be done in the 0.3.X branch.  And by making master
>> 1.0.0 it nudges people in that direction.  Also, I'm assuming that 0.3.X
>> will just be bug fixes at this point, and our next 'major' release will be
>> 1.0.0.   Is that a fair assumption to make?  If not, I might be more
>> inclined to agree with your original suggestion, although keeping that long
>> living branch up to date with all the changes from master might be a
>> maintenance nightmare.
>>
>> Dan Bress
>> Software Engineer
>> ONYX Consulting Services
>>
>> 
>> From: Aldrin Piri 
>> Sent: Thursday, September 24, 2015 10:49 AM
>> To: dev@nifi.apache.org
>> Subject: 1.0.0 Branch Guidance
>>
>> All,
>>
>> We are starting to receive more items that are "breaking" changes
>> (deprecation of components and code being those that immediately come to
>> mind) and are accordingly scheduled for a 1.0.0 release.  I would like to
>> solicit the community for best practices on the introduction and
>> maintenance of a 1.0.0 branch so we can do so in a practical manner.
>>
>> There are a few PRs/patches that are sitting in limbo because we do not
>> have an appropriate place to put them and would very much like to be able
>> to incorporate those changes and close them in lieu of waiting until master
>> reaches that juncture.
>>
>> Any input on caveats, items to note, and any other items to be mindful of
>> is greatly appreciated.
>>
>> Thanks!
>>



-- 
Sean

Re: What sort of major release support plan do we want to provide?

[Sean Busbey]

As the project matures, I expect at least part of major release line
lifetime will be driven by adoption since we're volunteer driven.

For example, HBase 0.94 (a major release line) has outlived HBase 0.96
and may outlive 0.98 yet (both major release lines). Part of this was
because the community failed to provide a zero-downtime option for
upgrading off of 0.94, but some of it is just a matter of timing and
upgrade cycles in large IT shops. Development on the line has
progressively slowed; it got new features for some time, but has been
locked into critical bug fixes for about a year now.

Part of the reason HBase has been able to use an adaptive lifetime for
release lines is that we get a volunteer to sign up as Release Manager
(originally over a major release's life and lately over a minor). That
RM then did the majority of work monitoring the activity of a branch
line. As incoming patches slow down they take the lead in 1) ensuring
that critical patches devs forget to backport still show up before
release, and 2) driving community expectations around the branch (e.g.
"I'm planning to slow the release cycle to quarterly, so anyone think
that'll adversely impact them too much?"). Full disclosure, HBase
hasn't worked out changes to major release lifecycle since it
transitioned to have patch releases (post 1.0.0) back in February; for
now we just wait for activity to slow on minor releases like we did
for the older major releases.



On Tue, Sep 29, 2015 at 9:31 AM, Joe Witt <joe.w...@gmail.com> wrote:
> Team,
>
> We should discuss an important topic that Sean Busbey raised which is
> what is our plan to support major release lines.
>
> We should identify:
> - how long to support the previous major release line(s)
> - what type of support we'd provide (feature porting, bugs only,
> critical bugs only, critical security bugs only)
>
> I would like to propose that we only offer critical security bug fixes
> to old release lines as a general philosophy.  But I am certainly ears
> open to understanding a broader view from users or those more
> experienced managing these sorts of projects.
>
> Look forward to seeing some thoughtful perspectives and inputs.
>
> Thanks
> Joe



-- 
Sean

Re: HDFS and Marathon

[Sean Busbey]

related: fuse over webhdfs: https://github.com/aw-altiscale/webhdfs

On Thu, Sep 17, 2015 at 8:04 PM, Tony Kurc  wrote:
> has anyone tried using fuse [1] to accomplish something like what Kathy was
> asking?
>
> [1] https://wiki.apache.org/hadoop/MountableHDFS
>
> On Thu, Sep 17, 2015 at 1:26 PM, Joe Witt  wrote:
>
>> Kathy,
>>
>> In short, no.  But this is a good idea.  Our content repository
>> implementations today expect access to a 'local' file system.  That
>> local file system could in theory be something like an NFS mount but
>> we'd need to implement it differently to work with HDFS.  This would
>> be a great JIRA to create if you're interested.
>>
>> Thanks
>> Joe
>>
>> On Thu, Sep 17, 2015 at 10:17 AM, Olsen, Katherine 
>> wrote:
>> > Hi,
>> >
>> >
>> >
>> > I'm attempting to deploy a NiFi instance on Mesos using Apache
>> Marathon.  Currently whenever the NiFi instance is re-started, a new Mesos
>> container is pulled down and serves as the run time environment. This is
>> all good except that the new container has an empty flow file repository.
>> My preferred solution to avoid this is to use a location on the distributed
>> file system (HDFS) as the flow file repository location.  However, when I
>> attempted to change the configuration file setting in nifi.properties to
>> write to HDFS as follows, it simply created a folder named hdfs:// with
>> associated sub-directories inside the local Mesos container that is running
>> NiFi.
>> >
>> >
>> >
>> > nifi.flowfile.repository.directory =
>> hdfs://ip_address:8020/nifi/flowfile_repository
>> >
>> >
>> >
>> > Is it currently possible to get Nifi to be respectful of the hdfs naming
>> convention and save its flow file repository to a distributed file system
>> like HDFS?  I'm using NiFi version 0.2.1, Mesos version 0.23.0, and
>> Marathon version 0.10.0.
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Kathy
>> >
>> > CONFIDENTIALITY NOTICE (v1.0): This e-mail message, including any
>> attachments, is for the sole use of the intended recipient(s) and may
>> contain proprietary, confidential or privileged information or otherwise be
>> protected by law. Any unauthorized review, use, disclosure or distribution
>> is prohibited. If you are not the intended recipient, please notify the
>> sender and destroy all copies and the original message.
>> >
>> >
>>



-- 
Sean

Re: [DISCUSS] move nifi-assembly into top level

[Sean Busbey]

It's been easier, in my experience, to properly work out maven tooling to
build assemblies with all of the needed legal bits via a dedicated assembly
submodule rather than having it at the top level.

Consider build order. The assembly should have a dependency on all the
modules it includes so that maven knows to build it last. If the assembly
is at the top pom and that pom is the parent for the submodules it
contains, how does that work out?

What if we just had a flag (default true) that controlled copying the
finished assembly to a top level target directory?

-- 
Sean
On Aug 15, 2015 11:32 AM, Joe Witt joe.w...@gmail.com wrote:

 Hello,

 With the work going on with NIFI-850 [1] and NIFI-857 [2] things will
 be much cleaner and simpler.  One thing that came up from a NiFi user
 was how weird they thought it was that we had a 'nifi-assembly'
 subproject that they had to go to so they could find the build.  Back
 when NiFi wasn't open sourced this approach made sense but at this
 point I couldn't actually make an argument for why we should still do
 that.

 I've created a JIRA to just move the assembly mechanism up to the root
 of the project NIFI-858 [3] but before I tackle that it seems like a
 good one to send out to the group and see if folks have thoughts on
 this.  Can anyone see a reason why this is a bad idea?

 Thanks
 Joe

 [1] https://issues.apache.org/jira/browse/NIFI-850
 [2] https://issues.apache.org/jira/browse/NIFI-857
 [3] https://issues.apache.org/jira/browse/NIFI-858