Resign from PMC

[James Wing]

I would like to resign from the PMC.  I have not been able to keep up
recently.

Thank you,

James Wing

Re: [VOTE] Release Apache NiFi 1.9.1 (rc1)

[James Wing]

+1 (binding) - Ran through the release helper, checked the signatures,
license/readme, and ran the full build.  Ran a simple test flow.

Thanks, Joe, for putting this release together!

On Tue, Mar 12, 2019 at 10:49 PM Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.9.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1138
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.9.1-rc1/
>
> The Git tag is nifi-1.9.1-RC1
> The Git commit ID is a5cedc4ad39b17bee97303b63b620f9ac3dddc79
>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=a5cedc4ad39b17bee97303b63b620f9ac3dddc79
>
> Checksums of nifi-1.9.1-source-release.zip:
> SHA256: 7099abb33e26445788630b69f38b8788117cdd787b7001752b4893d8b6c16f38
> SHA512:
>
> 678c2ee32f7db8c73393178f329c574315b1b892084b822f9b7a6dc5bc159d5e7e1169812d9676a72f738d03fd2f4366f2b67ddee152b56c8a77751fd5cbb218
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 19 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12345163
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.9.1
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.9.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.9.0 (rc2)

[James Wing]

+1 (binding) - Ran through the release helper, full build and test flow
with the resulting binary.

Thanks for putting this together, Joe.

On Sat, Feb 16, 2019 at 7:50 PM Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.9.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1136
>
> The Git tag is nifi-1.9.0-RC2
> The Git commit ID is 45bb53d2aafd6ec5cb6bb794b3f7f8fc8300a04b
>
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=45bb53d2aafd6ec5cb6bb794b3f7f8fc8300a04b
>
> Checksums of nifi-1.9.0-source-release.zip:
> SHA256: f8d2987a98903f0c00c50677f3a6ad361e417c6021f5179280cbe9ca838695da
> SHA512:
>
> 2e77c420f932514417693584b4708a534df398e344dac7c1471f55cc382b7493d73b10ebc0d9e58562eb989c1f0b72980d6d18a2555883267f0bc08f092f30fe
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 160 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12344357
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.9.0
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.9.0-rc2/
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.9.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [DISCUSS] Early, voluntary relocation to GitBox

[James Wing]

+1, thanks for volunteering.

> On Dec 7, 2018, at 13:39, Kevin Doran  wrote:
> 
> +1
> 
> On 12/7/18, 15:17, "Andy LoPresto"  wrote:
> 
>+1. 
> 
>Andy LoPresto
>alopre...@apache.org
>alopresto.apa...@gmail.com
>PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Dec 7, 2018, at 11:11 AM, Aldrin Piri  wrote:
>> 
>> Hey folks,
>> 
>> Daniel Gruno sent an email to the dev list about the deprecation of our
>> git-wip repositories [1] (these are the canonical repos that committers
>> push changes to and are currently mirrored to GitHub) and transitioning to
>> GitBox [2].
>> 
>> As highlighted in that email, this will not be an optional change, only in
>> when and how it happens.  There was a previous discussion over this topic
>> [3] where it was generally well received but I think some of the logistics
>> were never mapped out and thus the actual request to do so was never
>> executed.
>> 
>> I am proposing we volunteer for the early relocation.  The process looks to
>> be fairly straightforward and should ultimately only result in requiring
>> our contributors to add/replace their original git-wip based remotes.
>> 
>> If folks are in favor, I am happy to file the requisite INFRA ticket and
>> provide the associated communication/docs to the community to make this
>> happen.  Again, this is only about making the choice to perform this
>> migration now, in a coordinated manner, in lieu of the mass migration that
>> would happen later.
>> 
>> From a project management standpoint, I think it is a nice bit of
>> functionality that lets us better curate our open PRs and, given my prior
>> interest in the subject, would like to see it happen.
>> 
>> --aldrin
>> 
>> [1]
>> https://lists.apache.org/thread.html/8247bb17671d6131b1d7a646b4c018b21aac390c4f627d8fb1f421b2@%3Cdev.nifi.apache.org%3E
>> [2] https://gitbox.apache.org/
>> [3]
>> https://lists.apache.org/thread.html/de5e103994f356b1b8a572410938eef44af8cb352210e35305c04bc9@%3Cdev.nifi.apache.org%3E
> 
> 

Re: [VOTE] Release Apache NiFi 1.8.0 (RC3)

[James Wing]

+1 (binding).  Ran though the release helper, tested the resulting binary.
Thank you for your persistence, Jeff.


On Mon, Oct 22, 2018 at 10:56 PM Jeff  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.8.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1135
>
> The Git tag is nifi-1.8.0-RC3
> The Git commit ID is 98aabf2c50f857efc72fd6f2bfdd9965b97fa195
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=98aabf2c50f857efc72fd6f2bfdd9965b97fa195
>
> Checksums of nifi-1.8.0-source-release.zip:
> SHA256: 6ec21c36ebb232f344493a4aeb5086eed0c462c576e11a79abed8149bc8b65c3
> SHA512:
>
> 846aecd4eb497a3b7dee7d1911b02453b8162b6c87e39f3df837744a478212e2e3e3615921079d29c2804671f26ecd05b04ce46a4bb69e8911fc185e27be9c24
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jstorck.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 209 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12343482
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.8.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.8.0 (RC2)

[James Wing]

+1 (binding).  Thanks again, Jeff.

On Sat, Oct 20, 2018 at 8:11 PM Jeff  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.8.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1134
>
> The Git tag is nifi-1.8.0-RC2
> The Git commit ID is 19bdd375c32c97e2b7dfd41e5ffe65f5e1eb2435
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=19bdd375c32c97e2b7dfd41e5ffe65f5e1eb2435
>
> Checksums of nifi-1.8.0-source-release.zip:
> SHA256: 72dc2934f70f41e0c62e0aeb2bdc48e9feaa743dc06319cbed42da04bdc0f827
> SHA512:
>
> 012194f79d4bd5060032588e29f5e9c4240aa5e4758946a6cbcc89c0a1499de9db0c46d3f76e5ee694f0f9345c5f1bee3f3e315ef6fcc1194447958cb3f8b003
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jstorck.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 204 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12343482
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0
>
> The vote will be open for 96 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.8.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.8.0

[James Wing]

+1 Release this package as nifi-1.8.0.  Ran through the release helper,
tested the resulting binary, diffed some of the L and code files.

Thanks for putting this release together, Jeff.

On Wed, Oct 17, 2018 at 8:59 PM Jeff  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.8.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1133
>
> The Git tag is nifi-1.8.0-RC1
> The Git commit ID is 9b02d58626ca874ed2ed3e0bbe530512cfa0dbf8
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=9b02d58626ca874ed2ed3e0bbe530512cfa0dbf8
>
> Checksums of nifi-1.8.0-source-release.zip:
> SHA256: 3ec90a7f153e507d7bba2400d6dafac02641d6f7afc7a954fed959191073ce21
> SHA512:
>
> 8b9d944da1833bfb645f502107cab98a555e3b2a7602c5ff438407272c86defdeebe18625c5ad9dfb3f344397314569e97220a35f2438182a79a700caa90721e
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jstorck.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 204 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12343482
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.8.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.8.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.7.1

[James Wing]

+1 (binding).  Ran through the release helper, checked license, notice,
readme files, and tested the resulting binary.

Thanks for putting this together, Andy.

On Thu, Jul 12, 2018 at 2:00 PM Andy LoPresto  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.7.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1131
>
> The Git tag is nifi-1.7.1-RC1
> The Git commit ID is 241e5411a24435ca2c39ab7bea1c5eb9ed195cb3
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=241e5411a24435ca2c39ab7bea1c5eb9ed195cb3
>
> Checksums of nifi-1.7.1-source-release.zip:
> SHA1: 92428e219f8792b886a2826119d3f70c98de52fe
> SHA256: e951a0a2c6b7f8e742b8ab126532ab2e02876ba8c1db34b858fb72718a9fa0e6
> SHA512:
> a37add06c3c4f2ae5294c9f7668e2e837d6b92095d8e349d18cb13a1748c0e052632415e0fda148f55b61f6774bcdd487ee3d8b8125fe4d2345791da1fc7d8d9
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/alopresto.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 13 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12343647
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.7.1
>
> The vote will be open for 96 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.7.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because…
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>

Re: [DISCUSS] Tar + Gzip vs. Zip

[James Wing]

It's a great idea, Andy, I strongly support just one format.  I think Zip
is a good choice.

On Tue, Jun 26, 2018 at 11:16 AM Otto Fowler 
wrote:

> I end up using zip all the time.  zip +1
>
>
> On June 26, 2018 at 13:30:33, Tony Kurc (tk...@apache.org) wrote:
>
> My preference is zip.
>
> On Tue, Jun 26, 2018, 9:21 AM Josh Elser  wrote:
>
> >
> >
> > On 6/25/18 11:34 PM, Andy LoPresto wrote:
> > > Hi folks,
> > >
> > > I do not want to start a long-running argument or entrenched battle.
> > > However, having just performed the RM duties for the latest release, I
> > > believe I have identified a resource inefficiency in the fact that we
> > > generate, upload, host, and distribute two compressed archives of the
> > > binary which are functionally equivalent. For 1.7.0, both the .tar.gz
> > > and .zip files are 1.2 GB (1_224_352_000 bytes for tar.gz vs.
> > > 1_224_392_000 bytes for zip). The time to build and sign these is
> > > substantial, but the true cost comes in uploading and hosting them.
> > > While the fabled extension registry will save all of us from this
> > > burden, it isn’t arriving tomorrow, and I think we could drastically
> > > improve this before the next release.
> > >
> > > I have no personal preference between the two formats. In earlier days,
> > > there were platform inconsistencies and the tools weren’t available on
> > > all systems, but now they are pretty standard for all users. This [1]
> is
> > > an interesting article I found which had some good info on the origins,
> > > and here are some additional resources for anyone interested [2][3]. I
> > > don’t care which we pick, but I propose removing one of the options for
> > > the build going forward (toolkit as well).
> > >
> > > That said, if someone has a good reason that both are necessary, I
> would
> > > love to hear it. I didn’t find anything on the Apache Release Policy
> > > which stated we must offer both, but maybe I missed it. Thanks.
> >
> > I'm not aware of any ASF policy. I think it mostly stems from default
> > convention you get out of the maven-assembly-plugin.
> >
> > > [1] https://itsfoss.com/tar-vs-zip-vs-gz/
> > > [2] https://superuser.com/a/1257441/40003
> > > [3] https://superuser.com/a/173995/40003
> > > [4] https://www.apache.org/legal/release-policy.html#artifacts
> > >
> > >
> > > Andy LoPresto
> > > alopre...@apache.org 
> > > /alopresto.apa...@gmail.com /
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> >
>

Re: [VOTE] Release Apache NiFi 1.7.0

[James Wing]

+1 (binding).  Ran through the release helper, tested the binary, diffed
the LICENSE and NOTICE files.  Looks good to me.

Thanks, Andy, for putting this release together.

On Wed, Jun 20, 2018 at 12:16 AM Andy LoPresto  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.7.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1127
>
> and
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.7.0
>
> The Git tag is nifi-1.7.0-RC1
> The Git commit ID is 99bcd1f88dc826f857ae4ab33e842110bfc6ce21
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=99bcd1f88dc826f857ae4ab33e842110bfc6ce21
>
> Checksums of nifi-1.7.0-source-release.zip:
> SHA1: 11086ef532bb51462d7e1ac818f6308d4ac62f03
> SHA256: b616f985d486af3d05c04e375f952a4a5678f486017a2211657d5ba03aaaf563
> SHA512:
> d81e9c6eb7fc51905d6f6629b25151fc3d8af7a3cd7cbc3aa03be390c0561858d614b62d8379a90fdb736fcf5c1b4832f4e050fdcfcd786e9615a0b5cc1d563d
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/alopresto.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 194 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342979=12316020
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.7.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-1.7.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because…
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>

Re: Setting values as System properties

[James Wing]

I agree it is not what we would hope for.  But I have not found any
information to contradict you.  The proxy setting we use now for Google
Cloud Storage apparently does not apply to Google Cloud PubSub.

What user experience would you propose?  It seems reasonable that a user
might be able to configure the setting globally, if they know that is what
they are doing.  It would be bad if configuring the properties on one
processor set a global proxy setting that had a wide impact.  It might be
better to simply document that proxy use requires the configuration of
environment variables with a global impact.

On Tue, Jun 19, 2018 at 6:06 AM Sivaprasanna 
wrote:

> Team,
>
> As part of NIFI-5133, I started doing some bit of research on Google Cloud
> PubSub service and it's support on proxy configuration and came to know
> that the service uses 'gRPC' so the proxy configuration is expected to be
> at the System properties level. I know this approach is not good and
> believe it has to be avoided but wanted to know the community's thoughts on
> this.
>
> Thanks,
> Sivaprasanna
>

Re: [VOTE] Release Apache NiFi Registry 0.2.0

[James Wing]

+1 (binding).  Ran through the release helper, tested with a NiFi
1.7.0-SNAPSHOT build.

Thanks, Kevin!

On Fri, Jun 15, 2018 at 6:26 PM Kevin Doran  wrote:

> Hello,
>
> I am pleased to call this vote for the source release of Apache NiFi
> Registry 0.2.0.
>
> The source zip, including signatures, digests, etc. can be found at:
>
> https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.2.0/
>
> The Git tag is nifi-registry-0.2.0-RC1
> The Git commit ID is 7966c52edd7a468fa30d32a333ef907e2b9e374c
>
> https://git-wip-us.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=7966c52edd7a468fa30d32a333ef907e2b9e374c
>
> Checksums of nifi-registry-0.2.0-source-release.zip:
> SHA1: a3c2871b2e4dbadd537dec43ec6c28d1839f1706
> SHA256: ec824dcf39ce44cda8cb24c2049cc2920717161b0b6336fa4c3cfe759611637d
> SHA512:
> fe90234e9caa9e0bd7cc95f9a5f486e0f82a2b18e75d1df910b33c8d75fb965099d45f05dc094a41870f513b50219f8a6cdcf9d2fcd6b6f38e561c88d218
>
> Release artifacts are signed with the following key:
> Fingerprint = C09B A891 AED4 5B8C 2C23  1AFE 1FB6 6A91 F71B 6207
> Available at:
> https://people.apache.org/keys/committer/kdoran.asc
> https://pgp.mit.edu/pks/lookup?op=get=0x1FB66A91F71B6207
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/dev/nifi/KEYS
>
> 46 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12320920=12342369
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiRegistry0.2.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. Then please vote:
>
> [ ] +1 Release this package as nifi-registry-0.2.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
>
>

Re: [VOTE] Change the name Apache NiFi Fluid Design System to Apache NiFi Flow Design System

[James Wing]

+1 (binding)

> On May 18, 2018, at 6:30 AM, Rob Moran  wrote:
> 
> Following positive response discussing the name change of nifi-fds [1], I'd
> like to call a vote to officially change the name of the Apache NiFi Fluid
> Design System to the Apache NiFi Flow Design System.
> 
> The name change reflects that nifi-fds is meant to establish a set of
> reusable UI components for the NiFi ecosystem, and that these components
> are not tied to a single design system or UI platform.
> 
> I am a +1.
> 
> The vote will be open for 72 hours and will be a majority rule vote.
> 
> [ ] +1 Change the name to Apache NiFi Flow Design System
> [ ] +0 No opinion
> [ ] -1 Do not change the name because...
> 
> [1]
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201805.mbox/%3CCAGmW7OxAaazAYDs8_v%3DkwYPKtJp3tMDEJdyGztSFgcmdv%3DAS4w%40mail.gmail.com%3E

Re: Support with GetDynamoDB and PutDynamoDB

[James Wing]

The Json Document Attribute scopes the data retrieved to one of the child
attributes of the root document.  For example, you could specify "Data"
from your example object (without quotes).

It is required, which is to say that you cannot retrieve the entire root
document as you have it above.

On Thu, Apr 19, 2018 at 9:10 AM, Maurilio Lopez Martinez <
lopez.mtzm1...@gmail.com> wrote:

> Hi.
> Apache Nifi Team
> I would like you to help me in the configuration of the getDynamoDB
> processor, especially in the option Json Document Attribute my json has the
> following structure:
>
> {
>   "CreatedAt": "",
>   "CreatedBy": "",
>   "Data": "",
>   "Id": "",
>   "ModifiedAt": "",
>   "ModifiedBy": "",
>   "Owner": ""
> }
>
> Thank you very much for your support.
> Greetings.
>

Re: [VOTE] Release Apache NiFi 1.6.0 (RC3)

[James Wing]

+1 (binding) - Ran through the release helper checksums and build steps
(Amazon Linux). Upgraded a working NiFi out of unbounded optimism for RC3,
and it's looking good.  Thanks again for the release efforts!

On Tue, Apr 3, 2018 at 4:49 PM, Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1124
>
> The Git tag is nifi-1.6.0-RC3
> The Git commit ID is f8466cb16d6723ddc3bf5f0e7f8ce8a47d27cbe5
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> f8466cb16d6723ddc3bf5f0e7f8ce8a47d27cbe5
>
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: d1e1c24f9af809bf812982962b61d07df4f1131e
> SHA256: 1e5028d594bb402aa36460f1b826d4e8a664ad6f0538deed20286cbf3c621fb8
> SHA512: 8cb10cbafa6feeed712dbc0cf076496d6bc014276aab71383ff3481d8ea7
> 19cf1f39766abc76c75ba58ffca747df3bd6d9bac82e410de1c78673dcd16a5ddfee
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 162 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.6.0 (RC2)

[James Wing]

+1 (binding).  Ran through the release helper, worked with a test flow.
Thanks for putting this together.

On Mon, Mar 26, 2018 at 8:34 PM, Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1123
>
> The Git tag is nifi-1.6.0-RC2
> The Git commit ID is b5935ec81a7cbc048820781ac62cd96bbea5b232
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> b5935ec81a7cbc048820781ac62cd96bbea5b232
>
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: 009f1e2e3c17e38f21f27170b9c06228d11653c0
> SHA256: 39941a5b25427e2b4cc5ba8206084ff92df58863f29ddd097d4ac1e85424beb9
> SHA512: 1773417a48665e3cda22180ea7f401bc8190ebddbf3f7bc29831e46e7ab0
> a07694c6e478d252fa573209d4a3c8132a522a8507b6a8784669ab7364847a07e234
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 146 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12342422
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.6.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Establish Fluid Design System, a sub-project of Apache NiFi

[James Wing]

+1

On Fri, Mar 9, 2018 at 12:10 PM, Scott Aslan  wrote:

> All,
>
> Following a solid discussion for the past couple of weeks [1] regarding the
> establishment of Fluid Design System as a sub-project of Apache NiFi, I'd
> like to
> call a formal vote to record this important community decision and
> establish consensus.
>
> The scope of this project is to define a theme-able set of high quality UI
> components and utilities for use across the various Apache NiFi web
> applications in order to provide a more consistent user experience.
>
> I am a +1 and looking forward to the future work in this area.
>
> The vote will be open for 72 hours and be a majority rule vote.
>
> [ ] +1 Establish Fluid Design System, a subproject of Apache NiFi
> [ ]   0 Do not care
> [ ]  -1 Do not establish Fluid Design System, a subproject of Apache NiFi
>
> Thanks,
>
> ScottyA
>
> [1] *http://mail-archives.apache.org/mod_mbox/nifi-dev/201802.mbox/%
> 3CCAKeSr4ibXX9xzGN1GhdVv5uTmWvfB3QULXF9orzw4FYD0n7taQ%40mail.gmail.com%3E
>  3CCAKeSr4ibXX9xzGN1GhdVv5uTmWvfB3QULXF9orzw4FYD0n7taQ%40mail.gmail.com%3E
> >*
>

Re: 答复: Re: Is there a REST API to run a dataflow on demand?

[James Wing]

The NiFi API can be used to start and stop processors or process groups,
and this might solve your use case.  But NiFi does not have an API to run a
processor only once, immediately, separate from its configured schedule.  I
have solved similar problems in the past by creating two separate upstream
sources - one for scheduled operation, and one for ad-hoc operation.
GenerateFlowFile, GetFile, or similar processors can be used to inject a
flowfile where you need to kick off the flow.

Thanks,

James

On Wed, Feb 21, 2018 at 5:57 PM,  wrote:

> Thanks a lot.
>
> But I want to know if there is a REST API that triggers a dataflow on
> demand?
> I don't find the API in the page.
>
>
>
>
> 发件人:
> Charlie Meyer 
> 收件人:
> dev@nifi.apache.org
> 日期:
> 2018/02/22 09:36
> 主题:
> Re: Is there a REST API to run a dataflow on demand?
>
>
>
> Yep, when you make the changes in the UI, open developer tools in your
> browser and see what calls to the nifi api it is making then mimic those
> with code.
>
> The nifi team also kindly publishes
> https://nifi.apache.org/docs/nifi-docs/rest-api/index.html which help a
> lot.
>
> Best of luck!
>
> -Charlie
>
> On Wed, Feb 21, 2018 at 7:34 PM,  wrote:
>
> > Hi, team,
> >
> > We set up several NiFi dataflows for data processing.
> > These dataflows are configured to run once per day in the midnight.
> >
> > But sometimes, some dataflows are failed,I want to run the dataflow
> again
> > immediately after fixing the issue instead of waiting for running it in
> > the midnight to
> > make sure that the issue is really fixed.
> >
> > The only way I know to do this is to change the time of running the
> > dataflow to the 5 mintutes from now for example
> > and then change it back to midnight.
> >
> > It's a little inconvenient.
> >
> > Is there any REST API that I can use to trigger the dataflow on demand
> > i.e. without change the time back and forth?
> >
> > Thanks
> >
> > Boying
> >
> >
> >
> > 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对
> 外
> > 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发
> 件
> > 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
> >
> >
> > This email message may contain confidential and/or privileged
> information.
> > If you are not the intended recipient, please do not read, save,
> forward,
> > disclose or copy the contents of this email or open any file attached to
> > this email. We will be grateful if you could advise the sender
> immediately
> > by replying this email, and delete this email and any attachment or
> links
> > to this email completely and immediately from your computer system.
> >
> >
> >
> >
>
>
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人，请您不要阅读、保存、对外
> 披露或复制本邮件的任何内容，或者打开本邮件的任何附件。请即回复邮件告知发件
> 人，并立刻将该邮件及其附件从您的电脑系统中全部删除，不胜感激。
>
>
> This email message may contain confidential and/or privileged information.
> If you are not the intended recipient, please do not read, save, forward,
> disclose or copy the contents of this email or open any file attached to
> this email. We will be grateful if you could advise the sender immediately
> by replying this email, and delete this email and any attachment or links
> to this email completely and immediately from your computer system.
>
>
>
>

Re: [DISCUSS] Addressing Lingering Pull Requests

[James Wing]

This is a great idea, Mark, thanks for proposing it.  30 days after last
review comment seems like a good, enforceable standard.

James

On Mon, Jan 29, 2018 at 8:29 AM, Mark Payne  wrote:

> All,
>
> We do from time to time go through the backlog of PR's that need to be
> reviewed and
> start a "cleansing" process, closing out any old PR's that appear to have
> stalled out.
> When we do this, though, we typically will start sending out e-mails
> asking if there are
> any stalled PR's that we shouldn't close and start trying to decipher
> which ones are okay
> to close out and which ones are not. This puts quite an onus on the
> committer who is
> trying to clean this up. It also can result in having a large number of
> outstanding Pull Requests,
> which I believe makes the community look bad because it gives the
> appearance that we are
> not doing a good job of being responsive to Pull Requests that are
> submitted.
>
> I would like to propose that we set a new "standard" that is: if we have
> any Pull Request
> that has been stalled (and by "stalled" I mean a committer has reviewed
> the PR and did
> not merge but asked for clarifications or modifications and the
> contributor has not pushed
> any new commit or responded to the comments) for at least 30 days, that we
> go ahead
> and close the Pull Request (after commenting on the PR that it is being
> closed due to a lack
> of activity and that the contributor is more than welcome to open a new PR
> if necessary).
>
> I feel like this gives contributors enough time to address concerns and it
> is simple enough
> to create a new Pull Request if the need arises. Alternatively, if the
> contributor realizes that
> they need more time, they can simply comment on the PR that they are still
> interested in
> working on it but just need more time, and the simple act of commenting
> will mean that the
> PR is no longer stalled, as defined above.
>
> Any thoughts on such a proposal? Any better alternatives that people have
> in mind?
>
> Thanks
> -Mark

Re: [DISCUSS] Apache NiFi distribution has grown too large

[James Wing]

I think a reduced build is a good way forward until the extension registry
is ready.  If we can publish the remaining processors in one or more
additional artifacts, that would be ideal.  The admin burden of more git
repositories or separate releases does not appeal to me, especially since
we do not believe it to be our long-term path.

It's not going to be easy to decide on a "core" build with "extras" sold
separately. But we will have to confront the division for the registry
solution in any case, we might as well get started on it.

On Sun, Jan 14, 2018 at 1:37 PM, Mike Thomsen 
wrote:

> Since the limit was bumped to 1.6GB, it might be prudent to not do too much
> NiFi 1.X and instead focus on a comprehensive solution that coincides with
> 2.0. I think that would be a time when a lot of users might expect and be
> tolerant of breaking changes on issues like this.
>
> Also, is there a clear process for deprecating processors? If not, there
> should be because it would be really helpful for doing cleanup.
>
> On Sat, Jan 13, 2018 at 7:53 PM, Brett Ryan  wrote:
>
> > Why are core modules not listing everything as provided?
> >
> > IDE’s solve this problem with the use of dependency libraries. As an
> > example NetBeans nbm’s have a single purpose, you must export the
> packages
> > to be exposed.
> >
> > We do the same with confluence modules using felix.
> >
> > Why is NiFi doing things different just so the person who wants to
> install
> > many custom nars can be lazy?
> >
> > > On 14 Jan 2018, at 08:59, Tony Kurc  wrote:
> > >
> > > I added some more stats to the wiki page, trying to determine what
> > > dependencies are included in jars. It seems like there is opportunity.
> > >
> > > Highlights, 50 copies of what appears to be some version of
> bcprov-jdk15
> > > for a total of 162M. 51 copies of jackson-databind.
> > >
> > > total size   copies  jar
> > > 30.97MB 65 META-INF/bundled-dependencies/
> > commons-lang3-XXX.jar
> > > 32.53MB 50 META-INF/bundled-dependencies/
> > bcpkix-jdk15on-XXX.jar
> > > 33.55MB 16 META-INF/bundled-dependencies/guava-XXX.jar
> > > 39.62MB  1 META-INF/bundled-dependencies/
> > jython-shaded-XXX.jar
> > > 63.06MB 51
> > > META-INF/bundled-dependencies/jackson-databind-XXX.jar
> > >162.07MB 50 META-INF/bundled-dependencies/
> > bcprov-jdk15on-XXX.jar
> > >
> > >
> > >> On Sat, Jan 13, 2018 at 2:09 PM, Joey Frazee 
> > wrote:
> > >>
> > >> I tend to have feelings similar to Michael about a multi-repo
> approach.
> > >> I’ve rarely seen it help and more often seen it hurt — it’s confusing
> > >> (especially to newcomers), stuff gets neglected because it’s easier to
> > >> ignore, you need another master project or some such to do an entire
> > build.
> > >>
> > >> Maybe git submodules could help mitigate this, but creating
> independent
> > >> assemblies or using different build profiles to enable building and
> > >> packaging the binaries in different ways would satisfy everything
> except
> > >> disentangling the releases.
> > >>
> > >> -joey
> > >>
> > >>> On Jan 13, 2018, 12:40 PM -0600, Brandon DeVries ,
> wrote:
> > >>> I agree... Long term extension registry, short term one repo with
> > >> different
> > >>> assemblies (e.g. standard, slim, analytic, etc...).
> > >>>
> > >>> Brandon
> > >>>
> > >>> On Sat, Jan 13, 2018 at 1:35 PM Pierre Villard <
> > >> pierre.villard...@gmail.com
> > >>> wrote:
> > >>>
> >  Option #3 also has my preference. But it's probably a good idea to
> > only
> >  keep one git repo and play with the assembly and Maven profiles for
> > the
> >  releases, no? It'd be certainly easier for release management
> process.
> > >> But
> >  this decision could also depend on how the option #3 is going to be
> >  implemented I guess.
> > 
> >  2018-01-13 6:36 GMT-07:00 Joe Witt :
> > 
> > > thanks tony!
> > >
> > >> On Jan 12, 2018 10:48 PM, "Tony Kurc"  wrote:
> > >>
> > >> I put some of the data I was working with on the wiki -
> > >>
> > >> https://cwiki.apache.org/confluence/display/NIFI/NiFi+
> > >> 1.5.0+nar+files
> > >>
> > >> On Fri, Jan 12, 2018 at 10:28 PM, Jeremy Dyer  >  wrote:
> > >>
> > >>> So my favorite option is Bryan’s option number “three” of using
> > >> the
> > >>> extension registry. Now my thought is do we really need to add
> > > complexity
> > >>> and do anything in the mean time or just focus on that? Meaning
> > >> we
> >  have
> > >>> roughly 500mb of available capacity today so why don’t we spend
> > >> those
> > > man
> > >>> hours we would spend on getting the second repo up on the
> > >> extension
> > >>> registry instead?
> > >>>
> > >>> @Bryan do you have thoughts about the 

Re: [VOTE] Release Apache NiFi 1.5.0 (RC1)

[James Wing]

+1 (binding).  Ran through the release helper, checked the signature,
hashes, the license/notice changes, etc.  Tested the binary for a while.

Great work!

On Tue, Jan 9, 2018 at 2:19 AM, Joe Witt  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.5.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1116
>
> The Git tag is nifi-1.5.0-RC1
> The Git commit ID is 46d30c7e92f0ad034d9b35bf1d05c350ab5547ed
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 46d30c7e92f0ad034d9b35bf1d05c350ab5547ed
>
> Checksums of nifi-1.5.0-source-release.zip:
> MD5: 046f2dde4af592dd8c05e55c2bbb3c4f
> SHA1: 63b9a68b9f89200fd31f5561956a15b45b1b9c8c
> SHA256: 40b155c4911414907835f2eb0d5a4da798935f27f1e5134218d904fe6c942d13
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 195 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12341668
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.5.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.5.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi Registry 0.1.0

[James Wing]

+1 (binding).  Ran through the release helper, verified hashes, LICENSE,
NOTICE, and README.  Built the source and performed light testing of the
resulting binary.

On Thu, Dec 28, 2017 at 10:09 AM, Bryan Bende  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi Registry 0.1.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1115/
>
> The Git tag is nifi-registry-0.1.0-RC1
> The Git commit ID is 81b99e7b04491eabb72ddf30754053ca12d0fcca
> https://git-wip-us.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=
> 81b99e7b04491eabb72ddf30754053ca12d0fcca
>
> Checksums of nifi-registry-0.1.0-source-release.zip:
> MD5: 56244c3c296cdc9c3fcc6d22590b80d1
> SHA1: 6354e91f868f40d6656ec2467bde307260ad63ca
> SHA256: 2c680e441e6c4bfa2381bf004e9b19a6a79401a6a83e04597d0a714a95efd301
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/bbende.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 65 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12320920=12340217
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-NiFiRegistry0.1.0
>
> The vote will be open for 96 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
>
> The please vote:
>
> [ ] +1 Release this package as nifi-registry-0.1.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: hello

[James Wing]

A great place to start is the NiFi Development Guide (
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html).  The
Development Guide explains how the NiFi API works, and the extension
components you may develop to customize NiFi.

If you prefer to learn from examples, there are many processors included in
the NiFi project, and it is likely that one or more of them do something
similar to your goals.  You will find the processors under the
nifi-nar-bundles subproject at
https://github.com/apache/nifi/tree/master/nifi-nar-bundles.

Last, I recommend starting with a scripted component using the
ExecuteScript processor.  ExecuteScript is a built-in NiFi processor that
executes your custom logic written in a scripting language like Groovy,
Javascript, Python, etc.  It is extremely helpful to prototype your
processor behavior, and understand how it should behave within a flow
before investing in Java code.  You may find that scripting is good
enough.  There are a lot of articles about scripted processors, but you
might start with
http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
.

Thanks,

James

On Fri, Nov 10, 2017 at 10:48 PM, 时间煮鱼 <1976574...@qq.com> wrote:

> Dear sir: I come from China.I would like to kown how to develop my own
> processor.
>
> Yours sincerely,
>
>Chen Yu.

Re: Confluence Wiki Access

[James Wing]

Thank you, Aldrin, that seems to have worked.

On Thu, Oct 26, 2017 at 9:12 PM, Aldrin Piri <aldrinp...@gmail.com> wrote:

> Hi James,
>
> You should be good to go now.  Please let me know if that is not the case.
>
> On Thu, Oct 26, 2017 at 11:51 PM, James Wing <jvw...@gmail.com> wrote:
>
> > May I please have edit access to the Confluence Wiki?
> >
> > Thanks,
> >
> > James
> >
>

Confluence Wiki Access

[James Wing]

May I please have edit access to the Confluence Wiki?

Thanks,

James

Re: [DRAFT][REPORT] Apache NiFi Board Report Oct 2017

[James Wing]

Thanks, Joe. It looks good to me, too.

On Mon, Oct 9, 2017 at 7:12 AM, Joe Witt  wrote:

> Team,
>
> Please see draft board report.  Am looking for feedback/edits quickly
> as this needs to be submitted by Wed.  I'll probably send it in
> tonight though as I might not have time over the next couple of days.
>
> Thanks
> Joe
>
> **==**==**==**==**
>
> ## Description:
>  - Apache NiFi is an easy to use, powerful, and reliable system to
>process and distribute data.
>  - Apache NiFi MiNiFi is an edge data collection agent built to seamlessly
>integrate with and leverage the command and control of NiFi. There are
>both Java and C++ implementations.
>  - Apache NiFi Registry is a centralized registry for key configuration
> items
>including flow versions, assets, and extensions for Apache NiFi
>and Apache MiNiFi.
>  - Apache NiFi Nar Maven Plugin is a release artifact used for supporting
>the NiFi classloader isolation model.
>
> ## Issues:
>  - There are no issues requiring board attention at this time.
>
> ## Activity:
>  - Released Apache NiFi 1.4.0 which is another very heavy feature,
> stability,
>and bug fix release.  It includes improvements on the web UI coming
> from new
>contributors which is a great sign.
>  - The Apache NiFi downloads have been quite large and growing for some
> time.
>The groundwork to solve this is the Apache NiFi Registry effort which is
>coming along nicely and which the foundational elements of NiFi to
> integrate
>with it are present.  We're probably looking at two more releases for
> that
>to land and for which we can drop the NiFi build size down.
>  - We continue to hover around the 100 outstanding pull requests number.
> What
>is encouraging is we've seen an uptick in contributions both in code and
>review feedback from new community members.
>  - The Apache NiFi MiNiFi C++ agent has rapidly matured over the past
> couple of
>months and is serving as a new source for contributions and
> contributors.
>
> ## Health report:
>  - As stated in our past report, health of the community is strong and
>indicators of strength continue trending in the right direction. Mailing
>list and JIRA activity is strong. ASF Hipchat is serving as an on-ramp
> for
>new users to our mailing list and JIRA systems. We continue to see new
>users and contributors.
>  - The PMC and committer ranks did not grow during the reporting period.
> This
>is a reflection of the PMC not focusing enough on the topic though as
> we're
>seeing a very strong pipeline to both both committer and PMC. Expect
> both
>new committers and PMC members during our next reporting period.
>
> ## PMC changes:
>
>  - Currently 22 PMC members.
>  - Last PMC member added Fri May 26 2017.
>
> ## Committer base changes:
>
>  - Currently 34 committers.
>  - Last committer added Thu Jun 01 2017.
>
> ## Releases:
>
>  - Apache NiFi 1.4.0 was released Oct 1 2017.
>
> ## Mailing list activity:
>
>  - Activity on the mailing lists remains extremely high with a mixture
>of new users, contributors, and deeper more experienced users and
>contributors sparking discussion and questions and filing bugs or
>new features.
>
>  - us...@nifi.apache.org:
> - 553 subscribers (up 34 in the last 3 months):
> - 748 emails sent to list (862 in previous quarter)
>
>  - dev@nifi.apache.org:
> - 383 subscribers (up 12 in the last 3 months):
> - 604 emails sent to list (947 in previous quarter)
>
>  - iss...@nifi.apache.org:
> - 46 subscribers (up 4 in the last 3 months):
> - 5775 emails sent to list (7457 in previous quarter)
>
>
> ## JIRA activity:
>
>  - 335 JIRA tickets created in the last 3 months
>  - 263 JIRA tickets closed/resolved in the last 3 months
>

Re: Re: dose nifi suport Regular Expression capturing groups text using Nifi Expression Language

[James Wing]

This email list is usually helpful.  Can you provide some more detail about
the flow you are building?

On Sun, Oct 1, 2017 at 8:10 AM, YuNing <bel...@163.com> wrote:

>
> it realy helps me, thanks James. can you give some tips on how can i
> get more specific document about how to use a processor, as the doc on
> nifi-website is very simple.
>
>
>
> Best Regards
> YuNing
>
> From: James Wing
> Date: 2017-10-01 00:18
> To: NiFi Dev List
> Subject: Re: dose nifi suport Regular Expression capturing groups text
> using Nifi Expression Language
> I believe the Replacement Value field should be set to something like this:
>
> ${'$1':jsonPath("$.data.orgcode")}
>
> We may be able to help more if you can provide some sample content and
> regular expression you are using.
>
> Thanks,
>
> James
>
>
> On Fri, Sep 29, 2017 at 7:16 PM, YuNing <bel...@163.com> wrote:
>
> >
> > when i use replace text i want to use  :jsonPath('')  funciton for
> > Regular Expression capturing group; the Replacement Value   i set is
> > "$1:jsonPath('$.data.orgcode')", but it didn't work.
> > i didn't want  to use EvaluateJsonPath , because it couldn't evalue
> > mulit-line  json sting,can anyone help me.
> >
> > wait for your early replay!
> >
> >
> >
> >
> > Best Regards
> > YuNing
> >
>

Re: dose nifi suport Regular Expression capturing groups text using Nifi Expression Language

[James Wing]

I believe the Replacement Value field should be set to something like this:

${'$1':jsonPath("$.data.orgcode")}

We may be able to help more if you can provide some sample content and
regular expression you are using.

Thanks,

James


On Fri, Sep 29, 2017 at 7:16 PM, YuNing  wrote:

>
> when i use replace text i want to use  :jsonPath('')  funciton for
> Regular Expression capturing group; the Replacement Value   i set is
> "$1:jsonPath('$.data.orgcode')", but it didn't work.
> i didn't want  to use EvaluateJsonPath , because it couldn't evalue
> mulit-line  json sting,can anyone help me.
>
> wait for your early replay!
>
>
>
>
> Best Regards
> YuNing
>

Re: [VOTE] Release Apache NiFi 1.4.0 (RC2)

[James Wing]

Jeff, I agree the updated KEYS file has been published.  Thanks.

On Fri, Sep 29, 2017 at 6:00 AM, Jeff <jtsw...@gmail.com> wrote:

> James,
>
> I had to do a hard reload of the page in Chrome, since the browser kept
> showing me a cached version without my key.  After the hard reload, I can
> see my key at https://dist.apache.org/repos/dist/dev/nifi/KEYS.  Could you
> try opening the KEYS link in incognito mode and verify that my key is
> there?
>
> Thanks,
> Jeff
>
> On Fri, Sep 29, 2017 at 1:06 AM James Wing <jvw...@gmail.com> wrote:
>
> > +1 (binding). I ran through the release helper including signature,
> hashes,
> > build, and testing the binary.  I checked the LICENSE and NOTICE files.
> > Everything looks good to me.
> >
> > One thing I noted is that Jeff's GPG key is not yet in the public KEYS
> file
> > at https://dist.apache.org/repos/dist/dev/nifi/KEYS, but it is added in
> > the
> > master branch KEYS file to be published with the release.  I believe that
> > is OK for the signature, we've done this before, and perhaps we should
> > consider changing the helper text in the future.
> >
> > Thanks, Jeff, for putting this release together.
> >
> >
> > On Thu, Sep 28, 2017 at 12:54 PM, Jeff <jsto...@apache.org> wrote:
> >
> > > Hello,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > nifi-1.4.0.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-
> > >
> > > The Git tag is nifi-1.4.0-RC2
> > > The Git commit ID is e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
> > > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > > e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
> > >
> > > Checksums of nifi-1.4.0-source-release.zip:
> > > MD5: 41e4083e602883a3e180032f32913414
> > > SHA1: 26770625138126f45bed4989adb0a6b65a767aa2
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/jstorck.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 199 issues were closed/resolved for this release:
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > projectId=12316020=12340589
> > >
> > > Release note highlights can be found here:
> > > https://cwiki.apache.org/confluence/display/NIFI/
> > > Release+Notes#ReleaseNotes-Version-1.4.0
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build
> > > from source, and test.  The please vote:
> > >
> > > [ ] +1 Release this package as nifi-1.4.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> >
>

Re: [VOTE] Release Apache NiFi 1.4.0 (RC2)

[James Wing]

+1 (binding). I ran through the release helper including signature, hashes,
build, and testing the binary.  I checked the LICENSE and NOTICE files.
Everything looks good to me.

One thing I noted is that Jeff's GPG key is not yet in the public KEYS file
at https://dist.apache.org/repos/dist/dev/nifi/KEYS, but it is added in the
master branch KEYS file to be published with the release.  I believe that
is OK for the signature, we've done this before, and perhaps we should
consider changing the helper text in the future.

Thanks, Jeff, for putting this release together.


On Thu, Sep 28, 2017 at 12:54 PM, Jeff  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.4.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-
>
> The Git tag is nifi-1.4.0-RC2
> The Git commit ID is e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
>
> Checksums of nifi-1.4.0-source-release.zip:
> MD5: 41e4083e602883a3e180032f32913414
> SHA1: 26770625138126f45bed4989adb0a6b65a767aa2
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jstorck.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 199 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12340589
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version-1.4.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.4.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: Url encoding with S3 processors

[James Wing]

Joe,

The good news is that you can use expression language directly in
FetchS3Object's Object Key property to decode your keys.  For example,

${filename:urlDecode()}

However, I'm not sure that could be done without you specifying that the
key should be decoded. As I understand it, the following two keys:

file with space.txt
file+with+space.txt

are both valid, but distinctly separate keys. If we decoded by default, we
could create confusion with unexpected key changes.

Thanks,

James

On Thu, Sep 7, 2017 at 6:59 AM, Joe Gresock  wrote:

> Has anyone run into any URL encoding issues with the AWS nifi processors in
> version 1.3.0?  My flow has GetSQS and then FetchS3Object, along with the
> AWS feature of automated SQS events upon S3 file upload.  Some of my S3
> objects apparently have characters in their object keys that need to be URL
> encoded (e.g., %28 for open paren and + for space), and they come across
> encoded in the SQS message.  However, FetchS3Object apparently doesn't like
> this encoded URL, and it complains that the key does not exist.
>
> I suppose I could add an UpdateAttributes processor to translate certain
> urls in the keys, but this seems like a workaround.
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*
>

Re: Dynamic attributes on repeating capture groups

[James Wing]

ExecuteScript will certainly support a clustered NiFi environment.  Is the
processor actually processing files on both nodes, but only failing on
one?  Is it possible that the cluster's flow routes files through the
processor only on one node?

Also, can you share your script?  It sounds like you have an
InputStreamCallback not getting closed, or maybe an out of date flowfile
reference.

On Thu, Aug 3, 2017 at 6:54 AM, wildo  wrote:

> James- one last question. Does ExecuteScript support clustered NIFI
> instances? I keep getting this exception ONLY from the second node (2 of 2)
> in our cluster:
>
> "StandardFlowFileRecord ... already in use for active callback or an
> InputStream created by ProcessSession.read(FlowFile) has not been closed in
> 

Re: Dynamic attributes on repeating capture groups

[James Wing]

An exception thrown in your script and not caught will cause the process
session to roll back, meaning that the flowfile will be go back to the
input queue and retry.  If you have the processor set to run on schedule
every 0 seconds, it might retry a lot :).  You are correct that routing to
REL_FAILURE is usually better than retrying, but I suppose it depends on
the particular flow.

ExecuteScript provides a local script variable, 'log', which you can use to
log messages to the regular NiFi log.  The messages get sent to the same
log files, following the rules outlined in conf/logback.xml, and will show
up in the UI as bulletins if the severity deserves it.  So you can do

log.error("Something bad happpened")

And it will both log and display as you would expect a processor error to
display.  It is an instance of ComponentLog, so there are also debug(),
info(), warn(), etc.

James

On Wed, Aug 2, 2017 at 12:22 PM, wildo  wrote:

> James- thanks for the comment. I've been hacking on this all day and think
> I
> have the python written to do what I need. I have just one follow up
> question as I'm new to the ExecuteScript processor.
>
>
> -What do I do on exception? Some of the cookbook example scripts seem to
> print to traceback:
>
> except:
> traceback.print_exec(file=sys.stdout)
> raise
>
>
> ...but I'm curious what you should do in a production environment. Instead
> of raising the exception, should you instead transfer the session to
> REL_FAILURE? Or maybe raising the exception automatically transfers to
> REL_FAILURE?
>
> Also- I'm guessing that in the real world you'd probably write that
> exception to some logger, rather than stdout correct?
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-l
> ist.39713.n7.nabble.com/Dynamic-attributes-on-repeating-
> capture-groups-tp16540p16565.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Dynamic attributes on repeating capture groups

[James Wing]

I think you will need to use an ExecuteScript processor to nicely format
your data into attributes into the foo1=bar1 pattern.  I'm assuming here
that you cannot predict what the key names 'foo1', 'foo2', etc. will
actually be.  If you could predict those field names, an UpdateAttribute
processor with a brute-force list of keys might be ok if you can handle the
misses downstream using expressions with isEmpty() or something similar.

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-scripting-nar/1.3.0/org.apache.nifi.processors.script.ExecuteScript/index.html

Thanks,

James

On Mon, Jul 31, 2017 at 1:26 PM, wildo  wrote:

> I am parsing a text file with semi-formatted data. I use the SplitContent
> processor to get me each "block" of data- basically each object, and then
> the ExtractText processor to get all the individual properties that will be
> associated to that object.
>
> So I might have a flowfile with data:
>
> foo1: bar1
> foo2: bar2
> foo3: bar3
>
> When I push that flowfile into the ExtractText processor (which has "Enable
> repeating capture group" set to true) and run my capture group regex on it,
> I'm left with dynamic, numerated attributes. If I call my dynamic attribute
> "fields" then my flowfile attrs will have:
>
> fields.1 --> foo1
> fields.2 --> bar1
> fields.3 --> foo2
> fields.4 --> bar2
> fields.5 --> foo3
> fields.6 --> bar3
>
> Now I'd like to push this flowfile into an UpdateAttribute and turn my
> dynamic list into real, accessible attributes. The intention would then be
> to go into an AttributesToJSON processor.
>
> The question is: how do I configure the UpdateAttribute processor to have a
> new attribute of "foo1" using "fields.n" with a value of "bar1" using
> "field.n+1" and then dynamically add the other five as well?
>
> It seems like the ability to store repeating capture groups as attributes
> must imply that theirs a way to actually USE those attributes dynamically.
> What am I missing here?
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Dynamic-attributes-on-repeating-capture-groups-
> tp16540.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Expression Language support in AmazonS3

[James Wing]

The processors were written by different developers, at different times,
with different use cases in mind.  The end result is different approaches
to creating storage service client objects and managing credentials.
Although the current S3 processors are not designed with per-flowfile
expressions in mind, it would be possible to modify them to follow a
pattern more like the Azure Blob processors.

Thanks,

James

On Mon, Jul 17, 2017 at 2:38 AM, Venkateswara Rachapudi <
venkateswara.rachap...@eaiesb.com> wrote:

> Hello,
> I saw that  Access Key and Secret key does not support expression language
> https://stackoverflow.com/questions/43646105/how-to-
> configure-nifi-puts3object-processor-to-use-flow-attributes-for-s3-creden
>
> My question is why AmazonS3 Processor doesn’t support expression language
> where as I have configured the Access key and Secret Key using Expression
> Language for Azure Blob in NIFI.
>
>

Re: Start Contributing

[James Wing]

Grant,

Welcome to NiFi, and thanks for being willing to contribute!

Wes is right, you can go ahead and submit a pull request (or patch) with
your proposed changes.  You don't need to wait for the JIRA to be
reviewed.

Thanks,

James

On Wed, Jul 12, 2017 at 8:26 PM, Wes Lawrence  wrote:

> Hey Grant.
>
> I'm also new to this, but I recommend writing the solution and providing a
> GitHub PR and/or patch.
>
> I've only fixed some minor bugs I've found, and only today added a very
> minor feature, but I've found that approaching with a solution is the best
> approach.
>
> After all, if you were in charge of something, what do you prefer; I person
> approaching you and saying "This needs fixing", or "This needs fixing, and
> he's a potential fix".
>
> Worst case, if the fix isn't what the commiters/PMC has is mind, providing
> some solution gets the ball rolling on the problem.
>
> --Wes
>
> On Wed, Jul 12, 2017 at 11:10 PM, Grant Langlois <
> grant.r.langl...@gmail.com
> > wrote:
>
> > Hello all,
> >
> > First and foremost, my apologies if this message breaks protocol, this is
> > my first time hitting a dev mailing list.
> >
> > I'm interested in contributing to this project and have a specific
> feature
> > in mind to tackle first. I read through the developer's guide which
> > recommended first submitting a Jira (which I did here
> > .)
> >
> > My question is what happens next? Should I wait for someone to comment on
> > it as to whether or not it's a feasible addition? I'm more than willing
> to
> > write the code, assuming it's an acceptable addition.
> >
> > Thanks for your patience in helping out a newbie.
> >
> > Grant
> >
>

Re: RTC clarification

[James Wing]

We should definitely encourage review feedback from non-committers.
Getting additional perspectives, interest, and enthusiasm from users is
critical for any project, doubly so for an integrating application where
committers cannot be experts in all the systems we are integrating with.  I
believe NiFi could use more review bandwidth.  Are missing out on potential
reviewers because of the current policy?

I do not have any experience with non-committer "binding reviews" as
described in the Apache Gossip thread.  How would that work?  Wouldn't a
committer have to review the review and decide to commit?  If we knew the
reviewer well enough to accept their judgement, why not make them a
committer?

My expectation is that many non-committer reviews are helpful and
constructive, but not necessarily 100% comprehensive.  Reviewers might
comment on the JIRA ticket without working with the PR, or try the proposed
change without reviewing the code, tests, etc.  All great stuff, but
backstopped by committers.

Thanks,

James

On Thu, Jul 6, 2017 at 7:30 PM, Joe Witt  wrote:

> It is undefined at this point and I agree we should reach consensus
> and document it.
>
> I am in favor making non-committer reviews binding.
>
> Why do we do RTC:
> - To help bring along new committers/grow the community
> - To help promote quality by having peer reviews
>
> Enabling non-committer reviews to be binding still allows both of
> those to be true.
>
> On Thu, Jul 6, 2017 at 10:10 PM, Tony Kurc  wrote:
> > All, I was having a discussion with Mike Hogue - a recent contributor -
> off
> > list, and he had some questions about the review process. And largely the
> > root of the question was, if a non-committer reviews a patch or PR (which
> > Mike has spent some time doing), is that considered a "review"? I didn't
> > have the answers, so I went on a hunt for documentation. I started with
> the
> > Contributor Guide [1]. The guide describes reviewing, and calls out a
> > Reviewer role, but doesn't specifically point out that Reviewer is a
> > committer, just that a committer "can actively promote contributions into
> > the repository", and goes on to imply the non-committers can review.
> >
> > Given this, I was unable to answer this question:
> > If a committer "X" submits a patch or PR, it is reviewed by a
> non-committer
> > "Y", does that review satisfy the RTC requirement, and "X" may merge in
> the
> > patch?
> >
> > I found a related discussion on the email list in March [2], which I
> think
> > implies at least some of the community assumed the canonical review must
> be
> > by a committer. I also went back a bit to early days [3], where Benson
> > implied a much less "formal" review process.
> >
> > What I'm hoping for is hopefully come to a consensus of what our project
> > expectations are and document in the Contributor Guide. My expectation
> was
> > that non-committers could review, similar to what Sean discussed on this
> > thread for Apache Gossip (incubating) [4]. However, I don't believe this
> is
> > the current consensus.
> >
> > Thoughts?
> >
> > Tony
> >
> > 1.
> > https://cwiki.apache.org/confluence/display/NIFI/Contributor
> +Guide#ContributorGuide-CodeReviewProcess
> > 2.
> > https://mail-archives.apache.org/mod_mbox/nifi-dev/201703.mb
> ox/%3CCALJK9a7onOKo%3DXtAEmL7PSBUEBEqOOhcT9WSz-RZaNxqV6q%
> 3Dmg%40mail.gmail.com%3E
> > 3.
> > https://mail-archives.apache.org/mod_mbox/nifi-dev/201501.mb
> ox/%3CCALhtWkf67UpOe9W9HQ3RSn00xb_=C6ZMxN+_SEfWthpQrU5fsg@
> mail.gmail.com%3E
> > 4.
> > http://mail-archives.apache.org/mod_mbox/incubator-gossip-de
> v/201606.mbox/%3CCAN5cbe6P8aEEOLMA%2BPrfpQg9c_AWeSfvvmom8jAp%3Dk7wUpoVgQ%
> 40mail.gmail.com%3E
>

Re: File Filter regular expression - GetFile

[James Wing]

The .* in your regular expression is allowing all characters.  You can
escape this and specify a literal dot with a backslash, like \.

Try [ab]+\..* for example.


On Wed, Jun 28, 2017 at 2:55 AM, shankhamajumdar <
shankha.majum...@lexmark.com> wrote:

> Hi,
>
> Just a small modification, I am using ListFile processor. For example in
> the
> directory I have 5 files which are
> test.txt, ab.txt, ab1.txt. If I use [ab].* in the regular expression I am
> getting both the files i.e. ab.txt and ab1.txt. But I just want to get
> ab.txt not ab1.txt. Please let me know how to resolve this.
>
> Regards,
> Shankha
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/File-Filter-regular-expression-
> GetFile-tp16279p16280.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: [VOTE] Release Apache NiFi 0.7.4

[James Wing]

+1 (binding).  Signature and hashes verified.  License, notice, and readme
look good.  Build succeeded.  Testing with a basic flow succeeded.

Thanks again for the release.

James

On Mon, Jun 5, 2017 at 1:12 PM, Matt Gilman  wrote:

> Hello,
>
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-0.7.4.
>
>
> The source zip, including signatures, digests, etc. can be found at:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1109
>
>
> The Git tag is nifi-0.7.4-RC1
>
> The Git commit ID is aafa9dc025d4a750dac1b02447ff8cedb8e78827
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> aafa9dc025d4a750dac1b02447ff8cedb8e78827
>
>
> Checksums of nifi-0.7.4-source-release.zip:
>
> MD5: 7f901538cfc169b6a08afbf7f8c8dea6
>
> SHA1: 7123a2ebe74c432769fd02644eb6ea3f3bae6da8
>
> SHA256: 5bc91fd13d3bcdfc7e949a9fa2cff85c26821e20425b1466484d2f0602bfbbff
>
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/mcgilman.asc
>
>
> KEYS file available here:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
>
> 3 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12340590
>
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version0.7.4
>
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.  The
> please vote:
>
>
> [ ] +1 Release this package as nifi-0.7.4
>
> [ ] +0 no opinion
>
> [ ] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.3.0

[James Wing]

+1 (binding).  I ran through the release helper -- verified signature and
hashes, ran the full build, checked readme, license, and notice in source
and output.  I ran a simple flow and verified that it still works.

Thanks for putting this release together, Matt.

James


On Mon, Jun 5, 2017 at 10:54 AM, Matt Gilman  wrote:

> Hello,
>
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.3.0.
>
>
> The source zip, including signatures, digests, etc. can be found at:
>
> https://repository.apache.org/content/repositories/orgapachenifi-1108
>
>
> The Git tag is nifi-1.3.0-RC1
>
> The Git commit ID is ddb73612bd1512d8b2151b81f9aa40811bca2aaa
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> ddb73612bd1512d8b2151b81f9aa40811bca2aaa
>
>
> Checksums of nifi-1.3.0-source-release.zip:
>
> MD5: 8b115682ac392342b9edff3bf0658ecb
>
> SHA1: f11cdebbabdc0d8f1f0dd4c5b880ded39d17f234
>
> SHA256: 9ba5565729d98c472c31a1fdbc44e9dc1eee87a2cf5184e8428743f753145b7f
>
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/mcgilman.asc
>
>
> KEYS file available here:
>
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
>
> 110 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12340498
>
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.3.0
>
>
> The vote will be open for 72 hours.
>
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.  The
> please vote:
>
>
> [ ] +1 Release this package as nifi-1.3.0
>
> [ ] +0 no opinion
>
> [ ] -1 Do not release this package because...
>

Re: Connecting Nifi 1.2.0 with Google Cloud Storage issue

[James Wing]

Martin,

If you ran "gsutil ls" against the bucket from the instance using the
service account, is it able to list the bucket?  Can you describe a bit
more about how the service account's access to the bucket was configured?
Is it added to one of the project roles, explicitly granted on the bucket,
or something else?

Thanks,

James

On Fri, May 26, 2017 at 2:04 AM, Martin Eden 
wrote:

> Hi all,
>
> I have a Google Compute instance with a service account configured.
>
> I have installed Nifi 1.2.0 on it.
>
> In the Nifi UI I am describing a flow.
>
> The first processor is a ListGCSBucket processor. It is pointing to a
> bucket for which the service account is added as owner and I am not setting
> a Prefix.
>
> For the GCPCredentialsControllerService I have enabled, one by one all the
> possible properties
> Use Application Default Credentials
> Use Compute Engine Credentials
> Service Account JSON File
> Service Account JSON
>
> to no avail.
>
> In any configuration I get the below.
>
> Any suggestions on how to make this work would be much appreciated,
> M
>
> 2017-05-26 06:59:59,882 ERROR [Timer-Driven Process Thread-3]
> o.a.n.p.gcp.storage.ListGCSBucket ListGCSBucket[id=xxx-xxx-xxx-xxx]
> ListGCSBucket[id=xxx-xxx-xxx-xxx] failed to process session due to
> com.google.cloud.storage.StorageException: Not Found: {}
>
> com.google.cloud.storage.StorageException: Not Found
>
> at
> com.google.cloud.storage.spi.DefaultStorageRpc.translate(
> DefaultStorageRpc.java:202)
>
> at
> com.google.cloud.storage.spi.DefaultStorageRpc.list(
> DefaultStorageRpc.java:294)
>
> at com.google.cloud.storage.StorageImpl$8.call(
> StorageImpl.java:297)
>
> at com.google.cloud.storage.StorageImpl$8.call(
> StorageImpl.java:294)
>
> at com.google.cloud.RetryHelper.doRetry(RetryHelper.java:179)
>
> at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.
> java:244)
>
> at
> com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:293)
>
> at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:260)
>
> at
> org.apache.nifi.processors.gcp.storage.ListGCSBucket.
> onTrigger(ListGCSBucket.java:262)
>
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
>
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1118)
>
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:144)
>
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:47)
>
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> TimerDrivenSchedulingAgent.java:132)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:294)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by:
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not
> Found
>
> {
>
>   "code" : 404,
>
>   "errors" : [ {
>
> "domain" : "global",
>
> "message" : "Not Found",
>
> "reason" : "notFound"
>
>   } ],
>
>   "message" : "Not Found"
>
> }
>
> at
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(
> GoogleJsonResponseException.java:146)
>
> at
> com.google.api.client.googleapis.services.json.
> AbstractGoogleJsonClientRequest.newExceptionOnError(
> AbstractGoogleJsonClientRequest.java:113)
>
> at
> com.google.api.client.googleapis.services.json.
> AbstractGoogleJsonClientRequest.newExceptionOnError(
> AbstractGoogleJsonClientRequest.java:40)
>
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.
> interceptResponse(AbstractGoogleClientRequest.java:321)
>
> at
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1049)
>
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.
> executeUnparsed(AbstractGoogleClientRequest.java:419)
>
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.
> executeUnparsed(AbstractGoogleClientRequest.java:352)
>
> at
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.
> execute(AbstractGoogleClientRequest.java:469)
>
> at
> com.google.cloud.storage.spi.DefaultStorageRpc.list(
> 

Re: Duplicated processors when using nifi processors dependency

[James Wing]

I created a JIRA for the nifi-aws-bundle structure,
https://issues.apache.org/jira/browse/NIFI-3950.

Is there a best practice for the abstract classes like Robert describes?
Are they a better fit for the service API bundle or the processors bundle?
Or does that become irrelevant once the NAR dependencies are aligned and
they can be references as Java libraries?

Thanks,

James

On Sun, May 21, 2017 at 10:28 AM, Bryan Bende  wrote:

> Currently the AWS NAR contains the processors, controller service API, and
> controller service implementation, all in one NAR.
>
> Typically the controller service API should be in its own NAR, so there
> should be something like:
>
> nifi-aws-bunde
> nifi-aws-service-api
> nifi-aws-service-api-nar
> nifi-aws-processors
> nifi-aws-nar
>
> The nifi-aws-service-api module would have the
> AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR
> dependency on nifi-aws-service-api-nar.
>
> Setting it up this way, you nifi-aws-utils project could have a provided
> dependency on the nifi-aws-service-api, and then your custom NAR could have
> a NAR dependency on nifi-aws-service-api-nar.
>
> There is some more information here about setting up controller services:
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+
> Projects+for+Extensions#MavenProjectsforExtensions-
> LinkingProcessorsandControllerServices  confluence/display/NIFI/Maven+Projects+for+Extensions#
> MavenProjectsforExtensions-LinkingProcessorsandControllerServices>
>
> https://github.com/bbende/nifi-dependency-example <
> https://github.com/bbende/nifi-dependency-example>
>
>
> > On May 20, 2017, at 10:06 AM, Robert  wrote:
> >
> > I tired to do move abstract aws processors from
> > nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created
> > nifi-aws-utils.
> >
> > Problem that I face now is with AWSCredentialsProviderControllerService
> that
> > imo should stay under nifi-aws-bundle, but it is required by
> > AbstractAWSCredentialsProviderProcessor that should be moved to
> > nifi-aws-utils as it is used by abstract aws processors.
> >
> > What is the preferred way to solve this?
> > Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and
> > both nifi-aws-processors and nifi-aws-utils can depend on it?
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-
> dependency-tp15902p15937.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
>

Re: [VOTE] Release Apache NiFi 0.7.3

[James Wing]

+1 (binding), to be clearer

On Mon, May 15, 2017 at 1:08 PM, James Wing <jvw...@gmail.com> wrote:

> +1 - Ran through the release helper, verified hashes and signature (with
> provided mosermw.asc key).  Full build succeeded.  Ran a simple test flow.
> License, notice, readme all look good.
>
> Thanks for putting this release together.
>
> James
>
> On Sun, May 14, 2017 at 5:33 PM, Michael Moser <mose...@apache.org> wrote:
>
>> Hello Apache NiFi community,
>>
>> I am pleased to be calling this vote for the source release of Apache
>> NiFi nifi-0.7.3.
>>
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1105
>>
>> The Git tag is nifi-0.7.3-RC1
>> The Git commit ID is 78532ef44fb8b830a5316ce10a5cc7862bc3b5b2
>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;
>> h=78532ef44fb8b830a5316ce10a5cc7862bc3b5b2
>>
>> Checksums of nifi-0.7.3-source-release.zip:
>> MD5: bcd2b37ca38a15ed181879e0b6fa9ba9
>> SHA1: 68cae664b54d49c046a2c8e41bdafca404084acf
>> SHA256: 7d6d9bae4ccae9a4d249fc7830b5abed1ecdd4f2699f846f1829f306cda0
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/mosermw.asc
>>
>> KEYS file available here (NOTE: the key for mosermw is not yet in the
>> KEYS file):
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>
>> 29 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>> ctId=12316020=12340527
>>
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/Release+
>> Notes#ReleaseNotes-Version0.7.3
>>
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build
>> from source, and test.  The please vote:
>>
>> [ ] +1 Release this package as nifi-${NIFI_VERSION}
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because because...
>>
>
>

Re: Apache NiFi 0.7.3 RC1 Release Helper Guide

[James Wing]

I think we're good.  We need his signing key is in the KEYS file at the
time we publish the release.  It sounds like we're on track for that - his
key is checked into the git KEYS file, and someone has been enlisted to
publish the KEYS file.  I'm sorry to promote confusion, I wasn't sure what
the process for that was, or if it had been started.

Thanks,

James

On Mon, May 15, 2017 at 2:53 PM, Tony Kurc <trk...@gmail.com> wrote:

> Joe, James, Mike,
> I think I got confused. Do we need to get Mike's gpg public key which he
> used to sign in that KEYS file, or is the key Mike put in the helper
> sufficient? If it isn't sufficient, we need someone with access to svn to
> add Mike, correct?
>
> Tony
>
> On Mon, May 15, 2017 at 4:34 PM, Michael Moser <moser...@gmail.com> wrote:
>
> > Yeah, I didn't have write permission to SVN
> > https://dist.apache.org/repos/dist/ which I assumed was because I'm a
> > committer and not on the PMC.  I've enlisted a PMC member's help to
> > put the release up there if this vote passes.
> >
> >
> > On Mon, May 15, 2017 at 3:13 PM, Joe Witt <joe.w...@gmail.com> wrote:
> > > just need to update that dist key file is all.  I think mike did
> > > update the git entry.
> > >
> > > On Mon, May 15, 2017 at 3:12 PM, James Wing <jvw...@gmail.com> wrote:
> > >> Michael,
> > >>
> > >> What's the plan for the PGP key distribution, or how do we get your
> key
> > >> into the KEYS file?
> > >>
> > >> Thanks,
> > >>
> > >> James
> > >>
> > >> On Sun, May 14, 2017 at 5:36 PM, Michael Moser <mose...@apache.org>
> > wrote:
> > >>
> > >>> Hello Apache NiFi community,
> > >>>
> > >>> Please find the associated guidance to help those interested in
> > >>> validating/verifying the 0.7.3 release so they can vote.
> > >>>
> > >>> # Download latest KEYS file:
> > >>> https://dist.apache.org/repos/dist/dev/nifi/KEYS
> > >>>
> > >>> # Download the key used to sign this release:
> > >>> https://people.apache.org/keys/committer/mosermw.asc
> > >>>
> > >>> # Import keys file:
> > >>> gpg --import KEYS
> > >>>
> > >>> # Import key used to sign this release:
> > >>> gpg --import mosermw.asc
> > >>>
> > >>> # [optional] Clear out local maven artifact repository
> > >>>
> > >>> # Pull down nifi-0.7.3 source release artifacts for review:
> > >>> wget https://repository.apache.org/content/repositories/
> > >>> orgapachenifi-1105/org/apache/nifi/nifi/0.7.3/nifi-0.7.3-
> > >>> source-release.zip
> > >>> wget https://repository.apache.org/content/repositories/
> > >>> orgapachenifi-1105/org/apache/nifi/nifi/0.7.3/nifi-0.7.3-
> > >>> source-release.zip.asc
> > >>> wget https://repository.apache.org/content/repositories/
> > >>> orgapachenifi-1105/org/apache/nifi/nifi/0.7.3/nifi-0.7.3-
> > >>> source-release.zip.md5
> > >>> wget https://repository.apache.org/content/repositories/
> > >>> orgapachenifi-1105/org/apache/nifi/nifi/0.7.3/nifi-0.7.3-
> > >>> source-release.zip.sha1
> > >>>
> > >>> # Verify the signature
> > >>> gpg --verify nifi-0.7.3-source-release.zip.asc
> > >>>
> > >>> # Verify the hashes (md5, sha1, sha256) match the source and what was
> > >>> provided in the vote email thread
> > >>> # NOTE: the repository does not have the
> > >>> nifi-0.7.3-source-release.zip.sha256 file, please find that hash in
> > >>> the vote email thread
> > >>> md5sum nifi-0.7.3-source-release.zip
> > >>> sha1sum nifi-0.7.3-source-release.zip
> > >>> sha256sum nifi-0.7.3-source-release.zip
> > >>>
> > >>> # Unzip nifi-0.7.3-source-release.zip
> > >>>
> > >>> # Verify the build works including release audit tool (RAT) checks
> > >>> cd nifi-0.7.3
> > >>> mvn clean install -Pcontrib-check
> > >>>
> > >>> # Verify the contents contain a good README, NOTICE, and LICENSE.
> > >>>
> > >>> # Verify the git commit ID is correct
> > >>>
> > >>> # Verify the RC was branched off the correct git commit ID
> > >>>
> > >>> # Look at the resulting convenience binary as found in
> > nifi-assembly/target
> > >>>
> > >>> # Make sure the README, NOTICE, and LICENSE are present and correct
> > >>>
> > >>> # Run the resulting convenience binary and make sure it works as
> > expected
> > >>>
> > >>> # Send a response to the vote thread indicating a +1, 0, -1 based on
> > >>> your findings.
> > >>>
> > >>> Thank you for your time and effort to validate the release!
> > >>>
> >
>

Re: [VOTE] Release Apache NiFi 0.7.3

[James Wing]

+1 - Ran through the release helper, verified hashes and signature (with
provided mosermw.asc key).  Full build succeeded.  Ran a simple test flow.
License, notice, readme all look good.

Thanks for putting this release together.

James

On Sun, May 14, 2017 at 5:33 PM, Michael Moser  wrote:

> Hello Apache NiFi community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-0.7.3.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1105
>
> The Git tag is nifi-0.7.3-RC1
> The Git commit ID is 78532ef44fb8b830a5316ce10a5cc7862bc3b5b2
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 78532ef44fb8b830a5316ce10a5cc7862bc3b5b2
>
> Checksums of nifi-0.7.3-source-release.zip:
> MD5: bcd2b37ca38a15ed181879e0b6fa9ba9
> SHA1: 68cae664b54d49c046a2c8e41bdafca404084acf
> SHA256: 7d6d9bae4ccae9a4d249fc7830b5abed1ecdd4f2699f846f1829f306cda0
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/mosermw.asc
>
> KEYS file available here (NOTE: the key for mosermw is not yet in the
> KEYS file):
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 29 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12340527
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version0.7.3
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-${NIFI_VERSION}
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...
>

Re: Apache NiFi 0.7.3 RC1 Release Helper Guide

[James Wing]

Michael,

What's the plan for the PGP key distribution, or how do we get your key
into the KEYS file?

Thanks,

James

On Sun, May 14, 2017 at 5:36 PM, Michael Moser  wrote:

> Hello Apache NiFi community,
>
> Please find the associated guidance to help those interested in
> validating/verifying the 0.7.3 release so they can vote.
>
> # Download latest KEYS file:
> https://dist.apache.org/repos/dist/dev/nifi/KEYS
>
> # Download the key used to sign this release:
> https://people.apache.org/keys/committer/mosermw.asc
>
> # Import keys file:
> gpg --import KEYS
>
> # Import key used to sign this release:
> gpg --import mosermw.asc
>
> # [optional] Clear out local maven artifact repository
>
> # Pull down nifi-0.7.3 source release artifacts for review:
> wget https://repository.apache.org/content/repositories/
> orgapachenifi-1105/org/apache/nifi/nifi/0.7.3/nifi-0.7.3-
> source-release.zip
> wget https://repository.apache.org/content/repositories/
> orgapachenifi-1105/org/apache/nifi/nifi/0.7.3/nifi-0.7.3-
> source-release.zip.asc
> wget https://repository.apache.org/content/repositories/
> orgapachenifi-1105/org/apache/nifi/nifi/0.7.3/nifi-0.7.3-
> source-release.zip.md5
> wget https://repository.apache.org/content/repositories/
> orgapachenifi-1105/org/apache/nifi/nifi/0.7.3/nifi-0.7.3-
> source-release.zip.sha1
>
> # Verify the signature
> gpg --verify nifi-0.7.3-source-release.zip.asc
>
> # Verify the hashes (md5, sha1, sha256) match the source and what was
> provided in the vote email thread
> # NOTE: the repository does not have the
> nifi-0.7.3-source-release.zip.sha256 file, please find that hash in
> the vote email thread
> md5sum nifi-0.7.3-source-release.zip
> sha1sum nifi-0.7.3-source-release.zip
> sha256sum nifi-0.7.3-source-release.zip
>
> # Unzip nifi-0.7.3-source-release.zip
>
> # Verify the build works including release audit tool (RAT) checks
> cd nifi-0.7.3
> mvn clean install -Pcontrib-check
>
> # Verify the contents contain a good README, NOTICE, and LICENSE.
>
> # Verify the git commit ID is correct
>
> # Verify the RC was branched off the correct git commit ID
>
> # Look at the resulting convenience binary as found in nifi-assembly/target
>
> # Make sure the README, NOTICE, and LICENSE are present and correct
>
> # Run the resulting convenience binary and make sure it works as expected
>
> # Send a response to the vote thread indicating a +1, 0, -1 based on
> your findings.
>
> Thank you for your time and effort to validate the release!
>

Re: [VOTE] Release Apache NiFi 1.2.0 (RC2)

[James Wing]

+1 (binding).  Ran through the release helper, checked hashes, did the full
build successfully.  Checked the license, notice, and readme.  Tested a
basic flow.

Thanks for putting this together, Bryan.

James

On Fri, May 5, 2017 at 7:07 PM, Bryan Bende  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.2.0 (RC2).
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1104
>
> The Git tag is nifi-1.2.0-RC2
> The Git commit ID is 3a605af8e0ac024fb0ba67262d49dab2727b2576
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 3a605af8e0ac024fb0ba67262d49dab2727b2576
>
> Checksums of nifi-1.2.0-source-release.zip:
> MD5: 90e298a9e23a9dab65358daddd8b5990
> SHA1: e138941f576bdb1dff17df6674c19ffae3ef6719
> SHA256: c18398800c435dabff9f45ec55a450ca78c3c1aec222aa295c0e2057912feeb6
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/bbende.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 381 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12338432
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.2.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.2.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...
>

Error building RPM on master

[James Wing]

I'm having trouble building an RPM on the latest master.  I used to do this
with the command:

mvn clean install -DskipTests -Prpm

But this fails at the nifi-assembly step with an error:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-assembly-plugin:2.5.2:single (make shared
resource) on project nifi-assembly: No formats specified in the execution
parameters or the assembly descriptor. -> [Help 1]

The same "mvn clean install -DskipTests -Prpm" works on the 0.x branch,
resulting in a successful build with an rpm in the nifi-assembly/target
tree.

Am I doing something dumb?

$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
2015-11-10T16:41:47+00:00)
Maven home: /opt/apache-maven-3.3.9
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.29.amzn1.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.9.20-10.30.amzn1.x86_64", arch: "amd64",
family: "unix"

Thanks,

James

Re: AWS processor credentials question

[James Wing]

The AWSCredentialsProviderControllerService provides Instance Profile
credentials via the Default Credentials Provider Chain.  No file
configuration required, just the controller service.  Is that not working?
Which version of NiFi do you have?

On Thu, Apr 13, 2017 at 9:04 AM, Joe Gresock <jgres...@gmail.com> wrote:

> My thought was to make the default value be set to "True" (to allow
> anonymous credentials), which would make the default behavior exactly the
> same as the current behavior.
>
> We'd prefer to use the EC2 instance profile instead of having the
> controller service, since it relies on a file sitting on the file system
> and requires you to manage rotating the keys.  With EC2 instance profiles,
> they manage the keys for you and it's theoretically more secure.
>
>
>
> On Thu, Apr 13, 2017 at 11:41 AM, James Wing <jvw...@gmail.com> wrote:
>
> > Joe,
> >
> > Does the AWSCredentialsProviderControllerService work for you?  It can
> > provide default credentials.
> >
> > I agree with you that it would be better if individual processors used
> > "Default Credentials" as their unconfigured default rather than
> "Anonymous
> > Credentials".  The difficulty here is that changing the unconfigured
> > default will theoretically change the behavior of any flow that upgrades,
> > without notice.  I'm certainly skeptical about Anonymous credentials, if
> > anyone uses them, and if this is an actual problem for actual users.  But
> > it's a theoretical migration problem, possibly incompatible with a minor
> > release.
> >
> > Thanks,
> >
> > James
> >
> > On Thu, Apr 13, 2017 at 6:40 AM, Joe Gresock <jgres...@gmail.com> wrote:
> >
> > > Ok, I submitted a PR [1] for this ticket.  The change is only additive
> > and
> > > should be non-invasive to existing processors.  If one of you could
> > review
> > > it, it might even be nice to get into the 1.2.0 build.
> > >
> > > [1] https://github.com/apache/nifi/pull/1671
> > >
> > > On Thu, Apr 13, 2017 at 10:33 AM, Joe Gresock <jgres...@gmail.com>
> > wrote:
> > >
> > > > Actually it's a little more complicated than the code I suggested,
> > since
> > > > DefaultAWSCredentialsProviderChain.getInstance() is an
> > > > AWSCredentialsProvider, not an AWSCredentials object.  But the idea
> > would
> > > > be similar.
> > > >
> > > > On Thu, Apr 13, 2017 at 10:21 AM, Joe Gresock <jgres...@gmail.com>
> > > wrote:
> > > >
> > > >> AWS-related devs (I'm looking at James Wing and Adam Lamar),
> > > >>
> > > >> I just added [1] to propose allowing AWS processors to specify
> whether
> > > to
> > > >> use anonymous credentials or to use the default AWS client
> credentials
> > > >> provider chain.  The driver for this is that we would like to use
> EC2
> > > >> instance profiles with the NiFi AWS processors, but this is not
> > > currently
> > > >> possible because NiFi passes in AnonymousAWSCredentials to the AWS
> > > client
> > > >> constructor if no credentials are explicitly configured in the
> > > processor.
> > > >> If we use the DefaultAWSCredentialsProviderChain.getInstance(), I
> > think
> > > >> it would find our EC2 instance profile.
> > > >>
> > > >> My question to you is how you'd like to see this implemented.  Shall
> > we
> > > >> add a property to AbstractAWSProcessor called "Allow Anonymous
> > > >> Credentials"?  Then in AbstractAWSProcessor.
> > > getCredentials(ProcessContext
> > > >> context), the return statement could be:
> > > >>
> > > >> return allowAnonymousCredentials
> > > >>  ? new AnonymousAWSCredentials
> > > >>  : DefaultAWSCredentialsProviderChain.getInstance();
> > > >>
> > > >> I thought I'd ask here first, because I see many of the AWS methods
> > > >> marked Deprecated, and I didn't want to start adding code if it was
> > > going
> > > >> to go away soon.
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/NIFI-3703
> > > >>
> > > >> Thanks,
> > > >> Joe
> > > >>
> > > >> --
> > > >> I know what it is to be in need, and I know what it is to have
> plenty.
> > &g

Re: AWS processor credentials question

[James Wing]

Joe,

Does the AWSCredentialsProviderControllerService work for you?  It can
provide default credentials.

I agree with you that it would be better if individual processors used
"Default Credentials" as their unconfigured default rather than "Anonymous
Credentials".  The difficulty here is that changing the unconfigured
default will theoretically change the behavior of any flow that upgrades,
without notice.  I'm certainly skeptical about Anonymous credentials, if
anyone uses them, and if this is an actual problem for actual users.  But
it's a theoretical migration problem, possibly incompatible with a minor
release.

Thanks,

James

On Thu, Apr 13, 2017 at 6:40 AM, Joe Gresock <jgres...@gmail.com> wrote:

> Ok, I submitted a PR [1] for this ticket.  The change is only additive and
> should be non-invasive to existing processors.  If one of you could review
> it, it might even be nice to get into the 1.2.0 build.
>
> [1] https://github.com/apache/nifi/pull/1671
>
> On Thu, Apr 13, 2017 at 10:33 AM, Joe Gresock <jgres...@gmail.com> wrote:
>
> > Actually it's a little more complicated than the code I suggested, since
> > DefaultAWSCredentialsProviderChain.getInstance() is an
> > AWSCredentialsProvider, not an AWSCredentials object.  But the idea would
> > be similar.
> >
> > On Thu, Apr 13, 2017 at 10:21 AM, Joe Gresock <jgres...@gmail.com>
> wrote:
> >
> >> AWS-related devs (I'm looking at James Wing and Adam Lamar),
> >>
> >> I just added [1] to propose allowing AWS processors to specify whether
> to
> >> use anonymous credentials or to use the default AWS client credentials
> >> provider chain.  The driver for this is that we would like to use EC2
> >> instance profiles with the NiFi AWS processors, but this is not
> currently
> >> possible because NiFi passes in AnonymousAWSCredentials to the AWS
> client
> >> constructor if no credentials are explicitly configured in the
> processor.
> >> If we use the DefaultAWSCredentialsProviderChain.getInstance(), I think
> >> it would find our EC2 instance profile.
> >>
> >> My question to you is how you'd like to see this implemented.  Shall we
> >> add a property to AbstractAWSProcessor called "Allow Anonymous
> >> Credentials"?  Then in AbstractAWSProcessor.
> getCredentials(ProcessContext
> >> context), the return statement could be:
> >>
> >> return allowAnonymousCredentials
> >>  ? new AnonymousAWSCredentials
> >>  : DefaultAWSCredentialsProviderChain.getInstance();
> >>
> >> I thought I'd ask here first, because I see many of the AWS methods
> >> marked Deprecated, and I didn't want to start adding code if it was
> going
> >> to go away soon.
> >>
> >> [1] https://issues.apache.org/jira/browse/NIFI-3703
> >>
> >> Thanks,
> >> Joe
> >>
> >> --
> >> I know what it is to be in need, and I know what it is to have plenty.
> I
> >> have learned the secret of being content in any and every situation,
> >> whether well fed or hungry, whether living in plenty or in want.  I can
> >> do all this through him who gives me strength.*-Philippians 4:12-13*
> >>
> >
> >
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> > do all this through him who gives me strength.*-Philippians 4:12-13*
> >
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*
>

Re: Documentation for 1.x

[James Wing]

Mark,

Would you please open a JIRA ticket for known doc issues and feedback (
https://issues.apache.org/jira/browse/NIFI/)?  I do not know if anyone is
actively doing such a review at this minute, but the documentation is
frequently updated as contributions come in.

By the way, we are always looking for contributors :). Even if you don't
know the specific answers that should go into the documentation, sharing
your experiences of frustration are definitely valuable feedback for
improvements.  A good ticket is a great contribution on its own.

Thanks,

James

On Wed, Apr 12, 2017 at 8:54 AM, Mark Bean  wrote:

> I have been asking many questions lately regarding NiFi 1.x because we are
> in the process of more deliberately evaluating 1.x as it relates to our
> current production level flows. In the process, I have found the
> documentation to be incomplete, sometimes incorrect (e.g. typo), contain
> references to 0.x-specific items and generally lacking information (e.g.
> "In the future, we hope to provide supplemental documentation that covers
> the NiFi Cluster Architecture in depth.") IMO, the guides are not always
> sufficient to a novice user, but require pre-existing knowledge or
> experience to read between the lines.
>
> In general, I think the provided documentation (starting with the Admin and
> User Guides) could use a thorough review and update. Is this something that
> is on anyone's the "hit list"? If someone picks this up, I could provide
> feedback in a more directed way rather than spamming the mailing list.
>
> Thanks,
> Mark
>

Re: FW: Loading nifi server within a Java process

[James Wing]

Jamie,

Sorry you didn't get a response.  I'm happy to confirm that this is the
correct list for your question.  There is no well-beaten path for running
NiFi in another process, NiFi assumes that it is deployed as it's own
process.  NiFi contains a number of features for classloader isolation,
thread management, and performance monitoring that may mean something
different in another process, although I can't say for sure they will break.

Hopefully, some others can chime in on the pitfalls.

Thanks,

James

On Wed, Apr 5, 2017 at 2:25 PM, Jamie Wang  wrote:

> Hi,
>
> I sent the message below to the user group and unfortunately I didn't get
> any response. I am forwarding the message to this list and am hoping to get
> some feedback. If this is the wrong list for this message, I apologize and
> please let me know the right place to post. Thank you in advance.
>
> Jamie
>
> From: Jamie Wang [mailto:jam...@opentext.com]
> Sent: Tuesday, April 04, 2017 12:53 PM
> To: us...@nifi.apache.org
> Subject: [EXTERNAL] - Loading nifi server within a Java process
>
> I have a situation where we have constraint on the number of processes we
> can use and hence, is it possible to load nifi server within another Java
> process? Looking at the org.apache.nifi.bootstrap.RunNiFi.java, it seems
> that I can create a Java object that does most of what
> org.apache.nifi.bootstrap.RunNiFi.main() do. Any thoughts if this will
> work? What are some of the things I need to watch out? And is there another
> better way to do this or an example? Thank you for your time.
>
> Jamie
>

Re: FetchS3Object leak

[James Wing]

David,

Can you clarify which part of the FetchS3Object code looks problematic to
you?  From a quick look, I found one use of S3Object in FetchS3Object.java,
line ~106:

try (final S3Object s3Object = client.getObject(request)) {
flowFile = session.importFrom(s3Object.getObjectContent(),
flowFile);
attributes.put("s3.bucket", s3Object.getBucketName());

I believe declaring the variable within the try block will lead to its
proper and certain closure, but I'm not 100% on all the fine print with
that.  Is this what you are referring to, and does it not work as I hope?

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/FetchS3Object.java#L106

Thanks,

James


On Wed, Mar 22, 2017 at 12:41 PM, David Hesson  wrote:

> Greetings,
>
> In investigating a connection pool issue we were having during development,
> I was checking the FetchS3Object code to see how it reads content from S3.
> I don't see a close()
>  amazonaws/services/s3/model/S3Object.html#close-->invocation
> on the S3Object in the FetchS3Object processor. I believe this can lead to
> leaks on that object.
>
> We we're seeing logs like the following after trying to process some 90k
> objects from S3:
> INFO [Timer-Driven Process Thread-55] com.amazonaws.http.AmazonHttpClient
> Unable to execute HTTP request: Timeout waiting for connection from pool
>
> Is the S3Object not closed because the stream content is lazily loaded
> later in the flow (when accessed)? I didn't check the processSession
> implementation which reads the input stream. Just figured I'd ask and see
> if you all were aware, or that this is for some reason by design.
>
> Thanks,
> dh
>

Re: All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual interventio

[James Wing]

Would it be possible to configure EvaluateJsonPath to place the selected
JSON fragment in the flowfile content instead of an attribute?  Or to break
the selection across multiple attributes rather than one big one?

Thanks,

James

On Thu, Mar 16, 2017 at 11:57 AM, srini  wrote:

> Hi James,
>
> Yes, EvaluateJsonPath is creating attributes exceeding 64 KB. What should I
> do to avoid this?
> thanks
> Srini
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/All-Partitions-have-been-
> blacklisted-due-to-failures-when-attempting-to-update-If-
> the-Write-Ahead-Lo-tp15167p15169.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual interventio

[James Wing]

Srini,

The error message "FlowFile Repository failed to update" matches a known
issue where NiFi has trouble persisting attributes larger than 64 KB (
https://issues.apache.org/jira/browse/NIFI-3389).  Is it possible that your
EvaluateJsonPath is creating attributes exceeding 64 KB?  From your error
file:

o.a.n.p.standard.EvaluateJsonPath
EvaluateJsonPath[id=b71b17e5-1046-115a-5d5c-e4f368141f13] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException:
FlowFile Repository failed to update:
org.apache.nifi.processor.exception.ProcessException: FlowFile Repository
failed to update


Thanks,

James

On Thu, Mar 16, 2017 at 11:40 AM, srini  wrote:

> Hi,
> We have single nifi instance, and this is our production environment.
> Yesterday I increased from 512m to 4096m in bootstrap.conf for both Xms and
> Xmx, and restarted nifi.
> Then I see this error [1]. Then I deleted the folder
> ../nifi-1.1.0/flowfile_repository and restarted the nifi.
> Then everything is fine, and it has been working close 20 hours without any
> issue.
>
> Now all of sudden, I am seeing the same error. What should I do?
>
> [1] All Partitions have been blacklisted due to failures when attempting to
> update. If the Write-Ahead Log is able to perform a checkpoint, this issue
> may resolve itself. Otherwise, manual intervention will be required.
>
> thanks
> Srini error.txt
>  n15167/error.txt>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/All-Partitions-have-been-
> blacklisted-due-to-failures-when-attempting-to-update-If-
> the-Write-Ahead-Lo-tp15167.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: [VOTE] Release Apache NiFi nifi-nar-maven-plugin-1.2.0

[James Wing]

+1 Release this package as nifi-nar-maven-plugin-1.2.0

Went through the release helper, built NiFi with the new plugin, ran NiFi
to make sure it didn't explode (it didn't).


On Tue, Mar 14, 2017 at 9:21 AM, Bryan Bende  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-nar-maven-plugin-1.2.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1101
>
> The Git tag is nifi-nar-maven-plugin-1.2.0-RC1
> The Git commit ID is d0c9d46d25a3eb8d3dbeb2783477b1a7c5b2f345
> https://git-wip-us.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=
> d0c9d46d25a3eb8d3dbeb2783477b1a7c5b2f345
>
> Checksums of nifi-nar-maven-plugin-1.2.0-source-release.zip:
> MD5: a20b62075f79bb890c270445097dc337
> SHA1: 68e4739c9a4c4b2c69ff4adab8e1fdb0e7840923
> SHA256: f5d4acbaa38460bcf19e9b33f385aa643798788026875bd034ee837e5d9d45a8
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/bbende.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 3 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12339193
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.2.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
>
> Then please vote:
>
> [ ] +1 Release this package as nifi-nar-maven-plugin-1.2.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...
>

Re: ConvertJSONToAvro converts only last valid JASON record from an Input file with many JSON records

[James Wing]

Anshuman,

What kind of error messages are you seeing? If the input is
newline-delimited JSON records, I believe ConvertJSONToAvro should transfer
all valid records to the "success" relationship as a single Avro, and the
entire input flowfile to the "incompatible" output if any are invalid.
Does your "success" Avro contain more than one record?

One thing you could do is split the JSON records before conversion, so they
will succeed or fail individually.

Thanks,

James

On Tue, Mar 14, 2017 at 8:59 AM, Anshuman Ghosh <
anshuman.ghosh2...@gmail.com> wrote:

> Hello Group/ Bryan,
>
> Anshu here.
> While trying the "*ConvertJSONToAvro*" processor, I came across the
> following issue. I am not sure whether it is correct or not though.
> So I have this input JSON file which might contain one or more valid/
> invalid JSON records. Need to use this processor to convert them into Avro
> format before I write and process them onto HDFS.
> However strangely in the output FlowFile, I am getting only 1 record
> converted (last valid JSON record is only getting converted).
>
> Following "*Record Schem*a" I am using for this purpose.
>
> *{"type": "record",*
> * "name": "ClickData",*
> * "fields": [*
> * {"name": "SuperShopId", "type": ["string", "null"]},*
> * {"name": "LinkId",  "type": ["string", "null"]},*
> * {"name": "ClickId", "type": ["string", "null"]},  *
> * {"name": "VisitorId", "type": ["string", "null"]},*
> * {"name": "RawHTTPRequest", "type": ["string", "null"]},*
> * {"name": "RequestTimestamp", "type": "long"},*
> * {"name": "RedirectURL", "type": ["string", "null"]},*
> * {"name": "ResponseTimestamp", "type": "long"}*
> * ]*
> *}*
>
> Please let me know if I am doing something wrong!
> Thanking you in advance!
>
> ​
> __
>
> *Kind Regards,*
> *Anshuman Ghosh*
> *Contact - +49 179 9090964*
>

Re: Closing in on a NiFi 1.2.0 release?

[James Wing]

+1 for component versioning in 1.2.0, it will be a solid capstone feature.
And I agree it's probably not holding up the release.

Thanks,

James

On Wed, Mar 8, 2017 at 1:21 PM, Bryan Bende <bbe...@gmail.com> wrote:

> James,
>
> No problem :)
>
> I was mostly just suggesting an overall strategy...
>
> Usually when we start closing in on a release we go through the JIRAs
> tagged for that release and try to figure out which ones can be moved
> to a future release, and which ones the community is actually working
> on and close to being ready. Currently we have 39 unresolved JIRAs
> that are tagged as 1.2, one of which is NIFI-3380, and I figured if
> someone looked at the ticket it might look like no work had been done
> and figure that it can just be moved to next release, so I just wanted
> to mention that it is very close to being ready was still hoping for
> it be in 1.2, unless there was strong opinion to move on without it.
> Even if we moved on without it, I believe there is still a bit of work
> to do in that we still need a release manager and we need to decide
> what to do with the 39 JIRAs.
>
> As far as the new NAR plugin and how things will work...
>
> The changes to the NAR plugin add additional information to the
> MANIFEST file in the NAR. Technically existing NiFi would have no
> problem reading the new MANIFEST file because no entries are being
> removed, and the branch I have with the component versioning code for
> NiFi could also run against old NARs that don't have the new entries,
> you just see everything as being "unversioned" and can't actually make
> use of the new capabilities. We'll always have to be able to run older
> NARs because there are tons of custom NARs out there that have not
> been (and may never be) rebuilt with the newer version of the plugin,
> which is fine, they only need to be rebuilt if someone wants to run
> two versions of that NAR at the same time.
>
> Happy to elaborate more on any of the component versioning work if
> anyone is interested.
>
> Thanks,
>
> Bryan
>
>
> On Wed, Mar 8, 2017 at 2:43 PM, Russell Bateman <r...@windofkeltia.com>
> wrote:
> > +1 for component versioning in NiFi 1.2!
> >
> >
> > On 03/08/2017 12:40 PM, James Wing wrote:
> >>
> >> Bryan, I'm 100% in favor of you and Matt Gilman doing all the hard work.
> >> Oh, and uh... thanks! :)
> >>
> >> So the alternatives are:
> >> a.) Release 1.2.0 sooner (?), but without component versioning
> >> b.) Delay 1.2.0 (?) to incorporate component versioning
> >>
> >> Will the NAR plugin alone commit us to all of the component versioning
> >> work
> >> in 1.2, or will the new NAR format be backward-compatible?  Or is the
> >> question more about the strategy for 1.2.0?
> >>
> >>
> >> Thanks,
> >>
> >> James
> >>
> >> On Wed, Mar 8, 2017 at 9:06 AM, Bryan Bende <bbe...@gmail.com> wrote:
> >>
> >>> Just wanted to mention that one of the JIRAs tagged for 1.2.0 is
> >>> NIFI-3380 "support multiple versions of the same component" [1] and
> >>> I've been working with Matt Gilman on this [2]. The functionality is
> >>> very close to being done and I think we should get this into the 1.2.0
> >>> release.
> >>>
> >>> In order to fully leverage the versioned components we will need to
> >>> release an updated Maven NAR plugin, so we would do that first and
> >>> then release 1.2.0 using the new plugin. If everyone is on-board with
> >>> this plan then I can advise when we are ready to release the new
> >>> plugin which would be soon.
> >>>
> >>> [1] https://issues.apache.org/jira/browse/NIFI-3380
> >>> [2] https://github.com/bbende/nifi/tree/NIFI-3380
> >>>
> >>> On Fri, Mar 3, 2017 at 9:56 AM, Joe Gresock <jgres...@gmail.com>
> wrote:
> >>>>
> >>>> This is good discussion that should continue, but what about the
> >>>> original
> >>>> intent of Joe's post?  "Is there any reason folks can think of to hold
> >>>
> >>> off
> >>>>
> >>>> on a 1.2.0 release?"
> >>>>
> >>>> On Fri, Mar 3, 2017 at 1:45 PM, Mark Payne <marka...@hotmail.com>
> wrote:
> >>>>
> >>>>> Andy,
> >>>>>
> >>>>> Sorry, i haven't responded to this thread in over a week, but I think
> >>>
> >>> it's
> >>>>>
>

Re: Closing in on a NiFi 1.2.0 release?

[James Wing]

Bryan, I'm 100% in favor of you and Matt Gilman doing all the hard work.
Oh, and uh... thanks! :)

So the alternatives are:
a.) Release 1.2.0 sooner (?), but without component versioning
b.) Delay 1.2.0 (?) to incorporate component versioning

Will the NAR plugin alone commit us to all of the component versioning work
in 1.2, or will the new NAR format be backward-compatible?  Or is the
question more about the strategy for 1.2.0?


Thanks,

James

On Wed, Mar 8, 2017 at 9:06 AM, Bryan Bende  wrote:

> Just wanted to mention that one of the JIRAs tagged for 1.2.0 is
> NIFI-3380 "support multiple versions of the same component" [1] and
> I've been working with Matt Gilman on this [2]. The functionality is
> very close to being done and I think we should get this into the 1.2.0
> release.
>
> In order to fully leverage the versioned components we will need to
> release an updated Maven NAR plugin, so we would do that first and
> then release 1.2.0 using the new plugin. If everyone is on-board with
> this plan then I can advise when we are ready to release the new
> plugin which would be soon.
>
> [1] https://issues.apache.org/jira/browse/NIFI-3380
> [2] https://github.com/bbende/nifi/tree/NIFI-3380
>
> On Fri, Mar 3, 2017 at 9:56 AM, Joe Gresock  wrote:
> > This is good discussion that should continue, but what about the original
> > intent of Joe's post?  "Is there any reason folks can think of to hold
> off
> > on a 1.2.0 release?"
> >
> > On Fri, Mar 3, 2017 at 1:45 PM, Mark Payne  wrote:
> >
> >> Andy,
> >>
> >> Sorry, i haven't responded to this thread in over a week, but I think
> it's
> >> important to keep going.
> >>
> >> I just clicked "Cancel Patch" on one of my ticket that has a patch
> >> available to see which state it returned to.
> >> It did in fact go back to Open. Which I agree is less than ideal. Though
> >> we could certainly have a process
> >> by which we change the status to "In Progress" after canceling the
> patch.
> >>
> >> I guess where my viewpoint differs from yours is in the meaning of "In
> >> Review." Let's say that you submit a
> >> patch for a JIRA. I then review it and find that it needs some work -
> >> let's say there's an issue with licensing
> >> not being properly accounted for, for instance. At that point, I no
> longer
> >> consider the patch that you provided
> >> to be "In Review." I believe the patch should be canceled, and you will
> >> need to submit a new patch. I guess
> >> that I view a patch as being an immutable entity.
> >>
> >>
> >>
> >>
> >> On Feb 24, 2017, at 7:26 PM, Andy LoPresto   >> lopre...@apache.org>> wrote:
> >>
> >> Mark,
> >>
> >> Your understanding of “Patch Available” certainly makes sense and it
> >> explains why you approach the process the way you do. I have a slightly
> >> different personal understanding of “Patch Available” — I read it to
> mean
> >> “the person responsible for this Jira has contributed code they feel
> solves
> >> the issue.” A review will (hopefully) determine if that assertion is
> >> correct and complete. I think we kind of agree on "my viewpoint is
> simply
> >> that "Patch Available" means "Awaiting Review" or "In Review.”” but I
> see
> >> “In Review” as a potentially iterative process — it could be on the
> second
> >> pass of the contributor responding to comments, but it’s still “In
> Review”
> >> in my eyes. I don’t know that the granularity of Jira supports the
> specific
> >> workflow states of “been reviewed once but not complete/accepted yet”.
> >>
> >> What state does “Cancel Patch” result in? If it just reverts to “Open”,
> I
> >> don’t see the value because that obfuscates the difference between a
> Jira
> >> that hasn’t even been touched and one that has 90% of the code done. I
> >> agree we should make the RM’s job easier, but I also think it doesn’t
> help
> >> the visibility for reviewers to see a Jira marked as “open” when there
> is
> >> the potential for that patch to be ready for merge in a very short
> amount
> >> of time.
> >>
> >> I think these conversations will ultimately help us narrow in on shared
> >> definitions that make sense to everyone though, so I’m glad we’re
> talking
> >> about it.
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> alopresto.apa...@gmail.com
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >> On Feb 24, 2017, at 1:07 PM, Mark Payne > arka...@hotmail.com>> wrote:
> >>
> >> Andy,
> >>
> >> If the reviewer is looking for clarification, then it may make sense to
> >> leave the JIRA in "Patch Available" state
> >> as you suggest. If there are minor fixes needed, though, then the patch
> is
> >> not ready. In JIRA, the verbiage for
> >> Cancel Patch says "The patch is not yet ready to be committed." So if
> >> minor fixes are needed, then I believe
> >> it is 

Re: [REMINDER] Please signoff when committing other people's changes

[James Wing]

I recommend the practice.  Although the signoff may not be authoritative,
it requires a positive action that suggests you purposefully merged the
commit, as opposed to commits you might have accidentally merged and pushed.

Thanks,

James


On Thu, Mar 2, 2017 at 7:49 AM, Joe Witt  wrote:

> "If this not the expected process, we should definitely update the
> Contributor Guide."
>
> I think it is fine to encourage it.  It is not a requirement though.
>
> The signoff is not an apache thing.  Committer privileges to push code
> to a given repo is an apache thing.
>
> We're an RTC community and the only thing we do require is a +1 from
> another committer before the patch/PR is merged.  It is perfectly
> sufficient for the reviewer to add a +1 on github comments or a JIRA
> comment and the author could commit directly.  No signoff required.
>
> Thanks
> Joe
>
> On Thu, Mar 2, 2017 at 10:42 AM, Joe Skora  wrote:
> > Like Andre, I originally got the requirement for signoff from the
> > Contributor Guide[1] when I started working on the project and later from
> > this email thread[2].  If this not the expected process, we should
> > definitely update the Contributor Guide.
> >
> > From the Apache perspective the signoff confirms that the contribution
> was
> > reviewed for Apache compliance, but that's incumbent on anyone committing
> > to the repository.  Since the committer identity is available from the
> > repository the signoff seems redundant.
> >
> > [1]https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#
> > ContributorGuide-Stepstomerge/closepullrequestswithtwomainbranches
> > [2]
> > https://lists.apache.org/thread.html/1ce165a4172f67ce08683d3eb1c825
> 3319a97dadd0ccc3fbc598f639@1446565288@%3Cdev.nifi.apache.org%3E
> >
> >
> > On Thu, Mar 2, 2017 at 9:33 AM, Joe Witt  wrote:
> >
> >> For what it is worth this is definitely not a requirement and not
> >> something I knew anything of so I never do it.
> >>
> >> I think it is a perfectly fine idea and a good practice to follow so
> >> occasional reminders of its utility are fair game.  That said, to
> >> Bryan's point I rely on the JIRA/issues history if i need to know who
> >> did a given review.  So we have a couple of options.
> >>
> >> But we should probably stop short of calling this a requirement.  In
> >> an apache sense it is not.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, Mar 2, 2017 at 8:51 AM, Matt Burgess 
> wrote:
> >> > I didn't realize it was required either, I usually only sign off
> >> > (using the same thing Bryan Bende does) if the PR author couldn't
> >> > merge it on their own (i.e. not a NiFi committer/PMC). Certainly I can
> >> > start always signing off commits.
> >> >
> >> > Regards,
> >> > Matt
> >> >
> >> > On Thu, Mar 2, 2017 at 8:35 AM, Oleg Zhurakousky
> >> >  wrote:
> >> >> Thanks Bryan.
> >> >>
> >> >> If ‘-s’ is only for showcasing the committer I don’t believe anyone
> >> would have any issues with it, but my concern at the moment is purely
> >> legal, so I am not sure who is the right person to answer that, but
> figured
> >> raising the concern is the least I can do.
> >> >>
> >> >> Cheers
> >> >> Oleg
> >> >>
> >> >>
> >> >>> On Mar 2, 2017, at 8:16 AM, Bryan Bende  wrote:
> >> >>>
> >> >>> The sign-off is so we can easily see who the reviewer/merger was
> from
> >> >>> the git history.
> >> >>>
> >> >>> We can always go back to the JIRA or PR and the reviewer/merger
> should
> >> >>> have commented there, but its convenient to see it in the git
> history
> >> >>> in my opinion.
> >> >>>
> >> >>> Personally, whenever merging someones contribution I use "git am
> >> >>> --signoff < patchfile" which I guess is equivalent to doing the
> ammend
> >> >>> after applying the patch.
> >> >>>
> >> >>>
> >> >>> On Thu, Mar 2, 2017 at 8:05 AM, Oleg Zhurakousky
> >> >>>  wrote:
> >>  Andre
> >> 
> >>  Thanks for the reminder. I admit that I did not know that we
> require
> >> it in the Contributor Guide, so thanks for pointing it out.
> >>  However, your email did prompt me to look at the purpose and origin
> >> of the ‘-s’ flag and led me to this thread on Stack Overflow -
> >> http://stackoverflow.com/questions/1962094/what-is-the-
> >> sign-off-feature-in-git-for.
> >> 
> >>  And I am now wondering if we should require it or even use it in
> the
> >> first place, since it’s origin, history and purpose appears to have more
> >> “individual” legal implications then showcasing the actual committer.
> >> 
> >>  Thoughts?
> >> 
> >>  Cheers
> >>  Oleg
> >> 
> >>  On Mar 2, 2017, at 6:35 AM, Andre > ndre-li...@fucs.org>> wrote:
> >> 
> >>  dev,
> >> 
> >>  May I remind you to ensure we follow the Contributor Guide and use:
> >> 
> >>  git commit --amend -s
> >> 

Re: Quick question on LICENSE/NOTICE when bundling controllers and processors

[James Wing]

It may already be in the Licensing Guide (
https://nifi.apache.org/licensing-guide.html).  I'll have to read it again
to identify net-new material.  I'll open a PR if there is.

Thanks,

James

On Mon, Feb 27, 2017 at 4:23 PM, Tony Kurc <trk...@gmail.com> wrote:

> Joe,
> This is awesome and useful - should this guidance go on the wiki somewhere?
>
> On Mon, Feb 27, 2017 at 4:59 PM, Joe Witt <joe.w...@gmail.com> wrote:
>
> > 1)  All nars once built do need to contain a LICENSE and NOTICE file
> > to cover what ends up in them as an archive of binary dependencies and
> > also it should cover any specific source dependencies they might have
> > (like MIT javascript libs in nifi-web-ui).
> >
> > 2)  We need a LICENSE/NOTICE in every nar.  A LICENSE and/or NOTICE
> > will be auto generated if not already present anyway.  What we need
> > them to cover is mentioned in #1.
> >
> > Also, for any source or binary dependency in a given nar we must also
> > ensure all source dependencies are captured appropriately in the top
> > level nifi.git/LICENSE & NOTICE and any source & binary dependencies
> > are captured in the nifi.git/nifi-assembly/LICENSE and NOTICE.
> >
> > As we progress toward separating the application from the nars by way
> > of some extension registry we'll be able to have a far smaller/simple
> > LICENSE and NOTICE at the top level but the above nar L
> > considerations will still apply per L
> >
> > Does that help at all James?
> >
> > On Mon, Feb 27, 2017 at 4:31 PM, James Wing <jvw...@gmail.com> wrote:
> > > Thank you both for bringing up this discussion.  I have a few follow-up
> > > questions:
> > >
> > > 1.) Is it true all of the NAR bundles in the NiFi source should have
> > their
> > > own LICENSE and NOTICE files, without exception?  In looking through
> the
> > > source, most nifi-*-nar projects have both files for binary
> dependencies.
> > > I understand these binary dependencies should roll up to nifi-assembly
> > > LICENSE/NOTICE.
> > >
> > > 2.) Is it true we do not need source LICENSE/NOTICE for nar bundle
> > projects
> > > unless they have distinctive source dependency terms that roll up to
> the
> > > root LICENSE/NOTICE?  nifi-web-ui is the only example I've found so
> > far.  I
> > > assume the ASLv2 file headers cover most of the source.
> > >
> > > 3.) Where dependencies also distinguish between their source
> > LICENSE/NOTICE
> > > and binary LICENSE/NOTICE, should we match to our dependency
> > relationship?
> > > For example, a binary dependency would result in the inclusion of the
> > > binary NOTICE terms?
> > >
> > >
> > > Thanks,
> > >
> > > James
> > >
> > > On Sat, Feb 25, 2017 at 7:33 PM, Aldrin Piri <aldrinp...@gmail.com>
> > wrote:
> > >
> > >> Hey Andre,
> > >>
> > >> I may be off, but to help you along, I will give you my take on things
> > to
> > >> the best of my understanding.  If there are any wrong points, I hope
> > >> someone can further clarify.
> > >>
> > >> For your case, it looks like simply you are simply using binary
> > >> dependencies.  For that case, you have to consider where these items
> are
> > >> showing up and how they are released.  Your inclusion of a dependency
> > will
> > >> affect the generated NAR (nifi-irc-processors-nar) and, while it seems
> > to
> > >> be missing in the current PR, the zip and tgz nifi-assembly artifacts.
> > You
> > >> shouldn't need to include it in levels lower than this assuming you
> are
> > >> talking about JARs that compose the overall NAR.  While you are
> linking
> > >> these against dependencies, you are not explicitly bringing them into
> > the
> > >> project through the JARs incorporated in the NAR.
> > >>
> > >> Source inclusions are handled similarly but do go a level deeper as
> they
> > >> are also bundled with the JARs and are present in the root LICENSE and
> > >> NOTICE where applicable.  Again, both are for similar reasons with the
> > >> generated source package and JARs bundling this work including the
> > source.
> > >>
> > >> Do keep in mind the transitive dependencies.  Looking quickly through
> > the
> > >> pom for kitteh, I see usage of some netty libraries as well as
> > mbassador.
> > >>

Re: Quick question on LICENSE/NOTICE when bundling controllers and processors

[James Wing]

Yes, thank you, that does help.  I'm slowly sneaking up on an understanding
of how it works.

James

On Mon, Feb 27, 2017 at 1:59 PM, Joe Witt <joe.w...@gmail.com> wrote:

> 1)  All nars once built do need to contain a LICENSE and NOTICE file
> to cover what ends up in them as an archive of binary dependencies and
> also it should cover any specific source dependencies they might have
> (like MIT javascript libs in nifi-web-ui).
>
> 2)  We need a LICENSE/NOTICE in every nar.  A LICENSE and/or NOTICE
> will be auto generated if not already present anyway.  What we need
> them to cover is mentioned in #1.
>
> Also, for any source or binary dependency in a given nar we must also
> ensure all source dependencies are captured appropriately in the top
> level nifi.git/LICENSE & NOTICE and any source & binary dependencies
> are captured in the nifi.git/nifi-assembly/LICENSE and NOTICE.
>
> As we progress toward separating the application from the nars by way
> of some extension registry we'll be able to have a far smaller/simple
> LICENSE and NOTICE at the top level but the above nar L
> considerations will still apply per L
>
> Does that help at all James?
>
> On Mon, Feb 27, 2017 at 4:31 PM, James Wing <jvw...@gmail.com> wrote:
> > Thank you both for bringing up this discussion.  I have a few follow-up
> > questions:
> >
> > 1.) Is it true all of the NAR bundles in the NiFi source should have
> their
> > own LICENSE and NOTICE files, without exception?  In looking through the
> > source, most nifi-*-nar projects have both files for binary dependencies.
> > I understand these binary dependencies should roll up to nifi-assembly
> > LICENSE/NOTICE.
> >
> > 2.) Is it true we do not need source LICENSE/NOTICE for nar bundle
> projects
> > unless they have distinctive source dependency terms that roll up to the
> > root LICENSE/NOTICE?  nifi-web-ui is the only example I've found so
> far.  I
> > assume the ASLv2 file headers cover most of the source.
> >
> > 3.) Where dependencies also distinguish between their source
> LICENSE/NOTICE
> > and binary LICENSE/NOTICE, should we match to our dependency
> relationship?
> > For example, a binary dependency would result in the inclusion of the
> > binary NOTICE terms?
> >
> >
> > Thanks,
> >
> > James
> >
> > On Sat, Feb 25, 2017 at 7:33 PM, Aldrin Piri <aldrinp...@gmail.com>
> wrote:
> >
> >> Hey Andre,
> >>
> >> I may be off, but to help you along, I will give you my take on things
> to
> >> the best of my understanding.  If there are any wrong points, I hope
> >> someone can further clarify.
> >>
> >> For your case, it looks like simply you are simply using binary
> >> dependencies.  For that case, you have to consider where these items are
> >> showing up and how they are released.  Your inclusion of a dependency
> will
> >> affect the generated NAR (nifi-irc-processors-nar) and, while it seems
> to
> >> be missing in the current PR, the zip and tgz nifi-assembly artifacts.
> You
> >> shouldn't need to include it in levels lower than this assuming you are
> >> talking about JARs that compose the overall NAR.  While you are linking
> >> these against dependencies, you are not explicitly bringing them into
> the
> >> project through the JARs incorporated in the NAR.
> >>
> >> Source inclusions are handled similarly but do go a level deeper as they
> >> are also bundled with the JARs and are present in the root LICENSE and
> >> NOTICE where applicable.  Again, both are for similar reasons with the
> >> generated source package and JARs bundling this work including the
> source.
> >>
> >> Do keep in mind the transitive dependencies.  Looking quickly through
> the
> >> pom for kitteh, I see usage of some netty libraries as well as
> mbassador.
> >> These would presumably also be collected upon building the NAR.
> >>
> >> Of course, the docs we have on the site are quite nice if you need some
> >> light reading material ;)  https://nifi.apache.org/licensing-guide.html
> >> Both the guide and the links from it are good information and a nice
> >> reference to revisit when working through these things.
> >>
> >> Let us know if there are additional questions or if some additional
> >> clarification is needed.  Hopefully my anecdotal thoughts are both
> somewhat
> >> helpful and mostly correct!
> >>
> >> On Sat, Feb 25, 2017 at 9:39 PM, Afonso Murakami <murakam...@icloud.co

Re: Quick question on LICENSE/NOTICE when bundling controllers and processors

[James Wing]

Thank you both for bringing up this discussion.  I have a few follow-up
questions:

1.) Is it true all of the NAR bundles in the NiFi source should have their
own LICENSE and NOTICE files, without exception?  In looking through the
source, most nifi-*-nar projects have both files for binary dependencies.
I understand these binary dependencies should roll up to nifi-assembly
LICENSE/NOTICE.

2.) Is it true we do not need source LICENSE/NOTICE for nar bundle projects
unless they have distinctive source dependency terms that roll up to the
root LICENSE/NOTICE?  nifi-web-ui is the only example I've found so far.  I
assume the ASLv2 file headers cover most of the source.

3.) Where dependencies also distinguish between their source LICENSE/NOTICE
and binary LICENSE/NOTICE, should we match to our dependency relationship?
For example, a binary dependency would result in the inclusion of the
binary NOTICE terms?


Thanks,

James

On Sat, Feb 25, 2017 at 7:33 PM, Aldrin Piri  wrote:

> Hey Andre,
>
> I may be off, but to help you along, I will give you my take on things to
> the best of my understanding.  If there are any wrong points, I hope
> someone can further clarify.
>
> For your case, it looks like simply you are simply using binary
> dependencies.  For that case, you have to consider where these items are
> showing up and how they are released.  Your inclusion of a dependency will
> affect the generated NAR (nifi-irc-processors-nar) and, while it seems to
> be missing in the current PR, the zip and tgz nifi-assembly artifacts.  You
> shouldn't need to include it in levels lower than this assuming you are
> talking about JARs that compose the overall NAR.  While you are linking
> these against dependencies, you are not explicitly bringing them into the
> project through the JARs incorporated in the NAR.
>
> Source inclusions are handled similarly but do go a level deeper as they
> are also bundled with the JARs and are present in the root LICENSE and
> NOTICE where applicable.  Again, both are for similar reasons with the
> generated source package and JARs bundling this work including the source.
>
> Do keep in mind the transitive dependencies.  Looking quickly through the
> pom for kitteh, I see usage of some netty libraries as well as mbassador.
> These would presumably also be collected upon building the NAR.
>
> Of course, the docs we have on the site are quite nice if you need some
> light reading material ;)  https://nifi.apache.org/licensing-guide.html
> Both the guide and the links from it are good information and a nice
> reference to revisit when working through these things.
>
> Let us know if there are additional questions or if some additional
> clarification is needed.  Hopefully my anecdotal thoughts are both somewhat
> helpful and mostly correct!
>
> On Sat, Feb 25, 2017 at 9:39 PM, Afonso Murakami 
> wrote:
>
> > I just start and I really don’t know much so let see what I can learn
> when
> > time pass by and hope I can learn as much as you, thank’s
> > > On Feb 25, 2017, at 5:12 PM, Andre  wrote:
> > >
> > > Hi there,
> > >
> > > Quick question on proper licensing:
> > >
> > > When bundling Processors, Services and APIs, where should the NOTICES
> and
> > > LICENSES be added to?
> > >
> > > The PR in question is https://github.com/apache/nifi/pull/1541
> > >
> > > My current reading is that all NAR levels will have to include the
> proper
> > > references (although I may reduce a bit of the dependencies by
> excluding
> > > some of the deeper dependencies, specially at the
> > > nifi-irc-client-service-api-nar ).
> > >
> > > Would you agree?
> > >
> > > Cheers
> >
> >
>

Re: [ANNOUNCE] New Apache NiFi PMC Member - James Wing

[James Wing]

Thank you all

On Wed, Feb 22, 2017 at 6:58 AM, Aldrin Piri <ald...@apache.org> wrote:

> Team,
>
> On behalf of the Apache NiFi PMC, I am pleased to announce that James Wing
> has accepted the PMC's invitation to join the Apache NiFi PMC.  We
> greatly appreciate all of Joe's hard work and generous contributions to
> the project. We look forward to his continued involvement in the project.
>
> James started out contributing in January 2016 with review and assistance
> on all things AWS. After receiving committer status, James embraced the
> role and continued to assist in fostering and growing the community. Beyond
> code contributions, James is active in the community lists, votes, reviews
> and external sites helping solve questions about NiFi.
>
> Please join us in congratulating and welcoming James to the Apache NiFi
> PMC.
>
> Congratulations and welcome, James!
>

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[James Wing]

Congratulations, Jeff!

On Tue, Feb 21, 2017 at 11:41 AM, Aldrin Piri  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeff
> Storck has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate all of Jeff's hard work and
> generous contributions and look forward to continued involvement in the
> project.
>
> Jeff's contributions include significant efforts toward upgrade and
> migration processes inclusive of flow layout when upgrading from 0.x to 1.x
> and the ZooKeeper migration toolkit.
>
> Congrats, Jeff!
>

Re: [VOTE] Release Apache NiFi 1.1.2

[James Wing]

+1 (non-binding) - went through release helper.

On Thu, Feb 16, 2017 at 7:37 PM, Andy LoPresto  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-1.1.2.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1098
>
> The Git tag is nifi-1.1.2-RC1
> The Git commit ID is 51fad01f5daf33716b8b5379c32ee932d91c8c63
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 51fad01f5daf33716b8b5379c32ee932d91c8c63
>
> Checksums of nifi-1.1.2-source-release.zip:
> MD5: 31d20a09955fac32d5b4147c497deede
> SHA1: 9f8f2ce00397d990dfb0890f9b1ede70dca4f25e
> SHA256: 0f57b5ae7f5e1d7f1289d779df39f20d70af0ffd92cb01b265beb90b309b747e
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/alopresto.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 2 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> version=12339600=12316020
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.1.2
>
> The vote will be open for 96 hours (over holiday weekend).
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
>
> [ ] +1 Release this package as nifi-1.1.2
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because…
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>

Re: [VOTE] Release Apache NiFi 0.7.2

[James Wing]

The unifying theme of 0.7.2 and 1.1.2 appears to be NIFI-3487 "Refactor
user formatting".  Is this urgent?

https://issues.apache.org/jira/browse/NIFI-3487
https://github.com/apache/nifi/commit/bd88e4335ad151592f1310996e9a0513b7f0829a


Thanks,

James


On Fri, Feb 17, 2017 at 9:55 AM, Joe Witt  wrote:

> Mike,
>
> You're absolutely right that there are some good fixes to be included
> in an upcoming release.  There is no reason someone couldn't put
> together an 0.7.3.
>
> This is just something that we're wanting resolved and have the energy
> to push forward and doesn't require additional testing as inclusion of
> other items would.
>
> Please by no means see this as a deterrent to progression on an 0.7.3.
>
> Thanks
> Joe
>
> On Fri, Feb 17, 2017 at 12:38 PM, Michael Moser 
> wrote:
> > I'm sad that 0.7.2 does not contain some of the other bug fixes currently
> > on the 0.x branch.  It doesn't seem like it would have cost very much to
> > just include them.
> >
> > +0 (non binding)
> >
> > -- Mike
> >
> >
> > On Thu, Feb 16, 2017 at 11:42 PM, Andy LoPresto 
> > wrote:
> >
> >> Hello,
> >>
> >> I am pleased to be calling this vote for the source release of Apache
> NiFi
> >> nifi-0.7.2.
> >>
> >> The source zip, including signatures, digests, etc. can be found at:
> >> https://repository.apache.org/content/repositories/orgapachenifi-1099
> >>
> >> The Git tag is nifi-0.7.2-RC1
> >> The Git commit ID is 831ac6939df7d418cadb336eedb4e09fb97541a1
> >> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> >> 831ac6939df7d418cadb336eedb4e09fb97541a1
> >>
> >> Checksums of nifi-0.7.2-source-release.zip:
> >> MD5: efe9ea1cf0698a57a6829fe3829fc136
> >> SHA1: aee51164af34394c7017dae491b239a88b614865
> >> SHA256: 8cd02675003fca837ea0092b6225600a4700b905e5214a751ae7ff833263
> 193b
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/alopresto.asc
> >>
> >> KEYS file available here:
> >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >> 2 issues were closed/resolved for this release:
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> version=12339601=12316020
> >>
> >> Release note highlights can be found here:
> >> https://cwiki.apache.org/confluence/display/NIFI/
> >> Release+Notes#ReleaseNotes-Version0.7.2
> >>
> >> The vote will be open for 96 hours (over holiday weekend).
> >> Please download the release candidate and evaluate the necessary items
> >> including checking hashes, signatures, build
> >> from source, and test.  The please vote:
> >>
> >> [ ] +1 Release this package as nifi-0.7.2
> >> [ ] +0 no opinion
> >> [ ] -1 Do not release this package because because…
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> *alopresto.apa...@gmail.com *
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >>
>

Re: Legacy Authorized Users File approach not working?

[James Wing]

Russell,

I would not recommend the Legacy Authorized Users approach in 1.x.  It does
work, but the authorizations model in 1.x is bigger than it was in 0.x, and
the simple conversion will not provide all that you need.  I recommend
using the Initial Admin Identity, then configuring groups and users either
manually or with an API client.

That might make a good blog article...

James


On Tue, Dec 20, 2016 at 3:19 PM, Russell Bateman 
wrote:

> Thanks, Bryan. I will try this again. Meanwhile, I've been studying our
> 0.x LDAP twists and I am starting to feel sure I can replicate them in 1.x.
> Both avenues will be useful to us. (But, I really liked nifi-toolkit and
> your tutorial.)
>
> Best,
> Russ
>
> On 12/20/2016 03:41 PM, Bryan Bende wrote:
>
>> Russell,
>>
>> To verify the conversion of the legacy authorized users file, I just did
>> the following...
>>
>> 1) Took a fresh build off master (1.2-SNAPSHOT)
>>
>> 2) Created an authorized-users.xml with the following content as a test:
>>
>> 
>> 
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>>  
>> 
>>
>> 3) Configured authorizers.xml as follows:
>>
>> 
>>  file-provider
>>  org.apache.nifi.authorization.FileAuthorizer
>>  ./conf/authorizations.xml
>>  ./conf/users.xml
>>  
>>  
>> */path/to/authorized-users.xml*
>>  
>>
>> 4) Set the necessary TLS/SSL properties in nifi.properties
>>
>> 5) Started up successfully
>>
>> 6) users.xml got created and had this:
>>
>> 
>> 
>>  
>>  > name="group1">
>>  
>>  
>>  
>>  
>>  > identity="user1"/>
>>  > identity="user2"/>
>>  > identity="user3"/>
>>  > identity="user4"/>
>>  > identity="user5"/>
>>  > identity="user6"/>
>>  
>> 
>>
>> 7) authorizations.xml got created and had a bunch of policies for theses
>> users, not going to paste it because it was a bit long.
>>
>> Now at this point I don't have a real certificate for any of these fakes
>> users 1-6, but in a real migration those users would be in your LDAP or
>> you
>> would have a client certificate for one of them.
>>
>> One thing to keep in mind is that the legacy migration only runs when
>> there
>> are no other users or policies defined, it is meant to be a one time
>> initial start up task. So if you ever get into a weird state and want to
>> try it again, you will need to delete users.xml and authorizations.xml
>> from
>> your conf directory to get back to a clean state.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Tue, Dec 20, 2016 at 5:10 PM, Russell Bateman 
>> wrote:
>>
>> Bryan,
>>>
>>> As I'm new to setting this up (why I'm tackling this at all), I initially
>>> looked toJames Wing's tutorial >> onfiguring-ssl-auth.html>. At the end of that, I had a file,
>>> /authorized-users.xml/, which I wanted to provide via this "legacy"
>>> interface to NiFi 1.1.0. The Administration Guide told me how to do that
>>> and I did it, or so I thought. In the end, NiFi would not come up
>>> complaining about a missing /users.xml/, when I provided an empty one
>>> (with
>>> ) that this was wrong too.
>>>
>>> I am happy it works in this new way, after following your tutorial, but
>>> what we've really got is complicated, it uses LDAP, and we might want to
>>> take the "legacy" solution route to soften the impact on our /rpm/ and
>>> Ansible work by falling back on what we've already got.
>>>
>>> Sorting out that nightmare was going to be a subsequent step after
>>> getting
>>> James Wing's tutorial working in NiFi 1.1.0 since it's written for NiFi
>>> 0.x, producing 0.x artifacts that I could use in 1.1.0, but without the
>>> added LDAP complication that's present in our present, shipping 0.7.1
>>> artifacts.
>>>
>>> Does this make sense? I was trying to take a couple of paths in parallel
>>> to get ready for switching from 0.x to 1.x.
>>>
>>> Let's not waste your time. If there's value in my taking the output from
>>> James Wing's tutorial and trying to make it work in NiFi 1.x, and
>>> reporting
>>> what's not working to you guys, I'll do it. Otherwise, I was just going
>>> to
>>> move on. Ultimately though, we will have to tackle getting our LDAP
>>> solution rewritten for NiFi 1.x. If the legacy way of using 0.x artifacts
>>> had worked right off, we'd probably have resorted to that. We (my
>>> colleague, who originally injected our LDAP solution and I) will probably
>>> at least try that way again in a few weeks before starting over from
>>> scratch to shoehorn our LDAP stuff into NiFi 1.x.
>>>
>>> (I've probably repeated myself here, sorry.)
>>>
>>> Thanks for your help.
>>>
>>> Russ
>>>
>>>
>>>
>>> On 12/20/2016 01:32 PM, Bryan Bende wrote:
>>>
>>> For the 0.x instance, can you elaborate on "it was not working"?

Re: [VOTE] Release Apache NiFi 1.1.1 (RC1)

[James Wing]

+1 (non-binding). I ran through the release helper -- verified hashes,
license/notice/readme files, full build, and tested the resulting binary
without issues on JDK 1.8.0_101, Amazon Linux.

Thanks,

James

On Mon, Dec 19, 2016 at 2:35 PM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi, nifi-1.1.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1097
>
> The Git tag is nifi-1.1.1-RC1
> The Git commit hash is a92f2e36ed6be695e4dc6f624f6b3a96e6d1a57c
> * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> a92f2e36ed6be695e4dc6f624f6b3a96e6d1a57c
> * https://github.com/apache/nifi/commit/a92f2e36ed6be695e4dc6f624f6b3a
> 96e6d1a57c
>
> Checksums of nifi-1.1.1-source-release.zip:
> MD5: 74955060d8ee295d77a23607ac644a6e
> SHA1: 82efc0dc3141d0fad0205b33539e5928da87ad17
> SHA256: 25fab8d7abfecf4c0ccef1ed9cd5f0849c829c0741142ed4074bc8dd0781f7d0
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jpercivall
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 16 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12338797
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.1.1
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.1.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
> Thanks!

Re: [VOTE] Release Apache NiFi 1.0.1 (RC1)

[James Wing]

+1 (non-binding).  I ran through the release helper and did some basic
testing of the built binary (Java 1.8.0_101, AWS Linux).

Thanks for putting this together, Joe.


James

On Fri, Dec 16, 2016 at 7:28 AM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> I apologize for the improperly formatted message. Hopefully the below
> message is better.
>
> 
>
>
> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache NiFi,
> nifi-1.0.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1096
>
> The Git tag is nifi-1.0.1-RC1
> The Git commit hash is 1890f6c522514027ae46f86601f4771f62cadc6d
> * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 1890f6c522514027ae46f86601f4771f62cadc6d
> * https://github.com/apache/nifi/commit/1890f6c522514027ae46f86601f477
> 1f62cadc6d
>
> Checksums of nifi-1.0.1-source-release.zip:
> MD5: cc7fea9a22c0b48f87dd7152ab83c28c
> SHA1: 88c35d5d3ff9d350473a742cdd8c38204628d343
> SHA256: d9d9628ced5bf3f0f3e0eae7729f4eb507120b072e883e287198fee80fbf9d15
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jpercivall
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 6 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12338865
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.0.1
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.0.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
> Thanks!
>
>
>
> On Friday, December 16, 2016 10:23 AM, Joe Percivall <
> joeperciv...@yahoo.com.INVALID> wrote:
>
>
>
> Hello Apache NiFi Community,
> I am pleased to be calling this vote for the source release of Apache
> NiFi,nifi-1.0.1.
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1096
> The Git tag is nifi-1.0.1-RC1The Git commit hash is
> 1890f6c522514027ae46f86601f4771f62cadc6d* https://git-wip-us.apache.org/
> repos/asf?p=nifi.git;a=commit;h=1890f6c522514027ae46f86601f4771f62cadc6d*
> https://github.com/apache/nifi/commit/1890f6c522514027ae46f86601f477
> 1f62cadc6d
> Checksums of nifi-1.0.1-source-release.zip:MD5:
> cc7fea9a22c0b48f87dd7152ab83c28cSHA1: 
> 88c35d5d3ff9d350473a742cdd8c38204628d343SHA256:
> d9d9628ced5bf3f0f3e0eae7729f4eb507120b072e883e287198fee80fbf9d15
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jpercivall
> KEYS file available here:https://dist.apache.org/
> repos/dist/release/nifi/KEYS
> 6 issues were closed/resolved for this release:https://issues.apache.
> org/jira/secure/ReleaseNote.jspa?projectId=12316020&
> version=12338865Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.0.1
> The vote will be open for 72 hours.Please download the release candidate
> and evaluate the necessary itemsincluding checking hashes, signatures,
> build from source, and test. Thenplease vote:
> [ ] +1 Release this package as nifi-1.0.1[ ] +0 no opinion[ ] -1 Do not
> release this package because...
> Thanks!
>

Re: how to aad a specified counter in nifi

[James Wing]

If you rename the ExecuteScript processor to something more specific like
"CountMyStuff", the Counters table will use that name, rather than strictly
the processor type.

The aggregate values for "All ExecuteScripts" are likely to be meaningless,
however.

Thanks,

James

On Fri, Dec 9, 2016 at 8:54 AM, Russell Bateman <r...@windofkeltia.com>
wrote:

> Ah, okay. The NiFi Counters summary lists the particulars including
> individual passage of flowfiles in a processor that adjusts the count and
> also totals for all flowfiles through a processor that calls
> adjustCounter(). It's less confusing for what my guys want if the processor
> doing this only occurs once in the flow, but this is good. We can probably
> make use of this via a property they can use to name their counter.
>
> Great, thank you!
>
> Russ
>
> On 12/09/2016 09:15 AM, James Wing wrote:
>
>> You can add a custom counter very easily using ExecuteScript:
>>
>> /* ECMAScript */
>> flowFile = session.get();
>> if (flowFile != null) {
>>  session.adjustCounter("SampleScriptCounter", 1, false);
>>  session.transfer(flowFile, REL_SUCCESS);
>> }
>>
>> The performance may not be optimal, but you can experiment with counters
>> immediately.
>>
>> Thanks,
>>
>> James
>>
>> On Fri, Dec 9, 2016 at 7:56 AM, Russell Bateman <r...@windofkeltia.com>
>> wrote:
>>
>> Aldrin,
>>>
>>> Forgive me for commandeering this thread.
>>>
>>> My down-streamers are asking for a custom "counting" processor because of
>>> what from their point of view is an unreliable volatility in the counts
>>> that the UI displays. In particular, they want to know how many documents
>>> they're feeding into our ETL flow and how many they're getting out (at
>>> least, in places where there is a one-to-one expectation which isn't all
>>> of
>>> our flows of course). The delta, which they are pretty sure to be
>>> non-zero,
>>> would be tantamount to lost/dropped documents. (So, they want me to
>>> manufacture the pistol they intend aiming at my head.  ;-)   )
>>>
>>> Are you saying that ProcessSession provides me, as a custom processor
>>> writer, the ability to do this (ostensibly in already existing custom
>>> processors I've written) in preference to adding yet another dedicated,
>>> custom processor?
>>>
>>> As new as NiFi is, great functionality is still hidden behind the fact
>>> that no one's using it and no one's writing about it, giving examples,
>>> etc.
>>> I was surprised to read this thread just now and wondered if it's another
>>> cool thing I've missed.
>>>
>>> Thanks for any confirmation, other comments, etc.
>>>
>>> Russ
>>>
>>> On 12/09/2016 08:42 AM, Aldrin Piri wrote:
>>>
>>> Hello,
>>>>
>>>> Counters are a framework level item that allows processors to provide
>>>> counts on things whilst processing.  This functionality must be
>>>> exercised
>>>> via the ProcessSession[1] within a processor's code.  To add one, you
>>>> would
>>>> need to invoke the adjustCounter method in your processor.
>>>>
>>>> [1]
>>>> https://github.com/apache/nifi/blob/master/nifi-api/src/main
>>>> /java/org/apache/nifi/processor/ProcessSession.java#L161
>>>>
>>>> On Fri, Dec 9, 2016 at 3:40 AM, bingogo1986 <bingogo1...@163.com>
>>>> wrote:
>>>>
>>>> hi
>>>>
>>>>> I noticed 'Counter' button on top right of Nifi UI ，how to aad a
>>>>> specified
>>>>> counter ，thanks.
>>>>> Best regards.
>>>>>
>>>>>
>

Re: how to aad a specified counter in nifi

[James Wing]

You can add a custom counter very easily using ExecuteScript:

/* ECMAScript */
flowFile = session.get();
if (flowFile != null) {
session.adjustCounter("SampleScriptCounter", 1, false);
session.transfer(flowFile, REL_SUCCESS);
}

The performance may not be optimal, but you can experiment with counters
immediately.

Thanks,

James

On Fri, Dec 9, 2016 at 7:56 AM, Russell Bateman 
wrote:

> Aldrin,
>
> Forgive me for commandeering this thread.
>
> My down-streamers are asking for a custom "counting" processor because of
> what from their point of view is an unreliable volatility in the counts
> that the UI displays. In particular, they want to know how many documents
> they're feeding into our ETL flow and how many they're getting out (at
> least, in places where there is a one-to-one expectation which isn't all of
> our flows of course). The delta, which they are pretty sure to be non-zero,
> would be tantamount to lost/dropped documents. (So, they want me to
> manufacture the pistol they intend aiming at my head.  ;-)   )
>
> Are you saying that ProcessSession provides me, as a custom processor
> writer, the ability to do this (ostensibly in already existing custom
> processors I've written) in preference to adding yet another dedicated,
> custom processor?
>
> As new as NiFi is, great functionality is still hidden behind the fact
> that no one's using it and no one's writing about it, giving examples, etc.
> I was surprised to read this thread just now and wondered if it's another
> cool thing I've missed.
>
> Thanks for any confirmation, other comments, etc.
>
> Russ
>
> On 12/09/2016 08:42 AM, Aldrin Piri wrote:
>
>> Hello,
>>
>> Counters are a framework level item that allows processors to provide
>> counts on things whilst processing.  This functionality must be exercised
>> via the ProcessSession[1] within a processor's code.  To add one, you
>> would
>> need to invoke the adjustCounter method in your processor.
>>
>> [1]
>> https://github.com/apache/nifi/blob/master/nifi-api/src/main
>> /java/org/apache/nifi/processor/ProcessSession.java#L161
>>
>> On Fri, Dec 9, 2016 at 3:40 AM, bingogo1986  wrote:
>>
>> hi
>>> I noticed 'Counter' button on top right of Nifi UI ，how to aad a
>>> specified
>>> counter ，thanks.
>>> Best regards.
>>>
>>
>

Re: ValidateCSV

[James Wing]

Shashi,

The ValidateCsv processor's "Header" property allows the header to be
ignored in validation, but it does not have a special feature to validate
the header itself.  I do not believe there is a similar feature for
ignoring a footer.

Can you expand a bit more on your use case and what kind of validation you
want to do?

Thanks,

James

On Wed, Dec 7, 2016 at 4:59 AM, Shashi  wrote:

> Hi Team,
>
> i have a file with header , footer and delimited with '|'.
>
> Assit me on how to provide schema in ValidateCSV processor  to validate my
> header and footer.
>
>
>
> Regards,
> Shashi.
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/ValidateCSV-tp14153.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: [DISCUSS] NiFi 1.1.0 release

[James Wing]

e down fairly quickly as long as we don't let the
> >>>> list grow.
> >>>>
> >>>> Thanks
> >>>> joe
> >>>>
> >>>> On Fri, Oct 14, 2016 at 4:39 PM, Edgardo Vega <edgardo.v...@gmail.com
> >
> >>>>
> >>>> wrote:
> >>>>
> >>>> Joe,
> >>>>
> >>>> Appreciate the offer it isn't my PR. I was just using it as an
> >>>>
> >>>> example.
> >>>>
> >>>> All
> >>>>
> >>>> mine are currently closed, which I greatly appreciate.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Edgardo
> >>>>
> >>>> On Friday, October 14, 2016, Joe Witt <joe.w...@gmail.com> wrote:
> >>>>
> >>>> Edgardo,
> >>>>
> >>>> You mentioned a PR from August. I'd be happy to help you work that
> >>>> through review.
> >>>>
> >>>> Thanks
> >>>> Joe
> >>>>
> >>>> On Fri, Oct 14, 2016 at 10:45 AM, Edgardo Vega <
> >>>>
> >>>> edgardo.v...@gmail.com
> >>>>
> >>>> <javascript:;>> wrote:
> >>>>
> >>>> I have agreed that at this point a release is important. My goal
> >>>>
> >>>> was
> >>>>
> >>>> try
> >>>>
> >>>> to
> >>>>
> >>>> squeeze in a much goodness as possible into the release, but the
> >>>>
> >>>> important
> >>>>
> >>>> bug fixes should come first. Getting 1.x into a state where the
> >>>>
> >>>> release
> >>>>
> >>>> notes don't say that it is geared toward developers and testers is
> >>>>
> >>>> really
> >>>>
> >>>> huge.
> >>>>
> >>>> I think Nifi is a great community otherwise I would participate in
> >>>>
> >>>> the
> >>>>
> >>>> mailing list, create Jira tickets and pull requests. I am only
> >>>>
> >>>> trying to
> >>>>
> >>>> strengthen the great thing that is going on here. We can always do
> >>>>
> >>>> better.
> >>>>
> >>>> I was not trying to put down this community only to participate and
> >>>>
> >>>> make
> >>>>
> >>>> it
> >>>>
> >>>> better. I think this conversation is an indication of how great
> >>>>
> >>>> this
> >>>>
> >>>> community is.
> >>>>
> >>>> Maybe I am being sensitive about this issue and trying to
> >>>>
> >>>> strengthen
> >>>>
> >>>> the
> >>>>
> >>>> nifi community even more, after coming from a conference where it
> >>>>
> >>>> was
> >>>>
> >>>> reported there was lots of excitement at first and now the
> >>>>
> >>>> participation
> >>>>
> >>>> in
> >>>>
> >>>> the community has really died down and they are struggling. I don't
> >>>>
> >>>> want
> >>>>
> >>>> to
> >>>>
> >>>> see that happen here.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Edgardo
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Oct 14, 2016 at 9:37 AM, Andre <andre-li...@fucs.org
> >>>>
> >>>> <javascript:;>> wrote:
> >>>>
> >>>>
> >>>> Edgardo,
> >>>>
> >>>> Thank you for your feedback. We hear your comments and as a
> >>>>
> >>>> committer I
> >>>>
> >>>> can
> >>>>
> >>>> share we are constantly looking to improve the PR process, having
> >>>>
> >>>> already
> >>>>
> >>>> taken many of the steps you suggest.
> >>>>
> >>>> However, it is important to notice that the number of PRs should
> >>>>
> >>>

Re: [ANNOUNCE] New Apache NiFi PMC Member - Andre Fucs de Miranda

[James Wing]

Congratulations, Andre!

On Thu, Nov 3, 2016 at 7:37 PM, Joe Witt  wrote:

> Team,
>
> On behalf of the Apache NiFi PMC, I am pleased to announce that Andre
> Fucs de Miranda has accepted the PMC's invitation to join the Apache
> NiFi PMC.  Andre's excellent contributions to the project reflect the
> very best of the power of the Apache Way.  Andre started out in the
> project as an interested user in October of 2015.  He progressed from
> asking usage questions to contributing ideas, joined the developer
> list, started reviewing code and contributing new code, participated
> in votes and discussions and earned committer status.  Since then he
> has continued those contributions and gone well beyond by working to
> reduce our open PR and JIRA count and ensure hygiene in our Git
> branches.
>
> Please join us in congratulating and welcoming Andre to the Apache NiFi
> PMC.
>
> Congratulations and welcome, Andre!
>

Re: [VOTE] Release Apache NiFi 0.7.1 (RC1)

[James Wing]

+1, I was able to verify hashes, build with tests and checks, run NiFi and
do simple testing.  Thanks for putting this release together, Joe.


On Sun, Oct 16, 2016 at 8:32 PM, Joe Skora  wrote:

> Hello,
>
> I am pleased to be calling this vote for the source release of Apache NiFi
> nifi-0.7.1.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1091
>
> The Git tag is nifi-0.7.1-RC1
> The Git commit ID is 421d5e61553e5fa160af9e0cc9fdc237af46906d
> *
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 421d5e61553e5fa160af9e0cc9fdc237af46906d
> *
> https://github.com/apache/nifi/commit/421d5e61553e5fa160af9e0cc9fdc2
> 37af46906d
>
> Checksums of nifi-0.7.1-source-release.zip:
> MD5: a15fc40ec887d82440f2de05ef71f810
> SHA1: 1565f4e123478e91fd26022b939d9d2f6ea6a2cf
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jskora.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 41 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12338025
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version0.7.1
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.  The
> please vote:
>
> [ ] +1 Release this package as nifi-0.7.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...
>

Re: Closing on solving NIFI-1500 - NiFi requires too much write permissions to bootstrap

[James Wing]

Andre,

Is your target for NIFI-1500 the default installation permission scheme, or
just that NiFi does not fail to start without the permissions in a
customized scheme?  Would it be acceptable to distinguish between
permissions required to initialize NiFi the first time, and the permissions
required for ongoing use?

With respect to #3, conf directory permissions -- I like the
./conf/flows/... directory idea (or ./flows/...).  But there are other
writes to ./conf, including:

* Start-time initialization of users.xml, authorizations.xml, conversion
from legacy authorized-users.xml, etc.
* Existing UI for configuration of policies, users, and groups
* Any future UI-driven management options

I believe these can be optional to the experienced admin, but the default
installation requires write access to create and update these files.

Thanks,

James

On Tue, Sep 27, 2016 at 8:13 AM, Andre  wrote:

> devs,
>
> A while ago (0.4.0 IIRC) we had a brief exchange of messages around the
> permissions NiFi requires to run (NIFI-1500).
>
> The debate revolved mostly around 4 things:
>
> 1 - write access to $NIFI_HOME/bin
>
> 2. write access to $NIFI_HOME/lib - NIFI-2818  / #1059 (review is welcome)
>
> 3. write access to $NIFI_HOME/conf (i.e. by default the flow is saved under
> "conf/")
>
> 4. write access to $NIFI_HOME/. (i.e. creates the missing repo and working
> folders upon boot).
>
> Good news is that inspection of the code suggests #1 has been solved when
> we re-wrote the nifi.sh script and that #3 and #4 may be solved just by
> small changes to default configuration and documentation.
>
>
> Would anyone have thoughts on what would be the preferred approach to deal
> with issues #3 and #4?
>
>
> IMNSHO, the least impacting way of addressing #3 is to modify the default
> behaviour, and ship NiFi with the following settings:
>
> nifi.flow.configuration.file=./conf/flows/flow.xml.gz
> nifi.flow.configuration.archive.dir=./conf/flows/archive/
>
> instead of the currently used:
>
> nifi.flow.configuration.file=./conf/flow.xml.gz
> nifi.flow.configuration.archive.dir=./conf/archive/
>
> This way, ./conf and all files could be owned by root.root and have fs
> permissions set to 755 (drwxr-xr-x)
>
> while conf/flows could be set to runsasuser.runasgroup and fs permissions
> set to 700 (drwx--).
>
>
> #4 Can be solved by modifying the pom files to add the adequate directory
> structure to the  tar and gzip archives (i.e. pre-populating the directory
> structure) or by adjusting rpm and deb files
>
>
> We would also update the documentation to ensure people installing NiFi are
> informed of the ideal filesystem permissions.
>
>
> Would everyone be in agreement with this approach?
>
>
> On a related note:
>
> I would truly appreciate if we could get some eyes over PR-1059 as early as
> possible . Whilst a minor change to the code, I suspect it needs to be well
> thought off and tested before we commit.
>
>
> Cheers
>

Re: [ANNOUNCE] New Apache NiFi Committer Rob Moran

[James Wing]

Congratulations, Rob, and welcome!

On Mon, Sep 19, 2016 at 12:08 PM, Tony Kurc  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that Rob
> Moran has accepted the PMC's invitation to become a committer on the Apache
> NiFi project. We greatly appreciate all of Rob's hard work and generous
> contributions to the project. We look forward to his continued involvement
> in the project.
>
> Among other things, Rob has been responsible for a lot of the look and feel
> of the NiFi user interface, to include the awesome logos!
>
> Welcome and congratulations!
>
> Tony
>

Re: Nifi : FetchS3 object

[James Wing]

I do not believe FetchS3Object currently supports requester-pays buckets.
According to the documentation, we would have to include a header,
x-amz-request-payer, to signal approval of the download charges.  NiFi
currently has no option to include that header.

http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html

If you are interested in helping to develop this feature, I encourage you
to open a JIRA ticket at https://issues.apache.org/jira/browse/NIFI.

Thanks,

James

2016-09-14 6:16 GMT-07:00 Selvam Raman :

> Hi,
>
> Does the fetchs3 processor supports aws requester pays bucket.
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>

Re: Nifi Debugging AWS PutS3 processor

[James Wing]

I'm sorry to hear you're having trouble with PutS3.  A few questions:

   - Which version of NiFi are you using?
   - Have you configured the Proxy Host/Port or Endpoint Override URL
   settings on your PutS3 processor?
   - Are there larger exception stack traces in the nifi-app.log files you
   can share?
   - Some of the errors mention Multi-part Upload.  Are large flowfiles
   failing, but smaller files successful?
   - The errors in the screenshot reflect two IP addresses, but it may just
   be a small sample.  Are the errors evenly distributed across all the
   cluster nodes?

Thanks,


James


On Fri, Sep 9, 2016 at 11:19 AM, jmurakami 
wrote:

> Hi. I am currently using the putS3 processor and getting an
> AmazonClientException error: Connection Timed Out. We are running the flow
> on a Nifi cluster (of 3 instances and a cluster manager). We see the error
> attached below and about a third of the data not being put into the bucket.
>
> We were able to use the aws cli in an instance in our Nifi cluster and
> successfully put data into our S3 bucket, so the security groups, Nacl,
> subnets, and bucket policies should be okay. I don't believe it is a
> credential or key error since some of the data was put into the bucket
> already. We are using IAM roles to allow access to the S3 bucket.
>
> Has anyone else encountered this error?
>  n13311/Screen_Shot_2016-09-06_at_10.png>
>
> Or any advice about how to go about this error would help. Thanks!
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Nifi-Debugging-AWS-PutS3-processor-tp13311.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Nifi Cross Account Download With A Profile Flag

[James Wing]

Keren,

I'm certain cross-account access can work in 0.6.0, I've done it.

The timeout error calling sts:AssumeRole is not the same error you started
with, right?  Earlier, the error was 403 "Access Denied", which would have
been farther in the auth process.  Any idea what changed?  Are you using
the HTTP proxy settings on FetchS3Object?


Thanks,

James

On Thu, Sep 1, 2016 at 10:30 AM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Hey James,
>
> No problem. I’ve tried running a bunch of different ways to do it manually
> to try and avoid the CLI profile flag. But it seems like that is the only
> it works locally (without Nifi). My cross account role doesn’t require an
> External ID, and that feature doesn’t exist in 0.6.0 either.
>
> An small portion of the logs are below. The logs show that it starts to
> attempt to get the S3 object, it times out on assuming the
> role/credentials, and then fails.
>
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.startSession(STS
> AssumeRoleSessionCredentialsProvider.java:272)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:247)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:34)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> invoke(A
> WSSecurityTokenServiceClient.java:1098) [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> assumeRo
> le(AWSSecurityTokenServiceClient.java:1000)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.startSession(STS
> AssumeRoleSessionCredentialsProvider.java:272)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:247)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:34)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3589)
> [aws-java-sdk-s3-1.10.32.jar:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(
> AmazonS3Client.java:1116
> ) [aws-java-sdk-s3-1.10.32.jar:na]
> at
> org.apache.nifi.processors.aws.s3.FetchS3Object.
> onTrigger(FetchS3Object.jav
> a:105) [nifi-aws-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.jav
> a:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
>
>
> If you (or anyone scanning the thread) can think of a way to do this
> without upgrading that’d be awesome. Otherwise, I’ll start motivating the
> masses.
>
> Best,
> Keren
>
> On 9/1/16, 12:17 PM, "James Wing" <jvw...@gmail.com> wrote:
>
> >Keren,
> >
> >I'm sorry if my advice is a bit confusing, there have been some changes to
> >AWS credentials over the last few versions.  NiFi 0.6.0 does not have the
> >option to use a CLI profile in AWSCredentialsProviderControllerService, I
> >think that was introduced in 0.7.0.
> >
> >Would it be possible to share some of the log entries?  Was there a stack
> >trace associated with the timeout?
> >
> >Also, does your cross-account Role require an External ID?  I do not
> >believe that is supported in 0.6.0, but can be required to assume some
> >roles.
> >
> >
> >Thanks,
> >
> >James
> >
> >On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
> >keren.tseyt...@capitalone.com> wrote:
> >
> >> Thanks for your responses!
> >>
> >> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
> >>
> >> I¹ve set up debugging, and it shows me that it¹s trying to connect, but
> >>it
> >> times out on connecting. It would be awesome if it would also return the
> >> account ID of the credentials it is trying to use.
> >>
> >> Is there any way to see the exact keys/tokens that Nifi is trying to use
> >> to get the S3 object? I¹m not seeing it in the logs.
> >>
> >> I tried to set the Profile in Nifi, but it complains that it¹s not a
> >>valid
> >> property.
> >>
&g

Re: Nifi Cross Account Download With A Profile Flag

[James Wing]

Keren,

I'm sorry if my advice is a bit confusing, there have been some changes to
AWS credentials over the last few versions.  NiFi 0.6.0 does not have the
option to use a CLI profile in AWSCredentialsProviderControllerService, I
think that was introduced in 0.7.0.

Would it be possible to share some of the log entries?  Was there a stack
trace associated with the timeout?

Also, does your cross-account Role require an External ID?  I do not
believe that is supported in 0.6.0, but can be required to assume some
roles.


Thanks,

James

On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Thanks for your responses!
>
> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
>
> I¹ve set up debugging, and it shows me that it¹s trying to connect, but it
> times out on connecting. It would be awesome if it would also return the
> account ID of the credentials it is trying to use.
>
> Is there any way to see the exact keys/tokens that Nifi is trying to use
> to get the S3 object? I¹m not seeing it in the logs.
>
> I tried to set the Profile in Nifi, but it complains that it¹s not a valid
> property.
>
> Best,
> Keren
>
> On 8/31/16, 6:24 PM, "Andrew Grande" <apere...@gmail.com> wrote:
>
> >Debug logging can be set in a processor itself in the UI, too.
> >
> >On Wed, Aug 31, 2016, 5:34 PM James Wing <jvw...@gmail.com> wrote:
> >
> >> Keren,
> >>
> >> Which version of NiFi are you using?
> >>
> >> One thing I noticed in your configuration of FetchS3Object is you are
> >> setting both the Access Key and Secret Key properties with the AWS
> >> Credentials Provider.  When you are using the AWS Credentials Provider
> >> Service, you should not specify keys.
> >>
> >> A more certainly helpful thing to do is enable debug logging for the AWS
> >> processor package by adding a line like the following to
> >>conf/logback.xml:
> >>
> >> 
> >>
> >> With the debug logging enabled, there are messages indicating which
> >> credential type is being attempted.  Your settings for the AWS
> >>Credentials
> >> Provider look appropriate.  The controller service is indeed designed to
> >> refresh the STS token automagically using the AWS SDK classes for
> >>temporary
> >> credentials.
> >>
> >> Last, you might experiment with configuring
> >> AWSCredentialsProviderControllerService to use your named CLI profile
> >> "crossaccountrole", which should also work.
> >>
> >> Thanks,
> >>
> >> James
> >>
> >> On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
> >> keren.tseyt...@capitalone.com> wrote:
> >>
> >> > Hi All!
> >> >
> >> > Looking for some help on enabling Cross Account communication within
> >> Nifi!
> >> >
> >> > My goal: There are files stored from CloudTrail in an S3 bucket in
> >>VPC B.
> >> > My Nifi machines are in VPC A. I want Nifi to be able to get those
> >>files
> >> > from VPC B. VPC A and VPC B need to be communicating in the
> >>FetchS3Object
> >> > component.
> >> >
> >> > See this link for some additional info: http://docs.aws.amazon.com/
> >> >
> >>awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
> >> >
> >> > I have communication working manually on the Nifi machines in VPC A
> >>when
> >> I
> >> > use the AWS CLI. The process is as follows:
> >> >
> >> > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a
> >>role
> >> > I've created in VPC B that is configured to have access to the S3
> >>bucket
> >> in
> >> > VPC B.
> >> >
> >> > 2. This will generate temporary keys that need to be refreshed
> >>every
> >> > hour. There is no way to have assume role create permanent keys.
> >>Export
> >> the
> >> > keys as environment variables.
> >> >
> >> > 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
> >> > connects to the arn of the role created in VPC B.
> >> >
> >> > 4. Run the following command à "aws s3 cp s3://
> >> >> > name locally> --profile crossaccountrole"
> >> >
> >> > Most importantly, if I ever try to run this without the --profile
> >>f

Re: Nifi Cross Account Download With A Profile Flag

[James Wing]

Keren,

Which version of NiFi are you using?

One thing I noticed in your configuration of FetchS3Object is you are
setting both the Access Key and Secret Key properties with the AWS
Credentials Provider.  When you are using the AWS Credentials Provider
Service, you should not specify keys.

A more certainly helpful thing to do is enable debug logging for the AWS
processor package by adding a line like the following to conf/logback.xml:



With the debug logging enabled, there are messages indicating which
credential type is being attempted.  Your settings for the AWS Credentials
Provider look appropriate.  The controller service is indeed designed to
refresh the STS token automagically using the AWS SDK classes for temporary
credentials.

Last, you might experiment with configuring
AWSCredentialsProviderControllerService to use your named CLI profile
"crossaccountrole", which should also work.

Thanks,

James

On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Hi All!
>
> Looking for some help on enabling Cross Account communication within Nifi!
>
> My goal: There are files stored from CloudTrail in an S3 bucket in VPC B.
> My Nifi machines are in VPC A. I want Nifi to be able to get those files
> from VPC B. VPC A and VPC B need to be communicating in the FetchS3Object
> component.
>
> See this link for some additional info: http://docs.aws.amazon.com/
> awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
>
> I have communication working manually on the Nifi machines in VPC A when I
> use the AWS CLI. The process is as follows:
>
> 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a role
> I've created in VPC B that is configured to have access to the S3 bucket in
> VPC B.
>
> 2. This will generate temporary keys that need to be refreshed every
> hour. There is no way to have assume role create permanent keys. Export the
> keys as environment variables.
>
> 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
> connects to the arn of the role created in VPC B.
>
> 4. Run the following command à "aws s3 cp s3://  name locally> --profile crossaccountrole"
>
> Most importantly, if I ever try to run this without the --profile flag,
> then it will not allow me to download the file.  It seems like perhaps to
> get it to work with Nifi I need a place to pass in the profile that needs
> to be used in order for the communication to work.
>
> I've been trying to implement this in Nifi. Within the FetchS3Object, I
> have created an AWSCredentialsProviderService which has the following
> properties:
>
> ·  Access Key: VPC A access key
>
> ·  Secret Key: VPC A secret key
>
> ·  Assume Role ARN: VPC B role
>
> ·  Assume Role Session Name: crossaccountrole
>
> ·  Session Time: 3600
> The general properties in the FetchS3Object are as follows:
>
> ·  Bucket: VPC B bucket name
>
> ·  Object: Filename of VPC B bucket object
>
> ·  Access Key: VPC A access key
>
> ·  Secret Key: VPC A secret key
>
> ·  AWS Credentials Provider Service: 
>
> However, when this tries to run I get Access Denied. I've been going
> through the source code for Nifi and I'm not sure if short-lived tokens get
> passed through. Can anyone please provide me some guidance or suggestions
> on how to get this to work? J
>
> Best,
> Keren
> 
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Re: [VOTE] Release Apache NiFi 1.0.0 (RC1)

[James Wing]

+1 (non-binding).  I ran through the release helper, including hashes,
build, tests.  No showstopping issues for me.

James

On Fri, Aug 26, 2016 at 9:25 AM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache NiFi,
> nifi-1.0.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1090/
>
> Note: There is a second binary being distributed now, the NiFi Toolkit. It
> can be used to facilitate securing a NiFi instance.
>
> The Git tag is nifi-1.0.0-RC1
> The Git commit hash is 74d5224783dfdc513f6b3ad5ed96671d3c581707
> * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 74d5224783dfdc513f6b3ad5ed96671d3c581707
> * https://github.com/apache/nifi/commit/74d5224783dfdc513f6b3ad5ed9667
> 1d3c581707
>
> Checksums of nifi-1.0.0-source-release.zip:
> MD5: 8bdba49a73b94d036fad6c63b0ebe39d
> SHA1: 504c58f9b2fb305c41598a17f5b78f68f2b2fa3d
> SHA256: 22167ede5127683ca8de6dbd2fb9112cb1de650b7cfff7e640c905521447af92
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jpercivall
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 595 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12332640
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.0.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.0.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
> Thanks!
>

Re: How can we validate a data flow.?

[James Wing]

NiFi has several features to provide monitoring and verification of your
data flow:

1.) Provenance Data (
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance)
NiFi's Provenance subsystem tracks every flowfile, how it was modified, and
how it exited NiFi.  It is the "log of all activities" you mentioned.  The
NiFi UI includes a tool for browsing the provenance data to find and
examine individual flowfiles.  However, I am not aware of a tool to compile
aggregate statistics from provenance data in the form you describe.  But
there is an API to query provenance records, and a Reporting Task to export
them to another NiFi for further processing.

REST API - https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
SiteToSiteProvenanceReportingTask -
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.reporting.SiteToSiteProvenanceReportingTask/index.html

2.) Controller Status Reporting
The 5-minute rolling window status that you see in the NiFi UI can also be
logged by the ControllerStatusReportingTask.  It is very easy to set up,
and the ongoing series of these log entries can provide basic monitoring
for the health of your flow.
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controller.ControllerStatusReportingTask/index.html

3.) Flow Design
It is also possible to design some monitoring and validation into the flow
itself.  For example, there is a MonitorActivity processor to detect an
absence of flowfiles.  Also, you mentioned MergeContent, which can output
both the merged and the original flowfiles, so you could compile a list of
IDs from the original files to double-check later.

MonitorActivity -
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MonitorActivity/index.html


Thanks,


James

On Fri, Aug 12, 2016 at 10:48 AM, saikrishnat  wrote:

> Hi,
> I need to find out if there is a way to validate the data flow end-to-end
> and best practises for it.
> lets say if i have thousands of files that i want to move to a different
> destinations based on some attributes and\or contents. after the job is
> finished i want a complete track of all files like how many files are
> processed successfully , how many files are failed and what are those. if i
> use Merge process what files are merged under each bin etc. so that it will
> be easy to compare the source to destination.
> can anything along these lines be done in NiFi.?? kind of log of all
> activities.?
>
> Regards,
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/How-can-we-validate-a-data-flow-tp13036.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Question on template handling in 1.0

[James Wing]

Thanks for picking this up, Joe.  The short recap:

In 1.0, the import template option is provided only from the floating
Actions toolbar rather than the Templates dialog as it was in 0.x.  I asked
on the ticket why it moved.  If I understand Matt Gilman's explanation, the
reason is that importing templates is tied to the users's permissions
within the current process group context.  In the richer 1.0 permissions
model, a user might have permissions to upload a template into a particular
process group, but not globally.  The Actions toolbar is explicitly tied to
the current context rather than a global artifact like the Templates dialog.

-

This is not a critical issue by any means.  It works and even I was able to
figure it out.  I understand and approve of how the permission model
complicates template imports in a good way, but it was an abrupt change
from 0.x.  I won't be the last guy to stare at the Templates dialog and
wonder if I need an eye exam or a browser update. I'm curious how others
experience this and if it is worth further attention. I admit to being
short on good constructive suggestions that would do a better job either at
helping user migrate or helping them adopt the 1.0 mindset.

Helping Users Migrate:
* Same Old Import Button - Include the same old import button in the
Templates dialog out of unfettered optimism that the user has permissions
to import templates to the root process group.  Maybe we could invalidate
the button if the user does not have the right permissions?
* Hint in Templates Dialog - Add helpful text directly to the Templates
dialog that announces or explains the change, especially for the first one
or two 1.x releases.
* Help Documentation - Update the User Guide to explain the 1.0 concept and
the change from 0.x.  I wish I could say I looked at the help before
emailing, but I didn't :(.  It may need an update anyways, I'll create a
ticket.

Pushing the 1.0 Mindset:
* Templates by Process Group - I now notice the Process Group ID in the
Templates dialog, but I did not previously appreciate it's significance.
If the templates where listed in a "group by" style layout by process
group, this association might be more obvious.  Downside would be a lot
more UI work, and it might use a lot of dialog real estate in a complex
flow.
* Show Templates in Context - List the templates currently available to the
current context, as an extension to the Actions toolbar or yet another
floating window (yeah, I know).  This might help reinforce the association
and provide feedback as to where templates went.


Thanks,

James

On Tue, Aug 9, 2016 at 10:08 AM, Joe Witt <joe.w...@gmail.com> wrote:

> James Wing asked a good question on a closed JIRA that I didn't want
> to get lost (closed things get far less attention usually).
>
> https://issues.apache.org/jira/browse/NIFI-1781?page=
> com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel=15413792#comment-15413792
>
> Recommend for discussions on non-active items we bring them to dev so
> they have a better chance of being addressed.
>
> On this topic, James, I had wondered the same thing.  I've actually
> found I quite like the current approach now that I've become used to
> it but that said I think it is good to discuss it.
>
> Usability discussions are good to have and they don't have to be tied
> to any particular release.  We can keep improving this as we go.
>
> Thanks
> Joe
>

Re: [VOTE] Release Apache NiFi 1.0.0-BETA (RC1)

[James Wing]

+1, I think this is an appropriate Beta.  I ran through the Release Helper,
verified hashes and signatures, built and ran tests, and did some light
testing.

Thanks for putting this together, Joe.

James

On Fri, Aug 5, 2016 at 1:40 PM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache NiFi,
> nifi-1.0.0-BETA.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1089/
>
> The Git tag is nifi-1.0.0-BETA-RC1
> The Git commit hash is 8abcf2ef71334ce8861a81d50f1705b26852a06f
> * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 8abcf2ef71334ce8861a81d50f1705b26852a06f
> * https://github.com/apache/nifi/commit/8abcf2ef71334ce8861a81d50f1705
> b26852a06f
>
> Checksums of nifi-1.0.0-BETA-source-release.zip:
> MD5: 830a570cebf04aca8a95f3cd5d58594d
> SHA1: abdcf4655d5e237b94cab579de90db569b938bee
> SHA256: 2d3fdff4b284de3a9f9db654ad0c7a0c34760126fae919ca55ba58f309a4338a
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jpercivall
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 446 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12338066Release note highlights can be found
> here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.0.0-Beta
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-1.0.0-BETA
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
> Thanks!
>

0.x Build Breaking on nifi-kafka-pubsub-processors?

[James Wing]

It looks like the 0.x build is breaking on nifi-kafka-pubsub-processors.
Is this a known issue?
(see Travis build https://travis-ci.org/apache/nifi/jobs/150124534, for
https://issues.apache.org/jira/browse/NIFI-2423)

[INFO]

[INFO] Building nifi-kafka-pubsub-processors 0.8.0-SNAPSHOT
[INFO]

[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
nifi-kafka-pubsub-processors ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
nifi-kafka-pubsub-processors ---
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @
nifi-kafka-pubsub-processors ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @
nifi-kafka-pubsub-processors ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 6 source files to
/home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/target/classes
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR] bootstrap class path not set in conjunction with -source 1.7
/home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/AbstractKafkaProcessor.java:[361,52]
error: cannot find symbol
[ERROR]   symbol:   method getKeyPassword()
  location: variable sslContextService of type SSLContextService
/home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/AbstractKafkaProcessor.java:[361,139]
error: cannot find symbol
[INFO] 2 errors

Re: PutS3Object error

[James Wing]

ListS3 processor is a great place to start.  I can think of a couple of
likely causes:

* Credentials - you might want to double-check how you specified the
credentials to make sure it is right.  If you specify access key and secret
key, and the secret key is not correct, I believe you will get this error.

* Region - you should make sure the region you specify in the processors
matches the S3 bucket's region.  S3 is a bit weird about the rules for
buckets and regions, but it might help to check.

Thanks,

James

On Wed, Jul 20, 2016 at 12:57 PM, Nabegh  wrote:

> Hi James,
>
> I'm using nifi-0.7.0-RC2. I tried ListS3 processor and I received the same
> error.
>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/PutS3Object-error-tp12878p12880.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: PutS3Object error

[James Wing]

Nabegh,

Can you share which version of NiFi you are using?  Have you tried
GetS3Object?  I recommend also trying GetS3Object also to separate general
problems with authenticating and connecting to S3 from particular issues
putting objects, putting can be more complicated.

Thanks,

James


On Wed, Jul 20, 2016 at 12:09 PM, Nabegh  wrote:

> Hi .. PutS3Object processor is throwing the follow error .. Any idea why?
>
>
> 2016-07-20 15:25:41,922 ERROR [Timer-Driven Process Thread-6]
> o.a.nifi.processors.aws.s3.PutS3Object
> com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we
> calculated does not match the signature you provided. Check your key and
> signing method. (Service: Amazon S3; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID: D433C836E1584299)
> at
>
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1219)
> ~[aws-java-sdk-core-1.10.32.jar:na]
> at
>
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:803)
> ~[aws-java-sdk-core-1.10.32.jar:na]
> at
>
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:505)
> ~[aws-java-sdk-core-1.10.32.jar:na]
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317)
> ~[aws-java-sdk-core-1.10.32.jar:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3595)
> ~[aws-java-sdk-s3-1.10.32.jar:na]
> at
>
> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1382)
> ~[aws-java-sdk-s3-1.10.32.jar:na]
> at
>
> org.apache.nifi.processors.aws.s3.PutS3Object$1.process(PutS3Object.java:446)
> ~[nifi-aws-processors-0.7.0.jar:0.7.0]
> at
>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1851)
> ~[na:na]
> at
>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1822)
> ~[na:na]
> at
>
> org.apache.nifi.processors.aws.s3.PutS3Object.onTrigger(PutS3Object.java:400)
> ~[nifi-aws-processors-0.7.0.jar:0.7.0]
> at
>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-0.7.0.jar:0.7.0]
> at
>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
> [nifi-framework-core-0.7.0.jar:0.7.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_91]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_91]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_91]
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_91]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/PutS3Object-error-tp12878.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: [VOTE] Release Apache NiFi 0.7.0 (RC2)

[James Wing]

+1 (non-binding)

I ran through the release helper and lightly tested the generated binary.


On Sat, Jul 9, 2016 at 11:47 AM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache NiFi,
> nifi-0.7.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1088/
> The Git tag is nifi-0.7.0-RC2
> The Git commit hash is f5629062c5e2a6c55fb62255aee74c4f25d93e7b
> *
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=f5629062c5e2a6c55fb62255aee74c4f25d93e7b
> *
> https://github.com/apache/nifi/commit/f5629062c5e2a6c55fb62255aee74c4f25d93e7b
>
> Checksums of nifi-0.7.0-source-release.zip:
> MD5: 3a6af39c481fb0ad3eab5a4ea1a954fd
> SHA1: 33e16b0a73242ddda91f89591c82aa5d6e9c8d21
> SHA256: 56ab3132c1fef31dcf50b716b8e08272075f92e476849360280822e0cdc3e993
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jpercivall
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 138 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12335078
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.7.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-0.7.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
> Thanks!
>

Re: [VOTE] Release Apache NiFi 0.7.0 (RC1)

[James Wing]

This feels like a newbie question, but the release tag "nifi-0.7.0-RC1"
points to commit c9d9485, one commit ahead of the stated release commit
56dc8ae.  Is that expected as part of generating the RC?

Thanks,

James

On Fri, Jun 24, 2016 at 12:34 PM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Hello Apache NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache NiFi,
> nifi-0.7.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1083/
>
> The Git tag is nifi-0.7.0-RC1
> The Git commit hash is 56dc8aefac822454f2d4133e8cbde903a628ec5b
> *
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=56dc8aefac822454f2d4133e8cbde903a628ec5b
> *
> https://github.com/apache/nifi/commit/56dc8aefac822454f2d4133e8cbde903a628ec5b
>
> Checksums of nifi-0.7.0-source-release.zip:
> MD5: 8dc31d1f8103f9679cfe4b9533a4049b
> SHA1: dc160c9611dc18523c994cd9086fc7088cc520f4
> SHA256: 7df056fb6af0eb9bdb94c17bf5bc2db288ed2b0d0a8ad23127dab5cdf74ce767
>
> Release artifacts are signed with the following key:
> http://pgp.mit.edu/pks/lookup?op=get=0x3C6DBAFE22886FEE
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 131 issues were closed/resolved for this release:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12335078
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.7.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
>
> [ ] +1 Release this package as nifi-0.7.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>

Re: Using NiFi S3 Processors with S3-Compatible Services?

[James Wing]

Eric, I believe you are correct that Amazon is pushing for v4, and
compliance is only optional in certain regions, for a while.  I can see
NiFi getting caught between AWS users wanting latest-greatest support and
compatible service users preferring legacy support.  I believe signature v4
is mature enough at this point that most compatible services like Riak and
Ceph have had a chance to update, but I'm not very familiar with the space.

Joe, I like your thinking, I opened a JIRA and a PR for an AWS SDK update
in 1.0.  In recent SDKs, Amazon is in fact making signature v4 the default
in more (all?) regions, so upgrading may have a side effect for compatible
service users.
https://issues.apache.org/jira/browse/NIFI-2062
https://github.com/apache/nifi/pull/562

We also had a brief discussion in PR 362 about how to support SSE-KMS, but
keep the signature version configurable (
https://github.com/apache/nifi/pull/362).

Thanks,

James

On Thu, Jun 23, 2016 at 10:47 AM, Eric Olson <eol...@softnas.com> wrote:

> I will tell you that there is some AWS regions that will only work with V4
> when it comes to S3. I know for sure this is true of Frankfurt region for
> example. So it is something worth considering
>
> Eric
>
> On 6/23/16, 1:44 PM, "Joe Skora" <jsk...@gmail.com> wrote:
>
> >Jim,
> >
> >I don't have time to research this more fully at the moment, but the S3
> >processors use the AWS libraries that should know about the API versions.
> >
> >We may be bundling older library versions that pre-date the new API, but
> >that should be easy to verify.  It looks like we bundle
> >"com.amazonaws:aws-java-sdk:1.10.32" currently.  If that is too old and
> >updating to a current library requires a lot of code changes, that seems
> >like it would probably be a 1.0.0 change.
> >
> >Regards,
> >Joe
> >
> >On Mon, Jun 20, 2016 at 2:40 PM, Tony Kurc <trk...@gmail.com> wrote:
> >
> >> James,
> >> I briefly used s3 ninja, and experienced no issues, but I was chiefly
> >>using
> >> it for integration testing.
> >>
> >> Tony
> >>
> >> On Mon, Jun 20, 2016 at 2:05 PM, James Wing <jvw...@gmail.com> wrote:
> >>
> >> > Is anyone using NiFi's S3 processors to work with non-Amazon,
> >> S3-compatible
> >> > services?  Would you be willing to share which service you are using
> >>and
> >> if
> >> > you have experienced compatibility issues?
> >> >
> >> > Are you aware of the difference between S3's API signature versions 2
> >>and
> >> > 4?  NiFi's current default S3 signature version is version 2.  Would
> >>you
> >> be
> >> > concerned about upgrading to a more recent AWS SDK where more
> >>operations
> >> > default to version 4?
> >> >
> >> > Changes in Signature Version 4
> >> > http://docs.aws.amazon.com/general/latest/gr/sigv4_changes.html
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > James
> >> >
> >>
>
>

Re: 0.7.0 install bug centos 7

[James Wing]

Thanks for reporting this, Ryan, I see the same behavior.  I created a JIRA
for this issue (https://issues.apache.org/jira/browse/NIFI-2063).  If you
need a workaround, you can edit /etc/init.d/nifi to hard-code the path to
the script directory:

SCRIPT_DIR=/opt/nifi/current/bin
SCRIPT_NAME=$(basename "$0")
PROGNAME=nifi.sh

. "$SCRIPT_DIR"/nifi-env.sh

...

Thanks,

James

On Mon, Jun 20, 2016 at 2:33 PM, Ryan H  wrote:

> Looks like 0.7.0 doesn't install as a service quite right on linux/centos
> 7.
>
> I untar'd it in:
>  #/opt/nifi/current -> nifi-0.7.0-SNAPSHOT
>
> I followed these steps:
> #/opt/nifi/current/bin/nifi.sh install
> #chkconfig nifi on
> #service nifi start
>
> That copies the nifi.sh script into /etc/init.d correctly, however it can't
> start as a service because the "dirname" when starting as a service (at
> least in my case) appears to be /etc/init.d, instead of the nifi bin dir,
> resulting in this error:
>
> #$ sudo service nifi start
> #/etc/init.d/nifi: line 28: /etc/init.d/nifi-env.sh: No such file or
> directory
>
> I changed the SCRIPT_DIR variable to be my current nifi bin dir, to get it
> to work.
>
>1. From: # SCRIPT_DIR=$(dirname "$0")
>2. To: # SCRIPT_DIR=/opt/nifi/current/bin
>
>
> Thanks,
> Ryan
>

Using NiFi S3 Processors with S3-Compatible Services?

[James Wing]

Is anyone using NiFi's S3 processors to work with non-Amazon, S3-compatible
services?  Would you be willing to share which service you are using and if
you have experienced compatibility issues?

Are you aware of the difference between S3's API signature versions 2 and
4?  NiFi's current default S3 signature version is version 2.  Would you be
concerned about upgrading to a more recent AWS SDK where more operations
default to version 4?

Changes in Signature Version 4
http://docs.aws.amazon.com/general/latest/gr/sigv4_changes.html


Thanks,

James

Re: [DISCUSS] Apache NiFi 0.7.0 and 1.0.0

[James Wing]

I'm definitely in favor of releasing 0.7.0, but I don't think we need be
rigid about the schedule.  If delaying 0.7.0 a few weeks (2-4?) helps pace
us towards a 1.0 in mid- to late-Summer, that seems reasonable to me.  Do
we believe that is still a likely target?

Thanks,

James

On Tue, May 17, 2016 at 7:30 AM, Joe Witt  wrote:

> Team,
>
> Want to start zeroing in on the details of the next releases.  We had
> a good set of discussions around this back in January and have since
> been executing along this general path [1].
>
> On the 0.x line the next release would be 0.7.0.  There does appear to
> be a lot of useful improvements/features/fixes there now and it is
> time to do a release according to our general 6-8 week approach.
> However, given all the effort going into 1.x I'd like to get a sense
> of what the community preference is.
>
> On the 1.0 line the release is coming into focus.  Some things have
> moved into 1.x and some things look like they'd slide to the right of
> 1.x as is to be expected.  For example distributed durability (HA
> Data) looks like a good thing to do post 1.0 given the substantive
> changes present from the new HA clustering approach and multi-tenant
> authorization.  I'd also like to dive in and liberally apply Apache
> Yetus annotations [2] to all the things so we can be really explicit
> about what parts we can more freely evolve going forward.  We've been
> a bit awkwardly hamstrung thus far without these so they should help
> greatly to better convey intent.
>
> For those really interested in things coming in the 1.0 release please
> take a look through the JIRAs currently there and provide comments on
> what is important to you, what you'd like to see moved out, in, etc..
> [3].  At this point there are still a lot of things which will likely
> need to move out to allow the release to occur in a timely fashion.
>
> Also, keep in mind our stated release line/support model as found here [4].
>
> [1]
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201601.mbox/%3CCALJK9a4dMw9PyrrihpPwM7DH3R_4v8b%3Dr--LDhK7y5scob-0og%40mail.gmail.com%3E
>
> [2]
> https://yetus.apache.org/documentation/0.2.1/audience-annotations-apidocs/
>
> [3]
> https://issues.apache.org/jira/browse/NIFI-1887?jql=fixVersion%20%3D%201.0.0%20AND%20project%20%3D%20NIFI
>
> [4]
> https://cwiki.apache.org/confluence/display/NIFI/Git+Branching+and+Release+Line+Management
>
> Thanks
> Joe
>

Re: [ANNOUNCE] New Apache NiFi Committer James Wing

[James Wing]

Thank you, I'm excited to be contributing to a great project, and promise
to use this position for good.  Mostly.

On Wed, May 11, 2016 at 1:10 PM, Joe Witt <joew...@apache.org> wrote:

> On behalf of the Apache NiFi PMC, I am pleased to announce that James
> Wing has accepted the PMC's inviation to become a committer on the
> Apache NiFi project.  James has been an excellent contributor to a
> number of code submissions, peer reviews, mailing list discussions and
> decision making and it is greatly appreciated.  We look forward to his
> continued involvement in the project.
>
> Welcome and congratulations!
>

Re: ControllerStatusReportingTask bug

[James Wing]

Chris,

I believe there is a JIRA for this issue, NIFI-1738 (
https://issues.apache.org/jira/browse/NIFI-1738), which has a fix waiting
for the next release.  You may want to confirm that it does what you think
it should.

Thanks,

James

On Sat, Apr 30, 2016 at 7:58 AM, Chris Conklin 
wrote:

> These loggers are not defined correctly:
>
> private static final Logger processorLogger =
> LoggerFactory.getLogger(ControllerStatusReportingTask.class +
> ".Processors");
> private static final Logger connectionLogger =
> LoggerFactory.getLogger(ControllerStatusReportingTask.class +
> ".Connections");
>
> In order to grab the logger you need to find:
> "class org.apache.nifi.controller.ControllerStatusReportingTask.Processors"
> "class
> org.apache.nifi.controller.ControllerStatusReportingTask.Connections"
> instead of:
> "org.apache.nifi.controller.ControllerStatusReportingTask.Processors"
> "org.apache.nifi.controller.ControllerStatusReportingTask.Connections"
>
>
>
>

Re: [ANNOUNCE] Apache NiFi 0.6.1 release

[James Wing]

Thanks, Joe.  Will the 0.6.1 git tag be published as part of the release?

On Mon, Apr 18, 2016 at 5:23 AM, Joe Witt  wrote:

> Hello
>
> The Apache NiFi team would like to announce the release of Apache NiFi
> 0.6.1.
>
> Apache NiFi is an easy to use, powerful, and reliable system to
> process and distribute data.  Apache NiFi was made for dataflow.  It
> supports highly configurable directed graphs of data routing,
> transformation, and system mediation logic.
>
> More details on Apache NiFi can be found here:
>   http://nifi.apache.org/
>
> The release artifacts can be downloaded from here:
>   http://nifi.apache.org/download.html
>
> Maven artifacts have been made available here:
>
> https://repository.apache.org/content/repositories/releases/org/apache/nifi/
>
> Release note highlights can be found here:
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.6.1
>
> Thank you
> The Apache NiFi team
>

Re: [DISCUSS] git branching model

[James Wing]

I have some rhetorical questions for discussion of the branching model:

1.) Commits merged to master today are destined for the next minor release,
currently 0.6.0, by default?

2.) Is master always open for merging new code, or are there restrictions
before or after releases?

3.) How long will support/0.5.x be maintained?

4.) Where is compatibility-breaking code destined for a future major
release stored?  Is it visible anywhere?

5.) A critical data/security bug found after 1.0 would eligible to be
backported only to the last minor release in the 0.x line, or to all minor
releases in the 0.x line?




On Thu, Feb 25, 2016 at 8:01 AM, Joe Witt  wrote:

> Given the discussion has stalled i'd like to turn it more toward a
> proposal as we're at a point now where we need to start executing some
> of these approaches.  We're actually already seeing it take form in
> the support/0.5.x branch and the master branch (which is for 0.6.0 at
> this point).
>
> The proposal then for Git processes based on the other thread [1]
> where we outline a support model:
>
> - We will have a branch for each major release line
>
> - The branch designated 'master' will be for the latest major release
> line under active development
>
> - Commits against master should be evaluated for whether they should
> be cherry-picked to other still supported major release lines
> consistent with the community support model
>
> - When a release occurs a signed tag will be generated and the version
> for that major line will be bumped to the next incremental release
> snapshot
>
> - The next commit on a given major release line that requires a minor
> version change should increment the minor version number and reset
> incremental to zero
>
> - Major version changes should only ever be prompted from the master
> branch and should only occur when a commit warrants changing the major
> version at which point a major release line branch should be created
> off of master for the previous major release line
>
> [1]
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201602.mbox/%3CCALJK9a7bWjff7xXGmUtp3nFV3HRmqbLL1StwkXcf5JdohNPRmg%40mail.gmail.com%3E
>
> Thanks
> Joe
>
> On Tue, Feb 16, 2016 at 3:13 PM, Joe Witt  wrote:
> > I don't want to kill this thread.  It is good to discuss specific
> > tooling/procedures.  But I do want to get some consensus discussion
> > around Tony's original intent (as I read it).  So kicked off a
> > discussion back at that level.
> >
> >
> >
> > On Tue, Feb 16, 2016 at 8:34 AM, Tony Kurc  wrote:
> >> While I like gitflow, I can't say I like any of the plugins that are
> used.
> >> I have worked on some other projects (unfortunately not open source)
> that
> >> use a gitflow inspired workflow, without ever using a plugin. Nice side
> >> effect is that I believe this got me better at using git, and generally
> we
> >> all got better at managing merge pain.
> >>
> >> On merge problems, I think the reason we're operating the way we are
> now is
> >> to avoid merge mayhem. I think the initial bar for a patch is "can be
> >> merged into master", and we have our friend Travis to make this even
> easier
> >> to know upfront. This greatly simplifies things. If a bugfix is "patch
> >> needs to be able to apply onto the current release in progress, master,
> and
> >> several other versions we're supporting, with possibly drastically
> >> different code", well then things get interesting.
> >>
> >>
> >>
> >> On Mon, Feb 15, 2016 at 12:04 PM, Benson Margulies <
> bimargul...@gmail.com>
> >> wrote:
> >>
> >>> The issue tracker
> >>>
> >>>
> https://ecosystem.atlassian.net/projects/MJF/issues/MJF-259?filter=allopenissues
> >>> might also prove useful in evaluating it.
> >>>
> >>> On Mon, Feb 15, 2016 at 12:03 PM, Benson Margulies
> >>>  wrote:
> >>> > I tried to use the bitbucket gitflow plugin. It worked great, until
> it
> >>> > didn't. It would get into terrible, inexplicable, merge problems. No
> >>> > one seemed to be maintaining it.
> >>> >
> >>> > There's a new offering in this dept:
> >>> > https://github.com/egineering-llc/gitflow-helper-maven-plugin.
> >>> >
> >>> > On Mon, Feb 15, 2016 at 11:41 AM, Adam Taft 
> wrote:
> >>> >> One of the harder things with gitflow is using it in combination
> with
> >>> >> maven.  It's ideal that the tags and releases are tracking closely
> with
> >>> the
> >>> >> maven pom.xml version.  gitflow, on its own, doesn't keep the pom
> >>> version
> >>> >> updated with the git release names.
> >>> >>
> >>> >> Because of the general importance of keeping releases and tags
> >>> synchronized
> >>> >> with the pom version, I think whatever we do, it needs to be
> approached
> >>> >> with tools that are available through maven rather than from git.
> The
> >>> >> git-flow plugin (referenced by Thad) doesn't directly help deal with
> >>> this
> >>> >> synchronization, since it's a git tool, not