Re: [VOTE] Release Apache NiFi 2.0.0-M1 (RC1)

[Otto Fowler]

+1 (non-binding)

Validated hashes, signatures, build from source, and test run.


❯ mvn --version
Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546)
Maven home: /opt/homebrew/Cellar/maven/3.9.5/libexec
Java version: 21.0.1, vendor: Eclipse Adoptium, runtime:
/Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.0", arch: "aarch64", family: “mac"



On November 13, 2023 at 8:00:29 PM, David Handermann (
exceptionfact...@apache.org) wrote:

Team,

I am pleased to be calling this vote for the source release of Apache
NiFi 2.0.0-M1.

Please review the following guide for how to verify a release candidate
build:

https://cwiki.apache.org/confluence/display/NIFI/Release+Candidate+Verification

The source being voted on the and the convenience binaries are
available on the Apache Distribution Repository:

https://dist.apache.org/repos/dist/dev/nifi/nifi-2.0.0-M1

The build artifacts are available on the Apache Nexus Repository:

https://repository.apache.org/content/repositories/orgapachenifi-1236

Git Tag: nifi-2.0.0-M1-RC1
Git Commit ID: 0a7ba3722001bb8c3f09755792c4db2b2ab61f36
GitHub Commit Link:
https://github.com/apache/nifi/commit/0a7ba3722001bb8c3f09755792c4db2b2ab61f36

Checksums of nifi-2.0.0-M1-source-release.zip

SHA256: ba870d51e2362c2f07935634e30d469cb90cd71a65cc4ea205760a2192551696
SHA512:
bc25e1b1d27477950490e250297e74ec969b2ab1b351f318d0908e6c579e919972be5470acc9e147ea87a6185951c5c4f2d22d8f532eab0b045a2d7b922ad594


Release artifacts are signed with the following key:

https://people.apache.org/keys/committer/exceptionfactory.asc

KEYS file is available on the Apache Distribution Repository:

https://dist.apache.org/repos/dist/release/nifi/KEYS

Issues resolved for this version: 934

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12339599

Release note highlights can be found on the project wiki:

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version2.0.0-M1

The vote will be open for 72 hours.

Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test.
Then please vote:

[] +1 Release this package as nifi-2.0.0-M1
[] +0 no opinion
[] -1 Do not release this package because...

Re: [DISCUSS] Release Timing for NiFi 2.0.0 M1 and 1.24.0

[Otto Fowler]

 ~/t/nifi-release-2/nifi-/nifi-as/target/nifi-2.0.0-M1-bin/nifi-2.0.0-M1/logs
─── at 09:16:20
❯ grep Generated nifi-app_2023-11-14_08.0.log
 ~/t/nifi-release-2/nifi-/nifi-as/target/nifi-2.0.0-M1-bin/nifi-2.0.0-M1/logs
─── at 09:16:50
❯\


Sorry, it rolled



On November 14, 2023 at 9:15:18 AM, Otto Fowler (ottobackwa...@gmail.com)
wrote:

 ~/t/nifi-release-2/nifi-/nifi-as/target/nifi-2.0.0-M1-bin/nifi-2.0.0-M1/logs
─── at 09:13:24
❯ grep Generated nifi-app.log
 ~/t/nifi-release-2/nifi-/nifi-as/target/nifi-2.0.0-M1-bin/nifi-2.0.0-M1/logs
─── at 09:13:27
❯




On November 14, 2023 at 9:06:47 AM, David Handermann (
exceptionfact...@apache.org) wrote:

Otto,

Can you provide the exact command you are running? Running the
following command from the NiFi home directory should work on a new
installation:

grep Generated logs/nifi-app.log

Regards,
David Handermann

On Tue, Nov 14, 2023 at 7:57 AM Otto Fowler 
wrote:
>
> I”m trying to verify the 2.0.0-M1 release and it is not generating the
> default user and password per the README ( ie. The grep fails and I
cannot
> find them ).
>
> Am I missing something?
>
> ❯ mvn --version
> Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546)
> Maven home: /opt/homebrew/Cellar/maven/3.9.5/libexec
> Java version: 21.0.1, vendor: Eclipse Adoptium, runtime:
> /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "14.0", arch: "aarch64", family: "mac"
>
>
> On November 13, 2023 at 2:01:40 PM, Joe Witt (joe.w...@gmail.com) wrote:
>
> Very cool - thanks David and Pierre for tackling RM duties!
>
> On Mon, Nov 13, 2023 at 11:59 AM Pierre Villard <
pierre.villard...@gmail.com>
>
> wrote:
>
> > Thanks David, sounds good, will get started on the 1.24 release soon.
> >
> > Le lun. 13 nov. 2023 à 19:37, David Handermann <
> > exceptionfact...@apache.org>
> > a écrit :
> >
> > > Thanks for replies Pierre and Mark!
> > >
> > > With several pull requests merged last week addressing the Python
> > > concerns Mark raised, I believe we are ready to proceed with the
> > > release candidate process.
> > >
> > > Based on a conversation Pierre, we should be able to collaborate on
> > > preparing candidate builds for both 2.0.0-M1 and 1.24.0.
> > >
> > > We will need to make a few website adjustments for the 2.0.0-M1
> > > version in terms of documentation and downloads, but it should be
> > > possible to handle that with some alternative navigation links for
> > > now.
> > >
> > > I plan on starting the release candidate build process for 2.0.0-M1
> soon.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Wed, Nov 8, 2023 at 5:53 PM Mark Payne 
wrote:
> > > >
> > > > Hey Pierre,
> > > >
> > > > I ran into a couple of issues (they appear to be timing issues)
> around
> > > the Python stuff on startup. I think we need to hold off on the
release
> > > until these are resolved but I think I should have a PR up some time
> > > tomorrow hopefully.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > >
> > > >
> > > > > On Nov 8, 2023, at 10:30 AM, Pierre Villard <
> > > pierre.villard...@gmail.com> wrote:
> > > > >
> > > > > Hey David,
> > > > >
> > > > > With all of the recent changes we merged, I believe we are in the
> > right
> > > > > spot to start the process with 1.24 and 2.0M1 releases.
> > > > >
> > > > > Happy to help with this,
> > > > > Pierre
> > > > >
> > > > > Le ven. 3 nov. 2023 à 22:46, David Handermann <
> > > exceptionfact...@apache.org>
> > > > > a écrit :
> > > > >
> > > > >> Team,
> > > > >>
> > > > >> We have made great progress on a large number of features and
> fixes
> > > > >> for NiFi 2.0.0. The Jira version page for 2.0.0 [1] lists almost
> 900
> > > > >> issues! At the same time, we have 250 issues tagged for version
> > 1.24.0
> > > > >> [2].
> > > > >>
> > > > >> Although there is still more work to do for a full release of
> > 2.0.0, I
> > > > >> believe we are at a good point to release a milestone version.
> > > > >> Releasing 

Re: [DISCUSS] Release Timing for NiFi 2.0.0 M1 and 1.24.0

[Otto Fowler]

 ~/t/nifi-release-2/nifi-/nifi-as/target/nifi-2.0.0-M1-bin/nifi-2.0.0-M1/logs
─── at 09:13:24
❯ grep Generated nifi-app.log
 ~/t/nifi-release-2/nifi-/nifi-as/target/nifi-2.0.0-M1-bin/nifi-2.0.0-M1/logs
─── at 09:13:27
❯




On November 14, 2023 at 9:06:47 AM, David Handermann (
exceptionfact...@apache.org) wrote:

Otto,

Can you provide the exact command you are running? Running the
following command from the NiFi home directory should work on a new
installation:

grep Generated logs/nifi-app.log

Regards,
David Handermann

On Tue, Nov 14, 2023 at 7:57 AM Otto Fowler 
wrote:
>
> I”m trying to verify the 2.0.0-M1 release and it is not generating the
> default user and password per the README ( ie. The grep fails and I
cannot
> find them ).
>
> Am I missing something?
>
> ❯ mvn --version
> Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546)
> Maven home: /opt/homebrew/Cellar/maven/3.9.5/libexec
> Java version: 21.0.1, vendor: Eclipse Adoptium, runtime:
> /Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "14.0", arch: "aarch64", family: "mac"
>
>
> On November 13, 2023 at 2:01:40 PM, Joe Witt (joe.w...@gmail.com) wrote:
>
> Very cool - thanks David and Pierre for tackling RM duties!
>
> On Mon, Nov 13, 2023 at 11:59 AM Pierre Villard <
pierre.villard...@gmail.com>
>
> wrote:
>
> > Thanks David, sounds good, will get started on the 1.24 release soon.
> >
> > Le lun. 13 nov. 2023 à 19:37, David Handermann <
> > exceptionfact...@apache.org>
> > a écrit :
> >
> > > Thanks for replies Pierre and Mark!
> > >
> > > With several pull requests merged last week addressing the Python
> > > concerns Mark raised, I believe we are ready to proceed with the
> > > release candidate process.
> > >
> > > Based on a conversation Pierre, we should be able to collaborate on
> > > preparing candidate builds for both 2.0.0-M1 and 1.24.0.
> > >
> > > We will need to make a few website adjustments for the 2.0.0-M1
> > > version in terms of documentation and downloads, but it should be
> > > possible to handle that with some alternative navigation links for
> > > now.
> > >
> > > I plan on starting the release candidate build process for 2.0.0-M1
> soon.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Wed, Nov 8, 2023 at 5:53 PM Mark Payne 
wrote:
> > > >
> > > > Hey Pierre,
> > > >
> > > > I ran into a couple of issues (they appear to be timing issues)
> around
> > > the Python stuff on startup. I think we need to hold off on the
release
> > > until these are resolved but I think I should have a PR up some time
> > > tomorrow hopefully.
> > > >
> > > > Thanks
> > > > -Mark
> > > >
> > > >
> > > >
> > > > > On Nov 8, 2023, at 10:30 AM, Pierre Villard <
> > > pierre.villard...@gmail.com> wrote:
> > > > >
> > > > > Hey David,
> > > > >
> > > > > With all of the recent changes we merged, I believe we are in the
> > right
> > > > > spot to start the process with 1.24 and 2.0M1 releases.
> > > > >
> > > > > Happy to help with this,
> > > > > Pierre
> > > > >
> > > > > Le ven. 3 nov. 2023 à 22:46, David Handermann <
> > > exceptionfact...@apache.org>
> > > > > a écrit :
> > > > >
> > > > >> Team,
> > > > >>
> > > > >> We have made great progress on a large number of features and
> fixes
> > > > >> for NiFi 2.0.0. The Jira version page for 2.0.0 [1] lists almost
> 900
> > > > >> issues! At the same time, we have 250 issues tagged for version
> > 1.24.0
> > > > >> [2].
> > > > >>
> > > > >> Although there is still more work to do for a full release of
> > 2.0.0, I
> > > > >> believe we are at a good point to release a milestone version.
> > > > >> Releasing 1.24.0 at the same time will also provide the latest
> > > > >> upgrades, plus new features and deprecations that will help
> prepare
> > > > >> for upgrading to 2.0.0.
> > > > >>
> > > > >> We have informally referred to the initial release as a
milestone
> > > > >> version, so it seems like officially naming it 2.0.0-M1 would be
> the
> > > > >> most straightforward option.
> > > > >>
> > > > >> There are a couple more open issues that I believe we can wrap
up
> in
> > > > >> the next few days, but otherwise we should be able to target
early
> > > > >> next week for preparing a release build.
> > > > >>
> > > > >> With the goal of coordinating a release of multiple versions, I
> > would
> > > > >> be glad to handle release manager duties for one of the
versions.
> > > > >>
> > > > >> Regards,
> > > > >> David Handermann
> > > > >>
> > > > >> [1]
https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > > > >> [2]
https://issues.apache.org/jira/projects/NIFI/versions/12353443
> > > > >>
> > > >
> > >
> >

Re: [DISCUSS] Release Timing for NiFi 2.0.0 M1 and 1.24.0

[Otto Fowler]

I”m trying to verify the 2.0.0-M1 release and it is not generating the
default user and password per the README ( ie. The grep fails and I cannot
find them ).

Am I missing something?

❯ mvn --version
Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546)
Maven home: /opt/homebrew/Cellar/maven/3.9.5/libexec
Java version: 21.0.1, vendor: Eclipse Adoptium, runtime:
/Library/Java/JavaVirtualMachines/temurin-21.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.0", arch: "aarch64", family: "mac"


On November 13, 2023 at 2:01:40 PM, Joe Witt (joe.w...@gmail.com) wrote:

Very cool - thanks David and Pierre for tackling RM duties!

On Mon, Nov 13, 2023 at 11:59 AM Pierre Villard 

wrote:

> Thanks David, sounds good, will get started on the 1.24 release soon.
>
> Le lun. 13 nov. 2023 à 19:37, David Handermann <
> exceptionfact...@apache.org>
> a écrit :
>
> > Thanks for replies Pierre and Mark!
> >
> > With several pull requests merged last week addressing the Python
> > concerns Mark raised, I believe we are ready to proceed with the
> > release candidate process.
> >
> > Based on a conversation Pierre, we should be able to collaborate on
> > preparing candidate builds for both 2.0.0-M1 and 1.24.0.
> >
> > We will need to make a few website adjustments for the 2.0.0-M1
> > version in terms of documentation and downloads, but it should be
> > possible to handle that with some alternative navigation links for
> > now.
> >
> > I plan on starting the release candidate build process for 2.0.0-M1
soon.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Nov 8, 2023 at 5:53 PM Mark Payne  wrote:
> > >
> > > Hey Pierre,
> > >
> > > I ran into a couple of issues (they appear to be timing issues)
around
> > the Python stuff on startup. I think we need to hold off on the release
> > until these are resolved but I think I should have a PR up some time
> > tomorrow hopefully.
> > >
> > > Thanks
> > > -Mark
> > >
> > >
> > >
> > > > On Nov 8, 2023, at 10:30 AM, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> > > >
> > > > Hey David,
> > > >
> > > > With all of the recent changes we merged, I believe we are in the
> right
> > > > spot to start the process with 1.24 and 2.0M1 releases.
> > > >
> > > > Happy to help with this,
> > > > Pierre
> > > >
> > > > Le ven. 3 nov. 2023 à 22:46, David Handermann <
> > exceptionfact...@apache.org>
> > > > a écrit :
> > > >
> > > >> Team,
> > > >>
> > > >> We have made great progress on a large number of features and
fixes
> > > >> for NiFi 2.0.0. The Jira version page for 2.0.0 [1] lists almost
900
> > > >> issues! At the same time, we have 250 issues tagged for version
> 1.24.0
> > > >> [2].
> > > >>
> > > >> Although there is still more work to do for a full release of
> 2.0.0, I
> > > >> believe we are at a good point to release a milestone version.
> > > >> Releasing 1.24.0 at the same time will also provide the latest
> > > >> upgrades, plus new features and deprecations that will help
prepare
> > > >> for upgrading to 2.0.0.
> > > >>
> > > >> We have informally referred to the initial release as a milestone
> > > >> version, so it seems like officially naming it 2.0.0-M1 would be
the
> > > >> most straightforward option.
> > > >>
> > > >> There are a couple more open issues that I believe we can wrap up
in
> > > >> the next few days, but otherwise we should be able to target early
> > > >> next week for preparing a release build.
> > > >>
> > > >> With the goal of coordinating a release of multiple versions, I
> would
> > > >> be glad to handle release manager duties for one of the versions.
> > > >>
> > > >> Regards,
> > > >> David Handermann
> > > >>
> > > >> [1] https://issues.apache.org/jira/projects/NIFI/versions/12339599
> > > >> [2] https://issues.apache.org/jira/projects/NIFI/versions/12353443
> > > >>
> > >
> >
>

Re: Custom-processor configuration suggestions

[Otto Fowler]

```

   ┌──┐
   │  │
   │  processor instance  │┐
   │  ││
   └──┘│
  .─.
   │
   ,─'   '─.
   │
 ,' `.
   │
╱ ╲
   │
   ╱   ╲
   │
  ; :
   │
┌─┐ │  configuration
│
   │ │
│  ┌─▶│authority│
   │ │
Configruation service │  │  : ;
   ├┬───▶│
│──┘   ╲   ╱
   ┌──┐│││
│   ╲ ╱
   │  │││
└─┘╲   ╱
   │  processor instance  │┘│
  `.   ,'
   │  │ │
'─. ,─'
   └──┘ │
   `───'
│
│
│
│
│
│
│
┌──┐│
│  ││
│  processor instance  │┤
│  ││
└──┘│
│
│
│
│
┌──┐│
│  ││
│  processor instance  │┘
│  │
└──┘
```

You can also make a shared service component that loads the configurations
by some means and serves them to the processors.
The service can get the configurations however makes sense for you ( from
REST API  like Joe is saying, to reading from disk or something ).


```



On September 27, 2023 at 4:04:56 PM, Joe Witt (joe.w...@gmail.com) wrote:

Russ

It sounds like what you have is a case of significant reference data you
need made available to various instances of this processor that knows how
to use that reference state to do its function.

This is similar to cases like IP geo enrichment where the dataset on which
you'd make the decision is larger and more importantly subject to change
over time. In such cases the ideal state is:
(A) The reference dataset(s) is hosted at a RESTful endpoint and can be
periodically pulled and stored some place local/easily accessible.
(B) The processor knows where to look for this reference dataset download
and is able to hot reload it on the fly to include understanding that the
needed datasets might not yet be made available and it should yield until
it sees them and loads them.

Thanks

On Wed, Sep 27, 2023 at 11:51 AM Russell Bateman 
wrote:

> I'm posting this plea for suggestions as I'm short on imagination here.
>
> We have some custom processors that need extraordinary amounts of
> configuration of the sort a flow writer would have to copy and paste
> in--huge amounts of Yaml, regular expressions, etc. This is what our
> flow writers are already doing. It would be easier to insert a filename
> or -path, but...
>
> ...asking a custom processor to perform filesystem I/O is icky because
> of unpredictable filesystem access post installation. Thinking about how
> installation is beyond my control, I don't want to make installation
> messy, etc. Containers, Kubernetes deployment, etc. complicate this.
>
> I thought of wiring /GetFile/ to a subdirectory (problematic, but less
> so?) and accepting files as input to pass on to needy processors who
> would recognize, adopt and incorporate configuration based on
> higher-level and simpler cues posted by flow writers as property values.
>
> Assuming you both grok and are interested in what I'm asking, do you
> have thoughts, cautionary statements or even cat-calls to 

Re: [VOTE] Release Apache NiFi 1.23.0 (RC2)

[Otto Fowler]

Apache Maven 3.9.3 (21122926829f1ead511c958d89bd2f672198ae9f)
Maven home: /opt/homebrew/Cellar/maven/3.9.3/libexec
Java version: 11.0.18, vendor: Eclipse Adoptium, runtime:
/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4.1", arch: "aarch64", family: "mac"


On July 21, 2023 at 8:05:49 PM, Otto Fowler (ottobackwa...@gmail.com) wrote:

+1 (non-binding)



On July 20, 2023 at 8:16:12 PM, David Handermann (
exceptionfact...@apache.org) wrote:

Team,

I am pleased to be calling this vote for the source release of Apache NiFi
1.23.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1231

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.23.0-RC2
The Git commit ID is 2c707ad4e35fa7086024b966f279c8d4da486144
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=2c707ad4e35fa7086024b966f279c8d4da486144
https://github.com/apache/nifi/commit/2c707ad4e35fa7086024b966f279c8d4da486144

Checksums of nifi-1.23.0-source-release.zip:
SHA256: 79d8a2cc6f3c2e2b6a31398d2347a0fa5e2849cd47058f6ee7798b231c114b94
SHA512:
d80ecab0eb5721ceca1f8840902841fd33ff173ca2c6aff8beefeac6dddc3fe865f2dca9d1e985b339ad75a19ebafffc95d637840957258ae80961bf8db5b0ac


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/exceptionfactory.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

115 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353320

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.23.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.23.0 (RC2)

[Otto Fowler]

 +1 (non-binding)



On July 20, 2023 at 8:16:12 PM, David Handermann (
exceptionfact...@apache.org) wrote:

Team,

I am pleased to be calling this vote for the source release of Apache NiFi
1.23.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1231

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.23.0-RC2
The Git commit ID is 2c707ad4e35fa7086024b966f279c8d4da486144
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=2c707ad4e35fa7086024b966f279c8d4da486144
https://github.com/apache/nifi/commit/2c707ad4e35fa7086024b966f279c8d4da486144

Checksums of nifi-1.23.0-source-release.zip:
SHA256: 79d8a2cc6f3c2e2b6a31398d2347a0fa5e2849cd47058f6ee7798b231c114b94
SHA512:
d80ecab0eb5721ceca1f8840902841fd33ff173ca2c6aff8beefeac6dddc3fe865f2dca9d1e985b339ad75a19ebafffc95d637840957258ae80961bf8db5b0ac


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/exceptionfactory.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

115 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353320

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.23.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.23.0 (RC1)

[Otto Fowler]

 +1

Ran through verification and ran app.

```
Apache Maven 3.9.3 (21122926829f1ead511c958d89bd2f672198ae9f)
Maven home: /opt/homebrew/Cellar/maven/3.9.3/libexec
Java version: 11.0.18, vendor: Eclipse Adoptium, runtime:
/Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.4.1", arch: "aarch64", family: “Mac”
```



On July 20, 2023 at 12:45:43 AM, David Handermann (
exceptionfact...@apache.org) wrote:

Team,

I am pleased to be calling this vote for the source release of Apache NiFi
1.23.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1230

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.23.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.23.0-RC1
The Git commit ID is 551ed1079ceb0c96cafd6d984ce879ac793016aa
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=551ed1079ceb0c96cafd6d984ce879ac793016aa
https://github.com/apache/nifi/commit/551ed1079ceb0c96cafd6d984ce879ac793016aa

Checksums of nifi-1.23.0-source-release.zip:
SHA256: bd5c3e6bb20834bc116347c46668630769b4af64553afa0420d00119044bab1b
SHA512:
d9a59ba293b0a0823efbe2a4a7212191735ef58ea94ffc72c3d89b77f1ccfdc46ada43eb014eeaab38b9b228ab371851d7cde6ffde33410cfb35d14d8244c8d0


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/exceptionfactory.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

110 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12353320

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.23.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.23.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [ANNOUNCE] New nifi PMC Member Nandor Soma Abonyi

[Otto Fowler]

 Congratulations!

On June 26, 2023 at 7:11:16 PM, Joe Witt (joe.w...@gmail.com) wrote:

On behalf of the Apache NiFI PMC, I am very pleased to announce that Nandor
has accepted the PMC's invitation to join the Apache NiFi PMC. We greatly
appreciate the PR reviews and code contributions and release management
work of Nandor.

Please join us in congratulating and welcoming Nandor to the Apache NiFi
PMC.

Congratulations Nandor!

Re: Implementation question: PLC4X integration - Subscription operation

[Otto Fowler]

Unai, I have been able to talk this through with some of the NIF SME’s and
the TL;DR is:

There is ample precedence for this and it can be done, though it does open
up the possibility for problems with resource consumption.

We can take this back to your PLC4x pr.



On May 17, 2023 at 3:55:53 PM, Unai Leria (unai.le...@zylk.net.invalid)
wrote:

Hi Otto,

At the moment all available PLC4X subscriptions work when the protocol used
supports them natively.

The processor would start a single thread listening for incoming PLC
messages.

The PLC4X community was planning to add subscription to all protocols when
not natively supported via polling, but it is not currently implemented and
seems like it will take time.

In this case I suppose 2 to 3 threads could be launched per processor.
However to mimic the subscription with the current PLC4X integration (with
only the reading processor) would require around 3 more standard processors
so I don't see a big difference.

Unai


On 2023/05/17 15:32:24 Otto Fowler wrote:
 Hi Unai!

 Would this be a single thread, just listening? Is this pure exception
based
 processing ( where you only listen for things ) or is it possible to
have
 to simulate this by polling some of the endpoints?

 On May 17, 2023 at 10:28:24 AM, Unai Leria (unai.le...@zylk.net.invalid)

 wrote:

 Hi NiFi community! br/ 

 I'm working on a PLC subscriber/listener processor to improve the
 Apache/PLC4X's NiFi integration and a questions about the possible
 implementation have raised on the PLC4x community br/The processor
can be
 found here: [

https://github.com/zylklab/plc4x/blob/feature/nifi-integration-record-listener/plc4j/integrations/apache-nifi/nifi-plc4x-processors/src/main/java/org/apache/plc4x/nifi/Plc4xListenRecordProcessor.java
 |

https://github.com/zylklab/plc4x/blob/feature/nifi-integration-record-listener/plc4j/integrations/apache-nifi/nifi-plc4x-processors/src/main/java/org/apache/plc4x/nifi/Plc4xListenRecordProcessor.java
 ] br/ 
 At the moment I have made the processor trying to follow the already
 implemented listeners in NiFi: starting a daemon thread with a
dispatcher
 that connects to the PCL and waits for messages. br/That
disppatcher puts
 events into a queue from which the processor reads and creates
FlowFiles.
 br/ 
 The main question is if creating a thread for every processor that are
not
 in the thread pools and governed by the Nifi scheduler is a bad idea.
And
 what could be the effects be on a NiFi instance of multiple of this
 processors. br/ 

 Any thoughts would be greatly appreciated! br/ 
 Thank you for your time, br/ 
 Unai Lería br/ 

Re: Implementation question: PLC4X integration - Subscription operation

[Otto Fowler]

 Hi Unai!

Would this be a single thread, just listening? Is this pure exception based
processing ( where you only listen for things ) or is it possible to have
to simulate this by polling some of the endpoints?

On May 17, 2023 at 10:28:24 AM, Unai Leria (unai.le...@zylk.net.invalid)
wrote:

Hi NiFi community! br/> <

I'm working on a PLC subscriber/listener processor to improve the
Apache/PLC4X's NiFi integration and a questions about the possible
implementation have raised on the PLC4x community br/>The processor can be
found here: [
https://github.com/zylklab/plc4x/blob/feature/nifi-integration-record-listener/plc4j/integrations/apache-nifi/nifi-plc4x-processors/src/main/java/org/apache/plc4x/nifi/Plc4xListenRecordProcessor.java
|
https://github.com/zylklab/plc4x/blob/feature/nifi-integration-record-listener/plc4j/integrations/apache-nifi/nifi-plc4x-processors/src/main/java/org/apache/plc4x/nifi/Plc4xListenRecordProcessor.java
] br/> <
At the moment I have made the processor trying to follow the already
implemented listeners in NiFi: starting a daemon thread with a dispatcher
that connects to the PCL and waits for messages. br/>That disppatcher puts
events into a queue from which the processor reads and creates FlowFiles.
br/> <
The main question is if creating a thread for every processor that are not
in the thread pools and governed by the Nifi scheduler is a bad idea. And
what could be the effects be on a NiFi instance of multiple of this
processors. br/> <

Any thoughts would be greatly appreciated! br/> <
Thank you for your time, br/> <
Unai Lería br/> <

Re: [discuss] NiFi 1.20 and NiFi 2.0

[Otto Fowler]

ers
>>>>>>>> staying on 1.x or creating a schism of 1.x and 2.x users.
>>>>>>>> br/>>>>>>>;>> Good migration tooling will take a while to develop
and test, and the
>>>>>>>> core contributors to that effort may not have sufficient variety
of
>>>>>>>> flows to evaluate when the migration tools are "done" for the
majority
>>>>>>>> of the community to have success upgrading to 2.x. A milestone
release
>>>>>>>> would allow us get more feedback on migration over a longer period
>>>>>>>> than the vote window of an RC candidate.
>>>>>>>> br/>>>>>>>;>> Perhaps we could continue to release from the 1.x
line (including
>>>>>>>> minor releases with some features) until we are ready to drop the
"milestone"
>>>>>>>> qualifier from 2.0.0, and only then put 1.x into maintenance-only
status.
>>>>>>>> It would be the same proposal to move main to target 2.0.0-M1, but
>>>>>>>> relax restrictions for what can land on the 1.x branch and be open
to
>>>>>>>> a 1.21, 1.22, etc. if 2.0.0 work takes longer than anticipated.
For
>>>>>>>> example, maybe we would be open to landing new/backported
processors
>>>>>>>> on the 1.x branch, but not core framework features or API changes.
>>>>>>>> br/>>>>>>>;>> This might not be necessary, but I think it is fair
that saying "no
>>>>>>>> new features on 1.x" and also "no new features in 2.0.0" puts the
>>>>>>>> project in a rough place if 2.0.0 takes longer than a few months,
so
>>>>>>>> if we go that route, we need to commit to a quick release of 2.0.0
>>>>>>>> that most users can move to easily.
>>>>>>>> br/>>>>>>>;>> Thanks,
>>>>>>>> Kevin
>>>>>>>> br/>>>>>>>;>> On Jan 11, 2023 at 12:32:46, Joe Witt <
joe.w...@gmail.com> wrote:
>>>>>>>> br/>>>>>>>;>>> Kevin,
>>>>>>>>> br/>>>>>>;>>>> Yeah we can do whatever we want as far as
'releases' of 2.0 that are
>>>>>>>> prior
>>>>>>>>> to us officially considering it 2.0/stable.
>>>>>>>>> br/>>>>>>;>>>> That said - the migration tooling will be key to
provide as we need
>>>>>>>>> to
>>>>>>>> make
>>>>>>>>> the bridge as solid and stable as possible to help someone move
from
>>>>>>>>> 1.x
>>>>>>>> to
>>>>>>>>> 2.x. I dont know how related these two concepts (milestone
releases
>>>>>>>>> and 1.x to 2.x ease really are).
>>>>>>>>> br/>>>>>>;>>>> Thanks
>>>>>>>>> br/>>>>>>;>>>> On Wed, Jan 11, 2023 at 10:27 AM Kevin Doran <
kdo...@apache.org> wrote:
>>>>>>>>> br/>>>>>>;>>>> [hit the wrong keyboard shortcut, here is the rest
of my thoughts]
>>>>>>>>> br/>>>>>>;>>>> br/>>>>>>>>>> On this pooint from David:
>>>>>>>>> br/>>>>>>;>>>> br/>>>>>>>>>> We may neeed to have a longer
release candidate period, or more
>>>>>>>> incremental
>>>>>>>>> br/>>>>>>;>>>>> fix releases
>>>>>>>>> br/>>>>>>;>>>>> for the initial 2.0.0 release train, but I do not
expect delaying
>>>>>>>>>> a
>>>>>>>> 2.0.0
>>>>>>>>> br/>>>>>>;>>>>> release for new features, as that is not part of
the release goals.
>>>>>>>>> br/>>>>>>;>>>>> br/>>>>>>>>>> br/>>>>>>>>>> br/>>>>>>>>>> Would
the community benefit from one or more milestone releases of
>>>>>>>>> 2.0,
>>>>>>>> to
>>>>>>&

Re: [discuss] NiFi 1.20 and NiFi 2.0

[Otto Fowler]

 Super, is that written down anywhere we can point people to?

From: Bryan Bende  
Reply: dev@nifi.apache.org  
Date: January 10, 2023 at 09:32:51
To: dev@nifi.apache.org  
Subject:  Re: [discuss] NiFi 1.20 and NiFi 2.0

The plan as I understand it is not to diverge and create separate feature
development on the 1.x line, so I would expect all PRs to continue to be
submitted only to main. We would release 1.x as needed with major bug fixes
or critical security updates, and these would be cherry-picked and/or
backported as necessary, mostly without the need for PRs, the same as we
would do if we were bringing fixes from main (1.20.0-SNAPSHOT) back to a
maintenance line like (1.19.x). For precedent, we followed this same
approach going from the 0.x line to 1.0.0 and there wasn't any major issue.

On Tue, Jan 10, 2023 at 7:07 AM Otto Fowler 
wrote:

> It was also mentioned in another thread that we need to have agreement on
> our explicit strategy and support for 1.x going forward, did that happen?
>
> From: Otto Fowler  
> Reply: Otto Fowler  
> Date: January 10, 2023 at 07:02:34
> To: dev@nifi.apache.org  
> Subject: Re: [discuss] NiFi 1.20 and NiFi 2.0
>
> There needs to be an update to the contributing guide as to how to submit
> PRs to 1.x or 2.x etc.
>
> From: Joe Witt  
> Reply: dev@nifi.apache.org  
> Date: January 9, 2023 at 15:53:16
> To: dev@nifi.apache.org  
> Subject: [discuss] NiFi 1.20 and NiFi 2.0
>
> Team,
>
> As David mentioned in [1] following a successful NiFi 2.0 release goal
> planning - we are now going to start moving the 'main' line to be the
NiFi
> 2.0 line which will allow for the key work to take place. We will also
> move niFi 1.x to its appropriate support line.
>
> It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we
have
> work in there including security items so it is time to make a release.
> The intent then is to initiate 1.20 and immediate after that change
'main'
> to 2.0.
>
> Going forward then all work on the 1.x line should be focused on
> maintaining existing features, dependencies, and helping 1.x users
migrate
> to the 2.x line. Otherwise, new feature work will happen on 'main' as it
> normally does and will come out in the NiFi 2.x release line.
>
> Please flag key outstanding items as we narrow down the release candidate
> for NiFi 1.20.
>
> Thanks
> Joe
>
> [1] https://lists.apache.org/thread/qo4vvdw46235y7vy2crcd6l4m11wl7jz
>

Re: [discuss] NiFi 1.20 and NiFi 2.0

[Otto Fowler]

 It was also mentioned in another thread that we need to have agreement on
our explicit strategy and support for 1.x going forward, did that happen?

From: Otto Fowler  
Reply: Otto Fowler  
Date: January 10, 2023 at 07:02:34
To: dev@nifi.apache.org  
Subject:  Re: [discuss] NiFi 1.20 and NiFi 2.0

There needs to be an update to the contributing guide as to how to submit
PRs to 1.x or 2.x etc.

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: January 9, 2023 at 15:53:16
To: dev@nifi.apache.org  
Subject:  [discuss] NiFi 1.20 and NiFi 2.0

Team,

As David mentioned in [1] following a successful NiFi 2.0 release goal
planning - we are now going to start moving the 'main' line to be the NiFi
2.0 line which will allow for the key work to take place. We will also
move niFi 1.x to its appropriate support line.

It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we have
work in there including security items so it is time to make a release.
The intent then is to initiate 1.20 and immediate after that change 'main'
to 2.0.

Going forward then all work on the 1.x line should be focused on
maintaining existing features, dependencies, and helping 1.x users migrate
to the 2.x line. Otherwise, new feature work will happen on 'main' as it
normally does and will come out in the NiFi 2.x release line.

Please flag key outstanding items as we narrow down the release candidate
for NiFi 1.20.

Thanks
Joe

[1] https://lists.apache.org/thread/qo4vvdw46235y7vy2crcd6l4m11wl7jz

Re: [discuss] NiFi 1.20 and NiFi 2.0

[Otto Fowler]

 There needs to be an update to the contributing guide as to how to submit
PRs to 1.x or 2.x etc.

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: January 9, 2023 at 15:53:16
To: dev@nifi.apache.org  
Subject:  [discuss] NiFi 1.20 and NiFi 2.0

Team,

As David mentioned in [1] following a successful NiFi 2.0 release goal
planning - we are now going to start moving the 'main' line to be the NiFi
2.0 line which will allow for the key work to take place. We will also
move niFi 1.x to its appropriate support line.

It is also the case that we have nearly 100 JIRAs on NiFi 1.20 and we have
work in there including security items so it is time to make a release.
The intent then is to initiate 1.20 and immediate after that change 'main'
to 2.0.

Going forward then all work on the 1.x line should be focused on
maintaining existing features, dependencies, and helping 1.x users migrate
to the 2.x line. Otherwise, new feature work will happen on 'main' as it
normally does and will come out in the NiFi 2.x release line.

Please flag key outstanding items as we narrow down the release candidate
for NiFi 1.20.

Thanks
Joe

[1] https://lists.apache.org/thread/qo4vvdw46235y7vy2crcd6l4m11wl7jz

Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

[Otto Fowler]

Sorry to be late to this, the goals seem great. The question that comes to
my mind is will the current 1.x line will be maintained?

That may be a parallel issue to the goals, but it is important if we are
dropping support for Java versions.

I would think that *some* position on that has to be decided and
communicated ( if not voted on ).




From: David Handermann 

Reply: dev@nifi.apache.org  
Date: December 10, 2022 at 10:46:51
To: dev@nifi.apache.org  
Subject:  Re: [DISCUSS] Finalizing Release Goals for NiFi 2.0

Thanks for the additional feedback Ryan and Kevin!

There appears to be general agreement on the path forward, so I will
initiate a vote thread soon. I'm sure there are additional details to be
worked out, and we can address those following a vote on the general goals.

Regards,
David Handermannn

On Wed, Dec 7, 2022 at 5:41 PM Kevin Doran  wrote:

> Thanks, David. The updated wiki page looks good and I’m very supportive
of
> the proposal
>
> I support a narrower scope for 2.x and an eventual 3.x line sooner rather
> than later. It takes some pressure off trying to fit everything into this
> 2.x change / migration
>
> Kevin
>
> On Dec 7, 2022 at 18:07:35, Ryan Hendrickson <
> ryan.andrew.hendrick...@gmail.com> wrote:
>
> > The Proposed Release Goals and Deprecated Components and Features pages
> > look great.
> >
> > I appreciate the minor leap of Java 11 as a 2.x requirement vs Java 17.
> >
> > Maybe once there is a timeline, the 2.x branch could be scheduled only
to
> > be alive for a minor amount of time... a year, etc. Then a later 2.x or
> 3.0
> > release would bring about Java 17.
> >
> > Ryan
> >
> > On Wed, Dec 7, 2022 at 3:09 PM Mike Thomsen 
> > wrote:
> >
> > > Mike - what do you mean by "controller service-based configuration
for
> >
> >
> > Using controller services for configuring bundles that connect an
> >
> > external service such as Cassandra, Elasticsearch, etc. and removing
> >
> > the option to configure connections on the processor.
> >
> >
> > > I don't think you were suggesting the minimum version be Java 17,
were
> >
> > you?
> >
> >
> > I was. Partly as devil's advocate, partly because I actually want to
> >
> > use Java 17 as a daily driver.
> >
> >
> > On Wed, Dec 7, 2022 at 2:20 PM Mark Bean  wrote:
> >
> > >
> >
> > > I agree this is a great start to a discussion with pointers to
> important
> >
> > > docs for the 2.0 transition. Thanks David!
> >
> > >
> >
> > > Mike - what do you mean by "controller service-based configuration
for
> >
> > > connection details"?
> >
> > >
> >
> > > Also, the transition from Java 11 to 17 is not without potential
> issues.
> >
> > > I've discovered one already. [1] I support stepping up on Java
version
> >
> > > requirements. Perhaps rather than the currently stated "Requires Java
8
> >
> > or
> >
> > > Java 11", the requirement can be "Requires Java 11 or Java 17". I
don't
> >
> > > think you were suggesting the minimum version be Java 17, were you?
> >
> > Either
> >
> > > way, the issue with Java 17 needs to be identified and fixed as well
as
> >
> > > more thorough testing to find other possible edge cases before we
move
> >
> > > forward too aggressively.
> >
> > >
> >
> > > [1] https://issues.apache.org/jira/browse/NIFI-10958
> >
> > >
> >
> > > On Wed, Dec 7, 2022 at 1:33 PM Mike Thomsen 
> >
> > wrote:
> >
> > >
> >
> > > > Really good start on the discussion. One thing I'm curious about is
> >
> > > > Java 11 vs 17. Java 8 -> 11 is major jump that I can understand why
> >
> > > > businesses scoffed at that for a long time, but the lift from 11 to
> 17
> >
> > > > was about like 7 -> 8. A 2.0 release seems like a good time to jump
> >
> > > > straight to the latest official LTS for Java and start
greenlighting
> >
> > > > new language features that might simplify things.
> >
> > > >
> >
> > > > I would also add (since I didn't see it) a design goal of forcing a
> >
> > > > complete shift in all bundles to using controller service-based
> >
> > > > configurations for connection details. 2.0 feels like a really good
> >
> > > > time for us to establish a community-wide best practice of
> >
> > > > centralizing configurations in dedicated components.
> >
> > > >
> >
> > > > On Wed, Dec 7, 2022 at 9:13 AM Mark Payne 
> >
> > wrote:
> >
> > > > >
> >
> > > > > Yeah, agreed. I am very supportive, as well.
> >
> > > > >
> >
> > > > > Thanks for taking the time to put this together, David.
> >
> > > > >
> >
> > > > > -Mark
> >
> > > > >
> >
> > > > >
> >
> > > > > > On Dec 7, 2022, at 4:07 AM, Pierre Villard <
> >
> > > > pierre.villard...@gmail.com> wrote:
> >
> > > > > >
> >
> > > > > > Thanks for putting this together David. This is an excellent
> >
> > writeup
> >
> > > > and
> >
> > > > > > it's great to have a release where we focus on tech debt as
well
> as
> >
> > > > making
> >
> > > > > > sure we stay up to date with our dependencies and what we
> support.
> >
> > > > This is
> >
> > > > > > a great opportunity for us 

Re: [VOTE] Release Apache NiFi 1.18.0 (RC3)

[Otto Fowler]

 +1 ( non-binding)
ran through release steps

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /usr/local/Cellar/maven/3.8.6/libexec
Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime:
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: September 29, 2022 at 15:22:21
To: dev@nifi.apache.org  
Subject:  [VOTE] Release Apache NiFi 1.18.0 (RC3)

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi 1.18.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1213

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.18.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.18.0-RC3
The Git commit ID is 5bc64c812b2c76ee2879d8081ceadf62d5e3c702
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=5bc64c812b2c76ee2879d8081ceadf62d5e3c702

Checksums of nifi-1.18.0-source-release.zip:
SHA256: bd1b675f17dbf712089a79f7bc043eae2df63bcc2e08b2012a6431641037679f
SHA512:
cea43af57089128ff65bb53e76b2fdfa8dec7397e2bf45d41e35b758b731355075839b9c018ee6284cb15e293b105e248d88748148960ad80ae387824139f52b


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

171 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.18.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.18.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [discuss] Apache NiFi 1.18 release candidate

[Otto Fowler]

I am having issues building right at the start:

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for
org.apache.nifi:nifi-ranger-bundle:1.18.0-SNAPSHOT: Could not find
artifact org.apache.nifi:nifi-nar-bundles:pom:1.18.0-SNAPSHOT and
'parent.relativePath' points at wrong local POM @ line 19, column 13
[FATAL] Non-resolvable parent POM for
org.apache.nifi.registry:nifi-registry-ranger:1.18.0-SNAPSHOT: Could
not find artifact
org.apache.nifi.registry:nifi-registry-extensions:pom:1.18.0-SNAPSHOT
and 'parent.relativePath' points at wrong local POM @ line 17, column
13
 @
[ERROR] The build could not read 2 projects -> [Help 1]
[ERROR]
[ERROR]   The project
org.apache.nifi:nifi-ranger-bundle:1.18.0-SNAPSHOT
(/Users/ottofowler/tmp/nifi-release-1.18.0/nifi-1.18.0/nifi-nar-bundles/nifi-ranger-bundle/pom.xml)
has 1 error
[ERROR] Non-resolvable parent POM for
org.apache.nifi:nifi-ranger-bundle:1.18.0-SNAPSHOT: Could not find
artifact org.apache.nifi:nifi-nar-bundles:pom:1.18.0-SNAPSHOT and
'parent.relativePath' points at wrong local POM @ line 19, column 13
-> [Help 2]
[ERROR]
[ERROR]   The project
org.apache.nifi.registry:nifi-registry-ranger:1.18.0-SNAPSHOT
(/Users/ottofowler/tmp/nifi-release-1.18.0/nifi-1.18.0/nifi-registry/nifi-registry-extensions/nifi-registry-ranger/pom.xml)
has 1 error
[ERROR] Non-resolvable parent POM for
org.apache.nifi.registry:nifi-registry-ranger:1.18.0-SNAPSHOT: Could
not find artifact
org.apache.nifi.registry:nifi-registry-extensions:pom:1.18.0-SNAPSHOT
and 'parent.relativePath' points at wrong local POM @ line 17, column
13 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with
the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException

Any ideas?




From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: September 19, 2022 at 12:50:27
To: dev@nifi.apache.org  
Subject:  [discuss] Apache NiFi 1.18 release candidate

Team,

Amazingly we're back at the point where it is time for a release in
terms of progress and time since 1.17.

Apache NiFi 1.18 is packed and some in flight work makes it really
solid and we seem to have found our groove again on processor
development! I will try to start pulling RC1 together next week.
Will keep a close eye on JIRAs/PRs that are really close as well.
Happy to snag RM duties but always welcome others who have such
interest as well for this release or ones down the line.

Specifically, I want to thank several members of the community for
their efforts to drive out dependencies with vulnerabilities and
general security focus. Awesome to see!

All currently 1.18 Tagged JIRAs
https://issues.apache.org/jira/projects/NIFI/versions/12352150

The 1.18 release notes
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150

Thanks
Joe

Re: New NiFi PMC Member Nathan Gough

[Otto Fowler]

Congratulations, thanks for all you do




From: David Handermann 

Reply: dev@nifi.apache.org  
Date: September 7, 2022 at 12:10:21
To: dev@nifi.apache.org  
Subject:  New NiFi PMC Member Nathan Gough

Apache NiFi Community,

On behalf of the Apache NiFi Project Management Committee, I am very
pleased to announce that Nathan Gough has accepted the invitation to join
the PMC.

Nathan has been a consistent contributor to the project since 2018. In
addition to developing features and fixes, he regularly reviews pull
requests and assists others through mailing lists and Slack conversations.
He also plays an active role in reviewing and updating security reports for
the project.

Please join us in congratulating and welcoming Nathan to the Apache NiFi
PMC. Congratulations Nathan!

Re: [VOTE] Release Apache NiFi 1.17.0 (rc1)

[Otto Fowler]

 +1 (non-binding)
Followed release helper
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /usr/local/Cellar/maven/3.8.6/libexec
Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime:
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: July 25, 2022 at 22:57:16
To: dev@nifi.apache.org  
Subject:  [VOTE] Release Apache NiFi 1.17.0 (rc1)

Hello,

I am pleased to be calling this vote for the source release of Apache NiFi
1.17.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1209

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.17.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.17.0-RC1
The Git commit ID is cfa99de02ab5a440e94a2bb0044822ae9d0d99e7
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=cfa99de02ab5a440e94a2bb0044822ae9d0d99e7

Checksums of nifi-1.17.0-source-release.zip:
SHA256: 060ac40bbd6f3bac08f724bf040a68d52f4449d2e529af9c1d61db8932aa7b1b
SHA512:
97440184d0c34e1f287eba648eb47e5c276df46d6d89c376616939380d95afb6c03b4e4e9ea4170ba62d300637387d532314381225d000d529a6cc5bc4e8b436


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

305 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351438

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.17.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test. Then
please vote:

[ ] +1 Release this package as nifi-1.17.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: How to manage security artifacts from a custom processor

[Otto Fowler]

 Usually, you would write you custom processor to support the
StandardSSLSocketService:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.16.3/org.apache.nifi.ssl.StandardSSLContextService/index.html




From: Russell Bateman  
Reply: dev@nifi.apache.org  
Date: July 5, 2022 at 18:30:46
To: NiFi Developers List  
Subject:  How to manage security artifacts from a custom processor

>From a custom processor, I intend to interface with a third-party
br/>servicee (via simple HTTP client), however, I would need as I
understand br//>it to

a) maintain a private key by which I can identify myself to that
third-party service and
b) maintain a trusted-store certificate by which I can guarantee the
identity of the service.

This is pretty far outside my own experience. I have been reading on how
br/>this is achieved in Java, but in my mind a complication aarises from
the br/>fact that a custom NiFFi processor lives within NiFi's JVM. My
question br/>is therefore, how can I control the ceertificates and
authorities for my br/>use in or associated with NiFFi's JVM. Clearly, I
don't grok this well br/>enough even to ask the qquestion; I'm hoping
someone can see through what br/>I'm asking and point me in a good
direction to study.

I've written a pile of successful and useful custom NiFi processors to
br/>cover proprietary needs, so custom-processor writing isn''t a mystery.
br/>Certificates, keys, trusts and security in general still is.

Profuse thanks,

Russ

Re: Missing maven dependencies when building nifi

[Otto Fowler]

 What country are you in?  Are you under export controls?

From: Phil H  
Reply: dev@nifi.apache.org  
Date: April 6, 2022 at 07:08:45
To: dev@nifi.apache.org  
Subject:  Re: Missing maven dependencies when building nifi

So, I restarted the whole process inside a CentOS VM. The only way I could
get the build to complete was by using -DskipTests in the maven command.

Trying to run the built nifi fails after maybe 15 seconds (see below):

[phil@localhost nifi-1.17.0-SNAPSHOT]$ bin/nifi.sh run

Java home: /usr/java/jdk1.8.0_131/
NiFi home: /home/phil/nifi-1.17.0-SNAPSHOT

Bootstrap Config File: /home/phil/nifi-1.17.0-SNAPSHOT/conf/bootstrap.conf

2022-04-06 21:02:19,292 INFO [main] org.apache.nifi.bootstrap.Command
Generating Self-Signed Certificate: Expires on 2022-06-05
2022-04-06 21:02:20,426 ERROR [main] org.apache.nifi.bootstrap.Command
Self-Signed Certificate Generation Failed
java.io.IOException: exception encrypting data -
java.security.InvalidKeyException: Illegal key size
at
org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.wrapKey(Unknown

Source)
at
org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.doStore(Unknown

Source)
at
org.bouncycastle.jcajce.provider.keystore.pkcs12.PKCS12KeyStoreSpi.engineStore(Unknown

Source)
at
org.bouncycastle.jcajce.provider.keystore.util.AdaptingKeyStoreSpi.engineStore(Unknown

Source)
at java.security.KeyStore.store(KeyStore.java:1377)
at
org.apache.nifi.security.util.KeyStoreUtils.createKeyStoreAndGetX509Certificate(KeyStoreUtils.java:540)

at
org.apache.nifi.security.util.KeyStoreUtils.createTlsConfigAndNewKeystoreTruststore(KeyStoreUtils.java:228)

at
org.apache.nifi.bootstrap.util.SecureNiFiConfigUtil.configureSecureNiFiProperties(SecureNiFiConfigUtil.java:123)

at org.apache.nifi.bootstrap.RunNiFi.start(RunNiFi.java:1249)
at org.apache.nifi.bootstrap.RunNiFi.main(RunNiFi.java:294)
2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command
Starting Apache NiFi...
2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command
Working Directory: /home/phil/nifi-1.17.0-SNAPSHOT
2022-04-06 21:02:20,428 INFO [main] org.apache.nifi.bootstrap.Command
Command: /usr/java/jdk1.8.0_131/bin/java -classpath
/home/phil/nifi-1.17.0-SNAPSHOT/./conf:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/javax.servlet-api-3.1.0.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/jetty-schemas-5.2.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/logback-classic-1.2.11.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/logback-core-1.2.11.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/jcl-over-slf4j-1.7.36.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/jul-to-slf4j-1.7.36.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/log4j-over-slf4j-1.7.36.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/slf4j-api-1.7.36.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-api-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-framework-api-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-server-api-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-runtime-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-nar-utils-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-properties-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-property-utils-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-stateless-bootstrap-1.17.0-SNAPSHOT.jar:/home/phil/nifi-1.17.0-SNAPSHOT/./lib/nifi-stateless-api-1.17.0-SNAPSHOT.jar

-Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m
-Dcurator-log-only-first-connection-issue-as-error-level=true
-Djavax.security.auth.useSubjectCredsOnly=true
-Djava.security.egd=file:/dev/urandom -Dzookeeper.admin.enableServer=false
-Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol
-Dnifi.properties.file.path=/home/phil/nifi-1.17.0-SNAPSHOT/./conf/nifi.properties

-Dnifi.bootstrap.listen.port=36117 -Dapp=NiFi
-Dorg.apache.nifi.bootstrap.config.log.dir=/home/phil/nifi-1.17.0-SNAPSHOT/logs

org.apache.nifi.NiFi
2022-04-06 21:02:20,435 INFO [main] org.apache.nifi.bootstrap.Command
Launched Apache NiFi with Process ID 21977
[phil@localhost nifi-1.17.0-SNAPSHOT]$

If I try and run the tests as part of the build process, I get similar
complaints about key size (here's one example)

[INFO] Running org.apache.nifi.security.kms.FileBasedKeyProviderTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
0.003 s <<< FAILURE! - in
org.apache.nifi.security.kms.FileBasedKeyProviderTest
[ERROR] org.apache.nifi.security.kms.FileBasedKeyProviderTest.testGetKey
Time elapsed: 0.001 s <<< ERROR!
java.security.InvalidKeyException: Illegal key size
at
org.apache.nifi.security.kms.FileBasedKeyProviderTest.getSecretKeysPath(FileBasedKeyProviderTest.java:60)

at
org.apache.nifi.security.kms.FileBasedKeyProviderTest.testGetKey(FileBasedKeyProviderTest.java:47)




On Wed, Apr 6, 2022 at 5:08 PM Phil H  wrote:

> 

Re: [ANNOUNCE] New Apache NiFi Committer Adam Markovics

[Otto Fowler]

 Congratulations!

From: Marton Szasz  
Reply: dev@nifi.apache.org  
Date: March 25, 2022 at 08:50:07
To: dev@nifi.apache.org  
Subject:  [ANNOUNCE] New Apache NiFi Committer Adam Markovics

Apache NiFi community,

On behalf of the Apache NiFi PMC, I am very pleased to announce that
Adam Markovics has accepted the PMC's invitation to become a committer
on the Apache NiFi project. We greatly appreciate all of Adam's hard
work and generous contributions to the project. We look forward to
continued involvement in the project.

Adam has been a consistent contributor to MiNiFi C++ for almost 2
years. His contributions vary from user experience improvements, for
example better diagnostics for incorrect flow designs, and developer
quality of life improvements like improved static analysis and better
test coverage.

Welcome Adam and congratulations!

Re: SplitContent doesn’t support regex?

[Otto Fowler]

Joe, I don’t know if we can make the case for a stand alone processor for doing 
this on top of that, if so, I’d be willing to take a look at that.




From: Otto Fowler 
Reply: Otto Fowler 
Date: March 19, 2022 at 13:40:07
To: dev@nifi.apache.org , Phil H 
Subject:  Re: SplitContent doesn’t support regex?  

In the Apache Metron Project (in the attic now) we used 
https://github.com/nishihatapalmer/byteseek to do pcap searches, maybe you can 
check that out.




From: Phil H 
Reply: dev@nifi.apache.org 
Date: March 16, 2022 at 20:04:58
To: dev@nifi.apache.org 
Subject:  Re: SplitContent doesn’t support regex?  

I dunno about a good implementation…  

I did a similar extension of GetTCP to allow for a regex EOM rather than a  
single byte. It works, but I don’t feel like it was done in the spirit of  
the existing processor!  

On Thu, 17 Mar 2022 at 09:12, Joe Witt  wrote:  

> Phil  
>  
> I'd say if you have a good implementation in mind you should go for it.  
> Sounds interesting.  
>  
> Thanks  
>  
> On Wed, Mar 16, 2022 at 3:59 PM Phil H  wrote:  
>  
> > Hi,  
> >  
> > This seems like an odd omission - aside from performance (presumably?) is  
> > there a reason why there isn’t a regex option for the byte sequence? I  
> need  
> > one but thought I’d ask before I built my own.  
> >  
> > Thanks  
> > Phil  
> >  
>  

Re: SplitContent doesn’t support regex?

[Otto Fowler]

In the Apache Metron Project (in the attic now) we used
https://github.com/nishihatapalmer/byteseek to do pcap searches, maybe you
can check that out.




From: Phil H  
Reply: dev@nifi.apache.org  
Date: March 16, 2022 at 20:04:58
To: dev@nifi.apache.org  
Subject:  Re: SplitContent doesn’t support regex?

I dunno about a good implementation…

I did a similar extension of GetTCP to allow for a regex EOM rather than a
single byte. It works, but I don’t feel like it was done in the spirit of
the existing processor!

On Thu, 17 Mar 2022 at 09:12, Joe Witt  wrote:

> Phil
>
> I'd say if you have a good implementation in mind you should go for it.
> Sounds interesting.
>
> Thanks
>
> On Wed, Mar 16, 2022 at 3:59 PM Phil H  wrote:
>
> > Hi,
> >
> > This seems like an odd omission - aside from performance (presumably?)
is
> > there a reason why there isn’t a regex option for the byte sequence? I
> need
> > one but thought I’d ask before I built my own.
> >
> > Thanks
> > Phil
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Paul Grey

[Otto Fowler]

 Congratulations!

From: David Handermann 

Reply: dev@nifi.apache.org  
Date: March 16, 2022 at 18:46:09
To: dev@nifi.apache.org  
Subject:  [ANNOUNCE] New Apache NiFi Committer Paul Grey

Apache NiFi community,

On behalf of the Apache NiFi PMC, I am very pleased to announce that Paul
Grey
has accepted the PMC's invitation to become a committer on the Apache NiFi
project.

Paul has contributed a number of pull requests and code reviews over the
past year, improving project security and test stability in a number of
areas. We appreciate Paul's work and look forward to continued
contributions!

Welcome Paul, and congratulations!

Re: [VOTE] Release Apache NiFi 1.15.3 (rc1)

[Otto Fowler]

 +1 ( non-binding )
built and tested with java 11.0.12
followed release guide

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: January 13, 2022 at 15:44:42
To: dev@nifi.apache.org  
Subject:  [VOTE] Release Apache NiFi 1.15.3 (rc1)

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi 1.15.3.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1194

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.3/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.15.3-RC1
The Git commit ID is 753c311382005acadc16c64952d7b1eaaf2550d5
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=753c311382005acadc16c64952d7b1eaaf2550d5

Checksums of nifi-1.15.3-source-release.zip:
SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b
SHA512:
cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

11 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351203

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.3

The vote will be open
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test.
Then please vote:

[ ] +1 Release this package as nifi-1.15.3
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [DISCUSS] Release Apache NiFi 1.15.3 (rc1)

[Otto Fowler]

 So, i updated Java 11 and this time attended the build and saw that what
was breaking it was a firewall i’m running that was blocking java listening
on a port.
Sorry for the noise.

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: January 14, 2022 at 11:16:46
To: dev@nifi.apache.org  
Subject:  Re: [DISCUSS] Release Apache NiFi 1.15.3 (rc1)

Otto

Having built on osx/linux, and the github builds I have seen nothing
but stability generally. See here for the results of this specific
release branch on github
https://github.com/apache/nifi/actions/runs/1694607921

That said, try using a more recent Java 11 and see if results differ.

Thanks

On Fri, Jan 14, 2022 at 9:10 AM Otto Fowler 
wrote:
>
> I am seeing errors building the release candidate with failed tests. Does
> anyone have any insight into these failures?
>
> INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [ERROR] WARNING: An illegal reflective access operation has occurred
> [ERROR] WARNING: Illegal reflective access by
> org.codehaus.groovy.reflection.CachedClass
>
(file:/Users/ottofowler/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar)

> to method java.lang.Object.finalize()
> [ERROR] WARNING: Please consider reporting this to the maintainers of
> org.codehaus.groovy.reflection.CachedClass
> [ERROR] WARNING: Use --illegal-access=warn to enable warnings of
> further illegal reflective access operations
> [ERROR] WARNING: All illegal access operations will be denied in a
> future release
> [INFO] Running org.apache.nifi.remote.util.TestSiteToSiteRestApiClient
> [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 0.085 s - in
> org.apache.nifi.remote.util.TestSiteToSiteRestApiClient
> [INFO] Running org.apache.nifi.remote.util.TestExtendTransactionCommand
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.977 s - in org.apache.nifi.remote.util.TestExtendTransactionCommand
> [INFO] Running
org.apache.nifi.remote.protocol.http.TestHttpClientTransaction
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.079 s - in
org.apache.nifi.remote.protocol.http.TestHttpClientTransaction
> [INFO] Running
org.apache.nifi.remote.protocol.socket.TestSocketClientTransaction
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.088 s - in
org.apache.nifi.remote.protocol.socket.TestSocketClientTransaction
> [INFO] Running org.apache.nifi.remote.client.TestSiteInfoProvider
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 0.077 s - in org.apache.nifi.remote.client.TestSiteInfoProvider
> [INFO] Running org.apache.nifi.remote.client.PeerSelectorTest
> [INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 3.024 s - in org.apache.nifi.remote.client.PeerSelectorTest
> [INFO] Running org.apache.nifi.remote.client.http.TestHttpClient
> [ERROR] Tests run: 32, Failures: 0, Errors: 2, Skipped: 0, Time
> elapsed: 19.05 s <<< FAILURE! - in
> org.apache.nifi.remote.client.http.TestHttpClient
> [ERROR]
org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS
> Time elapsed: 0.598 s <<< ERROR!
> java.io.IOException: Failed to complete transaction with
> Peer[url=https://localhost:64409/nifi-api] due to java.io.IOException:
> [Url=https://localhost:64409/nifi-api, TransferDirection=SEND,
> State=TRANSACTION_CONFIRMED] Failed to receive a response from
> Peer[url=https://localhost:64409/nifi-api] when expecting a
> TransactionFinished Indicator. It is unknown whether or not the peer
> successfully received/processed the data.
> javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
> at
org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:942)

> at
org.apache.nifi.remote.client.http.TestHttpClient.testSend(TestHttpClient.java:805)

> at
org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS(TestHttpClient.java:916)

> Caused by: java.io.IOException: [Url=https://localhost:64409/nifi-api,
> TransferDirection=SEND, State=TRANSACTION_CONFIRMED] Failed to receive
> a response from Peer[url=https://localhost:64409/nifi-api] when
> expecting a TransactionFinished Indicator. It is unknown whether or
> not the peer successfully received/processed the data.
> javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
> at
org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:942)

> at
org.apache.nifi.remote.client.http.TestHttpClient.testSend(TestHttpClient.java:805)

> at
org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTT

[DISCUSS] Release Apache NiFi 1.15.3 (rc1)

[Otto Fowler]

I am seeing errors building the release candidate with failed tests. Does
anyone have any insight into these failures?

INFO]
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[ERROR] WARNING: An illegal reflective access operation has occurred
[ERROR] WARNING: Illegal reflective access by
org.codehaus.groovy.reflection.CachedClass
(file:/Users/ottofowler/.m2/repository/org/codehaus/groovy/groovy/2.5.14/groovy-2.5.14.jar)
to method java.lang.Object.finalize()
[ERROR] WARNING: Please consider reporting this to the maintainers of
org.codehaus.groovy.reflection.CachedClass
[ERROR] WARNING: Use --illegal-access=warn to enable warnings of
further illegal reflective access operations
[ERROR] WARNING: All illegal access operations will be denied in a
future release
[INFO] Running org.apache.nifi.remote.util.TestSiteToSiteRestApiClient
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 0.085 s - in
org.apache.nifi.remote.util.TestSiteToSiteRestApiClient
[INFO] Running org.apache.nifi.remote.util.TestExtendTransactionCommand
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.977 s - in org.apache.nifi.remote.util.TestExtendTransactionCommand
[INFO] Running org.apache.nifi.remote.protocol.http.TestHttpClientTransaction
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.079 s - in org.apache.nifi.remote.protocol.http.TestHttpClientTransaction
[INFO] Running 
org.apache.nifi.remote.protocol.socket.TestSocketClientTransaction
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.088 s - in org.apache.nifi.remote.protocol.socket.TestSocketClientTransaction
[INFO] Running org.apache.nifi.remote.client.TestSiteInfoProvider
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
0.077 s - in org.apache.nifi.remote.client.TestSiteInfoProvider
[INFO] Running org.apache.nifi.remote.client.PeerSelectorTest
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 3.024 s - in org.apache.nifi.remote.client.PeerSelectorTest
[INFO] Running org.apache.nifi.remote.client.http.TestHttpClient
[ERROR] Tests run: 32, Failures: 0, Errors: 2, Skipped: 0, Time
elapsed: 19.05 s <<< FAILURE! - in
org.apache.nifi.remote.client.http.TestHttpClient
[ERROR] org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS
 Time elapsed: 0.598 s  <<< ERROR!
java.io.IOException: Failed to complete transaction with
Peer[url=https://localhost:64409/nifi-api] due to java.io.IOException:
[Url=https://localhost:64409/nifi-api, TransferDirection=SEND,
State=TRANSACTION_CONFIRMED] Failed to receive a response from
Peer[url=https://localhost:64409/nifi-api] when expecting a
TransactionFinished Indicator. It is unknown whether or not the peer
successfully received/processed the data.
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:942)
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSend(TestHttpClient.java:805)
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS(TestHttpClient.java:916)
Caused by: java.io.IOException: [Url=https://localhost:64409/nifi-api,
TransferDirection=SEND, State=TRANSACTION_CONFIRMED] Failed to receive
a response from Peer[url=https://localhost:64409/nifi-api] when
expecting a TransactionFinished Indicator. It is unknown whether or
not the peer successfully received/processed the data.
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:942)
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSend(TestHttpClient.java:805)
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS(TestHttpClient.java:916)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSendIgnoreProxyError(TestHttpClient.java:942)
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSend(TestHttpClient.java:805)
at 
org.apache.nifi.remote.client.http.TestHttpClient.testSendSuccessHTTPS(TestHttpClient.java:916)

[ERROR] org.apache.nifi.remote.client.http.TestHttpClient.testSendLargeFileHTTPS
 Time elapsed: 0.554 s  <<< ERROR!
java.io.IOException: Failed to complete transaction with
Peer[url=https://localhost:64409/nifi-api] due to java.io.IOException:
[Url=https://localhost:64409/nifi-api, TransferDirection=SEND,
State=TRANSACTION_CONFIRMED] Failed to receive a response from
Peer[url=https://localhost:64409/nifi-api] when expecting a
TransactionFinished Indicator. It is unknown whether or not the peer
successfully received/processed the data.
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at 

Re: [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3

[Otto Fowler]

 +1 (non-binding)
ran through the testing guide

From: Bryan Bende  
Reply: dev@nifi.apache.org  
Date: November 30, 2021 at 15:56:38
To: dev@nifi.apache.org  
Subject:  [VOTE] Release Apache NiFi NAR Maven Plugin 1.3.3

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi NAR Maven Plugin 1.3.3.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1188

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-nar-maven-plugin-1.3.3/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-nar-maven-plugin-1.3.3-RC1
The Git commit ID is 99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2
https://gitbox.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=99f1ff7184e00cecc4763b7bcbdf0d12dfeffef2

Checksums of nifi-nar-maven-plugin-1.3.3-source-release.zip:
SHA256: 36d6579dfdbc4e1450a13457196ff399dbd344624348a62ee5e247cd5962dc77
SHA512:
0e90100d90e9dd142283aab729957e12cc6abbfdf2f86447f488bcbe5890d55564c19a0efc473307744f813bcdac2636b027ad4e5beebf1ce1ed6ee44deaccbe


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/bbende.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

1 issue was closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348815

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.3.3

The vote will be open for 72 hours. Please download the release
candidate and evaluate the necessary items including checking hashes,
signatures, build from source, and test. Then please vote:

[ ] +1 Release this package as nifi-nar-maven-plugin-1.3.3
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [DISCUSS] Docs from non packaged NARs

[Otto Fowler]

 This makes me wonder how documentation is going to work with the registry,
should the documentation show all the nars that are in the registry and
available or all the cars local? etc.

From: Pierre Villard 

Reply: dev@nifi.apache.org  
Date: November 9, 2021 at 18:17:30
To: dev@nifi.apache.org  
Subject:  [DISCUSS] Docs from non packaged NARs

Hi team,

I'd like to discuss the possibility to change the release process we have
for the next releases so that the documentation of all components we have
in the repository are published as part of the release process.

All of the documentation for components that are in specific profiles are
not only skipped for the final convenience binary but also for the
documentation. We have a tremendous amount of great components in this "by
default excluded" NARs and it would be helpful to have the corresponding
documentation to at least redirect users when we get some questions on the
users mailing list.

I feel like this should only be a release process thing but I could be
wrong.

Something that could be even better would be to enable all the profiles by
default but only for nifi-docs module? Not sure if that is even a
possibility.

Thoughts?

Thanks,
Pierre

Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)

[Otto Fowler]

 +1
Ran through verification, build and run

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: November 3, 2021 at 15:29:41
To: dev@nifi.apache.org  
Subject:  [VOTE] Release Apache NiFi 1.15.0 (rc3)

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi 1.15.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1186

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.15.0-RC3
The Git commit ID is 7fdc07cccdc0e23d4986557a9314e36859704a3b
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=7fdc07cccdc0e23d4986557a9314e36859704a3b

Checksums of nifi-1.15.0-source-release.zip:
SHA256: 56fe317248941fdbc6216ec973e6ce898d0f682a54e2d063edbabf8542f5288a
SHA512:
63f10d9127cf1613c29bf2306be3d7a5b931b31304920706e0df5ea2fe87b58c410efed6ac37afc38d12c65e69f14aec7afb926000fc90cc13f15c738cd15c7f


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

234 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350382

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test.
Then please vote:

[ ] +1 Release this package as nifi-1.15.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.15.0

[Otto Fowler]

 +1

Checked all the sources, hashes etc
Built and ran

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: November 1, 2021 at 20:18:04
To: dev@nifi.apache.org  
Subject:  [VOTE] Release Apache NiFi 1.15.0

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi 1.15.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1185

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.15.0-RC2
The Git commit ID is c48d10315976805f6e675fc7907dfe196787e4bc
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=c48d10315976805f6e675fc7907dfe196787e4bc

Checksums of nifi-1.15.0-source-release.zip:
SHA256: 9db0c22da32840320c06df3bca75c22440aa2b1bfae42a249b4850c3ce89f8e4
SHA512:
866a6dfdd9a843a27f34d1d4842f056cc82a1e17852f03fb90aa489d5e1adc26e8f7a6da07fdaf2de73c10580e0236eea528c3b5119d98fb9ddbd3f2cd9d2688


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

232 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350382

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test.
Then please vote:

[ ] +1 Release this package as nifi-1.15.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [discuss] NiFi 1.15

[Otto Fowler]

This release, we are verifying not only Nifi, but also Minifi Java. At
least that is my understanding.

This would be my first time having *anything* to do with minifi, i’ve not
even run it before.

As such, I think the RC validation guide needs to be update to include the
information about now validating nifi and minifi together.




From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: October 25, 2021 at 11:59:39
To: dev@nifi.apache.org  
Subject:  [discuss] NiFi 1.15

Team,

I thought I had already started such a thread but I dont see it so here
goes...

We have made a ton of progress again and there are some super
important fixes as well. It is definitely time to kick out a 1.15.
My intent will be to attempt to pull together an RC this week.
Haven't done the analysis yet of what is hanging out there but will do
so. A quick look at all the features and fixes already landed though
make it clear we have more than enough to work with.

Lets please use this thread for coordination on the RC rather than it
becoming the wish list. We have new features/fixes arriving all the
time - those can be addressed in normal channels.

Thanks

Re: [discuss] NiFi 1.15

[Otto Fowler]

Is there a discuss thread for the release? I only see the vote and I have a
question about the verification. My mail must be messed up, i have this
mail with no replies… then vote thread.




From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: October 25, 2021 at 11:59:39
To: dev@nifi.apache.org  
Subject:  [discuss] NiFi 1.15

Team,

I thought I had already started such a thread but I dont see it so here
goes...

We have made a ton of progress again and there are some super
important fixes as well. It is definitely time to kick out a 1.15.
My intent will be to attempt to pull together an RC this week.
Haven't done the analysis yet of what is hanging out there but will do
so. A quick look at all the features and fixes already landed though
make it clear we have more than enough to work with.

Lets please use this thread for coordination on the RC rather than it
becoming the wish list. We have new features/fixes arriving all the
time - those can be addressed in normal channels.

Thanks

Re: [ANNOUNCE] New NiFi PMC Member David Handermann

[Otto Fowler]

 Congratulations!

From: Bryan Bende  
Reply: dev@nifi.apache.org  
Date: September 17, 2021 at 08:56:43
To: dev@nifi.apache.org  
Subject:  [ANNOUNCE] New NiFi PMC Member David Handermann

NiFi Community,

On behalf of the Apache NiFI PMC, I am very pleased to announce that
David Handermann has accepted the PMC's invitation to join the Apache
NiFi PMC.

David has made significant improvements in a number of security
related areas, he continues to improve the stability of our tests and
CI builds, and regularly helps out the community through the mailing
lists, Slack, and PR reviews.

Please join us in congratulating and welcoming David to the Apache NiFi
PMC.

Congratulations David!

Re: Flood of JIRAs and presumably PRs to follow for junit-5 migration?

[Otto Fowler]

 perhaps we can have a jira label and PR naming convention for things that
fall under this as well, to set consistent expectations for review and work

From: Kevin Doran  
Reply: dev@nifi.apache.org  
Date: August 31, 2021 at 18:06:48
To: dev@nifi.apache.org  
Subject:  Re: Flood of JIRAs and presumably PRs to follow for junit-5
migration?

Hi all,

JUnit 5 certainly contains a number of benefits, so I’m glad to see some
interest and effort around it. br/> <
As long as we are taking the time to update tests in each module and line
up reviewers for these bulk refactoring, I agree with David that we should
make improvements aside from just JUnit 4 to 5 syntax/compatibility
migrations. This is a good opportunity to bring consistency and
improvements to our tests.

Just an idea, but maybe it would be a good idea to start a list of
anti-patterns and code smell in existing tests. This could be maintained on
the wiki or similar, and any PRs opened as part of this effort can address
those issues in addition to upgrading to JUnit 5. Here are examples of
things I would include:

Don’t:
- Leave dead/commented code in test classes
- @Ignored tests
- Using try/catch with fail() to expect exceptions
- Use thread.sleep() or other unreliable methods for asynchronous logic
assertions

Do:
- Use granular unit tests
- Use arrange/act/assert or given/when/then pattern where possible
- Use assertThrows() to check for expected exceptions
- Use libraries such as AssertJ or Awaitility to write clear, concise,
reliable tests.
- Use Spock where possible (Just joking! I know this is a heated issue. But
also, do use it! )

Thanks!
Kevin


> On Aug 25, 2021, at 4:01 PM, David Handermann 
wrote:
> br/>> Mike, <
> br/>> Thanks for moving things forward on the JUnit 5 migration. I posted
a
> couple comments on the PR for nifi-commons (NIFI-9080) with concerns
> related to breaking tests and perpetuating ignored unit tests.
> br/>> Summarizing in this thread for general discussiion, I am concerned
about
> breaking unit tests as part of the migration process. As long as NiFi
> supports both JUnit 4 and JUnit 5, the migration should improve the
overall
> testing environment. Breaking existing tests will require additional work
> to go back and fix, and spending the effort upgrading and reviewing test
> classes doesn't seem worth it if we continue marking tests as ignored. In
> some particular cases, it may be worthwhile to remove a test method or
test
> class. For test methods related to performance, migrating to JUnit 5
seems
> like an ideal opportunity to make tests conditional on environment
> variables or system properties. I would be glad to help with the
migration
> process, but it would be helpful to avoid having to revisit code multiple
> times to address these issues.
> br/>> Regards, <
> David Handermann
> br/>> On Wed, Aug 25, 2021 at 2:24 PM Mike Thomsen &
llt;mikerthom...@gmail.com> wrote:
> br/>>> I broke up the tickets because it is A LOT of individual tasks
that
>> can break the overall build and wanted to scope the work appropriately
>> so that people looking for a chance to contribute could snatch up a
>> few easy wins.
>> br/>>> I'm still going to take point on makingg the changes, but the
plan
>> going forward is to submit PRs that bundle a bunch of tickets so that
>> we don't overwhelm reviewers and the CI/CD pipeline.
>> br/>>> Most of the changes are automated by InntelliJ Ultimate's
migration
>> tool, which from my testing is really good at migrating about 95% of
>> our JUnit 4 unit and integration tests.
>> br/>>> Thanks, <
>> br/>>> Mike <
>> br/>>> On Wed, Aug 25, 2021 at 1:05 PM Joe Wiitt 
wrote:
>>> br/ Joey <
>>> br/ I personally dont care all thaat much about the number of
commits in PR
>>> - I think that rule is sort of soft already.
>>> br/ I dont think there is any inherrent value in having a single
module per
>>> JIRA (and PR or PR commit) on this. These can be done in much coarser
>>> grained chunks. It will have to be to get review cycles for instance
>>> (much less having the Github infra to run these builds).
>>> br/ Thanks <
>>> Joe
>>> br/ On Wed, Aug 25, 2021 at 9:44 AMM Joey Frazee
>>>  wrote:
 br/> Maybe this is an excepttion to the single squashed commit
guidance for
>> the initial pull?
 br/> I assume the intent is to make incremental progress and not
have a PR
>> with a hundred files affected, but if the different module changes
>> corresponded to a different commit, GH will make it easy enough to have
a
>> draft and review each commit in isolation.
 br/> Would that be a reasonaable approach?
 br/> -joey <
 br/>> On Aug 25, 2021, aat 9:36 AM, Joe Witt 
wrote:
> br/>> 

Re: Contributor

[Otto Fowler]

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#how-to-contribute-to-apache-nifi

From: Roberto Frederico Togo Santos  
Reply: dev@nifi.apache.org  
Date: August 20, 2021 at 21:04:20
To: dev@nifi.apache.org  
Subject:  Contributor

Hi.

I have added in Oracle12DatabaseAdapter.java a UPSERT capability, using
MERGE Statement, for Oracle12+ on Nifi 1.14.0 with it's unit tests.

How can I submit my code to you ?

Thank you for your attention.

-- br/>*Roberto FFrederico*

Re: [DISCUSS] NiFi 2.0 Release Goals

[Otto Fowler]

 You don’t have to assign it to yourself.  User submitted jiras are very
important, because they include use cases and details that may not occur to
a developer who just picks it up.  It is a great way to contribute!

From: Ryan Hendrickson 

Reply: dev@nifi.apache.org  
Date: August 6, 2021 at 03:23:43
To: dev@nifi.apache.org  
Subject:  Re: [DISCUSS] NiFi 2.0 Release Goals

If I put Jira's in, then I'd have to admit I'm a developer vs a user
:-)

I'm just a power user at heart :-) ...

Ryan

On Thu, Aug 5, 2021 at 2:01 PM Otto Fowler  wrote:

> Ryan, these are awesome, are there jiras for them? It would be best to
> capture the requirements / use cases
>
> From: Ryan Hendrickson 
> 
> Reply: dev@nifi.apache.org  
> Date: August 4, 2021 at 01:14:23
> To: dev@nifi.apache.org  
> Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
>
> I'll preface with what I'm about to say is not rooted in any real
> understanding of the lift required or other customer experiences... We
> have used NiFi for a long time and are excited for what new features may
> lay ahead in the 2.x line.
>
> ---After having written this, I think our succinct pain points focus
> around large data flow management, auto-scaling container environments,
and
> integration of external analytic/enrichment creation.---
>
> Thanks,
> Ryan
>
>
> 1 - Integration with auto-scaling container environments
> We often struggle with scaling custom processors in large flows. We've
> implemented a recent design pattern where we write custom python code
> packaged into a docker image, fronted with a web-service, and deployed
into
> a container environment. In NiFi we use InvokeHTTP to call out to the
> web-service. If the load becomes too great, we can easily spin-up more
> than one external web-service without sacrificing the NiFi's overall
> thruput and thread contention. In this scenario, we're using NiFi to
> manage the overall dataflow, routes, ETL services, etc, but not so much
> smarts around auto-scaling processing, etc.
>
> This makes it really easy to receive a dataflow analytic/enrichment from
an
> external team, wrap it in a docker image, and deploy it into an
> auto-scaling container environment. This also lets us avoid conversations
> about how performant the code is and if it's meeting a minimum throughput
> requirement - typically XXGB/5minutes.
>
> The request for 2.x would be to make interacting with container deployed
> services a more seamless, streamlined process, possibly through a
> normalized API or as a "Remote Container Environment" that can bind to a
> service with a health endpoint and verify it's alive and well, etc.
>
> 2 - Integration of S3 buckets with Remote Process Groups for outage
events
> and data overruns
> Another frequent struggle is the occasional death of a server in a NiFi
> flow. Example -- Our overall NiFi flow consists of ~15 stand-alone NiFis
> and 2 NiFi clusters. The stand-alone NiFi's are linearly chained together
> with Remote Process Groups to perform the overall dataflow goals. If any
> one of the 15 experience an outage, data begins to backup on the previous
> NiFi, hitting backpressure limits, etc. When the down server comes back
> online, it receives a flood of data, often overwhelming it, necessitating
> careful flowfile limits and backpressure points.
>
> The design pattern we're exploring, but would like to see baked into
NiFi,
> is that when a Remote Process Group becomes unavailable due to
> server-to-server comms failures AND the backpressure limit is reached on
a
> relationship, the Remote Process Group would begin sending data to an S3
> bucket as an automatic failover procedure. When the downed server comes
> back-online, the Remote Process Group port would know to pull data from
S3
> AND receive data via the port from the chained server.
>
> 3 - Large Canvas Management
> Beginning with a NiFi flow that consists of ~15 stand-alone NiFis and 2
> NiFi clusters, each canvas is managed individually. To see the entire
> canvas, systems engineering diagrams need to be kept up-to-date
identifying
> all the input/output ports. To move processors from server to server
often
> requires the verification of the existence of scripts, configuration
files,
> NiFi custom nars, and using the templating or Registry to migrate
portions
> of the dataflow.
>
> Our current design pattern to accomplish some of these tasks include
using
> Ansible, Jenkins, and Nexus to verify script & configuration files are up
> to date. We use Confluence's Gliffy to document the NiFi communication
> paths between the servers.
>
> Ideally, we'd be able to see an overall management view of the Canvases
> together in a single place - it would indicate which server the portions
of

Re: [DISCUSS] NiFi 2.0 Release Goals

[Otto Fowler]

 MiNiFi C++, both server-side and client-side, to allow deploying
to
> MiNiFi Java instances from a centralize flow definition. Thanks to the
> unification of MiNiFi Java and NiFi, this could likely also be added as a
> capability of NiFi at that same time.
> >
> > If we are able to do this on the 1.x line, then I would also support
> deprecating other methods of flow deployment to MiNiFi outlined in [1]
and
> removing them in a 2.x release.
> >
> > [1] https://github.com/apache/nifi/pull/4933#issuecomment-808411977 <
> https://github.com/apache/nifi/pull/4933#issuecomment-808411977>
> > [2] https://cwiki.apache.org/confluence/display/MINIFI/C2+Design <
> https://cwiki.apache.org/confluence/display/MINIFI/C2+Design>
> >
> > Thanks,
> > Kevin
> >
> > > On Aug 3, 2021, at 10:07 AM, David Handermann <
> exceptionfact...@gmail.com> wrote:
> > >
> > > Thanks for the continued discussion around what can or should be
> removed.
> > > Having a clear migration path from version 1 to version 2 will be
very
> > > important for supporting current deployments.
> > >
> > > Following the example of some other projects, one way to consider the
> > > upgrade is from the point of view of deprecation warnings. If
> implemented
> > > correctly, an installation of NiFi running the latest minor release
of
> > > version 1, and showing no deprecation warnings, should be upgradeable
> to
> > > version 2 without any changes. Making this work involves accurately
> tagging
> > > components and methods with deprecation indicators, and providing a
> clear
> > > way to observe deprecation warnings. With the current version
> containing
> > > both deprecated components and deprecated methods in various classes,
> this
> > > would involve some work to get right. A simple approach might be a
> named
> > > logger for deprecation warnings, which would write a separate log
> file. A
> > > more advanced approach might involve a tool to analyze the system and
> flow
> > > configurations. Either way, it seems that some additional work in a
> minor
> > > release version is necessary before considering a major release
> version.
> > >
> > > One other point on compatibility: as long as the core component API
> > > definition remains backwards compatible, it should be possible to run
> older
> > > components. This provides a transition option for customers using
> > > components such as PostHTTP, or any others that are not actively
> > > maintained. At some point it may become necessary to break
> compatibility at
> > > that level, but at least for version 2, maintaining component API
> > > compatibility is key.
> > >
> > > Regards,
> > > David Handermann
> > >
> > > On Sun, Aug 1, 2021 at 10:23 AM Mark Bean 
> wrote:
> > >
> > >> I created a JIRA ticket to investigate and improve the performance
of
> > >> InvokeHTTP. It includes a flow definition for benchmarking the
> performance
> > >> of both PostHTTP and InvokeHTTP.
> > >>
> > >> https://issues.apache.org/jira/browse/NIFI-8968
> > >>
> > >> Thanks,
> > >> Mark
> > >>
> > >> On Fri, Jul 30, 2021 at 6:41 PM Adam Taft  wrote:
> > >>
> > >>> I'm not seeing the side thread that was going to discuss
deprecation
> of
> > >>> PostHTTP. Has that thread started and I just don't see it?
> > >>>
> > >>> One (significant?) concern with PostHTTP is the smooth integration
of
> > >>> NiFi-to-NiFi communication that is very transparently enabled with
> the
> > >>> ListenHTTP and PostHTTP processors. There's some special logic in
> there
> > >> for
> > >>> handling flowfiles that InvokeHTTP doesn't really (nor should
really)
> > >> have.
> > >>>
> > >>> I know of several (large) NiFi installations that rely on the
> PostHTTP /
> > >>> ListenHTTP combination. It has enabled NiFi to NiFi communication
for
> > >> folks
> > >>> reluctant or unable to enable site-to-site type configuration.
> > >>>
> > >>> Honestly, I don't know why we'd want to "deprecate" any processor,
as
> > >>> opposed to just moving it to a new location. If these processors
can
> be
> > >>> ported and maintained to whatever the 2.0 API looks like, there's
> > >> possibly
> > >&

Re: [DISCUSS] NiFi 2.0 Release Goals

[Otto Fowler]

.) I understand the rationale for deprecating this
> > > > processor in that it eliminates a processor whose functionality is
> > > > available elsewhere, specifically in the more flexible InvokeHTTP.
> > > However,
> > > > in my experience and testing, PostHTTP performs transfers with far
> > > greater
> > > > throughput than InvokeHTTP. I would not be in favor of removing
> > PostHTTP
> > > > unless/until InvokeHTTP is refactored to increase its throughput
> > > > capability.
> > > >
> > > > Has anyone else continued to use PostHTTP over InvokeHTTP for
similar
> > > > reasons? Or, is there a performance-related configuration for
> > InvokeHTTP
> > > I
> > > > may have missed?
> > > >
> > > > Also, in order to help facilitate a smooth transition to 2.0 from a
> > user
> > > > perspective, would it be advisable to add some sort of visual
indicator
> > > in
> > > > the UI for components that are currently annotated as @Deprecated?
This
> > > > might help users proactively modify their flow prior to a release
that
> > > > would otherwise break it.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jul 26, 2021 at 5:02 PM Bryan Bende 
wrote:
> > > >
> > > > > With the merging of MiNiFi and registry into the NiFi repo, we've
> > > > > actually gone more towards a mono-repo where everything is
released
> > > > > together. Eventually it would still be nice to have a smaller
base
> > > > > distribution containing just the framework and standard NARs, but
it
> > > > > is hard to tackle that until we provide a way for users to easily
> > > > > obtain the NARs in some other way.
> > > > >
> > > > > On Mon, Jul 26, 2021 at 4:20 PM Edward Armes <
edward.ar...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Given the major version number shift and the spliting up of
> > > processors
> > > > > into
> > > > > > multiple NAR's. I'd like to suggest that we start individually
> > > > versioning
> > > > > > individual NARs/ bundles.
> > > > > >
> > > > > > I can see this bringing a large number of benifits including
making
> > > > Nifi
> > > > > > more deployable with things RPM, but also potentially allowing
> > those
> > > > that
> > > > > > have to deploy managed Nifi instances easier to upgrade.
> > > > > >
> > > > > > Edward
> > > > > >
> > > > > > On Mon, 26 Jul 2021, 20:42 Otto Fowler, 

> > > > wrote:
> > > > > >
> > > > > > > The issue with updating the aws sdk is if it breaks any one
of
> > the
> > > > > > > processors.
> > > > > > > the Web Gateway API invoke processor for example is not using
a
> > > high
> > > > > level
> > > > > > > purpose build client and may break.
> > > > > > >
> > > > > > > If we change the aws version, we need to coordinate in such a
way
> > > > that
> > > > > they
> > > > > > > all
> > > > > > > can come along reasonably.
> > > > > > > IE: what happens if 1 or 2 break but the rest or OK?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From: David Handermann 
> > > > > > > 
> > > > > > > Reply: dev@nifi.apache.org  <
> > > > dev@nifi.apache.org>
> > > > > > > Date: July 26, 2021 at 09:33:42
> > > > > > > To: dev@nifi.apache.org  <
> > dev@nifi.apache.org
> > > >
> > > > > > > Subject: Re: [DISCUSS] NiFi 2.0 Release Goals
> > > > > > >
> > > > > > > Chris,
> > > > > > >
> > > > > > > Thanks for the reply and recommendations. It seems like some
of
> > the
> > > > > work to
> > > > > > > reorganize the module structure could be done outside of a
major
> > > > > release,
> > > > > > > but it would be great to target any breaking changes for 2.0.
> > > > Perhaps a
> > > > > >

Re: [DISCUSS] NiFi 2.0 Release Goals

[Otto Fowler]

 The issue with updating the aws sdk is if it breaks any one of the
processors.
the Web Gateway API invoke processor for example is not using a high level
purpose build client and may break.

If we change the aws version, we need to coordinate in such a way that they
all
can come along reasonably.
IE:  what happens if 1 or 2 break but the rest or OK?



From: David Handermann 

Reply: dev@nifi.apache.org  
Date: July 26, 2021 at 09:33:42
To: dev@nifi.apache.org  
Subject:  Re: [DISCUSS] NiFi 2.0 Release Goals

Chris,

Thanks for the reply and recommendations. It seems like some of the work to
reorganize the module structure could be done outside of a major release,
but it would be great to target any breaking changes for 2.0. Perhaps a
separate feature proposal on module restructuring, with the goal of
supporting optimized builds, would be a helpful way to move that part of
the discussion forward.

Regarding updating AWS SDK to version 2, it seems like that might be
possible now. I haven't taken a close look at the referencing components,
so I'm not sure about the level of effort involved. Minor NiFi version
updates have incorporated major new versions of dependencies. For example,
NiFi 1.14 included an upgrade from Spring Framework 4 to 5. On the one
hand, including the AWS SDK update as part of a major release seems
helpful, but unless there are changes that break existing component
properties, upgrading the AWS SDK could be worked independently. Others may
have more insight into particular usage of that library.

Regards,
David Handermann

On Sun, Jul 25, 2021 at 2:12 AM Chris Sampson
 wrote:

> Might be worth considering refactoring the build as part of this work
too,
> e.g. only building the bits of the repo affected by a commit, etc. -
> discussed briefly in previous threads but don't think any changes made
yet.
> If NARs/components are likely to be split up and refactored then such
work
> around the build probably makes sense to consider.
>
> I've a couple of PRs open that include updates to Elasticsearch versions
> already, although I stopped at 7.10.2 (after which Elastic changed
licence)
> in case there were licence concerns. But more work can be done to tidy up
> the processors, absolutely.
>
> AWS libraries to v2 would seem a sensible move and a refactor of those
> processors as well.
>
>
> Cheers,
>
> Chris Sampson
>
> On Sat, 24 Jul 2021, 17:47 David Handermann, 

> wrote:
>
> > Thanks for pointing out the standard NAR bundles, Mark. There are a
> number
> > of components in the standard NAR bundles with particular dependencies
> that
> > would make more sense in separate NARs. Reorganizing the standard NAR
to
> > components with limited dependencies and wide applicability would
> > definitely help with future maintenance.
> >
> > Regards,
> > David Handermann
> >
> > On Sat, Jul 24, 2021 at 10:57 AM Mark Payne 
> wrote:
> >
> > > There’s also some code that exists in order to maintain backward
> > > compatibility in the repositories. I would very much like the
> > repositories
> > > to contain no unnecessary code. And swap file format supports really
> old
> > > formats. And the old impls of the repositories themselves, like
> > > PersistentProvRepo instead of WriteAheadProv Repo, etc. Lots of stuff
> > there
> > > that can be removed. And some methods in ProcessSession that are
never
> > used
> > > by any processor in the codebase but exists in the public API so
can’t
> be
> > > removed till 2.0.
> > >
> > > I think his is also a great time to clean up the “standard nar.” At
> this
> > > point, it’s something like 70 MB. And many of the components there
are
> > not
> > > really “standard” - things like connecting to FTP & SFTP servers, XML
> > > processing, Jolt transform, etc. could potentially be moved into
other
> > > nars. The nifi-standard-content-viewer-1.15.0-SNAPSHOT.war is 6.9 MB
is
> > not
> > > necessary for stateless or minifi java. Lots of things probably to
> > > reconsider within the standard nar.
> > >
> > > I definitely think this is a reasonable approach, to allow for a 2.0
> that
> > > is not a huge feature release but allows the project to be simpler
and
> > more
> > > nimble.
> > >
> > > Thanks
> > > -Mark
> > >
> > > On Jul 24, 2021, at 10:59 AM, Mike Thomsen  >  > > mikerthom...@gmail.com>> wrote:
> > >
> > > Russell,
> > >
> > > AFAICT from looking at Elastic's repos, the low level REST client is
> > > still fine.
> > >
> >
>
https://github.com/elastic/elasticsearch/blob/e5518e07f13701e3bb3dcc6842b9023966752497/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
> > >
> > > Our Elasticsearch support is spread over two NARs at present. One
uses
> > > OkHttp and the other uses that low level Elastic REST client.
> > > Therefore, I think we're fine on licensing for the moment.
> > >
> > > Mike
> > >
> > > On Fri, Jul 23, 2021 at 1:10 PM Russell Bateman  > > > wrote:
> > >
> > > Bringing up Elastic also reminds me that the 

Re: [DISCUSS] NiFi Registry -> NiFi consensus

[Otto Fowler]

 +1 (non-binding)

From: Matt Burgess  
Reply: dev@nifi.apache.org  
Date: July 16, 2021 at 14:20:20
To: dev@nifi.apache.org  
Subject:  [DISCUSS] NiFi Registry -> NiFi consensus

All,

We've been asked to record our consensus for the move of NiFi Registry
to the NiFi codebase in a mailing list thread for posterity. Most
discussion happened on the PR but INFRA would like a link to this
thread showing consensus from PMC members, committers, and the
community.

I didn't put my +1 on the PR but I am in favor of moving the NiFi
Registry codebase into NiFi :) Please feel free to share your thoughts
as well.

Thank you,
Matt

Re: [discuss] nifi 1.14.0

[Otto Fowler]

 PR up

From: Otto Fowler  
Reply: Otto Fowler  
Date: July 12, 2021 at 20:20:30
To: dev@nifi.apache.org  
Subject:  Re: [discuss] nifi 1.14.0

I’ll take care of it.  I’ll make the minimum 3.6.0

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: July 12, 2021 at 18:25:28
To: dev@nifi.apache.org  
Subject:  Re: [discuss] nifi 1.14.0

Yeah we just need to update the latest required version . We're going
all in on all the HTTPS requirements and such so we're probably really
tight on required version at this point anyway.

I'd not be sinking the RC for this. Just a good JIRA to work on.

Thanks

On Mon, Jul 12, 2021 at 12:59 PM Otto Fowler 
wrote:
>
> NIFI-8778
>
> From: Otto Fowler  
> Reply: Otto Fowler  
> Date: July 12, 2021 at 13:56:36
> To: dev@nifi.apache.org  
> Subject: Re: [discuss] nifi 1.14.0
>
> I see an issue, though I’m not sure if we want to cancel the RC so I’m
> asking before voting.
> The documentation states that we require Apache Maven 3.1.1 or newer, but
> the build fails with the error:
>
> [ERROR] Failed to execute goal
> com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm
> (install-node-and-npm) on project nifi-web-ui: The plugin
> com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version
> 3.6.0 -> [Help 1]
>
> I have maven version:
>
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-17T14:33:14-04:00)
> Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec
> Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime:
> /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
>
> So, the documentation on building nifi is wrong.
>
>
>
> From: Joe Witt  
> Reply: dev@nifi.apache.org  
> Date: July 9, 2021 at 10:24:58
> To: dev@nifi.apache.org  
> Subject: Re: [discuss] nifi 1.14.0
>
> Still working on the 1.14 RC. Has been a series of issues with the
> build. Will be up soon hopefully!
>
> On Thu, Jul 8, 2021 at 1:41 PM Mark Payne  wrote:
> >
> > Joe,
> >
> > I just filed a BUG Jira [1]. It’s a bit of a corner case, but when it
> occurs it can cause a pretty big problem. Should have a fix up very
> shortly. Will leave it up to you whether or not you think we should get
> this into 1.14.0.
> >
> > Thanks
> > -Mark
> >
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-8771
> >
> >
> > On Jul 8, 2021, at 10:26 AM, Joe Witt  joe.w...@gmail.com>> wrote:
> >
> > Team,
> >
> > The 1.14 RC1 build is underway. Hopefully I will email artifacts
> > today. Reminder this should now generate convenience binaries for
> > nifi, stateless nifi, minifi java, nifi registry and the associated
> > toolkits all in a single release process which also keeps these things
> > in sync.
> >
> > Please do not tag any further fix versions to 1.14.0. Use 1.15.0 from
> > here. I'll pull things in if RCs fail/etc.
> >
> > Thanks
> >
> > On Tue, Jul 6, 2021 at 8:21 AM Joe Witt  joe.w...@gmail.com>> wrote:
> >
> > Team
> >
> > Going to try to pull the rc together today. Havent looked at what
remains
> but it is time. Will look at what is hanging out/mergable and get after
it.
> >
> > Thanks
> >
> > On Thu, Jun 24, 2021 at 8:45 AM Nathan Gough  thena...@gmail.com>> wrote:
> >
> > Joe Gresock just pinged me about an issue that may have been introduced
> by
> > a dependency upgrade I did for lucene:
> > https://issues.apache.org/jira/browse/NIFI-8699 which appears to cause
an
> > issue for existing provenance repositories. I tested the upgrade on a
> fresh
> > install so I didn't notice the issue. There appears to be a way to add
a
> > backwards codec which should allow the new lucene to keep working with
> the
> > existing provenance repo. Looking into reproducing and a fix for it
now.
> >
> > On Wed, Jun 23, 2021 at 9:06 AM Mark Bean  mark.o.b...@gmail.com>> wrote:
> >
> > Putting out one more request for the following open PR's before 1.14
> >
> > https://github.com/apache/nifi/pull/5094
> > https://github.com/apache/nifi/pull/5061
> >
> > Both have been reviewed, but still need attention from a comitter.
> >
> > Thanks!
> > -Mark
> >
> >
> > On Mon, Jun 21, 2021 at 12:17 PM Joe Witt  wrote:
> >
> > Team,
> >
> > I'll start pulling 1.14 together more this week as time permits. As
> > far as specific commits/etc.. please work with reviewers/etc.. to help
> > nail that down. If anything doesn't make it in when I initiate the RC
> > line then we'll get it on the next one. There is a shock

Re: [VOTE] Release Apache NiFi 1.14.0 (rc2)

[Otto Fowler]

 +1 ( non binding )
Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
Maven home: /usr/local/Cellar/maven/3.8.1/libexec
Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime:
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"



From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: July 10, 2021 at 18:40:26
To: dev@nifi.apache.org  
Subject:  [VOTE] Release Apache NiFi 1.14.0 (rc2)

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi 1.14.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1183

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/1.14.0/

Please note that this release now includes the convenience binaries for
Apache NiFi, NiFi toolkit, MiNiFi, MiNiFi Toolkit, Registry, Registry
Toolkit,
and Stateless NiFi.

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.14.0-RC2
The Git commit ID is fcbf1d5f975dd984e34f3a543b9480c779b0dc2f
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=fcbf1d5f975dd984e34f3a543b9480c779b0dc2f

Checksums of nifi-1.14.0-source-release.zip:
SHA256: a96cf75c4f82d01e1c8e0b678d5ff23dec8c26824611c8d37f2ec245e9932b1c
SHA512:
2d23b1a2fae9f545f665c4ee5d9723cdf9c68a62a26d80287b96a55773594e1e80f689ec0f00ba74af92df164c6f4df73ac9b91db7678aaefd69ee8f1eed3f42


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

330+ issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12349644

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.14.0

The vote will be open for at least 72 hours.
Please download the release candidate and evaluate the necessary items
including checking
hashes, signatures, build from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.14.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [discuss] nifi 1.14.0

[Otto Fowler]

 I’ll take care of it.  I’ll make the minimum 3.6.0

From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: July 12, 2021 at 18:25:28
To: dev@nifi.apache.org  
Subject:  Re: [discuss] nifi 1.14.0

Yeah we just need to update the latest required version . We're going
all in on all the HTTPS requirements and such so we're probably really
tight on required version at this point anyway.

I'd not be sinking the RC for this. Just a good JIRA to work on.

Thanks

On Mon, Jul 12, 2021 at 12:59 PM Otto Fowler 
wrote:
>
> NIFI-8778
>
> From: Otto Fowler  
> Reply: Otto Fowler  
> Date: July 12, 2021 at 13:56:36
> To: dev@nifi.apache.org  
> Subject: Re: [discuss] nifi 1.14.0
>
> I see an issue, though I’m not sure if we want to cancel the RC so I’m
> asking before voting.
> The documentation states that we require Apache Maven 3.1.1 or newer, but
> the build fails with the error:
>
> [ERROR] Failed to execute goal
> com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm
> (install-node-and-npm) on project nifi-web-ui: The plugin
> com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version
> 3.6.0 -> [Help 1]
>
> I have maven version:
>
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-17T14:33:14-04:00)
> Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec
> Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime:
> /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
>
> So, the documentation on building nifi is wrong.
>
>
>
> From: Joe Witt  
> Reply: dev@nifi.apache.org  
> Date: July 9, 2021 at 10:24:58
> To: dev@nifi.apache.org  
> Subject: Re: [discuss] nifi 1.14.0
>
> Still working on the 1.14 RC. Has been a series of issues with the
> build. Will be up soon hopefully!
>
> On Thu, Jul 8, 2021 at 1:41 PM Mark Payne  wrote:
> >
> > Joe,
> >
> > I just filed a BUG Jira [1]. It’s a bit of a corner case, but when it
> occurs it can cause a pretty big problem. Should have a fix up very
> shortly. Will leave it up to you whether or not you think we should get
> this into 1.14.0.
> >
> > Thanks
> > -Mark
> >
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-8771
> >
> >
> > On Jul 8, 2021, at 10:26 AM, Joe Witt  joe.w...@gmail.com>> wrote:
> >
> > Team,
> >
> > The 1.14 RC1 build is underway. Hopefully I will email artifacts
> > today. Reminder this should now generate convenience binaries for
> > nifi, stateless nifi, minifi java, nifi registry and the associated
> > toolkits all in a single release process which also keeps these things
> > in sync.
> >
> > Please do not tag any further fix versions to 1.14.0. Use 1.15.0 from
> > here. I'll pull things in if RCs fail/etc.
> >
> > Thanks
> >
> > On Tue, Jul 6, 2021 at 8:21 AM Joe Witt  joe.w...@gmail.com>> wrote:
> >
> > Team
> >
> > Going to try to pull the rc together today. Havent looked at what
remains
> but it is time. Will look at what is hanging out/mergable and get after
it.
> >
> > Thanks
> >
> > On Thu, Jun 24, 2021 at 8:45 AM Nathan Gough  thena...@gmail.com>> wrote:
> >
> > Joe Gresock just pinged me about an issue that may have been introduced
> by
> > a dependency upgrade I did for lucene:
> > https://issues.apache.org/jira/browse/NIFI-8699 which appears to cause
an
> > issue for existing provenance repositories. I tested the upgrade on a
> fresh
> > install so I didn't notice the issue. There appears to be a way to add
a
> > backwards codec which should allow the new lucene to keep working with
> the
> > existing provenance repo. Looking into reproducing and a fix for it
now.
> >
> > On Wed, Jun 23, 2021 at 9:06 AM Mark Bean  mark.o.b...@gmail.com>> wrote:
> >
> > Putting out one more request for the following open PR's before 1.14
> >
> > https://github.com/apache/nifi/pull/5094
> > https://github.com/apache/nifi/pull/5061
> >
> > Both have been reviewed, but still need attention from a comitter.
> >
> > Thanks!
> > -Mark
> >
> >
> > On Mon, Jun 21, 2021 at 12:17 PM Joe Witt  wrote:
> >
> > Team,
> >
> > I'll start pulling 1.14 together more this week as time permits. As
> > far as specific commits/etc.. please work with reviewers/etc.. to help
> > nail that down. If anything doesn't make it in when I initiate the RC
> > line then we'll get it on the next one. There is a shocking amount of
> > goodness in here already.
> >
> > Thanks
> >
> > On Sun, Jun 13, 2021 at 2:27 PM Chris Sampson

Re: [discuss] nifi 1.14.0

[Otto Fowler]

 NIFI-8778

From: Otto Fowler  
Reply: Otto Fowler  
Date: July 12, 2021 at 13:56:36
To: dev@nifi.apache.org  
Subject:  Re: [discuss] nifi 1.14.0

I see an issue, though I’m not sure if we want to cancel the RC so I’m
asking before voting.
The documentation states that we require Apache Maven 3.1.1 or newer, but
the build fails with the error:

[ERROR] Failed to execute goal
com.github.eirslett:frontend-maven-plugin:1.12.0:install-node-and-npm
(install-node-and-npm) on project nifi-web-ui: The plugin
com.github.eirslett:frontend-maven-plugin:1.12.0 requires Maven version
3.6.0 -> [Help 1]

I have maven version:

Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
2018-06-17T14:33:14-04:00)
Maven home: /usr/local/Cellar/maven@3.5/3.5.4_1/libexec
Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime:
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre

So, the documentation on building nifi is wrong.



From: Joe Witt  
Reply: dev@nifi.apache.org  
Date: July 9, 2021 at 10:24:58
To: dev@nifi.apache.org  
Subject:  Re: [discuss] nifi 1.14.0

Still working on the 1.14 RC. Has been a series of issues with the
build. Will be up soon hopefully!

On Thu, Jul 8, 2021 at 1:41 PM Mark Payne  wrote:
>
> Joe,
>
> I just filed a BUG Jira [1]. It’s a bit of a corner case, but when it
occurs it can cause a pretty big problem. Should have a fix up very
shortly. Will leave it up to you whether or not you think we should get
this into 1.14.0.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-8771
>
>
> On Jul 8, 2021, at 10:26 AM, Joe Witt > wrote:
>
> Team,
>
> The 1.14 RC1 build is underway. Hopefully I will email artifacts
> today. Reminder this should now generate convenience binaries for
> nifi, stateless nifi, minifi java, nifi registry and the associated
> toolkits all in a single release process which also keeps these things
> in sync.
>
> Please do not tag any further fix versions to 1.14.0. Use 1.15.0 from
> here. I'll pull things in if RCs fail/etc.
>
> Thanks
>
> On Tue, Jul 6, 2021 at 8:21 AM Joe Witt > wrote:
>
> Team
>
> Going to try to pull the rc together today. Havent looked at what remains
but it is time. Will look at what is hanging out/mergable and get after it.
>
> Thanks
>
> On Thu, Jun 24, 2021 at 8:45 AM Nathan Gough > wrote:
>
> Joe Gresock just pinged me about an issue that may have been introduced
by
> a dependency upgrade I did for lucene:
> https://issues.apache.org/jira/browse/NIFI-8699 which appears to cause an
> issue for existing provenance repositories. I tested the upgrade on a
fresh
> install so I didn't notice the issue. There appears to be a way to add a
> backwards codec which should allow the new lucene to keep working with
the
> existing provenance repo. Looking into reproducing and a fix for it now.
>
> On Wed, Jun 23, 2021 at 9:06 AM Mark Bean > wrote:
>
> Putting out one more request for the following open PR's before 1.14
>
> https://github.com/apache/nifi/pull/5094
> https://github.com/apache/nifi/pull/5061
>
> Both have been reviewed, but still need attention from a comitter.
>
> Thanks!
> -Mark
>
>
> On Mon, Jun 21, 2021 at 12:17 PM Joe Witt  wrote:
>
> Team,
>
> I'll start pulling 1.14 together more this week as time permits. As
> far as specific commits/etc.. please work with reviewers/etc.. to help
> nail that down. If anything doesn't make it in when I initiate the RC
> line then we'll get it on the next one. There is a shocking amount of
> goodness in here already.
>
> Thanks
>
> On Sun, Jun 13, 2021 at 2:27 PM Chris Sampson
>  wrote:
>
> Joe,
>
> Yeah I thought it would be something like that (but didn't spend time
> looking at the moment, just thought I'd highlight the thread). Don't
> know
> whether there's anything to consider adding to/clarifying in the
> documentation in order to highlight that to (first time) users?
>
> Again, I figured this would probably be "as designed" and I've not
> spent
> the time reading the docs for this new default behaviour - so long as
> it
> should be clear to first time users (provided they read the docs), then
> all
> good.
>
>
> Cheers,
>
> Chris Sampson
>
> On Sun, 13 Jun 2021, 21:42 Joe Witt,  wrote:
>
> Chris
>
> I responded to the slack thread. Pretty sure it is doing exactly what
> is expected. We are not offering a user management and policy
> authoring experience for that. It is quite literally 'single user
> auth' and in that mode this single user we generate has all the
> authorizations. This is functionally equivalent to how it was with
> an
> unsecured instance with what is basically 'anonymous' user except in
> this case it is TLS

Re: [discuss] nifi 1.14.0

[Otto Fowler]

existing authentication and authorization plugin options.
>
> Thanks
>
> On Sun, Jun 13, 2021 at 11:26 AM Chris Sampson
>  wrote:
>
> FYI, there's a new thread in slack about the new
> single-user-authoriser
> setup - user has https but no users/policy screen for setting up
> AuthZ.
>
> Might be worth someone taking a look before an RC to see whether
> there's
> documentation (or functionality) that needs clarifying.
>
>
> Cheers,
>
> Chris Sampson
>
> On Sun, 13 Jun 2021, 13:57 Mark Bean, 
> wrote:
>
> There are three open PR's I would appreciate some eyes on before
> the RC
> process is kicked off. Two of the three have been reviewed, but
> not
> yet by
> a committer.
>
> https://github.com/apache/nifi/pull/5094
> https://github.com/apache/nifi/pull/5061
> https://github.com/apache/nifi/pull/5064
>
> Thanks in advance!
> -Mark
>
> On Fri, Jun 11, 2021 at 4:05 PM Joe Witt 
> wrote:
>
> So. Dang. Cool. I just built from latest main and poof - I'm
> on
> https
> with username/password.
>
> Will start whipping up the process for an RC. Probably will
> be a
> little slow going with dayjob factors but will get on it.
>
> Thanks
>
> On Fri, Jun 11, 2021 at 12:14 PM David Handermann
>  wrote:
>
> Thanks to Mark Payne, NIFI-8516 is now merged, so that covers
> current
> open
> issues around securing the default configuration.
>
> Regards,
> David Handermann
>
> On Fri, Jun 11, 2021 at 11:55 AM David Handermann <
> exceptionfact...@apache.org> wrote:
>
> Joe,
>
> Thanks for following up. The PR for NIFI-8516 has gone
> through
> several
> rounds of feedback, I believe it is about ready to go,
> pending
> confirmation
> that the ability to set custom credentials addresses the
> ease of
> use
> concern.
>
> Regards,
> David Handermann
>
> On Fri, Jun 11, 2021 at 11:41 AM Joe Witt <
> joe.w...@gmail.com>
> wrote:
>
> David,
>
> Ok thanks - do you have a sense of when what you see as
> good
> 1.14
> specific work will be merged? Do you have the
> reviewers/engagement
> you need?
>
> This 1.14 is already pretty packed but definitely agree we
> need
> to
> make real progress on secure by default and this release
> is
> a
> great
> time to take the first big step.
>
> Thanks
>
> On Mon, May 31, 2021 at 5:52 AM David Handermann
>  wrote:
>
> Thanks for kicking off the discussion Joe!
>
> Of the many items that could be included in the next
> release,
> securing
> the
> default configuration as described in NIFI-8220
> <https://issues.apache.org/jira/browse/NIFI-8220> would
> be
> great
> to
> have
> completed. Most of the elements are in place, and the
> current
> Pull
> Request
> for NIFI-8516 <https://github.com/apache/nifi/pull/5068
>
> is
> under
> review.
> If there are any other achievable items that should be
> included as
> part
> of
> a secure default installation for NiFi, it would be
> helpful
> to add
> sub-tasks to NIFI-8220. The current scope is limited
> to a
> standalone
> installation, so issues regarding clustered deployments
> can be
> handled
> separately. If others are interested in evaluating the
> proposed
> new
> default configuration that requires HTTPS and leverages
> a
> generated
> username and password, feel free to provide feedback on
> NIFI-8516.
>
> Regards,
> David Handermann
>
> On Thu, May 27, 2021 at 6:51 PM Otto Fowler <
> ottobackwa...@gmail.com>
> wrote:
>
> I think NIFI-8625 and NIFI-8461 need to be understood
> and
> addressed.
>
>
> On May 27, 2021, at 13:29, Joe Witt <
> joe.w...@gmail.com>
> wrote:
>
> Team,
>
> There has been a tremendous amount of work already
> on
> the
> 1.14
> line
> as
> shown:
>
>
> https://issues.apache.org/jira/projects/NIFI/versions/12349644
>
> These include merging the nifi registry and minifi
> java
> into
> the
> nifi
> line itself. So when we release these things stay
> in
> sync and
> maintained. The release will now produce things
> like
> Apache
> NiFi,
> the
> Apache NiFi toolkit, Apache NiFi Registry, and
> Apache
> NiFi
> MiNiFi
> Java
> and the Apache NiFi stateless runtime as well.
> There
> have
> been
> many
> improvements to core nifi and stateless nifi now
> meaning
> we
> have the
> traditional execution form factor and this new
> stateless
> mode.
> We
> can
> now hot load nars from HDFS storage locations which
> could
> mean
> HDFS,
> blob storage in the cloud, etc.. There is a lot
> more.
>
> Anyway, I wanted to start circling the wagons for a
> 1.14
> release.
> I'm
> happy to take on RM duties especially since there
> will be
> new
> elements
> to the release process.
>
> Thanks
>
>
>
>
>
>
>
>
>
>

Re: [ANNOUNCE] New NiFi PMC Member Marton Szasz

[Otto Fowler]

Congratulations!




From: Arpad Boda  
Reply: dev@nifi.apache.org  
Date: June 24, 2021 at 15:01:31
To: NiFi Developers List  
Subject:  [ANNOUNCE] New NiFi PMC Member Marton Szasz

On behalf of the Apache NiFI PMC, I am very pleased to announce that Marton
has accepted the PMC's invitation to join the Apache NiFi PMC. We greatly
appreciate the PR reviews and code contributions of Marton.

Please join us in congratulating and welcoming Marton to the Apache NiFi
PMC.

Congratulations Marton!

Re: [ANNOUNCE] New Apache NiFi Committer Denes Arvay

[Otto Fowler]

Congratulations!




From: Arpad Boda  
Reply: dev@nifi.apache.org  
Date: June 24, 2021 at 15:01:09
To: NiFi Developers List  
Subject:  [ANNOUNCE] New Apache NiFi Committer Denes Arvay

On behalf of the Apache NiFI PMC, I am very pleased to announce that
Denes has accepted the PMC's invitation to become a committer on the
Apache NiFi project. We greatly appreciate all of Denes' hard work and
generous contributions to the project and look forward to the
continued involvement.

Welcome and congratulations!

Re: [discuss] nifi 1.14.0

[Otto Fowler]

I think NIFI-8625 and NIFI-8461 need to be understood and addressed.


> On May 27, 2021, at 13:29, Joe Witt  wrote:
> 
> Team,
> 
> There has been a tremendous amount of work already on the 1.14 line as shown:
> 
> https://issues.apache.org/jira/projects/NIFI/versions/12349644
> 
> These include merging the nifi registry and minifi java into the nifi
> line itself.  So when we release these things stay in sync and
> maintained.  The release will now produce things like Apache NiFi, the
> Apache NiFi toolkit, Apache NiFi Registry, and Apache NiFi MiNiFi Java
> and the Apache NiFi stateless runtime as well.  There have been many
> improvements to core nifi and stateless nifi now meaning we have the
> traditional execution form factor and this new stateless mode.  We can
> now hot load nars from HDFS storage locations which could mean HDFS,
> blob storage in the cloud, etc..  There is a lot more.
> 
> Anyway, I wanted to start circling the wagons for a 1.14 release.  I'm
> happy to take on RM duties especially since there will be new elements
> to the release process.
> 
> Thanks

Re: [ANNOUNCE] New Apache NiFi Committer Chris Sampson

[Otto Fowler]

Awesome!  Congratulations!

> On May 13, 2021, at 16:25, Joe Witt  wrote:
> 
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Chris has accepted the PMC's invitation to become a committer on the
> Apache NiFi project. We greatly appreciate the contributions from Chris
> on the mailing list, PR reviews, code contributions, and especially rich
> communications in the Apache NiFi slack channels.
> 
> Welcome and congratulations!

Re: MiNiFi Java -> NiFi

[Otto Fowler]

Awesome Job Matt!

> On Apr 28, 2021, at 14:08, Matt Burgess  wrote:
> 
> All,
> 
> Now that [1] is merged, MiNiFi Java is now officially in the NiFi
> codebase! Thanks to all who participated in discussions, reviews, and
> testing. Here are some notes on what this means going forward:
> 
> - All future releases of MiNiFi Java will be in lockstep with NiFi,
> the next release version will be 1.14.0 rather than 0.6.0. MiNiFi Java
> artifacts will be built and published as part of the NiFi release
> process. This will allow MiNiFi to immediately benefit from
> improvements/fixes in NiFi components instead of occasionally updating
> the NiFi dependency version in MiNiFi Java.
> 
> - Although we will continue to track issues in Jira under the MINIFI
> project, PRs should be submitted against the nifi repo's main branch.
> If you have an open PR against the nifi-minifi repo, you can port it
> to the nifi repo by adding ".patch" to the URL of your PR, downloading
> the patch, and applying it to a branch in your fork of the nifi repo,
> for example:
> 
> git checkout -b MINIFI-541
> git apply --directory='minifi' ~/195.patch
> 
> Sooner than later we'll make the nifi-minifi repo read-only so no more
> PRs can be submitted against it, and we can update NiFi's PR template
> to refer to MINIFI Jira cases as well.
> 
> There's still work to be done of course, contributions and PR reviews
> are most welcome!
> 
> Regards,
> Matt
> 
> [1] https://github.com/apache/nifi/pull/4933

Apache Project issues with available GitHub Action resources across projects

[Otto Fowler]

As you can probably imagine, it takes a lot of resources in order to CI for all 
the Apache projects.  Periodically this becomes an issue, as the donated 
resources from cloud CI providers ( Travis and now GitHub Actions ) end up 
queuing and delaying builds across Apache projects because of larger projects 
and their requirements.

The discussions center around a few common themes:

- the CI requirements of large complex projects dominate the Apache Queue
- how those projects can supplement their builds in a way acceptable to ASF 
INFRA
- how those projects can have per project metering such that a project can pay 
for hours over the ’norm’
- how to optimize projects 

This issue is currently being discussed again on the @builds apache list.  I’m 
sending this over to the Nifi Dev community because Nifi itself has been 
mentioned as one of the top users of GitHub action resources by some measure.   
Many of the other projects are actively taking measures to decrease or optimize 
their usage, and we should probably think about how we can do so as well.

Here is the *current* thread:
https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3cCAMFhwAbyoDPM8V7bt6=40m++GTdbXK64Q5LjwD=cvrghs7d...@mail.gmail.com%3e
 


Here is the message where 13 projects are listed
https://mail-archives.apache.org/mod_mbox/www-builds/202104.mbox/%3c86d8e65a-4943-cbf1-ec46-3980128b8...@apache.org%3e
 


There are many other threads regarding GitHub Action limits and resources if 
you start scrolling back through the archives.

I’m posting this to hopefully kick off some discussions.

Re: Stale PR automatic closure

[Otto Fowler]

https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features
What I mean is, beyond the GitHub settings it has, if it offered PR pruning
( if that were possible ).
Seems silly that everyone has to roll their own.

On Thu, Mar 25, 2021 at 3:05 PM Joe Witt  wrote:

> Otto
>
> I'm not clear on what you mean.  But here we're talking about our
> communities usage of Github in particular as a convenient vehicle to
> support contributions.  The ASF/Git (gitbox) is a different animal.
>
> Thanks
>
> On Thu, Mar 25, 2021 at 12:02 PM Otto Fowler 
> wrote:
> >
> > What would be nice is if this could be part of the ASF official .yml for
> git
> >
> > On Thu, Mar 25, 2021 at 2:42 PM Joe Witt  wrote:
> >
> > > Team,
> > >
> > > We've discussed it in various forums but I want to formally move to
> > > taking action on our PR situation.  As of a couple hours ago we had
> > > 280 or so open PRs.  Many many many of these are quite old (two or
> > > more years).  These aren't just abandoned PRs because they were not
> > > good enough to be clear.  There are actually a lot of good and
> > > interesting things in here.  I think though we simply cannot keep up
> > > with the PRs that come in.  The most difficult appear to be less about
> > > fixing bugs and more on adding features.  Anyway, we clearly get a lot
> > > of PRs and we clearly close a lot of PRs but many remain unaddressed.
> > > Other communities such as Apache Spark [1],[2] have provided an auto
> > > close mechanism.  It is intended to keep the PR queue tidy and not
> > > just go endlessly without a response.
> > >
> > > I am planning to implement the same with a 180 day auto-stale/closure
> > > using the same github/ci workflow [3] based on github action [4].  You
> > > can see the results of this here [5].
> > >
> > > If someone disagrees please share a workable alternative that doesn't
> > > leave the PRs or those that submitted them hanging indefinitely which
> > > you are willing to implement.
> > >
> > > [1]
> > >
> http://apache-spark-developers-list.1001551.n3.nabble.com/Handling-stale-PRs-td8015i20.html#a9684
> > > [2]
> > >
> http://apache-spark-developers-list.1001551.n3.nabble.com/Closing-stale-PRs-with-a-GitHub-Action-td28477.html
> > > [3]
> > >
> https://github.com/apache/spark/blob/master/.github/workflows/stale.yml
> > > [4] https://github.com/marketplace/actions/close-stale-issues
> > > [5]
> > >
> https://github.com/apache/spark/pulls?q=is%3Apr+is%3Aclosed+label%3AStale
> > >
> > > Thanks
> > > Joe
> > >
>

Re: Stale PR automatic closure

[Otto Fowler]

What would be nice is if this could be part of the ASF official .yml for git

On Thu, Mar 25, 2021 at 2:42 PM Joe Witt  wrote:

> Team,
>
> We've discussed it in various forums but I want to formally move to
> taking action on our PR situation.  As of a couple hours ago we had
> 280 or so open PRs.  Many many many of these are quite old (two or
> more years).  These aren't just abandoned PRs because they were not
> good enough to be clear.  There are actually a lot of good and
> interesting things in here.  I think though we simply cannot keep up
> with the PRs that come in.  The most difficult appear to be less about
> fixing bugs and more on adding features.  Anyway, we clearly get a lot
> of PRs and we clearly close a lot of PRs but many remain unaddressed.
> Other communities such as Apache Spark [1],[2] have provided an auto
> close mechanism.  It is intended to keep the PR queue tidy and not
> just go endlessly without a response.
>
> I am planning to implement the same with a 180 day auto-stale/closure
> using the same github/ci workflow [3] based on github action [4].  You
> can see the results of this here [5].
>
> If someone disagrees please share a workable alternative that doesn't
> leave the PRs or those that submitted them hanging indefinitely which
> you are willing to implement.
>
> [1]
> http://apache-spark-developers-list.1001551.n3.nabble.com/Handling-stale-PRs-td8015i20.html#a9684
> [2]
> http://apache-spark-developers-list.1001551.n3.nabble.com/Closing-stale-PRs-with-a-GitHub-Action-td28477.html
> [3]
> https://github.com/apache/spark/blob/master/.github/workflows/stale.yml
> [4] https://github.com/marketplace/actions/close-stale-issues
> [5]
> https://github.com/apache/spark/pulls?q=is%3Apr+is%3Aclosed+label%3AStale
>
> Thanks
> Joe
>

Re: [ANNOUNCE] New NiFi PMC Member Joey Frazee

[Otto Fowler]

Congratulations!

On Thu, Mar 25, 2021 at 2:54 PM Joe Witt  wrote:

> NiFi Community,
>
> On behalf of the Apache NiFi PMC, I am pleased to announce that Joey Frazee
> has accepted the PMC's invitation to join the Apache NiFi PMC.
>
> Joey has been a contributor and committer of Apache NiFi for several
> years and has been involved across the whole spectrum of code reviews,
> JIRAs, code contributions, maintained a great 'awesome nifi' github
> site, and more.
>
> Please join us in congratulating and welcoming Joey to the Apache NiFi PMC.
>
> Congratulations Joey!
>

Re: [EXT] [ANNOUNCE] New Apache NiFi Committer Otto Fowler

[Otto Fowler]

Thanks everyone!

On Mon, Mar 22, 2021 at 2:39 PM Mark Bean  wrote:

> Congrats Otto!
>
> -Mark
>
> On Mon, Mar 22, 2021 at 2:28 PM Peter Wicks (pwicks) 
> wrote:
>
> > Micron Confidential
> >
> > Congratulations Otto!
> >
> > From: Joe Witt 
> > Date: Monday, March 22, 2021 at 12:16 PM
> > To: dev@nifi.apache.org 
> > Subject: [EXT] [ANNOUNCE] New Apache NiFi Committer Otto Fowler
> > CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless
> you
> > recognize the sender and were expecting this message.
> >
> >
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> > Otto has accepted the PMC's invitation to become a committer on the
> > Apache NiFi project. We greatly appreciate all of Otto's hard work and
> > generous contributions to the project and look forward to the
> > continued involvement.
> >
> > Otto has been engaged for several years and is always helping with
> > release votes, takes on some of the more corner-casey (that is a word)
> > PRs/JIRAs, shares perspective, and is always willing to share thoughts
> > with people in the mailing list or slack which is vital to community
> > growth.
> >
> > Welcome and congratulations!
> >
> >
> > Micron Confidential
> >
>

Re: [VOTE] Release Apache NiFi 1.13.2

[Otto Fowler]

+1
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

> On Mar 18, 2021, at 01:29, Joe Witt  wrote:
> 
> Hello,
> 
> Please note this has a shorter than normal vote period as this
> corrects an important regression introduced in 1.13.1 and is purely a
> focused bug fix release.
> 
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.13.2.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1180
> 
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.2/
> 
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> 
> The Git tag is nifi-1.13.2-RC1
> The Git commit ID is 174938e5e3bfe36951d5607d0d53e78604b0e07b
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=174938e5e3bfe36951d5607d0d53e78604b0e07b
> 
> Checksums of nifi-1.13.2-source-release.zip:
> SHA256: db18f57b4629367f21f70ec9a332294ec6b6c090a661c974b0fa7a80b09a79be
> SHA512: 
> 279c5701fd7b6d76206f5e79bdf1340d432a8e4834b8e786559095781c77c96a4594e1da040c6b404d1bab615acf5aa9dc60d289e460e6d7261f454f6210c66b
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 5 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350008
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.2
> 
> The vote will be open for 36 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
> Then please vote:
> 
> [ ] +1 Release this package as nifi-1.13.2
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [RESULT][VOTE] Release Apache NiFi 1.13.1

[Otto Fowler]

Awesome job, thanks Joe!

> On Mar 15, 2021, at 11:11, Joe Witt  wrote:
> 
> Apache NiFi Community,
> 
> I am pleased to announce that the 1.13.1 release of Apache NiFi passes with
>8 +1 (binding) votes
>8 +1 (non-binding) votes
>0 0 votes
>0 -1 votes
> 
> Thanks to all who helped make this release possible. Several RCs many of
> you stuck with and some new voters.  Great stuff!
> 
> Here is the PMC vote thread:
> https://lists.apache.org/thread.html/rb92304f1e60918f18b41f1b9e1685f67c21e2d155512249228efed10%40%3Cdev.nifi.apache.org%3E
> 
> 
> On Mon, Mar 15, 2021 at 8:07 AM Joe Witt  wrote:
> 
>> +1 binding
>> 
>> On Sun, Mar 14, 2021 at 7:45 PM Kevin Doran  wrote:
>> 
>>> +1 (binding)
>>> 
>>> Verified hashes and sigs.
>>> Built and ran on Mac with simple test flow.
>>> Built docker image from the resulting binary and tested integration with
>>> NiFi Registry docker image.
>>> All working as expected - nice work everyone!
>>> 
>>> -Kevin
>>> 
 On Mar 11, 2021, at 11:29, Joe Witt  wrote:
 
 Hello,
 
 I am pleased to be calling this vote for the source release of Apache
>>> NiFi
 1.13.1.
 
 The source zip, including signatures, digests, etc. can be found at:
 https://repository.apache.org/content/repositories/orgapachenifi-1179
 
 The source being voted upon and the convenience binaries can be found
>>> at:
 https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.1/
 
 A helpful reminder on how the release candidate verification process
>>> works:
 
>>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
 
 The Git tag is nifi-1.13.1-RC1
 The Git commit ID is acbc217cb7002d7489421f4d346b995a43b6ea01
 
>>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=acbc217cb7002d7489421f4d346b995a43b6ea01
 
 Checksums of nifi-1.13.1-source-release.zip:
 SHA256: 0a397df640e579720e26699e38a3738c5be05af01aad8aaeefc04eb58591faac
 SHA512:
 
>>> 7f8df759d4345ccd6e75c169bd0aab1b7f4f64bf5a8b11b45bc1d7c8163acd0035922dcdbef232392279f4ea0710d4a97c55d480281bfe1d50b6401295633d48
 
 Release artifacts are signed with the following key:
 https://people.apache.org/keys/committer/joewitt.asc
 
 KEYS file available here:
 https://dist.apache.org/repos/dist/release/nifi/KEYS
 
 48 issues were closed/resolved for this release:
 
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
 
 Release note highlights can be found here:
 
>>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1
 
 The vote will be open for 72 hours.
 Please download the release candidate and evaluate the necessary items
 including checking hashes, signatures, build from source, and test. Then
 please vote:
 
 [ ] +1 Release this package as nifi-1.13.1
 [ ] +0 no opinion
 [ ] -1 Do not release this package because...
>>> 
>>> 

Re: [VOTE] Release Apache NiFi 1.13.1

[Otto Fowler]

+1

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"


> On Mar 11, 2021, at 11:29, Joe Witt  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.13.1.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1179
> 
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.1/
> 
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> 
> The Git tag is nifi-1.13.1-RC1
> The Git commit ID is acbc217cb7002d7489421f4d346b995a43b6ea01
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=acbc217cb7002d7489421f4d346b995a43b6ea01
> 
> Checksums of nifi-1.13.1-source-release.zip:
> SHA256: 0a397df640e579720e26699e38a3738c5be05af01aad8aaeefc04eb58591faac
> SHA512:
> 7f8df759d4345ccd6e75c169bd0aab1b7f4f64bf5a8b11b45bc1d7c8163acd0035922dcdbef232392279f4ea0710d4a97c55d480281bfe1d50b6401295633d48
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 48 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
> 
> [ ] +1 Release this package as nifi-1.13.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [RESULT][VOTE] Release Apache NiFi 1.13.0

[Otto Fowler]

Great job everyone, thanks Joe!

> On Feb 15, 2021, at 11:48, Joe Witt  wrote:
> 
> Apache NiFi Community,
> 
> I am pleased to announce that the 1.13.0 release of Apache NiFi passes with
>6 +1 (binding) votes
>7 +1 (non-binding) votes
>0 0 votes
>0 -1 votes
> 
> Thanks to all who helped make this release possible. Several RCs many of
> you stuck with and some new voters.  Great stuff!
> 
> Here is the PMC vote thread:
> https://lists.apache.org/thread.html/r7f8e56e60a9f2da7eb09da7ba696142a81c9dd19e374fca25a536cd1%40%3Cdev.nifi.apache.org%3E
> 
> On Mon, Feb 15, 2021 at 9:43 AM Joe Witt  wrote:
> 
>> +1 binding
>> 
>> On Sun, Feb 14, 2021 at 3:26 AM Pierre Villard <
>> pierre.villard...@gmail.com> wrote:
>> 
>>> +1 binding
>>> 
>>> Went through the usual verification process. Deployed secured cluster on
>>> Google Cloud and checked integration with the NiFi Registry.
>>> 
>>> Thanks,
>>> Pierre
>>> 
>>> Le dim. 14 févr. 2021 à 04:45, Peter Turcsanyi  a
>>> écrit :
>>> 
 +1 non-binding
 
 Went through the release helper guide.
 Verified full build with empty local maven repo with Java 8
>>> (AdoptOpenJDK
 1.8.0_282-b08) and 11 (AdoptOpenJDK 11.0.10+9) on Ubuntu 20.04.
 Ran full builds with different time zone settings to verify NIFI-8023,
>>> also
 tested it with flows.
 Ran some SSL related test flows with processors changed recently
 (ListenHTTP, GRPC, MQTT).
 Verified --wait-for-init on Ubuntu and CentOS.
 Verified NiFi web server listening on localhost only by default.
 
 Thanks,
 Peter
 
 On Sun, Feb 14, 2021 at 12:56 AM David Handermann <
 exceptionfact...@gmail.com> wrote:
 
> +1 non-binding
> 
> Verified release signatures and expected files.
> Verified build on Ubuntu 20.10 using Apache Maven 3.6.3 with Azul Zulu
 JDK
> 11.0.10 and AdoptOpenJDK 1.8.0_282.
> Verified absence of startup script shell warnings on Ubuntu 20.10.
> Verified service listening on localhost when using default
 nifi.properties.
> Configured and tested mutual TLS authentication on a standalone server
 with
> BCFKS key store and trust store.
> 
> Regards,
> David Handermann
> 
> On Fri, Feb 12, 2021 at 5:45 PM Muazma Zahid 
>>> wrote:
> 
>> +1 (non-binding)
>> 
>> - Ran build with OpenJDK 1.8.0_275 on Linux
>> - Deployed cluster on Azure and tested flows with Blob, ADLS, and
> CosmosDB
>> processors.
>> 
>> Looks good to me.
>> 
>> Thanks
>> Muazma
>> 
>> On Fri, Feb 12, 2021 at 3:38 PM Sushil Kumar 
 wrote:
>> 
>>> +1 (non-binding) Release this package as nifi-1.13.0
>>> 
>>> Deployed this via helm chart(
>>> https://github.com/sushilkm/nifi-chart)
> on
>>> kubernetes.
>>> Thank you to all the contributors and reviewers.
>>> 
>>> On Fri, Feb 12, 2021 at 2:13 PM Marc Parisi 
> wrote:
>>> 
 +1
 - Verified sigs and hashes
 - Built on PopOS w/ java 8 && 11
 - Tested with basic flow sending data to various devices w/
>>> Secured
 transport
 - Tested secured w/ secured nifi reg.
 
 Thanks,
 Marc
 
 On Fri, Feb 12, 2021 at 2:56 PM Andrew Lim <
> andrewlim.apa...@gmail.com
>>> 
 wrote:
 
> +1 (binding)
> 
> -Tested secure NiFi with secure NiFi Registry 0.8.0
> -Ran basic flows successfully
> -Tested basic versioned flow functionality
> -Reviewed and tested Core UI fixes and Documentation updates
> 
> Drew
> 
>> On Feb 10, 2021, at 11:37 PM, Joe Witt 
> wrote:
>> 
>> Hello,
>> 
>> I am pleased to be calling this vote for the source release
>>> of
>> Apache
> NiFi
>> 1.13.0.
>> 
>> The source zip, including signatures, digests, etc. can be
 found
>> at:
>> 
>>> 
 https://repository.apache.org/content/repositories/orgapachenifi-1178
>> 
>> The source being voted upon and the convenience binaries
>>> can be
>> found
 at:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
>> 
>> A helpful reminder on how the release candidate verification
>> process
> works:
>> 
> 
 
>>> 
>> 
> 
 
>>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>> 
>> The Git tag is nifi-1.13.0-RC4
>> The Git commit ID is
>>> 3bc6a122091214b33eee17a270163d7ca26e2a0c
>> 
> 
 
>>> 
>> 
> 
 
>>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=3bc6a122091214b33eee17a270163d7ca26e2a0c
>> 
>> Checksums of nifi-1.13.0-source-release.zip:
>> SHA256:
 

Re: [VOTE] Release Apache NiFi 1.13.0 (rc4)

[Otto Fowler]

+1

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

> On Feb 10, 2021, at 23:37, Joe Witt  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.13.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1178
> 
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
> 
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> 
> The Git tag is nifi-1.13.0-RC4
> The Git commit ID is 3bc6a122091214b33eee17a270163d7ca26e2a0c
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=3bc6a122091214b33eee17a270163d7ca26e2a0c
> 
> Checksums of nifi-1.13.0-source-release.zip:
> SHA256: 95efe5db38e973c9f4062a7b2c95fdc5dad31d7c5e1d36ce1776b9b227c89b9c
> SHA512:
> d7dd9e851341ebd605784142a7861935f6f814bc612499013456a15713bc9119e426df8f26445c260bdb25cbfc21822cf0d44985bf372a065c9d8597953a3c4a
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 260 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
> 
> [ ] +1 Release this package as nifi-1.13.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [discuss] we need to enable secure by default...

[Otto Fowler]

Aren’t most of these applications installed by an installer?
Maybe we can have one or more installers that setup a secure instance, and 
those installers
could be the *production* nifi, and keep the zip distribution open for 
developers?


> On Feb 10, 2021, at 10:04, David Handermann  
> wrote:
> 
> I agree that a generated password is the way to go, and the challenge is
> making it available to the user.  Depending on how it is stored for the
> identity provider, having a command line tool to read and print it could be
> a reasonable option.
> 
> Although this particular thread referenced a specific Twitter post, this
> general discussion is more of a reflection of the need to make things more
> secure by default as other products have followed similar approaches.
> 
> Regards,
> David Handermann
> 
> On Wed, Feb 10, 2021 at 8:53 AM Kevin Doran  wrote:
> 
>> I am in favor of requiring some authentication by default.
>> 
>> I’m in favor of an admin username and generated password. (It sounds li9ke
>> most of us are on the same page, but I don’t think a static, default
>> password buys us much against the types of abuse shown on the twitter
>> thread Joe shared.)
>> 
>> We need some way of making the generated password discoverable… Print to
>> the logs on first boot? Not great but are there other mechanisms? A setup
>> CLI utility?
>> 
>> To help not impede automated deployments, maybe we should change the
>> startup flow such that there is a way to provide this password. That would
>> be better than people just disabling whatever default authentication we set.
>> 
>> 
>>> On Feb 10, 2021, at 09:45, David Handermann 
>> wrote:
>>> 
>>> Mark,
>>> 
>>> Thanks for clarifying, that makes sense.
>>> 
>>> Regards,
>>> David Handermann
>>> 
>>> On Wed, Feb 10, 2021 at 8:41 AM Mark Payne  wrote:
>>> 
 David,
 
 My concern was purely around generating client certs and using mutual
>> TLS.
 I definitely think we should have a server cert if using username &
 password.
 
 Thanks
 -Mark
 
 
> On Feb 10, 2021, at 9:37 AM, David Handermann <
 exceptionfact...@gmail.com> wrote:
> 
> Mark,
> 
> Thanks for your perspective on certificate generation, I agree that
> troubleshooting certificates can be confusing due to unclear error
> messaging.  Generating self-signed certificates for one-way TLS still
> requires the user to accept the certificate presented, but at least
>> that
 is
> more common in various products.  Are you concerned about that
>> situation,
> or are you more concerned about the additional challenges of mutual TLS
> authentication?
> 
> Generating a random password in absence of any other configuration
>> would
> certainly be easier from a new user perspective.  Some of the security
> concerns with password authentication can be mitigated with one-way
>> TLS,
 so
> a blending of these approaches, as Joe describes in NIFI-8220, seems
 like a
> good way to go.
> 
> Regards,
> David Handermann
> 
> On Wed, Feb 10, 2021 at 8:23 AM Mark Payne 
>> wrote:
> 
>> I would be in favor of this as well. I agree that there is no need to
 wait
>> for a 2.0 version - there is no inherit backward compatibility
>> challenge
>> here.
>> Requiring that new application configs be set, etc. I think is
 completely
>> fair game between minor versions.
>> 
>> Personally, though, I would very much stray away from auto-generating
>> certificates. For those of us who have dealt with certificates
 significantly
>> In the past, minor configuration issues are easy to address. But for
>> someone who hasn’t spent much time dealing with certificates, the
>> error
>> messages
>> that are often intentionally vague can quickly result in users being
>> overwhelmed. This is especially true if browser configurations are
 already
>> setup to
>> automatically offer certificates that area already installed - this
>> can
>> result in weird error messages about untrusted certificates when the
 user
>> thinks
>> they haven’t even provided a certificate, etc. It gets really hairy.
>> 
>> I am more in favor of a default username/password personally. It would
>> require implementing a new auth provider. And it’s one that
 historically we
>> have
>> avoided implementing because a basic auth type of approach is less
 secure
>> than mutual auth, ldap, etc. But it’s more secure than nothing.
>> 
>> Thanks
>> -Mark
>> 
>> 
>>> On Feb 10, 2021, at 9:13 AM, Joe Witt  wrote:
>>> 
>>> There are a range of things to consider.  And yes whatever we do will
>>> involve writing code.  We also have to find the right place for the
>> bar
>>> here.
>>> 
>>> 1. Disable HTTP by default.  And if they want to enable HTTP they
 should
>>> also have to 

Re: [VOTE] Release Apache NiFi 1.13.0 (rc3)

[Otto Fowler]

+1

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"


> On Feb 7, 2021, at 01:09, Joe Witt  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.13.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1177
> 
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
> 
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> 
> The Git tag is nifi-1.13.0-RC3
> The Git commit ID is 547127c07aa9da2698b8b6f65e990b986065ef18
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=547127c07aa9da2698b8b6f65e990b986065ef18
> 
> Checksums of nifi-1.13.0-source-release.zip:
> SHA256: 670f6c97a5a2ea488b3006cd6267fa9f02aeeb0d5d011b02f43f76c51ee787bc
> SHA512:
> a9ed5d8b5380bedbb18c94c22ba577901392141ba5f07c0c812e548d88df76a365d99ab7cb890d1803acb2e0f0215c1d6ed9dc4704af1da7e0854ef96fa649ad
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 253 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
> 
> [ ] +1 Release this package as nifi-1.13.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.13.0 (rc2)

[Otto Fowler]

+1
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"

Thanks Joe!


> On Feb 1, 2021, at 20:10, Joe Witt  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.13.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1176
> 
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
> 
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> 
> The Git tag is nifi-1.13.0-RC2
> The Git commit ID is c27e59fc679a2e982102a75b8b8df2b0f062af23
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=c27e59fc679a2e982102a75b8b8df2b0f062af23
> 
> Checksums of nifi-1.13.0-source-release.zip:
> SHA256: 4913dd3d943afac710d1a277bf31beebf7a6207a20e1148849d69511f44fc97b
> SHA512:
> dc9935f0eb8692cd8493f5863bc8ae2ef0b52653fa69ff8b9a7e8db7dbd9ec6561f6ffdca4a1b55e43b289d04f5671f5ab4de30999838c5fca5c282c00a7c7f8
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 238 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
> 
> [ ] +1 Release this package as nifi-1.13.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.13.0

[Otto Fowler]

+1
Followed the guide, built and ran, verified some of the new processor 
documentation.

Note:  was able to get this to build with :

Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"

but NOT with:

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 1.8.0_171, vendor: Oracle Corporation, runtime: 
/Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac”


> On Jan 27, 2021, at 23:15, Joe Witt  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.13.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1175
> 
> The source being voted upon and the convenience binaries can be found at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
> 
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> 
> The Git tag is nifi-1.13.0-RC1
> The Git commit ID is be1ac1c49726f366423b28fe75a08cbe9885ada3
> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=be1ac1c49726f366423b28fe75a08cbe9885ada3
> 
> Checksums of nifi-1.13.0-source-release.zip:
> SHA256: 02f5cfff3a3c2a82f82270b03e3533449330d9fbc102da8e920ab0cb39361218
> SHA512:
> 84769ec5791b6af1e9bca3088d535a611457ac13d046191021faf7c40ea0f1fc31238de43b9349e8034c2ec7d0cfeb570880fd9d4e6575d88ec52042b9fd997a
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 228 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0
> 
> Please note the convenience binary tar.gz and zip provided for NiFi include
> one less nar (the 'nifi-ignite-nar') to ensure the total file size remains
> under the ASF infrastructure limit. The migration guide and nifi-assembly
> have been updated so explicit removal won't be necessary in the future.
> But we will need to remain careful about any growth of our binaries and
> will need to continue to prune out old nars until we move to another model
> for users to acquire the nars they need.
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test. Then
> please vote:
> 
> [ ] +1 Release this package as nifi-1.13.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Otto Fowler]

Although I _have_ been building with this jdk for a long time, I may not have 
built with it since the TLS changes when it.
Maybe that is a factor.


> On Jan 28, 2021, at 10:15, Otto Fowler  wrote:
> 
> Thanks Joe,
> 
> I’m started down that road right after I sent this.  I’ll update when I know.
> I _have_ been building nifi with this jdk all along though.
> 
> If there are jdk minimum requirements, should they be documented ( or are 
> they and I missed it)?
> 
>> On Jan 28, 2021, at 09:35, Joe Witt  wrote:
>> 
>> Otto
>> 
>> I dont know we can keep all variants of Java 8 happy.  That one is nearly
>> three years old.  I would say try with something more recent.  It has been
>> pretty shocking how many api changes we have seen across these builds.
>> 
>> Thanks
>> 
>> On Thu, Jan 28, 2021 at 6:12 AM Otto Fowler  wrote:
>> 
>>> I’m trying to validate RC-1 and I’m seeing this exception building /
>>> running tests
>>> 
>>> [INFO] Running org.apache.nifi.processors.standard.TestListenTCP
>>> [ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>>> 4.499 s <<< FAILURE! - in org.apache.nifi.processors.standard.TestListenTCP
>>> [ERROR]
>>> testTLSClientAuthRequiredAndClientCertNotProvided(org.apache.nifi.processors.standard.TestListenTCP)
>>> Time elapsed: 0.127 s  <<< FAILURE!
>>> java.lang.AssertionError: unexpected exception type thrown;
>>> expected: but was:
>>>   at
>>> org.apache.nifi.processors.standard.TestListenTCP.testTLSClientAuthRequiredAndClientCertNotProvided(TestListenTCP.java:159)
>>> Caused by: java.net.SocketException: Connection reset
>>>   at
>>> org.apache.nifi.processors.standard.TestListenTCP.runTCP(TestListenTCP.java:209)
>>>   at
>>> org.apache.nifi.processors.standard.TestListenTCP.lambda$testTLSClientAuthRequiredAndClientCertNotProvided$0(TestListenTCP.java:160)
>>>   at
>>> org.apache.nifi.processors.standard.TestListenTCP.testTLSClientAuthRequiredAndClientCertNotProvided(TestListenTCP.java:159)
>>> 
>>> 
>>> 
>>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>>> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
>>> Java version: 1.8.0_171, vendor: Oracle Corporation, runtime:
>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"
>>> 
>>> 
>>> 
>>>> On Jan 27, 2021, at 10:31, Joe Witt  wrote:
>>>> 
>>>> ...in this brutal game of wach a mole another couple brittle tests have
>>>> emerged on the Java 11 line.
>>>> https://issues.apache.org/jira/browse/NIFI-8177 will get that looked
>>> into.
>>>> And I'm also seeing if for some reason adding a bit more horsepower/cpu
>>>> usage to the github tests will help.  Things had been very stable for
>>> quite
>>>> a while until the update to the latest 11.0.10 azul jdk came out so kind
>>> of
>>>> strange the tests became less reliable then. Anyway dont want to initiate
>>>> the RC until things seem stable otherwise it makes the vote process more
>>>> difficult and further when the builds dont pass it makes the already
>>>> difficult task of staying apace with PRs more inivolved.
>>>> 
>>>> Thanks
>>>> 
>>>> On Wed, Jan 27, 2021 at 12:19 AM Pierre Villard <
>>> pierre.villard...@gmail.com>
>>>> wrote:
>>>> 
>>>>> Awesome, thanks Joe!
>>>>> 
>>>>> Le mer. 27 janv. 2021 à 10:16, Joe Witt  a écrit :
>>>>> 
>>>>>> Team,
>>>>>> 
>>>>>> Ok wow so the issue with build failures that emerged has been resolved
>>> it
>>>>>> appears thanks to a lot of effort by exceptionfactory.  The other
>>>>> lingering
>>>>>> commits/PRs are in and I think most of what folks asked for in this
>>>>>> thread.  I just put in a commit which drops a few older/less common
>>> nars
>>>>>> from convenience binary inclusion to get us back under the maximum size
>>>>>> again.  And now I think we can initiate RC1 build for 1.13.0.
>>>>>> 
>>>>>> That should be published in the AM.
>>>>>> 
>>>>

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Otto Fowler]

Thanks Joe,

I’m started down that road right after I sent this.  I’ll update when I know.
I _have_ been building nifi with this jdk all along though.

If there are jdk minimum requirements, should they be documented ( or are they 
and I missed it)?

> On Jan 28, 2021, at 09:35, Joe Witt  wrote:
> 
> Otto
> 
> I dont know we can keep all variants of Java 8 happy.  That one is nearly
> three years old.  I would say try with something more recent.  It has been
> pretty shocking how many api changes we have seen across these builds.
> 
> Thanks
> 
> On Thu, Jan 28, 2021 at 6:12 AM Otto Fowler  wrote:
> 
>> I’m trying to validate RC-1 and I’m seeing this exception building /
>> running tests
>> 
>> [INFO] Running org.apache.nifi.processors.standard.TestListenTCP
>> [ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
>> 4.499 s <<< FAILURE! - in org.apache.nifi.processors.standard.TestListenTCP
>> [ERROR]
>> testTLSClientAuthRequiredAndClientCertNotProvided(org.apache.nifi.processors.standard.TestListenTCP)
>> Time elapsed: 0.127 s  <<< FAILURE!
>> java.lang.AssertionError: unexpected exception type thrown;
>> expected: but was:
>>at
>> org.apache.nifi.processors.standard.TestListenTCP.testTLSClientAuthRequiredAndClientCertNotProvided(TestListenTCP.java:159)
>> Caused by: java.net.SocketException: Connection reset
>>at
>> org.apache.nifi.processors.standard.TestListenTCP.runTCP(TestListenTCP.java:209)
>>at
>> org.apache.nifi.processors.standard.TestListenTCP.lambda$testTLSClientAuthRequiredAndClientCertNotProvided$0(TestListenTCP.java:160)
>>at
>> org.apache.nifi.processors.standard.TestListenTCP.testTLSClientAuthRequiredAndClientCertNotProvided(TestListenTCP.java:159)
>> 
>> 
>> 
>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
>> Java version: 1.8.0_171, vendor: Oracle Corporation, runtime:
>> /Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"
>> 
>> 
>> 
>>> On Jan 27, 2021, at 10:31, Joe Witt  wrote:
>>> 
>>> ...in this brutal game of wach a mole another couple brittle tests have
>>> emerged on the Java 11 line.
>>> https://issues.apache.org/jira/browse/NIFI-8177 will get that looked
>> into.
>>> And I'm also seeing if for some reason adding a bit more horsepower/cpu
>>> usage to the github tests will help.  Things had been very stable for
>> quite
>>> a while until the update to the latest 11.0.10 azul jdk came out so kind
>> of
>>> strange the tests became less reliable then. Anyway dont want to initiate
>>> the RC until things seem stable otherwise it makes the vote process more
>>> difficult and further when the builds dont pass it makes the already
>>> difficult task of staying apace with PRs more inivolved.
>>> 
>>> Thanks
>>> 
>>> On Wed, Jan 27, 2021 at 12:19 AM Pierre Villard <
>> pierre.villard...@gmail.com>
>>> wrote:
>>> 
>>>> Awesome, thanks Joe!
>>>> 
>>>> Le mer. 27 janv. 2021 à 10:16, Joe Witt  a écrit :
>>>> 
>>>>> Team,
>>>>> 
>>>>> Ok wow so the issue with build failures that emerged has been resolved
>> it
>>>>> appears thanks to a lot of effort by exceptionfactory.  The other
>>>> lingering
>>>>> commits/PRs are in and I think most of what folks asked for in this
>>>>> thread.  I just put in a commit which drops a few older/less common
>> nars
>>>>> from convenience binary inclusion to get us back under the maximum size
>>>>> again.  And now I think we can initiate RC1 build for 1.13.0.
>>>>> 
>>>>> That should be published in the AM.
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> On Mon, Jan 25, 2021 at 6:40 AM Joe Witt  wrote:
>>>>> 
>>>>>> Team,
>>>>>> 
>>>>>> There is a TZ handling issue
>>>>>> https://issues.apache.org/jira/browse/NIFI-8023 that needs a
>>>> resolution
>>>>>> and there is also a new issue that has emerged on Github Actions/CI
>>>>>> https://github.com/apache/nifi/actions which is resulting in broken
>>>>>>

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Otto Fowler]

 machine, which might be worth looking at (although
>>>>>> I've not
>>>>>>>>> deleted
>>>>>>>>>>>> and re-created the binaries in a little while, so I could
>>>>>> have
>>>>>>>>> something
>>>>>>>>>>>> wrong in my /lib folder or such):
>>>>>>>>>>>> 
>>>>>>>>>>>> 2020-12-01 08:52:48,382 ERROR [NiFi logging handler]
>>>>>>>>>>> org.apache.nifi.StdErr
>>>>>>>>>>>> WARNING: An illegal reflective access operation has
>>> occurred
>>>>>>>>>>>> 2020-12-01 08:52:48,383 ERROR [NiFi logging handler]
>>>>>>>>>>> org.apache.nifi.StdErr
>>>>>>>>>>>> WARNING: Illegal reflective access by
>>>>>>>>>>>> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
>>>>>>>>>>>> 
>>>>>> (file:.../nifi-1.13.0-SNAPSHOT/lib/java11/jaxb-impl-2.3.0.jar) to
>>>>>>>>> method
>>>>>>>>>>>> 
>>>>>> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
>>>>>>>>>>>> 2020-12-01 08:52:48,383 ERROR [NiFi logging handler]
>>>>>>>>>>> org.apache.nifi.StdErr
>>>>>>>>>>>> WARNING: Please consider reporting this to the
>> maintainers
>>> of
>>>>>>>>>>>> com.sun.xml.bind.v2.runtime.reflect.opt.Injector
>>>>>>>>>>>> 2020-12-01 08:52:48,383 ERROR [NiFi logging handler]
>>>>>>>>>>> org.apache.nifi.StdErr
>>>>>>>>>>>> WARNING: Use --illegal-access=warn to enable warnings of
>>>>>> further
>>>>>>>>> illegal
>>>>>>>>>>>> reflective access operations
>>>>>>>>>>>> 2020-12-01 08:52:48,383 ERROR [NiFi logging handler]
>>>>>>>>>>> org.apache.nifi.StdErr
>>>>>>>>>>>> WARNING: All illegal access operations will be denied in
>> a
>>>>>> future
>>>>>>>>> release
>>>>>>>>>>>> 
>>>>>>>>>>>> Also:
>>>>>>>>>>>> 2020-12-01 08:52:52,486 WARN [main]
>>>>>>>>>>>> o.a.n.n.StandardExtensionDiscoveringManager Failed to
>>>>>> register
>>>>>>>>> extension
>>>>>>>>>>>> org.apache.nifi.provenance.VolatileProvenanceRepository
>> due
>>>>>> to:
>>>>>>>>> Attempt
>>>>>>>>>>> was
>>>>>>>>>>>> made to load
>>>>>> org.apache.nifi.provenance.VolatileProvenanceRepository
>>>>>>>>> from
>>>>>>>>>>>> 
>>>>>> org.apache.nifi:nifi-provenance-repository-nar:1.13.0-SNAPSHOT but
>>>>>>>>> that
>>>>>>>>>>>> class name is already loaded/registered from
>>>>>>>>>>>> org.apache.nifi:nifi-stateless-nar:1.13.0-SNAPSHOT and
>>>>>> multiple
>>>>>>>>> versions
>>>>>>>>>>>> are not supported for this type
>>>>>>>>>>>> 
>>>>>>>>>>>> These log messages appear just before the NAR extraction
>>>>>> logs during
>>>>>>>>>>>> instance startup.
>>>>>>>>>>>> 
>>>>>>>>>>>> Don't know whether these are things that should be fixed
>>>>>> before
>>>>>>>>> 1.13.0
>>>>>>>>>>>> and/or have already been looked into, but figured I'd
>>>>>> mention them so
>>>>>>>>>>>> they're at least known about.
>>>>>>>>>>>> 
>>>>>>>>>>>> ---
>>>>>>>>>>>> *Chris Sampson*
>>>>>>>>>>>> IT Consultant
>>>>>>>>>>>> chris.samp...@naimuri.com
>>>>>>>>>>>> <https://www.naimuri.com/>
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, 30 Nov 2020 at 18:50, Nissim Shiman
>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello Nifi Team,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> NIFI-7738 [1]
>> https://github.com/apache/nifi/pull/4563
>>>>>>>>>>>>> has already been reviewed/tested by a fellow
>> contributer
>>>>>>>>>>>>> I would greatly appreciate a committer moving this to
>>> main
>>>>>> for
>>>>>>>>> 1.13.0
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Nissim Shiman
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/NIFI-7738
>>>>>>>>>>>>>On Sunday, November 29, 2020, 07:12:12 PM EST, Matt
>>>>>> Burgess <
>>>>>>>>>>>>> mattyb...@apache.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I just merged Mike's Cassandra DMC PR, reviewing the
>>>>>> graph stuff
>>>>>>>>> now.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have a bunch of open PRs [1], I don't think any are
>>>>>> blockers for
>>>>>>>>>>>>> 1.13.0 but thought I'd take this opportunity to find
>> some
>>>>>>>>> reviewers :)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Matt
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [1] https://github.com/apache/nifi/pulls/mattyb149
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, Nov 28, 2020 at 10:13 AM Phillip Grenier
>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I wouldn't mind
>>> https://github.com/apache/nifi/pull/4554
>>>>>> making
>>>>>>>>> it
>>>>>>>>>>> in.
>>>>>>>>>>>>> Have
>>>>>>>>>>>>>> seen a couple more instances where people have
>> created
>>>>>> custom
>>>>>>>>>>> processors
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> work around this PR.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Nov 26, 2020 at 11:10 AM Mike Thomsen <
>>>>>>>>>>> mikerthom...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Also, for most of us (committers and community
>>>>>> members), it's a
>>>>>>>>>>>>>>> holiday season so might be good to delay a release
>> to
>>>>>> early
>>>>>>>>> January
>>>>>>>>>>>>>>> anyway. Just a thought.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Nov 26, 2020 at 11:08 AM Mike Thomsen <
>>>>>>>>>>> mikerthom...@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> https://github.com/apache/nifi/pull/4638 - This
>> is
>>>>>> a big
>>>>>>>>>>> increase in
>>>>>>>>>>>>>>>> graph functionality, and I want to give him a
>>> chance
>>>>>> to get
>>>>>>>>> his
>>>>>>>>>>> first
>>>>>>>>>>>>>>>> contribution in with 1.13 (we may have some more
>> of
>>>>>> his work
>>>>>>>>> we
>>>>>>>>>>> can
>>>>>>>>>>>>>>>> open source).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Matt's also reviewing some of my Cassandra and
>>> graph
>>>>>> tickets
>>>>>>>>> as
>>>>>>>>>>> well.
>>>>>>>>>>>>>>>> I wouldn't call those **blockers**, but having
>> the
>>>>>> Cassandra
>>>>>>>>> DMC
>>>>>>>>>>>>> fully
>>>>>>>>>>>>>>>> fleshed out would be nice.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, Nov 26, 2020 at 9:10 AM Otto Fowler <
>>>>>>>>>>> ottobackwa...@gmail.com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> NIFI-7795
>>>>>> https://github.com/apache/nifi/pull/4656
>>>>>>>>>>>>>>>>> NIFI-7761
>>> https://github.com/apache/nifi/pull/4513
>>>>>>>>>>>>>>>>> NIFI-2072
>>> https://github.com/apache/nifi/pull/4384
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> From: Pierre Villard <
>>> pierre.villard...@gmail.com>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Reply: dev@nifi.apache.org <
>> dev@nifi.apache.org>
>>> <
>>>>>>>>>>>>> dev@nifi.apache.org>
>>>>>>>>>>>>>>>>> Date: November 26, 2020 at 04:55:56
>>>>>>>>>>>>>>>>> To: dev@nifi.apache.org 
>> <
>>>>>>>>>>> dev@nifi.apache.org
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Subject:  [DISCUSS] Release of Apache NiFi
>> 1.13.0
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi community,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Starting a discussion thread around the release
>>> of
>>>>>> NiFi
>>>>>>>>>>> 1.13.0. We
>>>>>>>>>>>>>>> added
>>>>>>>>>>>>>>>>> quite a lot of significant improvements and
>>>>>> features since
>>>>>>>>> the
>>>>>>>>>>>>> release
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> 1.12.1 - I see 143 JIRAs with 1.13.0 as the fix
>>>>>> version. I
>>>>>>>>>>> think it
>>>>>>>>>>>>>>> makes
>>>>>>>>>>>>>>>>> sense to consider a new release.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Please share here the JIRAs with opened pull
>>>>>> requests that
>>>>>>>>> we
>>>>>>>>>>> would
>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>> look at to make this release happen.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Pierre
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> 

Re: Question about ListenTCP processor

[Otto Fowler]

It is in the base class
https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-extension-utils/nifi-processor-utils/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventBatchingProcessor.java#L50

> On Jan 26, 2021, at 21:31, Hodor, Paul 
>  wrote:
> 
> One of the constraints of the ListenTCP processor is that it delimites 
> incoming data by the “line feed” or “newline” character. I am looking at its 
> source code at 
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenTCP.java.
>  Could you give me a hint where this constraint is defined. I don’s see it in 
> the processor source code, so I assume it must come from another class that 
> is being used.
> 
> The reason I am asking is because we are trying to modify the processor to 
> use a different delimiter character.
> 
> Thanks,
> Paul
> 
> 
> CONFIDENTIALITY NOTICE: This e-mail, including any attachments, is for the 
> sole use of the intended recipient(s) and may contain confidential and 
> privileged information protected by law. Any unauthorized review, use, 
> disclosure or distribution is prohibited. If you are not the intended 
> recipient, please contact the sender by reply e-mail and destroy all copies 
> of the original message.

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Otto Fowler]

I would still like to get https://github.com/apache/nifi/pull/4384 
 in and not miss the release.

> On Jan 12, 2021, at 13:17, Pierre Villard  wrote:
> 
> Hi all,
> 
> Current status about what we want to get in based on this thread:
> - NIFI-8034 - PropertyValue.isExpressionLanguagePresent always returns true
> for non-null values - https://github.com/apache/nifi/pull/4746
> - NIFI-7989 - Add Hive "data drift" processor -
> https://github.com/apache/nifi/pull/4750
> - NIFI-7356 - TLS + embedded Zookeeper - pull request to be submitted by
> Nathan
> 
> If we can wrap up these JIRAs, we could get a release candidate for NiFi
> 1.13 soon.
> 
> Thanks,
> Pierre
> 
> 
> 
> Le mer. 6 janv. 2021 à 04:17, Joey Frazee 
> a écrit :
> 
>> I have https://github.com/apache/nifi/pull/4632 which fixes an OOME in
>> PutAzureBlobStorage reported in
>> https://lists.apache.org/x/thread.html/rdef82be24828277b85bdc94dc57a8fb9df6f73552daeda289c941a51%40%3Cusers.nifi.apache.org%3E
>> 
>> It’s a pretty small change.
>> 
>> -joey
>> 
>>> On Jan 5, 2021, at 3:14 PM, Matt Burgess  wrote:
>>> 
>>> I am reviewing a PR at the moment but intend to review the graph stuff
>> right afterwards.
>>> 
 On Jan 5, 2021, at 5:35 PM, Mike Thomsen 
>> wrote:
 
 I have a PR for graph that we need to close out because part of the
>> graph
 bundle will be broken without it.
 
>> On Tue, Jan 5, 2021, 11:50 Mark Bean  wrote:
> 
> I'd like to see this PR completed as well
> https://github.com/apache/nifi/pull/4225
> 
> There's been discussion on it, and I don't see any further objections.
> 
> Thanks,
> Mark
> 
> On Thu, Nov 26, 2020 at 4:55 AM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
> 
>> Hi community,
>> 
>> Starting a discussion thread around the release of NiFi 1.13.0. We
>> added
>> quite a lot of significant improvements and features since the
>> release of
>> 1.12.1 - I see 143 JIRAs with 1.13.0 as the fix version. I think it
>> makes
>> sense to consider a new release.
>> 
>> Please share here the JIRAs with opened pull requests that we would
>> need
> to
>> look at to make this release happen.
>> 
>> Thanks,
>> Pierre
>> 
> 
>> 

Re: Static processor design

[Otto Fowler]

*ListenDynamicPropertyChange

> On Jan 8, 2021, at 12:42, Otto Fowler  wrote:
> 
> So you would be implementing ListDynamicPropertyChange kind of.
> 
> 
>> On Jan 8, 2021, at 12:41, Otto Fowler > <mailto:ottobackwa...@gmail.com>> wrote:
>> 
>> What are the attributes of the processor?
>> Do you have : 
>> 
>> @InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
>> or anything?
>> 
>> You should take a look at the Listen** processors.
>> They do not have any inputs etc, and run processing queued input from a 
>> background thread.
>> You can have on modified queue a change and then have trigger process the 
>> queue of changes.
>> 
>> 
>> 
>>> On Jan 8, 2021, at 11:04, Russell Bateman >> <mailto:r...@windofkeltia.com>> wrote:
>>> 
>>> I only put the code I want to execute in onTrigger(), I suspected it would 
>>> not fire there. I know that this isn't what processors do. Configuration is 
>>> a messy problem to solve when your downstreamers want it made easy. This is 
>>> supposed to be a solution that allows them to remain in the NiFi UI and not 
>>> have to run off doing harder things to configure. I could put what I'm 
>>> doing in /HumanReadables/ into a real, running processor, but then, I would 
>>> kind of have to add them to several processors and I wanted to avoid the 
>>> confusion that created.
>>> 
>>> Here's the code. Thanks.
>>> 
>>> import com.windofkeltia.constants.HumanReadableMappings;
>>> 
>>> ...
>>> 
>>> @TriggerWhenEmpty
>>> @SideEffectFree
>>> @CapabilityDescription( "Dynamic properties can be created to specify (or 
>>> add to) static configuration"
>>>   + " of key-value substitution pairs. See additional 
>>> details." )
>>> public class HumanReadables extends AbstractProcessor
>>> {
>>>   @Override
>>>   public void onTrigger( final ProcessContext context, final ProcessSession 
>>> session ) throws ProcessException
>>>   {
>>> getLogger().info( "inside onTrigger()..." );
>>> 
>>> for( Map.Entry< PropertyDescriptor, String > entry : 
>>> context.getProperties().entrySet() )
>>> {
>>>   PropertyDescriptor property = entry.getKey();
>>> 
>>>   // do work here--maybe pre-wipe all key-value pairs if we're just 
>>> going to recreate them?
>>>   final String PROPERTY_NAME  = property.getName();
>>>   final String PROPERTY_VALUE = entry.getValue();
>>> 
>>>   logger.trace( "Processing configurable mappings titled \"" + 
>>> PROPERTY_NAME + "\"" );
>>> 
>>>   try
>>>   {
>>> harvestDynamicPropertyMappings( PROPERTY_VALUE );
>>>   }
>>>   catch( Exception e )
>>>   {
>>> getLogger().debug( e.getMessage() );
>>>   }
>>> }
>>>   }
>>> 
>>>   protected static void harvestDynamicPropertyMappings( final String 
>>> PROPERTY_VALUE )
>>>   {
>>> final String[] LINES  = PROPERTY_VALUE.split( "\n" );
>>> intlineNumber = 0;
>>> 
>>> if( LINES.length < 1 )
>>>   return;
>>> 
>>> final String WHICH_LIST = LINES[ 0 ];
>>> 
>>> for( final String VALUE_LINE : LINES )
>>> {
>>>   char delimiter = VALUE_LINE.charAt( 0 );
>>>   int  position = VALUE_LINE.indexOf( delimiter, 1 );
>>>   String key, value;
>>> 
>>>   key   = ( position < 0 ) ? VALUE_LINE.substring( 1 ) : 
>>> VALUE_LINE.substring( 1, position ).trim();
>>>   value = ( position > 0 ) ? VALUE_LINE.substring( position + 1 
>>> ).trim() : "";
>>> 
>>>   HumanReadableMappings.add( key, value );
>>> }
>>>   }
>>> 
>>>   @Override
>>>   protected PropertyDescriptor getSupportedDynamicPropertyDescriptor( final 
>>> String propertyDescriptorName )
>>>   {
>>> return new PropertyDescriptor.Builder()
>>>  .required( false )
>>>  .name( propertyDescriptorName )
>>>  .addValidator( 
>>> StandardValidators.NON_EMPTY_VALIDATOR )
>>>

Re: Static processor design

[Otto Fowler]

So you would be implementing ListDynamicPropertyChange kind of.


> On Jan 8, 2021, at 12:41, Otto Fowler  wrote:
> 
> What are the attributes of the processor?
> Do you have : 
> 
> @InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
> or anything?
> 
> You should take a look at the Listen** processors.
> They do not have any inputs etc, and run processing queued input from a 
> background thread.
> You can have on modified queue a change and then have trigger process the 
> queue of changes.
> 
> 
> 
>> On Jan 8, 2021, at 11:04, Russell Bateman > <mailto:r...@windofkeltia.com>> wrote:
>> 
>> I only put the code I want to execute in onTrigger(), I suspected it would 
>> not fire there. I know that this isn't what processors do. Configuration is 
>> a messy problem to solve when your downstreamers want it made easy. This is 
>> supposed to be a solution that allows them to remain in the NiFi UI and not 
>> have to run off doing harder things to configure. I could put what I'm doing 
>> in /HumanReadables/ into a real, running processor, but then, I would kind 
>> of have to add them to several processors and I wanted to avoid the 
>> confusion that created.
>> 
>> Here's the code. Thanks.
>> 
>> import com.windofkeltia.constants.HumanReadableMappings;
>> 
>> ...
>> 
>> @TriggerWhenEmpty
>> @SideEffectFree
>> @CapabilityDescription( "Dynamic properties can be created to specify (or 
>> add to) static configuration"
>>   + " of key-value substitution pairs. See additional 
>> details." )
>> public class HumanReadables extends AbstractProcessor
>> {
>>   @Override
>>   public void onTrigger( final ProcessContext context, final ProcessSession 
>> session ) throws ProcessException
>>   {
>> getLogger().info( "inside onTrigger()..." );
>> 
>> for( Map.Entry< PropertyDescriptor, String > entry : 
>> context.getProperties().entrySet() )
>> {
>>   PropertyDescriptor property = entry.getKey();
>> 
>>   // do work here--maybe pre-wipe all key-value pairs if we're just 
>> going to recreate them?
>>   final String PROPERTY_NAME  = property.getName();
>>   final String PROPERTY_VALUE = entry.getValue();
>> 
>>   logger.trace( "Processing configurable mappings titled \"" + 
>> PROPERTY_NAME + "\"" );
>> 
>>   try
>>   {
>> harvestDynamicPropertyMappings( PROPERTY_VALUE );
>>   }
>>   catch( Exception e )
>>   {
>> getLogger().debug( e.getMessage() );
>>   }
>> }
>>   }
>> 
>>   protected static void harvestDynamicPropertyMappings( final String 
>> PROPERTY_VALUE )
>>   {
>> final String[] LINES  = PROPERTY_VALUE.split( "\n" );
>> intlineNumber = 0;
>> 
>> if( LINES.length < 1 )
>>   return;
>> 
>> final String WHICH_LIST = LINES[ 0 ];
>> 
>> for( final String VALUE_LINE : LINES )
>> {
>>   char delimiter = VALUE_LINE.charAt( 0 );
>>   int  position = VALUE_LINE.indexOf( delimiter, 1 );
>>   String key, value;
>> 
>>   key   = ( position < 0 ) ? VALUE_LINE.substring( 1 ) : 
>> VALUE_LINE.substring( 1, position ).trim();
>>   value = ( position > 0 ) ? VALUE_LINE.substring( position + 1 ).trim() 
>> : "";
>> 
>>   HumanReadableMappings.add( key, value );
>> }
>>   }
>> 
>>   @Override
>>   protected PropertyDescriptor getSupportedDynamicPropertyDescriptor( final 
>> String propertyDescriptorName )
>>   {
>> return new PropertyDescriptor.Builder()
>>  .required( false )
>>  .name( propertyDescriptorName )
>>  .addValidator( 
>> StandardValidators.NON_EMPTY_VALIDATOR )
>>  // or .addValidator( Validator.VALID ) if 
>> you do not wish it validated!
>>  .dynamic( true )
>>  .build();
>>   }
>> 
>>   private volatile Set< String > dynamicPropertyNames = new HashSet<>();
>> 
>>   @Override
>>   public void onPropertyModified( final PropertyDescriptor descriptor, final 
>> String oldValue, final String newValue )
>>   {
>> getLogger().info( oldValue + " -> " + newValue );

Re: Static processor design

[Otto Fowler]

What are the attributes of the processor?
Do you have : 

@InputRequirement(InputRequirement.Requirement.INPUT_FORBIDDEN)
or anything?

You should take a look at the Listen** processors.
They do not have any inputs etc, and run processing queued input from a 
background thread.
You can have on modified queue a change and then have trigger process the queue 
of changes.



> On Jan 8, 2021, at 11:04, Russell Bateman  wrote:
> 
> I only put the code I want to execute in onTrigger(), I suspected it would 
> not fire there. I know that this isn't what processors do. Configuration is a 
> messy problem to solve when your downstreamers want it made easy. This is 
> supposed to be a solution that allows them to remain in the NiFi UI and not 
> have to run off doing harder things to configure. I could put what I'm doing 
> in /HumanReadables/ into a real, running processor, but then, I would kind of 
> have to add them to several processors and I wanted to avoid the confusion 
> that created.
> 
> Here's the code. Thanks.
> 
> import com.windofkeltia.constants.HumanReadableMappings;
> 
> ...
> 
> @TriggerWhenEmpty
> @SideEffectFree
> @CapabilityDescription( "Dynamic properties can be created to specify (or add 
> to) static configuration"
>   + " of key-value substitution pairs. See additional 
> details." )
> public class HumanReadables extends AbstractProcessor
> {
>   @Override
>   public void onTrigger( final ProcessContext context, final ProcessSession 
> session ) throws ProcessException
>   {
> getLogger().info( "inside onTrigger()..." );
> 
> for( Map.Entry< PropertyDescriptor, String > entry : 
> context.getProperties().entrySet() )
> {
>   PropertyDescriptor property = entry.getKey();
> 
>   // do work here--maybe pre-wipe all key-value pairs if we're just going 
> to recreate them?
>   final String PROPERTY_NAME  = property.getName();
>   final String PROPERTY_VALUE = entry.getValue();
> 
>   logger.trace( "Processing configurable mappings titled \"" + 
> PROPERTY_NAME + "\"" );
> 
>   try
>   {
> harvestDynamicPropertyMappings( PROPERTY_VALUE );
>   }
>   catch( Exception e )
>   {
> getLogger().debug( e.getMessage() );
>   }
> }
>   }
> 
>   protected static void harvestDynamicPropertyMappings( final String 
> PROPERTY_VALUE )
>   {
> final String[] LINES  = PROPERTY_VALUE.split( "\n" );
> intlineNumber = 0;
> 
> if( LINES.length < 1 )
>   return;
> 
> final String WHICH_LIST = LINES[ 0 ];
> 
> for( final String VALUE_LINE : LINES )
> {
>   char delimiter = VALUE_LINE.charAt( 0 );
>   int  position = VALUE_LINE.indexOf( delimiter, 1 );
>   String key, value;
> 
>   key   = ( position < 0 ) ? VALUE_LINE.substring( 1 ) : 
> VALUE_LINE.substring( 1, position ).trim();
>   value = ( position > 0 ) ? VALUE_LINE.substring( position + 1 ).trim() 
> : "";
> 
>   HumanReadableMappings.add( key, value );
> }
>   }
> 
>   @Override
>   protected PropertyDescriptor getSupportedDynamicPropertyDescriptor( final 
> String propertyDescriptorName )
>   {
> return new PropertyDescriptor.Builder()
>  .required( false )
>  .name( propertyDescriptorName )
>  .addValidator( 
> StandardValidators.NON_EMPTY_VALIDATOR )
>  // or .addValidator( Validator.VALID ) if 
> you do not wish it validated!
>  .dynamic( true )
>  .build();
>   }
> 
>   private volatile Set< String > dynamicPropertyNames = new HashSet<>();
> 
>   @Override
>   public void onPropertyModified( final PropertyDescriptor descriptor, final 
> String oldValue, final String newValue )
>   {
> getLogger().info( oldValue + " -> " + newValue );
> 
> final Set< String > newDynamicPropertyNames = new HashSet<>( 
> dynamicPropertyNames );
> 
> if( isNull( newValue ) )
>   newDynamicPropertyNames.remove( descriptor.getName() );
> else if( isNull( oldValue ) && descriptor.isDynamic() )
>   newDynamicPropertyNames.add( descriptor.getName() );
> 
> dynamicPropertyNames = Collections.unmodifiableSet( 
> newDynamicPropertyNames );
> 
> final Set< String > allDynamicProperties = dynamicPropertyNames;
>   }
> 
>   @OnScheduled public void processProperties( final ProcessContext context )
>   {
> for( Map.Entry< PropertyDescriptor, String > entry : 
> context.getProperties().entrySet() )
> {
>   PropertyDescriptor descriptor = entry.getKey();
> 
>   if( descriptor.isDynamic() )
> getLogger().debug( "Dynamic property named:\n" + 
> descriptor.getName()
> + ", value: " + entry.getValue().replaceAll( 
> "\n", " + " ) );
> }
>   }
> 
>   protected static final String DEFAULT_MAPPING_VALUE = ""
> + "|http://loinc.org |LOINC\n"
> + 

Re: Using record data with the graph bundle

[Otto Fowler]

 It is awesome that you guys are contributing all this awesome graph
functionality, thanks Mike and colleagues!

From: Mike Thomsen  
Reply: dev@nifi.apache.org  
Date: December 1, 2020 at 13:25:46
To: dev@nifi.apache.org  
Subject:  Using record data with the graph bundle

I just approved and merged a PR from a colleague of mine to bring over
some code that we've been building for our client. It's a new
processor called ExecuteGraphQueryRecord. It allows users to build a
parameter map from dynamic properties that specify mappings between a
key and a record path.

Example usage:

Input:

{
"name": "John"
}

Mapping:

name => /name

Cypher

MATCH (p:Person) WHERE p.name = $name RETURN p

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Otto Fowler]

 NIFI-7795 https://github.com/apache/nifi/pull/4656
NIFI-7761 https://github.com/apache/nifi/pull/4513
NIFI-2072 https://github.com/apache/nifi/pull/4384

From: Pierre Villard 

Reply: dev@nifi.apache.org  
Date: November 26, 2020 at 04:55:56
To: dev@nifi.apache.org  
Subject:  [DISCUSS] Release of Apache NiFi 1.13.0

Hi community,

Starting a discussion thread around the release of NiFi 1.13.0. We added
quite a lot of significant improvements and features since the release of
1.12.1 - I see 143 JIRAs with 1.13.0 as the fix version. I think it makes
sense to consider a new release.

Please share here the JIRAs with opened pull requests that we would need to
look at to make this release happen.

Thanks,
Pierre

Re: request to review NIFI-7738, Provenance Search Events - add reverse query capability

[Otto Fowler]

 Joe,

Thank you for expounding on your original response.  I think that clarifies
expectations a great deal. I totally agree, and do those types of reviews
myself.
As a non-committer who does pr’s and reviews I guess what I was saying is
that even if I and someone who entered the issue in jira review a pr, it
can still require
*some* committer check off to get merged.

It wasn’t my intention to convey that I was disagreeing with your
statement.

On October 8, 2020 at 14:33:50, Joe Witt (joe.w...@gmail.com) wrote:

Otto

I wrote: "Anyone can do a review of the code and build and feasibility of a
change. It only requires commit privileges to actually merge something."
You then wrote "Anyone can review, but you need at least one committer
approval / review to get a pr landed."

While similar, there is an important difference in these two notes. Anyone
can review/verify the build/consider the validity of the change and make
their assertion about its readiness (by giving a plus 1 or other
comments). That person does not have to be a committer and it is an
*excellent* way to demonstrate awareness and understanding of the codebase
and implications to the community when someone does so. This demonstration
is in turn a great way to earn committership. Once anyone (regardless of
their status as a contributor, committer, PMC) has done sufficient review
it is ready to be merged. Whomever has commit privileges (committer or
PMC) is the one that has to do the merge. Yes they should know what
they're committing. And yes they can make a judgement call whether
whomever already reviewed it did a good enough job. But we dont want
contributors assuming their reviews arent meaningful and that someone else
will do a real review later. That belief is self fulfilling and
problematic. We want them thinking "I need to think carefully here about
how this applies to the codebase, what it means to the community, what it
means for maintenance, testing, API, etc..". It is that process of
ownership that earns merit and committer status.

So the way I wrote it is important in how we think about this.

Thanks

On Thu, Oct 8, 2020 at 11:13 AM Otto Fowler 
wrote:

> Anyone can review, but you need at least one committer approval / review
> to get a pr landed.
>
>
> On October 8, 2020 at 12:17:17, Joe Witt (joe.w...@gmail.com) wrote:
>
> Hello
>
> Anyone can do a review of the code and build and feasibility of a change.
> It only requires commit privileges to actually merge something.
>
> Doing the hard work of reviewing/testing the PRs is a *great* way to earn
> merit.
>
> Much of the PRs that tend to hang around are from people contributing a
> specific thing and those are some of the most challenging. We as a
> community then take on support of those things for function, License and
> Notice adherence, etc.. On the other hand, these folks are also the ones
> that can often start contributing elsewhere and building merit and
earning
> commit and even PMC status. So we have to invest one way or another. But
> for sure this is not easy to do well as time has shown.
>
> Nissim, in your case the contribution is probably cool/good/useful. What
> could help you get more attention on it is helping with bug fixes or
> reviewing other peoples contributions/etc.. Not saying anyone is checking
> whether you do that or not but if they see more of your effort it *does*
> help improve likelihood of getting merge attention. Not required...just
> sharing another perspective.
>
> Regardless thanks for the contribution and hopefully someone is able to
> review soon.
>
> Thanks
>
> On Thu, Oct 8, 2020 at 9:00 AM Phillip Grenier 
wrote:
>
> > Pierre,
> >
> > I think this discussion brings up a valid conversation point. At some
> point
> > a PMC member needs to approve the merge request, so from a contributors
> > level what can we do to make that merge both easier and/or more likely
to
> > happen. That and how the community can help filter down the ever
growing
> > backlog of PRs.
> >
> > Thanks and look forward to helping,
> >
> > Phillip
> >
> > On Wed, Oct 7, 2020 at 2:04 PM Pierre Villard <
> pierre.villard...@gmail.com
> > >
> > wrote:
> >
> > > Nissim,
> > >
> > > Thanks a lot for your contribution but there are 277 pull requests
> opened
> > > at this time. This is a community effort, and if you expect people to
> > > review your PRs, you'd have to also try reviewing PRs opened by
others
> in
> > > the community. Otherwise this will need to wait for someone with some
> > > bandwidth to focus on this.
> > >
> > > Thanks,
> > > Pierre
> > >
> > > Le mer. 7 oct. 2020 à 19:59, Nissim Shiman 

> 

Re: request to review NIFI-7738, Provenance Search Events - add reverse query capability

[Otto Fowler]

 Anyone can review, but you need at least one committer approval / review
to get a pr landed.


On October 8, 2020 at 12:17:17, Joe Witt (joe.w...@gmail.com) wrote:

Hello

Anyone can do a review of the code and build and feasibility of a change.
It only requires commit privileges to actually merge something.

Doing the hard work of reviewing/testing the PRs is a *great* way to earn
merit.

Much of the PRs that tend to hang around are from people contributing a
specific thing and those are some of the most challenging. We as a
community then take on support of those things for function, License and
Notice adherence, etc.. On the other hand, these folks are also the ones
that can often start contributing elsewhere and building merit and earning
commit and even PMC status. So we have to invest one way or another. But
for sure this is not easy to do well as time has shown.

Nissim, in your case the contribution is probably cool/good/useful. What
could help you get more attention on it is helping with bug fixes or
reviewing other peoples contributions/etc.. Not saying anyone is checking
whether you do that or not but if they see more of your effort it *does*
help improve likelihood of getting merge attention. Not required...just
sharing another perspective.

Regardless thanks for the contribution and hopefully someone is able to
review soon.

Thanks

On Thu, Oct 8, 2020 at 9:00 AM Phillip Grenier  wrote:

> Pierre,
>
> I think this discussion brings up a valid conversation point. At some
point
> a PMC member needs to approve the merge request, so from a contributors
> level what can we do to make that merge both easier and/or more likely to
> happen. That and how the community can help filter down the ever growing
> backlog of PRs.
>
> Thanks and look forward to helping,
>
> Phillip
>
> On Wed, Oct 7, 2020 at 2:04 PM Pierre Villard  >
> wrote:
>
> > Nissim,
> >
> > Thanks a lot for your contribution but there are 277 pull requests
opened
> > at this time. This is a community effort, and if you expect people to
> > review your PRs, you'd have to also try reviewing PRs opened by others
in
> > the community. Otherwise this will need to wait for someone with some
> > bandwidth to focus on this.
> >
> > Thanks,
> > Pierre
> >
> > Le mer. 7 oct. 2020 à 19:59, Nissim Shiman 
a
> > écrit :
> >
> > > Second attempt...
> > >
> > >
> > > Hello NiFi team,
> > >
> > > Could someone respond to this email.
> > >
> > > Thanks,
> > > Nissim Shiman
> > > On Monday, October 5, 2020, 11:01:57 AM EDT, Nissim Shiman
> > >  wrote:
> > >
> > > Hello NiFi team,
> > > Could someone review the pull request for NIFI-7738 (
> > > https://issues.apache.org/jira/browse/NIFI-7738), that was put in 5
> days
> > > ago?
> > > (Provenance Search Events - add feature to allow inverse query)
> > >
> > > One case this will come in handy is if a particular flowfile has an
> issue
> > > and goes in circles through processors and ends up emitting a lot of
> > > provenance making it hard to follow other flowfiles.
> > >
> > > This feature will allow users to query for all provenance except for
> the
> > > uuid of the problematic flowfile.
> > >
> > >
> > >
> > > This can also be used for any indexed field or attribute as well.
> > > This has been compiled and tested on java 8 and java11 for all 4
> > > provenance repository implementations for both indexed fields and
> > indexed
> > > attributes
> > > The only "real" code changes were made to LuceneUtil.java (to handle
> > > WriteAhead, EncryptedWriteAhead and Persistent Provenance
Repostories)
> > and
> > > VolatileProvenanceRepository.java
> > > The rest are just involved in propagating the ability to query the
> "NOT"
> > > option for a given user entered provenance search value from the gui
to
> > the
> > > backend query.
> > >
> > > Thank You,
> > > Nissim Shiman
> >
>

Re: Teradata and Nifi

[Otto Fowler]

 I believe this still applies:
https://community.cloudera.com/t5/Support-Questions/Teradata-and-NIFI/td-p/211964



On August 28, 2020 at 10:54:34, Urooj, Harisa (harisa.ur...@teradata.com)
wrote:

We’re trying to do bulk loading of data in Teradata using NIFI. Is that
possible in NIFI?

Currently we noticed that row by row inserts to teradata is possible but
that is not scaling well when it comes to loading millions of records.
Teradata has their own set of utility operators: FASTLOAD, MULTILOAD,
TPTLOAD, TPTUPDATE, etc that we can use to do fast bulk loading of records

into Teradata.  Does NIFI support any of these operators? So far I only see
the option to run SQL against teradata not utility jobs.









*Harisa Urooj*
EMEA Cloud Architect
+44 7557 564489

[image: signature_1806873648] 

3 London Bridge Street

SE1 9SG, London, UK

teradata.com 



*Stop buying “analytics.” Invest in answers. *

This e-mail is from Teradata Corporation and may contain information that
is confidential or proprietary. If you are not the intended recipient, do
not read, copy or distribute the e-mail or any attachments. Instead, please
notify the sender and delete the e-mail and any attachments. Thank you.

Please consider the environment before printing.

Re: From one flowfile to two...

[Otto Fowler]

 Thanks for writing that up!

On August 27, 2020 at 17:49:07, Russell Bateman (r...@windofkeltia.com)
wrote:

In case anyone cares,
https://www.javahotchocolate.com/notes/nifi-custom.html#two-split-from-one

On 8/27/20 11:15 AM, Andy LoPresto wrote:
> Russell,
>
> Glad you found a working solution. Maybe it would be better for you to
write up your findings and share them with a broader audience. I have often
seen the best explanations are written by people who were recently in the
“how do I do X?” state, as they are closest to the problem and can walk
through their process of gathering understanding. Someone who works on
these methods day in and day out may not write for the appropriate audience
or explain the experience as well.
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Aug 27, 2020, at 10:10 AM, Russell Bateman 
wrote:
>>
>> I needed to get back here...
>>
>> I took this advice to heart and finished my processor. Thanks to Matt
and Mark for all their suggestions! They cleared up a few things. There was
one bug in the code that was mine, small, but significant in its effect on
the rest. That mistake also explained why I thought the uuidwas identical
between at least two of the cloned flowfiles. What I would wish for, and am
probably not strong enough to write, would be a synthesis of the session
methods read() and write() and how best to use them (one-to-one,
one-to-many, etc.). Javadoc is too paratactic by nature, the NiFi
Developer's Guide almost silent on these methods. If it were not for the
many existing examples using these methods, it would be hard to learn to do
even simple things. I did look for something closer to what I needed to do,
but unsuccessfully.
>>
>> Thanks again. If anything, the NiFi mailing lists are a place both for
great information and being treated well.
>>
>> Russ
>>
>> On 8/25/20 12:24 PM, Mark Payne wrote:
>>> Russ,
>>>
>>> Several comments here. I’ve included them inline, below.
>>>
>>> Hope it’s helpful.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
 On Aug 25, 2020, at 2:09 PM, Russell Bateman 
wrote:

 Thanks for your suggestions, Matt.

 I decided to keep the original flowfile only upon failure. So, I have
the embedded-document file and the serialized POJOs created from processing
the non embedded-document part as the result if successful. (Condensed code
at end...)

 Now I have three questions...

 1. I seem not to have placated NiFi with the assurance that I have
transferred or disposed of all three flowfiles suitably. I get:

 java.lang.AssertionError:
org.apache.nifi.processor.exception.FlowFileHandlingException: Cannot
commit session because the following FlowFiles have not been removed or
transferred: [2]
> This is probably because at the end of the block, you catch Exception
and then route the original FlowFile to failure. But you’ve already cloned
it and didn’t deal with the clone.
 *Which of the three flowfiles does [2] refer to? Or does it just mean
I botched two flowfiles. *

 2. session.clone()generates a new flowfile with the identical uuid. I
don't think I want the result to be two flowfiles with the same uuid. I am
binding them together so I can associate them later using attribute
embedded-document. *Should I/How do I force cloning to acquire new
**uuid**s?*
>> This appears to actually be a bug in the mock framework. It *should*
have a unique uuid, and would in a running NiFi instance. Feel free to file
a Jira for that.
 3. A question on theory... *Wouldn't all of this cloning be expensive*
and I should just clone for one of the new files and then mangle the
original flowfile to become the other?
> session.clone() is not particularly expensive. It’s just creating a
new FlowFile object. It doesn’t clone the FlowFile’s contents.
>>> That said, it is probably more appropriate to call
session.create(flowFile), rather than session.clone(flowFile). It makes
little difference in practice but what you’re really doing is forking a
child, and that will come across more cleanly in the Provenance lineage
that is generated if using session.create(flowFile).
>>>
>>> Additional comments in code below.
>>>
>>>
 Thanks,
 Russ


 @Override
 public void onTrigger( final ProcessContext context, final
ProcessSession session ) throws ProcessException
 {
 FlowFile flowfile = session.get();

 if( flowfile == null )
 {
 context.yield();
>> No need to yield here. Let the framework handle the scheduling.
ProcessContext.yield() is meant for cases where you’re communicating with
some external service, for instance, and you know the service is
unavailable or rate limiting you or something like that. You can’t make any
progress, so tell NiFi to not bother wasting CPU cycles with this
Processor.
 return;
 }

 try
 {
 final String UUID = 

Re: [VOTE] Release Apache NiFi 1.12.0

[Otto Fowler]

 +1

 - release helper
- ran built instance
- imported template
- ran external python program ( dotifi ) which uses nipyapi + rest api



On August 14, 2020 at 14:02:00, Bryan Bende (bbe...@gmail.com) wrote:

+1 (binding)

Verified everything in the release helper and ran a secure instance with a
secure registry to test saving and importing some flows, all looks good.



On Fri, Aug 14, 2020 at 1:37 PM Mark Payne  wrote:

> +1 (binding)
>
> Performed full bull with contrib-check using:
>
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
> Java version: 1.8.0_242, vendor: AdoptOpenJDK, runtime:
> /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.15.6", arch: "x86_64", family: “mac"
>
> Build succeeded without issue. Started up fine. Verified several of the
> newly added features and all appears to be working as anticipated.
>
> Thanks
> -Mark
>
> > On Aug 14, 2020, at 10:35 AM, Joe Witt  wrote:
> >
> > Correction to the git commit id and checksums.
> >
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > 1.12.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1165
> >
> > The source being voted upon and the convenience binaries can be found
at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.12.0/
> >
> > A helpful reminder on how the release candidate verification process
> works:
> >
>
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-1.12.0-RC1
> > The Git commit ID is 303d6c59ba79b1eed425ee03b807948c7c400e99
> >
>
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=303d6c59ba79b1eed425ee03b807948c7c400e99
> >
> > Checksums of nifi-1.12.0-source-release.zip:
> > SHA256:
0abf5d5273e00b62790f143cdf878128b637e57e9fcf9e7978555d386eba5fff
> > SHA512:
> >
>
76dd941a4c6a5cb4a5683eb04b1144488d5bc38ab1f8d68f7f1a05b1d83ad716b0b4737fc4a7d64b4d68f1a3a355c6b2413b9a7d9860018127609a0a0a443c59

> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 335 issues were closed/resolved for this release:
> >
>
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12346778
> >
> > Release note highlights can be found here:
> >
>
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.0
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-1.12.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
> >
> > On Thu, Aug 13, 2020 at 1:31 PM Joe Witt  wrote:
> >
> >> Hello,
> >>
> >> I am pleased to be calling this vote for the source release of Apache
> NiFi
> >> 1.12.0.
> >>
> >> The source zip, including signatures, digests, etc. can be found at:
> >> https://repository.apache.org/content/repositories/orgapachenifi-1165
> >>
> >> The source being voted upon and the convenience binaries can be found
> at:
> >> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.12.0/
> >>
> >> A helpful reminder on how the release candidate verification process
> works:
> >>
> >>
>
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >>
> >> The Git tag is nifi-1.12.0-RC1
> >> The Git commit ID is 4d940bb151eb8d250b0319318b96d23c4a9819ae
> >>
> >>
>
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=4d940bb151eb8d250b0319318b96d23c4a9819ae
> >>
> >> Checksums of nifi-1.12.0-source-release.zip:
> >> SHA256:
a67ecb34f32cc1f070ebb006b6f05456a2ac663b12f708eadac67754194a6c63
> >> SHA512:
> >>
>
2e04235c4d49a76319af7756289ce11554a412bf5f7dcb6dc3915fc931df9f067142820c297e83bc36cb1079fb8384794ef457a20dd00568761eed6621701953

> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/joewitt.asc
> >>
> >> KEYS file available here:
> >> https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >> 335 issues were closed/resolved for this release:
> >>
> >>
>
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12346778
> >>
> >> Release note highlights can be found here:
> >>
> >>
>
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.0
> >>
> >> The vote will be open for 72 hours.
> >> Please download the release candidate and evaluate the necessary items
> >> including checking hashes, signatures, build
> >> from source, and test. Then please vote:
> >>
> >> 

Re: [ANNOUNCE] New Apache NiFi Committer Marton Szasz

[Otto Fowler]

 Belated congratulations!



On August 3, 2020 at 04:02:17, Arpad Boda (ab...@apache.org) wrote:

Apache NiFi community,

On behalf of the Apache NiFI PMC, I am very pleased to announce that Marton
has accepted the PMC's invitation to become a committer on the Apache NiFi
project. We greatly appreciate all of Marton's hard work and generous
contributions to the project. We look forward to continued involvement in
the project.

Marton had more than 100 contributions to MiNiFi C++ this year in various
areas: from Windows-specific memory leaks to nice new features and a lot of
code reviews. He also showed active presence on the mailing list helping
out the community.

Welcome and congratulations!

Re: [nifi] branch main created (now 239a2e8)

[Otto Fowler]

 What I did was to rebase -i on upstream main and force push.
Then in the PR reset the base to main

On July 7, 2020 at 16:01:28, Andy LoPresto (alopre...@apache.org) wrote:

The PRs should be mergeable to main automatically as I _moved_ the branch
rather than copying it, so unless there are new code conflicts, there
shouldn’t be a problem. There are tools from other developers [1] which can
automate the process of rebasing, but in general I don’t think this is
necessary. br/> <
For anyone who wishes to relabel their local working copy branches, you can
follow these instructions from [2]. br/> <
If someone has a local clone, then can update their locals <
https://twitter.com/xunit/status/1269881005877256192> like this:

$ git checkout master
$ git branch -m master main
$ git fetch
$ git branch --unset-upstream
$ git branch -u origin/main
$ git symbolic-ref refs/remotes/origin/HEAD refs/remotes/origin/main


I will send new updates when the other NiFi subprojects (Registry, MiNiFi
Java, MiNiFi C++, and FDS) have had their default branches renamed. br/> <
[1] https://github.com/ethomson/retarget_prs <
https://github.com/ethomson/retarget_prs>
[2]
https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx
<
https://www.hanselman.com/blog/EasilyRenameYourGitDefaultBranchFromMasterToMain.aspx>



Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69

> On Jul 7, 2020, at 9:14 AM, Matt Burgess  wrote:
> br/>> IMO that's a yes, the branch is created and noww main has more
commits
> than master and is the "source of truth". We should be using main as
> the default branch (the way it is in Github now), rebasing PRs atop
> main and changing the PRs in Github to have main as the base branch
> (so no mistakes are made when merging). With any luck master just kind
> of fades away, or perhaps at some point we just delete it.
> br/>> With some scripting we should be able to automaate such PR updates,
but
> personally it felt a little heavy-handed so I haven't written said
> script(s). Instead I'm just updating my own PRs and when I review
> someone's, I'm leaving a comment to please switch over to main as the
> base branch (both in the code and the PR).
> br/>> That's just my two cents though, happy to discuuss further. I think
we
> really just need to avoid updating the master branch from here on out
> rather than trying to keep them in sync, or even worse having them
> diverge each with new commits that aren't in the other.
> br/>> Regards, <
> Matt
> br/>> On Tue, Jul 7, 2020 at 12:06 PM Otto FFowler <
ottobackwa...@gmail.com> wrote:
>> br/>>> So, is that a yes??
>> br/>>> On July 7, 2020 at 10:53:59, Matt Burgeess (mattyb...@apache.org)
wrote:
>> br/>>> Thanks Mike!! I rebased against main since that's our default
branch
>> from now on, so should be ok now.
>> br/>>> On Tue, Jul 7, 2020 at 10:52 AM Mike Thhomsen <
mikerthom...@gmail.com>
>> wrote:
>>> br/>>>> I'll take it. <
>>> br/>>>> On Tue, Jul 7, 2020 at 9:57 AMM Matt Burgess <
mattyb...@apache.org> wrote:
>>> br/>>>>> I've got a PR [[1] that's approaching 1.5 years since the last
update,
>>>> but it should be good to go and fairly straightforward to test, any
>>>> takers? :) I just rebased it against the laster master branch, should
>>>> I be using main as of this point, or is there more work to be done
>>>> before we switch over?
>>>> br/>>>>> Thanks, <
>>>> Matt
>>>> br/>>>>> [[1] https://github.com/apache/nifi/pull/2718
>>>> br/>>>>> On Tue, Jul 7, 2020 at 5:48 AM Mike Thomsen <
mikerthom...@gmail.com>
>>>> wrote:
>>>>> br/>>>>>> Might also be aa good time to think about closing out old
PRs (like
>> age >
>>>>> 1.5 years).
>>>>> br/>>>>>> On Mon, Jul 6, 2020 at 10:49 PM Otto Fowler <
ottobackwa...@gmail.com>
>>>> wrote:
>>>>> br/>>>>>>> So, should we rebase our outstanding PR’s? Should all
outstanding
>>>> PR’s
>>>>>> get a note?
>>>>>> br/>>>>>>> On Julyy 6, 2020 at 17:57:25, Andy LoPresto (
alopre...@apache.org)
>>>> wrote:
>>>>>> br/>>>>>>> Apache Infra completed the change. Everything looks ok
from my end
>> but
>>>> as I
>>>>>> made the initial change, can someone else verify that their local
>> 

Re: [nifi] branch main created (now 239a2e8)

[Otto Fowler]

 So, is that a yes?

On July 7, 2020 at 10:53:59, Matt Burgess (mattyb...@apache.org) wrote:

Thanks Mike! I rebased against main since that's our default branch
from now on, so should be ok now.

On Tue, Jul 7, 2020 at 10:52 AM Mike Thomsen 
wrote:
>
> I'll take it.
>
> On Tue, Jul 7, 2020 at 9:57 AM Matt Burgess  wrote:
>
> > I've got a PR [1] that's approaching 1.5 years since the last update,
> > but it should be good to go and fairly straightforward to test, any
> > takers? :) I just rebased it against the laster master branch, should
> > I be using main as of this point, or is there more work to be done
> > before we switch over?
> >
> > Thanks,
> > Matt
> >
> > [1] https://github.com/apache/nifi/pull/2718
> >
> > On Tue, Jul 7, 2020 at 5:48 AM Mike Thomsen 
> > wrote:
> > >
> > > Might also be a good time to think about closing out old PRs (like
age >
> > > 1.5 years).
> > >
> > > On Mon, Jul 6, 2020 at 10:49 PM Otto Fowler 
> > wrote:
> > >
> > > > So, should we rebase our outstanding PR’s? Should all outstanding
> > PR’s
> > > > get a note?
> > > >
> > > > On July 6, 2020 at 17:57:25, Andy LoPresto (alopre...@apache.org)
> > wrote:
> > > >
> > > > Apache Infra completed the change. Everything looks ok from my end
but
> > as I
> > > > made the initial change, can someone else verify that their local
repo
> > is
> > > > good to go? Once that’s done, I’ll update the committer guide and
> > GitHub PR
> > > > template with the new language, and check the CI/CD for GitHub
Actions
> > too.
> > > > Thanks. br/> <
> > > >
> > > > Andy LoPresto
> > > > alopre...@apache.org
> > > > alopresto.apa...@gmail.com
> > > > He/Him
> > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > >
> > > > > On Jul 6, 2020, at 11:02 AM, Joe Witt  wrote:
> > > > > br/>> Looking into this we might be fine. We use thee github
'default
> > > > branch'
> > > > > which I think you'll get changed by the INFRA ticket.
> > > > > br/>> We then might want to look at
> > > > https://github.coom/ethomson/retarget_prs
> > > > > br/>> Thanks <
> > > > > br/>> On Mon, Jul 6, 2020 at 9:21 AM Joe Witt 

> > > > wrote:
> > > > > br/>>> Andy <
> > > > >> br/>>> I *think* we need to change the githubb actions config a
> > little
> > > > bit as
> > > > >> well. It defaults to master I believe and we can specify an
> > alternative
> > > > >> branch of 'main'. Maybe we just keep an eye on it and see if
there
> > is a
> > > > >> material difference. One example is the 'build passing'
indicator
> > comes
> > > > >> from master unless you override it (as I've read previously).
> > > > >> br/>>> Thanks <
> > > > >> br/>>> On Mon, Jul 6, 2020 at 9:18 AM Andy LooPresto <
> > > > alopre...@apache.org> wrote:
> > > > >> br/>>>> It appears that we don’t havee the ability to switch the
> > default
> > > > branch
> > > > >>> ourselves; I’ve filed
> > > > https://issues.apache.org/jira/browse/INFRA-20487
> > > > <
> > > > >>> https://issues.apache.org/jira/browse/INFRA-20487> requesting
> > Apache
> > > > >>> Infra takes care of that. Until that time, committers _should_
> > merge
> > > > the
> > > > >>> code to main but I will monitor to ensure we keep the branches
in
> > sync.
> > > > The
> > > > >>> history was moved, the only remaining step is GitHub being
aware
> > of the
> > > > >>> default branch change.
> > > > >>> br/>>>> br/>>>> Andy LoPresto <
> > > > >>> alopre...@apache.org
> > > > >>> alopresto.apa...@gmail.com
> > > > >>> He/Him
> > > > >>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
EF69
> > > > >>> br/>>>>> On Jul 4, 2020, at 11:57 AMM, Joey Frazee
> > > > 
> > > > >>> wrote:
> > > > >>>> br/>>>>> @@alopresto Do you want everyone individually pushing
to
> > both
> > > > master and
> > > > >>> main or are you intending to keep it in sync until it’s made
the
> > > > default?
> > > > >>>> br/>>>>> -joey <
> > > > >>>> On Jul 2, 2020, 6:39 PM -0500, alopre...@apache.org, wrote:
> > > > >>>>> This is an automated email from the ASF dual-hosted git
> > repository.
> > > > >>>>> br/>>>>>> alopresto pusheed a change to branch main
> > > > >>>>> in repository https://gitbox.apache.org/repos/asf/nifi.git.
> > > > >>>>> br/>>>>>> br/>>>>>> at 239a2e8 NIFFI-7513 Added custom DNS
> > resolution
> > > > steps to walkthrough
> > > > >>> (#4359)
> > > > >>>>> br/>>>>>> No new revisionns were added by this update.
> > > > >>>>> br/>>>> br/>>>> br/> <
> > > >
> >

Re: [nifi] branch main created (now 239a2e8)

[Otto Fowler]

 So, should we rebase our outstanding PR’s?  Should all outstanding PR’s
get a note?

On July 6, 2020 at 17:57:25, Andy LoPresto (alopre...@apache.org) wrote:

Apache Infra completed the change. Everything looks ok from my end but as I
made the initial change, can someone else verify that their local repo is
good to go? Once that’s done, I’ll update the committer guide and GitHub PR
template with the new language, and check the CI/CD for GitHub Actions too.
Thanks. br/> <

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69

> On Jul 6, 2020, at 11:02 AM, Joe Witt  wrote:
> br/>> Looking into this we might be fine. We use thee github 'default
branch'
> which I think you'll get changed by the INFRA ticket.
> br/>> We then might want to look at
https://github.coom/ethomson/retarget_prs
> br/>> Thanks <
> br/>> On Mon, Jul 6, 2020 at 9:21 AM Joe Witt 
wrote:
> br/>>> Andy <
>> br/>>> I *think* we need to change the githubb actions config a little
bit as
>> well. It defaults to master I believe and we can specify an alternative
>> branch of 'main'. Maybe we just keep an eye on it and see if there is a
>> material difference. One example is the 'build passing' indicator comes
>> from master unless you override it (as I've read previously).
>> br/>>> Thanks <
>> br/>>> On Mon, Jul 6, 2020 at 9:18 AM Andy LooPresto <
alopre...@apache.org> wrote:
>> br/ It appears that we don’t havee the ability to switch the default
branch
>>> ourselves; I’ve filed https://issues.apache.org/jira/browse/INFRA-20487
<
>>> https://issues.apache.org/jira/browse/INFRA-20487> requesting Apache
>>> Infra takes care of that. Until that time, committers _should_ merge
the
>>> code to main but I will monitor to ensure we keep the branches in sync.
The
>>> history was moved, the only remaining step is GitHub being aware of the
>>> default branch change.
>>> br/ br/ Andy LoPresto <
>>> alopre...@apache.org
>>> alopresto.apa...@gmail.com
>>> He/Him
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>> br/> On Jul 4, 2020, at 11:57 AMM, Joey Frazee

>>> wrote:
 br/> @@alopresto Do you want everyone individually pushing to both
master and
>>> main or are you intending to keep it in sync until it’s made the
default?
 br/> -joey <
 On Jul 2, 2020, 6:39 PM -0500, alopre...@apache.org, wrote:
> This is an automated email from the ASF dual-hosted git repository.
> br/>> alopresto pusheed a change to branch main
> in repository https://gitbox.apache.org/repos/asf/nifi.git.
> br/>> br/>> at 239a2e8 NIFFI-7513 Added custom DNS resolution
steps to walkthrough
>>> (#4359)
> br/>> No new revisionns were added by this update.
> br/ br/ br/> <

Re: Informing users fields are deprecated

[Otto Fowler]

 There are jira issues out there like :
https://issues.apache.org/jira/browse/NIFI-6043



On July 6, 2020 at 09:28:35, Bryan Bende (bbe...@gmail.com) wrote:

Do you mean a property on a processor/controller service?

If so, there is no visual indicator that I'm aware of, it would require
adding something to the PropertyDescriptor API like ".deprecated()".

Right now you can probably just make a statement in the description
explaining that it is deprecated and what should be used instead.


On Mon, Jul 6, 2020 at 9:13 AM Mike Thomsen  wrote:

> Do we have any visual cues that inform users that a field has been marked
> @Deprecated?
>
> Thanks,
>
> Mike
>

Re: [DISCUSS] rename master branch, look through code for other related issues

[Otto Fowler]

 As long as it isn’t renamed to zeek or something, I think we should change
it and not look back.



On June 18, 2020 at 19:05:38, Mike Thomsen (mikerthom...@gmail.com) wrote:

> As teammates and friends, it was an easy change, even if code was
involved. And I assume much easier than having the courage to ask for it.

Ironically, around the same time I had a colleague who was like the evil
opposite of that. Friend is the last word any of us would use to describe
him. He was a cautionary tale in why teams have to also maintain defense
mechanisms against toxic people who exploit empathy as a power play; it's a
common tactic of abusers/toxic people to make demands on people to change
their behavior to see how compliant they are. That former colleague, if you
got them talking about their views, could wax eloquent about tolerance,
inclusiveness, etc. and then without a hint of irony turn around and wage a
one man war on everyone else.

On Thu, Jun 18, 2020 at 11:53 AM Joey Frazee 

wrote:

> +1
>
> I’m repeating this from elsewhere but I was on a team 7 years ago where a
> teammate asked us to stop using master and slave terminology, even master
> alone, because it made them uncomfortable. I can’t estimate how common
that
> feeling is but this isn’t a theoretical exercise. As teammates and
friends,
> it was an easy change, even if code was involved. And I assume much
easier
> than having the courage to ask for it.
>
> I’d say it’s also important to note that “but that’s not the original
> intended word sense” doesn’t alleviate that alienating experience. While
> potentially a matter of fact of the intent for some uses, “I want to use
> that word” is pretty unfriendly stacked against “that makes me feel
> unwelcome”.
>
> Two guidelines from the code of conduct seem particularly apropos:
>
> - Be empathetic, welcoming, friendly, and patient
>
> - Be careful in the words that we choose
>
> AFAICT there’s not an escape hatch for code, tools, or effort.
>
> -joey
>
> On Jun 18, 2020, 10:05 AM -0500, Edward Armes ,
> wrote:
> > I agree with this, and maybe that is the potential the step forward
here
> > is: issue a statement is issued saying something like this is a complex
> > issue and instead of making changes that could cause further division
> > within the community we are looking for those that are interested to
help
> > form a constructive working group that will help influence and resolve
> all
> > of these issues in a positive way for all not only for project but also
> > within the wider group of apache projects.
> >
> > Edward
> >
> >
> >
> > On Thu, 18 Jun 2020, 11:17 u...@moosheimer.com, 
> wrote:
> >
> > > Language is always changing and the meaning of words is changing,
> > > sometimes positively and sometimes negatively.
> > > I think that now is time for change again and we should discuss the
use
> > > of phrases and meanings.
> > >
> > > Of course we should change "Master Branch" to "Main Branch".
> > > But I also think that we shouldn't just make quick changes because
it's
> > > opportune and hastily change a few words.
> > >
> > > An example: We could change Master/Slave to Leader/Follower. This may
> be
> > > a perfect choice for most people in the world.
> > > In German Leader is the English word for "Führer". And it is
precisely
> > > this word that we in Germany do not actually want to use for it.
> > >
> > > What I mean is that every country and every group (e.g. religion
etc.)
> > > has its own history and certain words or phrases are just not a
perfect
> > > choice.
> > > We should try to go the ethically correct way worldwide.
> > >
> > > This concerns the adaptation of current words and phrases with a view
> to
> > > all: in English, Indian, Chinese, German etc. but also for indigenous
> > > peoples, different religions etc.
> > > And cultural differences should also be taken into account.
> > >
> > > What I would wish for:
> > > Apache.org should set up an "Ethics Board". A group of people of
> > > different genders, all colors, religions and from different countries
> > > and cultures all over our world.
> > > This Ethics Board should find good and for no one discriminating
words
> > > or phrases for all the areas that stand out today as offensive.
> > >
> > > And it would be nice if not only computer scientists participated,
but
> > > also ethicists, philosophers, engineers, various religious people,
> > > chemists, biologists, physicists, sociologists, etc.
> > >
> > > And this Council should set binding targets for all projects.
> > >
> > > Am 18.06.2020 um 09:36 schrieb Pierre Villard:
> > > > > In my perspective this should be an issue for the entire
community.
> > > Being
> > > > > able to identify an issue that directly affects another person
but
> not
> > > > > one’s self is the definition of privilege. If I can look at how
> the use
> > > of
> > > > > these words in someone’s daily life or career impacts them
> negatively,
> > > > when
> > > > > the change would not harm me at 

Re: [ANNOUNCE] New NiFi PMC member Drew Lim

[Otto Fowler]

 Congratulations!

On May 29, 2020 at 15:55:19, Tony Kurc (tk...@apache.org) wrote:

NiFi Community,

On behalf of the Apache NiFi PMC, I am pleased to announce that Drew Lim
has accepted the PMC's invitation to join the Apache NiFi PMC.

Drew has been doing an amazing job for several years making NiFi more
approachable and usable. He's contributed to almost every part of NiFi;
going over and beyond to improve documentation and usability.

Please join us in congratulating and welcoming Drew to the Apache NiFi PMC.

Congratulations Drew!

Tony

Re: [ANNOUNCE] New NiFi PMC member Arpad Boda

[Otto Fowler]

 Congratulations

On May 29, 2020 at 15:48:10, Tony Kurc (tk...@apache.org) wrote:

NiFi Community,

On behalf of the Apache NiFi PMC, I am pleased to announce that Arpad Boda
has accepted the PMC's invitation to join the Apache NiFi PMC.

In addition to being a regular code contributor and engaged community
member with the project, (especially MiNiFi C++!) for quite some time,
Arpad has done the arduous task of being a release manager for both NiFi
Registry and MiNiFi C++, which is a challenging and detail oriented task.

Please join us in congratulating and welcoming Arpad to the Apache NiFi
PMC.

Congratulations Arpad!

Tony

Re: Nifi tutorials

[Otto Fowler]

 When you post them, you should tweet them and reference @apachenifi !

On May 26, 2020 at 15:48:53, Jeremy Dyer (jdy...@gmail.com) wrote:

Of course! I think that would be very much appreciated by everyone.

Get Outlook for iOS

From: Russell Bateman 
Sent: Tuesday, May 26, 2020 3:30:04 PM
To: dev@nifi.apache.org 
Subject: Re: Nifi tutorials

Certainly!

On 5/26/20 12:32 PM, Anuj Jain wrote:
> Hi Team,
>
> I am currently working on Apache-Nifi in my company from past 1 year.
> I am planning to launch a series of videos for Nifi learners about what I
> have learned in Nifi in last 1 year on youtube. Am I allowed to do that?
>
>
> Regards,
> Anuj Jain
>

Re: Camel & Nifi

[Otto Fowler]

 Great!  Thanks for the blog!

On May 26, 2020 at 02:15:24, ski n (raymondmees...@gmail.com) wrote:

Wrote a tech blog on using Camel with Apache NiFi:

https://medium.com/@raymondmeester/using-camel-and-nifi-in-one-solution-c7668fafe451

I thought I share it here,

Raymond

Re: Persisting/loading NIFI flows for Docker deployment

[Otto Fowler]

 No, that is what I mean, I would guess that you could spin up _n_ number
of nifi nodes with registry parameters that tell it what to load etc so you
wouldn’t have to manage the flow on disk

On April 8, 2020 at 14:21:52, Chris Sampson (
chris.samp...@naimuri.com.invalid) wrote:

If you mean having a way of telling a new nifi instance about a registry,
bucket and flow to load at startup, then I'd say yes that would be good. Or
even to be able to directly import the flow json that can currently be
exported from registry (or nifi).

Is this effectively what NiFi serverless can do at the minute? On my radar
to investigate but haven't got there yet.

I'm currently in the middle of coding something to pull flows from a
registry at startup, but using a mix of NiPyAPI and NiFi Toolkit (for
creating users/policies, reporting tasks & controllers, flows & parameter
contexts).

Our have I misunderstood your "silly question"?


Cheers,

Chris Sampson

On Wed, 8 Apr 2020, 18:17 Otto Fowler,  wrote:

> Can I ask a silly question? Would the ability for nodes to load flows
> from the registry make this easier? Would that even make sense?
>
> On April 8, 2020 at 12:52:19, Chris Sampson (
> chris.samp...@naimuri.com.invalid) wrote:
>
> When first starting with NiFi (I've always used it in container form), I
> looked at taking a similar approach with the flow.xml.gz, but quickly
moved
> away from that after reading various tales online about it not always
being
> very portable and/or people having issues keeping it in sync (remembering
> to extract it from the container while still running and persisting it
> externally, then injecting it again on startup *or* trying to use a bind
> mount and relying on the container always being run by the same node,
> etc.).
>
> Instead, we went with using a persistent volume for the flow.xml.gz
> directory (having changed that from it's default location via the
> nifi.properties settings), which allows the instance to persist its Flow
> between restarts (running this first in a Docker Swarm, then subsequently
> in Kubernetes). Externalisation of this and promotion between
environments
> (e.g. if you have Dev, Test, Prod, etc.) was first using Templates, which
> can be exported directly from and imported directly into the NiFi UI as
XML
> files (which you can, of course, store in a source control repository as
> desired, although I'd not look to do much in the way of Pull Request
> reviews using them).
>
> Templates are now deprecated in NiFi and instead the normally recommended
> approach would be to spin up a NiFi Registry instance, have your Flows
> within one or more Process Groups and version control those between NiFi
UI
> and NiFi Registry. The Flows can also be exported as JSON either from
NiFi
> itself or Registry and again stored in source control then re-imported
> later (sometimes I've found code reviews possible on such files, but only
> for relatively minor Flow changes). Even better would be to use Git as
the
> Flow Persistence layer within Registry, which can then send the Flow
> definitions straight to a Git repo - Registry can also rebuild from this
if
> it loses its stored metadata, etc.
>
> Using Registry allows for some reviews directly in the NiFi UI as it
tells
> you what changes have been made to a Flow definition before they're
> committed to Registry. But, of course, you've then got another container
to
> look at spinngin up and orchestrating in your stack (I'd say it's
certainly
> been worth the effort having spent a bit of time using the tools now).
>
>
> Hope that helps.
>
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
>
>
>
> On Wed, 8 Apr 2020 at 17:36, Kevin Telford 
> wrote:
>
> > Thank you Mike. This makes sense.
> >
> > How do you get the flow.xml.gz file? Do you have NiFi installed locally
> on
> > bare metal or do you develop also in a container? Initially I was
> thinking
> > of adding a custom volume to facilitate both getting and deploying the
> flow
> > file.
> >
> > Best,
> > Kevin
> >
> > On 2020/04/08 14:37:43, Mike Thomsen  wrote:
> > > What I've done in the past looks something like this:
> > >
> > > FROM apache/nifi:1.11.4
> > > COPY flow.xml.gz /opt/nifi/nifi-current/conf/flow.xml.gz
> > >
> > > And that's it. The obvious caveat is that you need to follow good
> > practices
> > > with ensuring that your flow and the way you setup the container can
> > > replicate the environment you built it on on your host.
> > >
> > > On Wed, Apr 8, 2020 at 10:34 AM Kevin Telford  >
> > > wrote:
> > >
> > > > Hi all – I have a two part question.
> >

Re: ConsumeEWS

[Otto Fowler]

 The ConsumeEWS processor has no special handling for multipart mime
messages.  It would need to be extended to have more control over the
output.  It looks like all you can do is specify headers to include or
ignore.

On April 8, 2020 at 13:02:21, Jisson Dennis (jissonden...@gmail.com) wrote:

Hi ,

I have a usecase to monitor an exchange mailbox.
Using ConsumeEWS I am able to bind the inbox and receive the email flow
file. However, the message body alone cannot be extracted by the custom
processors next to ConsumeEWS.
All the mime parts including the attachment are parsed into a single body
part.
It looks like the multipart boundary is missing from the ConsumeEWS raw
message.
My flow is ConsumeEWS --> ExtractBody(ExecuteStreamCommand) -->
ExtractEmailAttachment --> 

Any help to parse just the body would be much appreciated.
Thanks in advance.


Regards

Jisson Dennis

Re: Persisting/loading NIFI flows for Docker deployment

[Otto Fowler]

 Can I ask a silly question?  Would the ability for nodes to load flows
from the registry make this easier?  Would that even make sense?

On April 8, 2020 at 12:52:19, Chris Sampson (
chris.samp...@naimuri.com.invalid) wrote:

When first starting with NiFi (I've always used it in container form), I
looked at taking a similar approach with the flow.xml.gz, but quickly moved
away from that after reading various tales online about it not always being
very portable and/or people having issues keeping it in sync (remembering
to extract it from the container while still running and persisting it
externally, then injecting it again on startup *or* trying to use a bind
mount and relying on the container always being run by the same node,
etc.).

Instead, we went with using a persistent volume for the flow.xml.gz
directory (having changed that from it's default location via the
nifi.properties settings), which allows the instance to persist its Flow
between restarts (running this first in a Docker Swarm, then subsequently
in Kubernetes). Externalisation of this and promotion between environments
(e.g. if you have Dev, Test, Prod, etc.) was first using Templates, which
can be exported directly from and imported directly into the NiFi UI as XML
files (which you can, of course, store in a source control repository as
desired, although I'd not look to do much in the way of Pull Request
reviews using them).

Templates are now deprecated in NiFi and instead the normally recommended
approach would be to spin up a NiFi Registry instance, have your Flows
within one or more Process Groups and version control those between NiFi UI
and NiFi Registry. The Flows can also be exported as JSON either from NiFi
itself or Registry and again stored in source control then re-imported
later (sometimes I've found code reviews possible on such files, but only
for relatively minor Flow changes). Even better would be to use Git as the
Flow Persistence layer within Registry, which can then send the Flow
definitions straight to a Git repo - Registry can also rebuild from this if
it loses its stored metadata, etc.

Using Registry allows for some reviews directly in the NiFi UI as it tells
you what changes have been made to a Flow definition before they're
committed to Registry. But, of course, you've then got another container to
look at spinngin up and orchestrating in your stack (I'd say it's certainly
been worth the effort having spent a bit of time using the tools now).


Hope that helps.

*Chris Sampson*
IT Consultant
chris.samp...@naimuri.com



On Wed, 8 Apr 2020 at 17:36, Kevin Telford  wrote:

> Thank you Mike. This makes sense.
>
> How do you get the flow.xml.gz file? Do you have NiFi installed locally
on
> bare metal or do you develop also in a container? Initially I was
thinking
> of adding a custom volume to facilitate both getting and deploying the
flow
> file.
>
> Best,
> Kevin
>
> On 2020/04/08 14:37:43, Mike Thomsen  wrote:
> > What I've done in the past looks something like this:
> >
> > FROM apache/nifi:1.11.4
> > COPY flow.xml.gz /opt/nifi/nifi-current/conf/flow.xml.gz
> >
> > And that's it. The obvious caveat is that you need to follow good
> practices
> > with ensuring that your flow and the way you setup the container can
> > replicate the environment you built it on on your host.
> >
> > On Wed, Apr 8, 2020 at 10:34 AM Kevin Telford 
> > wrote:
> >
> > > Hi all – I have a two part question.
> > >
> > >
> > >
> > > I’d like to run NiFi inside a container in order to deploy to various
> > > environments. As far as I can tell, the flow.xml.gz file is the main
> > > “source” if you will, for a NiFi data flow.
> > >
> > > Q1) Is the flow.xml.gz file the “source” of a NiFi data flow, and if
> so, is
> > > it best practice to copy it to a new env in order to “deploy” a
> prebuilt
> > > flow? Or how best is this handled?
> > >
> > >
> > >
> > > Given that Q1 is true, my challenge then becomes somewhat
> Docker-specific…
> > >
> > > Situation:
> > >
> > > - In the Dockerfile we unzip the NiFi source (L62
> > > <
> > >
>
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/Dockerfile#L62
> > > >)
> > > and then create Docker volumes (L75
> > > <
> > >
>
https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/Dockerfile#L75
> > > >
> > > specifically for the conf dir). Once the container starts all the
> normal
> > > NiFi startup things happen, and
> /opt/nifi/nifi-current/conf/flow.xml.gz
> > > created.
> > >
> > > Complication:
> > >
> > > - In order to persist flow.xml.gz outside of the container, I would
> > > normally mount the /opt/nifi/nifi-current/conf directory, however in
> > > this
> > > case I cannot mount it on initialization because that will overwrite
> > > conf
> > > config files with whatever directory I bind it to (Docker container
> > > isolation ensures host -> container file precedence).
> > > - I could mount to a running container, but this is less ideal due
> to
> > > the various ways a 

Re: Reading the incoming flowfile "twice"

[Otto Fowler]

Oh, sorry I did not understand that you needed to do that




On March 31, 2020 at 11:22:20, Russell Bateman (r...@windofkeltia.com)
wrote:

Yes, I though of that, but there's no way to insert completing XML
structure into the input stream ahead of (). SAX will choke if
I just start feeding it the flowfile where I left off from copying up to
.

On 3/30/20 8:25 PM, Otto Fowler wrote:
> Can I ask why you would consume the whole stream when doing the non-sax
> part? If you consume the stream right up to the sax part ( the stream POS
> is at the start of the xml ) then you can just pass the stream to sax as
is
> can’t you?
>
>
>
>
> On March 30, 2020 at 16:23:27, Russell Bateman (r...@windofkeltia.com)
> wrote:
>
> If I haven't worn out my welcome, here is the simplified code that should
> demonstrate either that I have miscoded your suggestions or that the API
> doesn't in fact work as advertised. First, the output. The code, both
JUnit
> test and processor are attached and the files are pretty small.
>
> Much thanks,
> Russ
>
> This is the input stream first time around (before copying)
> ===
> * * * session.read( flowfile );
> Here's what's in input stream:
> **
> * *
> * This is the original document.*
> * *
> * *
> * 2016-06-28 13:23*
> * *
> * *
> * 1980-07-01*
> * 36*
> * *
> **
>
> And now, let's copy some of the input stream to the output stream
> =
> * * * flowfile = session.write( flowfile, new StreamCallback() ...
> Copying input stream to output stream up to ...
> The output stream has in it at this point:
> **
> * *
> * This is the original document.*
> * *
>
> [1. When we examine the output stream, it has what we expect.]
>
> After copying, can we reopen input stream intact and does outputstream
have
> what we think? 
> * * * flowfile = session.write( flowfile, new StreamCallback() ...
> Here's what's in input stream:
> **
> * *
> * This is the original document.*
> * *
>
> [2. The input stream as reported just above is truncated by exactly the
> content we did
> not copy to the output stream. We expected to see the entire,
> original file, but the
> second half is gone.]
>
> Here's what's in the output stream at this point:
> * (nothing)*
>
> [3. The content we copied to the output stream has disappeared. Does it
> disappear simply
> because we looked at it (printed it out here)?]
>
>
> On 3/29/20 5:05 AM, Joe Witt wrote:
>
> Russell
>
> I recommend writing very simple code that does two successive read/write
> operations on basic data so you can make sure the api work/as expected.
> Then add the xml bits.
>
> Thanks
>
> On Sun, Mar 29, 2020 at 5:15 AM Mike Thomsen 
>  wrote:
>
>
> If these files are only a few MB at the most, you can also just export
them
> to a ByteArrayOutputStream. Just a thought.
>
> On Sun, Mar 29, 2020 at 12:16 AM Russell Bateman
>  
> wrote:
>
>
> Joe and Mike,
>
> Sadly, I was not able to get very far on this. It seems that the extend
> to which I copy the first half of the contents of the input stream, I
> lose what comes after when I try to read again, basically, the second
> half comprising the and elements which I was
> hoping to SAX-parse. Here's code and output. I have highlighted the
> output to make it easier to read.
>
> ? <#>
> |try|
> |{|
> |||InputStream inputStream = session.read( flowfile );|
> |||System.out.println( ||"This is the input stream first time around
> (before copying to output stream)..."| |);|
> |||System.out.println( StreamUtilities.fromStream( inputStream ) );|
> |||inputStream.close();|
> |}|
> |catch||( IOException e )|
> |{|
> |||e.printStackTrace();|
> |}|
> |flowfile = session.write( flowfile, ||new| |StreamCallback()|
> |{|
> |||@Override|
> |||public| |void| |process( InputStream inputStream, OutputStream
> outputStream ) ||throws| |IOException|
> |||{|
> |||System.out.println( ||"And now, let's copy..."| |);|
> |||CxmlStreamUtilities.copyCxmlHeaderAndDocumentToOutput( inputStream,
> outputStream );|
> |||}|
> |} );|
> |try|
> |{|
> |||InputStream inputStream = session.read( flowfile );|
> |||System.out.println( ||"This is the input stream second time around
> (after copying)..."| |);|
> |||System.out.println( StreamUtilities.fromStream( inputStream ) );|
> |||inputStream.close();|
> |}|
> |catch||( IOException e )|
> |{|
> |||e.printStackTrace();|
> |}|
> |// ...on to SAX parser which dies because the input has been truncated
>
> to|
>
> |// exactly what was written out to the output stream|
>

Re: Reading the incoming flowfile "twice"

[Otto Fowler]

Can I ask why you would consume the whole stream when doing the non-sax
part? If you consume the stream right up to the sax part ( the stream POS
is at the start of the xml ) then you can just pass the stream to sax as is
can’t you?




On March 30, 2020 at 16:23:27, Russell Bateman (r...@windofkeltia.com)
wrote:

If I haven't worn out my welcome, here is the simplified code that should
demonstrate either that I have miscoded your suggestions or that the API
doesn't in fact work as advertised. First, the output. The code, both JUnit
test and processor are attached and the files are pretty small.

Much thanks,
Russ

This is the input stream first time around (before copying)
===
* * * session.read( flowfile );
  Here's what's in input stream:
**
*  *
*This is the original document.*
*  *
*  *
*2016-06-28 13:23*
*  *
*  *
*1980-07-01*
*36*
*  *
**

And now, let's copy some of the input stream to the output stream
=
* * * flowfile = session.write( flowfile, new StreamCallback() ...
  Copying input stream to output stream up to ...
  The output stream has in it at this point:
**
*  *
*This is the original document.*
*  *

[1. When we examine the output stream, it has what we expect.]

After copying, can we reopen input stream intact and does outputstream have
what we think? 
* * * flowfile = session.write( flowfile, new StreamCallback() ...
  Here's what's in input stream:
**
*  *
*This is the original document.*
*  *

[2. The input stream as reported just above is truncated by exactly the
content we did
  not copy to the output stream. We expected to see the entire,
original file, but the
  second half is gone.]

  Here's what's in the output stream at this point:
* (nothing)*

[3. The content we copied to the output stream has disappeared. Does it
disappear simply
because we looked at it (printed it out here)?]


On 3/29/20 5:05 AM, Joe Witt wrote:

Russell

I recommend writing very simple code that does two successive read/write
operations on basic data so you can make sure the api work/as expected.
Then add the xml bits.

Thanks

On Sun, Mar 29, 2020 at 5:15 AM Mike Thomsen 
 wrote:


If these files are only a few MB at the most, you can also just export them
to a ByteArrayOutputStream. Just a thought.

On Sun, Mar 29, 2020 at 12:16 AM Russell Bateman
 
wrote:


Joe and Mike,

Sadly, I was not able to get very far on this. It seems that the extend
to which I copy the first half of the contents of the input stream, I
lose what comes after when I try to read again, basically, the second
half comprising the and elements which I was
hoping to SAX-parse. Here's code and output. I have highlighted the
output to make it easier to read.

? <#>
|try|
|{|
|||InputStream inputStream = session.read( flowfile );|
|||System.out.println( ||"This is the input stream first time around
(before copying to output stream)..."| |);|
|||System.out.println( StreamUtilities.fromStream( inputStream ) );|
|||inputStream.close();|
|}|
|catch||( IOException e )|
|{|
|||e.printStackTrace();|
|}|
|flowfile = session.write( flowfile, ||new| |StreamCallback()|
|{|
|||@Override|
|||public| |void| |process( InputStream inputStream, OutputStream
outputStream ) ||throws| |IOException|
|||{|
|||System.out.println( ||"And now, let's copy..."| |);|
|||CxmlStreamUtilities.copyCxmlHeaderAndDocumentToOutput( inputStream,
outputStream );|
|||}|
|} );|
|try|
|{|
|||InputStream inputStream = session.read( flowfile );|
|||System.out.println( ||"This is the input stream second time around
(after copying)..."| |);|
|||System.out.println( StreamUtilities.fromStream( inputStream ) );|
|||inputStream.close();|
|}|
|catch||( IOException e )|
|{|
|||e.printStackTrace();|
|}|
|// ...on to SAX parser which dies because the input has been truncated

to|

|// exactly what was written out to the output stream|


Output of above:

This is the input stream first time around (before copying to output
stream)...

   
 This is the original document.
   
   
 2016-06-28 13:23
   
   
 1980-07-01
 36
   


And now, let's copy...
This is the input stream second time around (after copying)...

   
 This is the original document.
   
And now, we'll go on to the SAX parser...
  This is the original document. 
[pool-1-thread-1] ERROR [...] SAX ruleparser error:
org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 14; XML
document structures must start and end within the same entity.


I left off the code that prints, "And now, we'll go on to the SAX
parser..." It's in the next flowfile = session.write( ... ). I have unit
tests that verify the good functioning of
copyCxmlHeaderAndDocumentToOutput(). The SAX error occurs because the
"file" is truncated; SAX finds the first "half" just fine, but there is
no second "half". If I comment out copying from input stream to output
stream, the error doesn't occur--the whole document is 

Re: Make invokehttp to process faster to process more than 20k records

[Otto Fowler]

Do you have some other client calling that proves your API endpoint isn’t
the bottleneck? Sorry if I missed that part and you already provided that.




On March 18, 2020 at 09:29:27, Midhun Mohan (midhun.mo...@esginc.us) wrote:

Hey Mike , I meant like when I try posting that many records it is taking
bit time. Just checking did you tweak around the thread count and what is
your instance size. CPU and RAM

On Wed, 18 Mar 2020 at 18:57, Mike Thomsen  wrote:

> By setting the HTTP verb to POST in InvokeHTTP.
>
> On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan 
> wrote:
>
> > How did you post 50k flowfiles, that is what am looking for
> >
> > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen 
> wrote:
> >
> > > That's probably a lot of the issue, especially if it's an evented
> service
> > > like a Node service running with one thread. I just did a simple test
> by
> > > posting 50k flowfiles with 4k of JSON in them to an Express hello
world
> > app
> > > and it was able to respond to 50k flowfiles in under 10s using
> InvokeHttp
> > > with only 3 threads. All on my MacBook Pro.
> > >
> > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan 
> > > wrote:
> > >
> > > > Endpoint does execute db query in a transaction, yes it is in the
> same
> > > > setup but different cluster
> > > >
> > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, 
> > > wrote:
> > > >
> > > > > How is the endpoint implemented and what does it do? Also, is it
> > > located
> > > > in
> > > > > the same data center as the EC2 instance running NiFi?
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <
> midhun.mo...@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Yeah the endpoint which am sending right now has plenty of
> > resources
> > > > > > available. Only thing is I need to send more records
> > > > > >
> > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky,  >
> > > > wrote:
> > > > > >
> > > > > > > I have not test throughput of InvokeHTTP so am not sure what
> the
> > > > > maximum
> > > > > > > is, but can give some general guidance.
> > > > > > >
> > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> figure
> > > out
> > > > > > system
> > > > > > > resources usage while your flow is running.
> > > > > > >
> > > > > > > You should be able to increase both your nifi count by more
and
> > > > adjust
> > > > > > your
> > > > > > > processors until you are limited by your system resources.
> > > > > > >
> > > > > > > Have you verified the endpoint you are sending to is not the
> > bottle
> > > > > neck?
> > > > > > >
> > > > > > > Chad
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > midhun.mo...@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Size of file around kb not more than that,
> > > > > > > > How can I find the bottle neck,
> > > > > > > >
> > > > > > > > Yes I adjust the count to 11 then all other processors
> stopped
> > > but
> > > > > > > > processing improved.
> > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > >
> > > > > > > > I just need a way to post more records to endpoint to make
it
> > > > > realtime
> > > > > > > >
> > > > > > > > Hope i was able to give more details
> > > > > > > >
> > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > czobri...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Midhun,
> > > > > > > > > A little more information would help.
> > > > > > > > >
> > > > > > > > > What size files are you sending?
> > > > > > > > > Have you looked at resource usage to see what the
> bottleneck
> > > is?
> > > > > > > > > Did you adjust your nifi system thread count from the
> > hamburger
> > > > > menu?
> > > > > > > > > How many threads are running total for nifi? It's the
> number
> > in
> > > > the
> > > > > > top
> > > > > > > > > left of the screen.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chad
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > midhun.mo...@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > >
> > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > midhun.mo...@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I have a queue which will be filling up with realtime
> > > records
> > > > > of
> > > > > > > > around
> > > > > > > > > > > 20k records. Which is posting to an endpoint using
> > > Invokehttp
> > > > > > > > > processor.
> > > > > > > > > > >
> > > > > > > > > > > currently the average throughput is for 10k records
it
> > > takes
> > > > > > around
> > > > > > > > 20
> > > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > > I Increased the concurrent thread to larger number
> which
> 

Re: [DISCUSS] Advanced search capabilities

[Otto Fowler]

+1 for the “bring your own golden hammer” approach




On February 24, 2020 at 11:46:14, Mike Thomsen (mikerthom...@gmail.com)
wrote:

Another thing I forgot to throw out there was that you have an issue of
latency if you use Janus or Neo4j. Lucene will almost certainly have
substantially lower latency for updating and querying the provenance data
if you were to do a bake off between the two to power a provenance
repository.

That said, if you care more about being able to query with Cypher or
Gremlin than having raw performance, you could write a custom provenance
repository. They are pluggable.

On Sat, Feb 22, 2020 at 7:00 AM Martin Ebert  wrote:

> Hi Mike,
> that is a fair point. You would actually raise the minimum requirements
of
> Nifi accordingly if you wanted to use a graph. As an additional
> application, as we are currently planning, Neo4j is nevertheless a good
> choice and there is nothing to be said against making it open source. The
> open source version of Neo4j should be sufficient for this.
>
>
> Mike Thomsen  schrieb am Sa., 22. Feb. 2020,
> 02:36:
>
> > Martin,
> >
> > In theory, a graph database would be superior here. Absolutely. In
> > practice, none of the tech out there is better than the current
> > Lucene-based approach in terms of ease of development and integration
and
> > low memory footprint. Adding Neo4J or JanusGraph would cause a huge
jump
> in
> > the minimum requirements to run NiFi. Possibly to the point where Xms
and
> > Xmx would have to start at 2GB for people getting started.
> >
> > It's been a long time since I've played with Atlas and the Atlas
> > integration, but if that doesn't work you can build in support for
Cypher
> > and Gremlin by adding -Pinclude-graph to a 1.10 or 1.11 build. In 1.10,
> one
> > of the NARs was overlooked in that profile, so you'd need to add it
back
> to
> > the profile. That was fixed in 1.11. The ExecuteGraphQuery processor
will
> > allow you to execute Cypher or Gremlin commands/scripts depending on
> which
> > controller service/driver you configure.
> >
> > On Fri, Feb 21, 2020 at 6:42 PM Martin Ebert 
> wrote:
> >
> > > We still think about building a graph based search (Neo4j) in top of
> > NiFi.
> > > Would be also fantastic to have it within NiFi.
> > >
> > > There are plenty of examples
> > >
> > >
> >
>
https://blog.grandstack.io/using-neo4js-full-text-search-with-graphql-e3fa484de2ea
> > > From the idea it could go in this direction - of course much more
> > > rudimentary. Then one would have the possibility to have only the
> results
> > > displayed as text or to find out exploratory connections (graph
> layout).
> > > The built-in data lineage function of NiFi would also benefit from
the
> > > power of Neo4j.
> > >
> > > Simon Bence  schrieb am Fr., 21. Feb. 2020,
> > > 19:00:
> > >
> > > > Dear Community,
> > > >
> > > > In my project, I do use relatively high number of processors and
> > process
> > > > groups. The current search function on the NiFi UI has no
> > capabilitites
> > > to
> > > > narrow the results based on the group, which would make the results
> > more
> > > > relevant, so I would like to propose a possible solution. Please if
> you
> > > > have any comment on this, do not hesitate to share it.
> > > >
> > > > The general approach would be to keep the current text box and
extend
> > the
> > > > server side capabilities to process search query in the similar
> manner
> > > for
> > > > example the Google search behaves.This extensions I would call
> > "filters".
> > > > For now I am interested in the ones I will mention below, but I
> think,
> > it
> > > > is only a matter of small work for further extend the solution with
> > > further
> > > > ones.
> > > >
> > > > In order to distinguish the filters from the rest of the search
> query,
> > I
> > > > propose to put them at the beginning of the query and use the
> > > > [a-zA-Z0-9\.]{1..n}\:[a-zA-Z0-9\.]{1..n} format. For example a
filter
> > > might
> > > > look the following: lorem:ipsum
> > > >
> > > > Adding this, the search query should look like the following:
> > > >
> > > > filter1:value filter2:value rest of the query
> > > >
> > > > As for processing the filters, I suggest the following behaviour:
> > > >
> > > > - Without filters the current behaviour should be kept
> > > > - Everything after the filters should be handled as the search term
> > > > - After the first "non filter word", anything should be considered
as
> > > part
> > > > of the search term (meaning: to keep the text parsing simple, I
would
> > not
> > > > go in the direction to support filters at the end of the query,
etc.)
> > > > - The ordering of the filters should have no effect on the result
> > > > - Filter duplications should be eliminated
> > > > - In case a filter appears multiple times in the query, the first
> > > occasion
> > > > will be used
> > > > - Unknown filters should be ignored
> > > > - Only adding filters will not end up with result, at least one
> > 

Re: [VOTE] Release Apache NiFi 1.11.3

[Otto Fowler]

+1

Followed the new confluence instructions @
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

ran app and simple flow
verified documentation open and links work



On February 21, 2020 at 22:21:41, Joe Witt (joew...@apache.org) wrote:

Hello,

I am pleased to be calling this vote for the source release of Apache NiFi
1.11.3.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1158

The source being voted upon and the convenience binaries can be found at:
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.11.3/

A helpful reminder on how the release candidate verification process works:
https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate

The Git tag is nifi-1.11.3-RC1
The Git commit ID is 10b509cf5bb88b12b0dd497543291f3c49a9903f
https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=10b509cf5bb88b12b0dd497543291f3c49a9903f

Checksums of nifi-1.11.3-source-release.zip:
SHA256: b2544719a8de02892b3436f1e6084b9b6011bf706a183fa2f059a9961634a7bd
SHA512:
353671e06dd60e186d2a9a29eb9b7c2d183757f6a0f76801b4ecb5d87aab99e701b9ebde932ee414d4b304afd2bd1d0a0e574906a385cfafc8de36cc47c2d0a3


Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

14 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12347022

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.3

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test. Then please vote:

[ ] +1 Release this package as nifi-1.11.3
[ ] +0 no opinion
[ ] -1 Do not release this package because...