Re: [VOTE] Release Apache NiFi 1.10.0 (rc3)
+1 binding. Tested the graph flow (added the include-graph profile) and a dockerized cluster. LGTM. On Tue, Oct 29, 2019 at 1:32 PM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > nifi-1.10.0. > > As they say 'third time's a charm'. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1151 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.10.0/ > > The Git tag is nifi-1.10.0-RC3 > The Git commit ID is b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > > Checksums of nifi-1.10.0-source-release.zip: > SHA256: e9b0a14b3029acd69c6693781b6b6487c14dda12676db8b4a015bce23b1029c1 > SHA512: > > b07258cbc21d2e529a1aa3098449917e2d059e6b45ffcfcb6df094931cf16caa8970576555164d3f2290cfe064b5780ba1a8bf63dad04d20100ed559a1cfe133 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 384 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.10.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Minifi C2 - connection to secured NIFI API
Hi All, I'm trying to setup Minifi C2 with a connection to the Nifi API to pull templates and distribute them to MINIFI nodes. In my case the NIFI server is running in certificate / SSL auth mode (using the NIFI toolkit for testing). Is it possible to get C2 to connect to the API in this case? I have been able to configure C2 to require ssl connections to it, but cannot find a way to get it to authenticate to NIFI in order to access the API. Any help would be appreciated, Ali This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation. Refer any FOIA queries to ncscinfo...@ncsc.gov.uk. All material is UK Crown Copyright (c)
Re: PULL ProvenanceEvent
Joe,
It is hard to say how much value transit URI would bring to clarify a RECEIVE.
For example a RECEIVE with transit URI of https: could be either a
GetHTTP (i.e. active) or ListenHTTP (i.e. passive)
but your idea of "a metadata item specifying active vs passive" is a very
clever way to make this work with mimimal disruptions.
My understanding of this is that the current receive() calls in
ProvenanceReporter [1] will remain the same, but news ones will be added with a
boolean parameter reflecting if the receive is active or passive.
This will allow the current list of Provenance Events [2] to remain the same.
So third party/custom processors can continue working as is
Does this sound like what you are thinking?
[1]
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46
[2]
https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java
Thanks,
Nissim
On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt
wrote:
Nissim
I like the idea to introduce a more refined type of event for how data is
brought into nifi (active - PULL, passive - RECEIVE).
That said it might be sufficient to simply have this distinction be on the
"RECEIVE" event as a metadata item specifying active vs passive. The
protocol utilized as mentioned in the transport URI should clarify this
though.
In short - i think there is a way here that is all opt-in for existing
users and components.
Thanks
On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman
wrote:
> Adam,
> good points...
> I missed a step in explaining the use case where Provenance Events is
> incomplete...
> Where the second nifi does a GetSFTP from the *filesytem* that the first
> nifi is located on
> So the second nifi currently sends a RECEIVE event, but there is no
> corresponding SEND event from the first nifi (nor should there be)
> If the second nifi sent a PULL event, it would be easier for a system
> overseer to know that there should be no corresponding SEND event
>
> Currently send/receive works well when nifi 1 does a PostHTTP and nifi 2
> does a ListenHTTP, but not in the case above.
>
> The ERROR case you mention is a nice point as well, although not my
> specific issue at the moment.
> Thanks,
> Nissim
>
>
>
>
>
> On Monday, October 28, 2019, 11:52:57 PM EDT, Adam Taft <
> a...@adamtaft.com> wrote:
>
> > But a flowfile that was PULLed by the second nifi (from the first nifi)
> will not necessarily have any provenance event generated by the first nifi.
>
> Isn't this the fault of the first NiFi to fail to emit a SEND event in
> response to the second NiFi's request? In this scenario, shouldn't the
> send/receive pair be:
> NiFi-1 [SEND] :: NIFI-2 [RECEIVE]?
>
> What you describe is an odd use case for NiFi. NiFi is usually not in the
> business of acting as a file server daemon in order to "send" flowfiles to
> other systems. As you mention, HandleHttpResponse may be a lone wolf
> example processor which generates a SEND event whose input originates from
> a "listener". [1] The other ListenXYZ processors generally issue RECEIVE
> events because they are receiving bytes, not generating them.
>
> Are there other processors in question? Something custom? Or is this
> related to site-to-site transfers?
>
> I still kind of question the motive of a provenance event pair that is
> trying to establish "who called who first". Honestly just trying to
> understand the use case where a matching SEND/RECEIVE pair doesn't give you
> what you need.
>
> The only thing I could see would be a processor that asks for data, but
> then doesn't receive it due to some error condition. In this case, adding
> some sort of ERROR event might be useful. "I attempted to retrieve data
> from ${uri}, but the transfer failed because of ${error condition}". That
> way, GetXYZ processors could report an error to provenance instead of as a
> bulletin.
>
> If the problem is related to a processor or the framework itself not
> generating an event, can we just fix that function to emit SEND in the
> scenario that you describe? Changing the provenance model itself (beyond
> possibly adding an ERROR event) feels like it would be the last scenario to
> consider.
>
> Thanks,
> Adam
>
> [1]
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/HandleHttpResponse.java#L191
>
>
>
>
> On Mon, Oct 28, 2019 at 4:47 PM Nissim Shiman
> wrote:
>
> > Adam,
> > I believe there is a need for more detailed ProvenanceEvents.
> > A use case would be a customer that is trying to track data passed
> between
> > two nifi's and trying to match up SENDs and RECEIVEs
> >
> > So a flowfile that has a SEND event on the first nifi should have a
> > RECEIVE event on the second nifi.
> > But a flowfile that was PULLed by the second nifi (from the first nifi)
> >
Need Advice on Nifi Cluster Using Separate Storage for Provenance Repo, Flowfile Repo and Content Repo.
Hi, My Name is Rifat. I am a Software Engineer at ESPN/Disney. I have been using Nifi for almost one year now and We have a 10 Node Nifi Cluster setup in our production environment. As per the best practices document: https://community.cloudera.com/t5/Community-Articles/NiFi-Sizing-Guide-Deployment-Best-Practices/ta-p/246781 I would want to have 5 separate Repos for Content Repo, 5 for Provenance Repo and 1 for Flowfile Repo per node. I need your expert advice on whether using EBS or EFS is the best approach to achieve this goal. I already tried EFS and I saw some problems with Load Balancing since I mounted an EFS Volume per partition(that's 11 EFS partitions per node). This is from the Response I got from AWS after raising a support ticket with them. Hello Rifat, Thank you for contacting AWS Premium Support. I am not familiar with Apache NiFi, as this is a third-party software not covered by our support policy [1]. That been said, I had a look at its documentation [2] and some related links, and it doesn't seem to me that these repositories are meant to be on shared storage accessible by all nodes. If you look at the NiFi Architecture section, it's suggested that this data is stored locally on each node, and the guidelines in the link you provided us with are aligned with that principle. I did find some connection between NiFi and Hadoop Distributed File System (HDFS), which has some fundamental differences to EFS, however it doesn't seem to have any relation with these repositories. While EFS itself provides strong durability and availability guarantees, the NFS protocol is meant to provide weaker cache coherence among its clients as a trade-off for higher performance. Characteristics such as Attribute Caching, Directory Entry Caching, Asynchronous writes, and the differences in how file timestamps are maintained lead to discrepancies in how each node sees data, potentially impacting clustered applications expecting strong consistency. You'll find a good write-up on that in the Linux NFS documentation [3], section "Data and Metadata Coherence". To see if one of these characteristics are causing the issue, I advise you to append 'sync' and 'noac' as mount options for all EFS resources in all nodes; the first one will cause all write I/O to become synchronous, and the second one will disable Attribute and Directory Entry caching. If that helps resolve the issues you are seeing, we'll know that NiFi is expecting strong cache coherence. However, you'll need to evaluate if the performance penalty of mounting with these options is bearable. It may be possible that EBS or even Instance Store are better options to host these repositories, provided that you understand the differences in performance and durability between the two. On a side note, you are missing a few of the recommended mount options for EFS. Although I don't expect them to cause an immediate impact for the issue described in this support case, it's a good idea to implement them to avoid other issues. Please check here [4] for details. Regarding your question on how to enable communication between directories that are mounted on a different EFS, this whole idea of inter-EFS communication does not apply. EFS is a file system, and there's no exchange of data between separate EFS resources; the only "communication" in that sense would be moving data from one EFS to another, which can be done within an instance having both file systems mounted. I believe that at this stage, testing the solution with the proposed mount options above is a good course of action to isolate the problem. With regard to your comment on logging into these machines and reading the contents of /var/log/nifi, please note that Support personnel is not allowed under any circumstances to access customer's instances. At this stage, I believe that these logs are not required for this case. To summarise, my first advice is that you seek advice from NiFi experts on whether using a distributed file system such as EFS to host cluster node's repositories is a valid approach. If using Cloudera Flow Management, you should be able to receive support from Cloudera, otherwise the NiFi Community is an option [6]. The second advice is to test EFS mounted with 'sync' and 'noac' to see if it helps resolve the issue; if the performance penalty is unbearable, consider switching repositories to EBS or Instance Store volumes. If you have questions on the above, please let me know. Please let me know the Best Approach to take to solve this problem. Best Regards, Rifat
Java 11 Compilation
While building 1.10.0-rc3, I wanted to experiment with the compilation and runtime variants using Java 8 and Java 11. The summary of this experiment was: Comp: Java 8 Run: Java 8 => SUCCESS Comp: Java 8 Run: Java 11 => SUCCESS Comp: Java 11 Run: Java 8 => FAILURE Comp: Java 11 Run: Java 11 => SUCCESS As introduced in JEP-247 [1], starting with Java 9, javac has the ability to compile to an older Java platform. This was not possible previously without having multiple JDKs installed and specifying the '-bootclasspath' option in javac for your target. The newly introduced "--release" option to Java 9+ javac allows you to specify the target version using the documented API for that platform. This means replacing the '-source' and '-target' parameters with the '--release' option instead. In maven, that manifests itself as a configuration option to the maven-compiler-plugin, which is the 'maven.compiler.release' property. [2] The discussion here would be consideration for using the "release" option as opposed to our current setup which uses "source" and "target". The benefit would be that all the deployment scenarios (above) could result in success. The downside though is that, without other changes, Java 11 would be required to compile NiFi. It would still target Java 8 as a supported runtime (specifying "release=8"), but you'd have to build/compile with Java 11. I don't think this should be a problem, but it's worth discussion. Thanks, Adam [1] https://openjdk.java.net/jeps/247 [2] https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#release
Re: [VOTE] Release Apache NiFi 1.10.0 (rc3)
+1 (binding) Signatures verified. Hashes verified. Tests pass, source builds cleanly. I used both Java 11 & Java 8 to build. I did run into a problem compiling with Java 11 and running with Java 8. I don't believe this was a goal of the Java 11 compatibility changes, so nothing unexpected about this. But it's possibly worth discussion in another thread (which I'll send out). The convenience binary was compiled with Java 8, so no problems with compatibility either way. On Tue, Oct 29, 2019 at 11:32 AM Joe Witt wrote: > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > nifi-1.10.0. > > As they say 'third time's a charm'. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1151 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.10.0/ > > The Git tag is nifi-1.10.0-RC3 > The Git commit ID is b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > > Checksums of nifi-1.10.0-source-release.zip: > SHA256: e9b0a14b3029acd69c6693781b6b6487c14dda12676db8b4a015bce23b1029c1 > SHA512: > > b07258cbc21d2e529a1aa3098449917e2d059e6b45ffcfcb6df094931cf16caa8970576555164d3f2290cfe064b5780ba1a8bf63dad04d20100ed559a1cfe133 > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 384 issues were closed/resolved for this release: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993 > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0 > > The vote will be open for 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.10.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because... >
Re: Java 11 Compilation
Adam Interesting. Id say though that where we are now for nifi 1.x is perfect. That matrix you shared as currently working seems ideal. For nifi 2.x we cut bait on Java 8 and go with latest stable Java at that time (11, 13) thanks On Wed, Oct 30, 2019 at 12:51 PM Adam Taft wrote: > While building 1.10.0-rc3, I wanted to experiment with the compilation and > runtime variants using Java 8 and Java 11. The summary of this experiment > was: > > Comp: Java 8 Run: Java 8 => SUCCESS > Comp: Java 8 Run: Java 11 => SUCCESS > Comp: Java 11 Run: Java 8 => FAILURE > Comp: Java 11 Run: Java 11 => SUCCESS > > As introduced in JEP-247 [1], starting with Java 9, javac has the ability > to compile to an older Java platform. This was not possible previously > without having multiple JDKs installed and specifying the '-bootclasspath' > option in javac for your target. > > The newly introduced "--release" option to Java 9+ javac allows you to > specify the target version using the documented API for that platform. > This means replacing the '-source' and '-target' parameters with the > '--release' option instead. > > In maven, that manifests itself as a configuration option to the > maven-compiler-plugin, which is the 'maven.compiler.release' property. [2] > > The discussion here would be consideration for using the "release" option > as opposed to our current setup which uses "source" and "target". The > benefit would be that all the deployment scenarios (above) could result in > success. > > The downside though is that, without other changes, Java 11 would be > required to compile NiFi. It would still target Java 8 as a supported > runtime (specifying "release=8"), but you'd have to build/compile with Java > 11. I don't think this should be a problem, but it's worth discussion. > > Thanks, > Adam > > [1] https://openjdk.java.net/jeps/247 > [2] > > https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#release >
Re: [VOTE] Release Apache NiFi 1.10.0 (rc3)
+1 (binding) Verified signatures, hashes, build, etc. Tested both standalone and clustered and secure and unsecured. Thanks for RMing Joe! Matt On Wed, Oct 30, 2019 at 1:51 PM Adam Taft wrote: > +1 (binding) > > Signatures verified. > Hashes verified. > Tests pass, source builds cleanly. > I used both Java 11 & Java 8 to build. > > I did run into a problem compiling with Java 11 and running with Java 8. I > don't believe this was a goal of the Java 11 compatibility changes, so > nothing unexpected about this. But it's possibly worth discussion in > another thread (which I'll send out). The convenience binary was compiled > with Java 8, so no problems with compatibility either way. > > On Tue, Oct 29, 2019 at 11:32 AM Joe Witt wrote: > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > NiFi > > nifi-1.10.0. > > > > As they say 'third time's a charm'. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1151 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.10.0/ > > > > The Git tag is nifi-1.10.0-RC3 > > The Git commit ID is b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > > > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > > > > Checksums of nifi-1.10.0-source-release.zip: > > SHA256: e9b0a14b3029acd69c6693781b6b6487c14dda12676db8b4a015bce23b1029c1 > > SHA512: > > > > > b07258cbc21d2e529a1aa3098449917e2d059e6b45ffcfcb6df094931cf16caa8970576555164d3f2290cfe064b5780ba1a8bf63dad04d20100ed559a1cfe133 > > > > Release artifacts are signed with the following key: > > https://people.apache.org/keys/committer/joewitt.asc > > > > KEYS file available here: > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > 384 issues were closed/resolved for this release: > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993 > > > > Release note highlights can be found here: > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0 > > > > The vote will be open for 72 hours. > > Please download the release candidate and evaluate the necessary items > > including checking hashes, signatures, build > > from source, and test. Then please vote: > > > > [ ] +1 Release this package as nifi-1.10.0 > > [ ] +0 no opinion > > [ ] -1 Do not release this package because... > > >
Re: Java 11 Compilation
Right, I agree with your perspective. Just note, however, that this stance will require the RM to create the 1.x convenience binary with Java 8 only. It will be incumbent on the RM to ensure they build with Java 8, because if they accidentally build with Java 11, the binary distribution won't run on Java 8. Adam On Wed, Oct 30, 2019 at 11:56 AM Joe Witt wrote: > Adam > > Interesting. Id say though that where we are now for nifi 1.x is perfect. > That matrix you shared as currently working seems ideal. > > For nifi 2.x we cut bait on Java 8 and go with latest stable Java at that > time (11, 13) > > thanks > > On Wed, Oct 30, 2019 at 12:51 PM Adam Taft wrote: > > > While building 1.10.0-rc3, I wanted to experiment with the compilation > and > > runtime variants using Java 8 and Java 11. The summary of this > experiment > > was: > > > > Comp: Java 8 Run: Java 8 => SUCCESS > > Comp: Java 8 Run: Java 11 => SUCCESS > > Comp: Java 11 Run: Java 8 => FAILURE > > Comp: Java 11 Run: Java 11 => SUCCESS > > > > As introduced in JEP-247 [1], starting with Java 9, javac has the ability > > to compile to an older Java platform. This was not possible previously > > without having multiple JDKs installed and specifying the > '-bootclasspath' > > option in javac for your target. > > > > The newly introduced "--release" option to Java 9+ javac allows you to > > specify the target version using the documented API for that platform. > > This means replacing the '-source' and '-target' parameters with the > > '--release' option instead. > > > > In maven, that manifests itself as a configuration option to the > > maven-compiler-plugin, which is the 'maven.compiler.release' property. > [2] > > > > The discussion here would be consideration for using the "release" option > > as opposed to our current setup which uses "source" and "target". The > > benefit would be that all the deployment scenarios (above) could result > in > > success. > > > > The downside though is that, without other changes, Java 11 would be > > required to compile NiFi. It would still target Java 8 as a supported > > runtime (specifying "release=8"), but you'd have to build/compile with > Java > > 11. I don't think this should be a problem, but it's worth discussion. > > > > Thanks, > > Adam > > > > [1] https://openjdk.java.net/jeps/247 > > [2] > > > > > https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#release > > >
Re: Java 11 Compilation
Ah yep good point. I think we have that somewhere but will def add that to release guide. thanks On Wed, Oct 30, 2019 at 2:26 PM Adam Taft wrote: > Right, I agree with your perspective. > > Just note, however, that this stance will require the RM to create the 1.x > convenience binary with Java 8 only. It will be incumbent on the RM to > ensure they build with Java 8, because if they accidentally build with Java > 11, the binary distribution won't run on Java 8. > > Adam > > > On Wed, Oct 30, 2019 at 11:56 AM Joe Witt wrote: > > > Adam > > > > Interesting. Id say though that where we are now for nifi 1.x is > perfect. > > That matrix you shared as currently working seems ideal. > > > > For nifi 2.x we cut bait on Java 8 and go with latest stable Java at that > > time (11, 13) > > > > thanks > > > > On Wed, Oct 30, 2019 at 12:51 PM Adam Taft wrote: > > > > > While building 1.10.0-rc3, I wanted to experiment with the compilation > > and > > > runtime variants using Java 8 and Java 11. The summary of this > > experiment > > > was: > > > > > > Comp: Java 8 Run: Java 8 => SUCCESS > > > Comp: Java 8 Run: Java 11 => SUCCESS > > > Comp: Java 11 Run: Java 8 => FAILURE > > > Comp: Java 11 Run: Java 11 => SUCCESS > > > > > > As introduced in JEP-247 [1], starting with Java 9, javac has the > ability > > > to compile to an older Java platform. This was not possible previously > > > without having multiple JDKs installed and specifying the > > '-bootclasspath' > > > option in javac for your target. > > > > > > The newly introduced "--release" option to Java 9+ javac allows you to > > > specify the target version using the documented API for that platform. > > > This means replacing the '-source' and '-target' parameters with the > > > '--release' option instead. > > > > > > In maven, that manifests itself as a configuration option to the > > > maven-compiler-plugin, which is the 'maven.compiler.release' property. > > [2] > > > > > > The discussion here would be consideration for using the "release" > option > > > as opposed to our current setup which uses "source" and "target". The > > > benefit would be that all the deployment scenarios (above) could result > > in > > > success. > > > > > > The downside though is that, without other changes, Java 11 would be > > > required to compile NiFi. It would still target Java 8 as a supported > > > runtime (specifying "release=8"), but you'd have to build/compile with > > Java > > > 11. I don't think this should be a problem, but it's worth discussion. > > > > > > Thanks, > > > Adam > > > > > > [1] https://openjdk.java.net/jeps/247 > > > [2] > > > > > > > > > https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#release > > > > > >
Re: [VOTE] Release Apache NiFi 1.10.0 (rc3)
+1 (binding) Was able to build and verify that all of the Jiras that I raised last time around have been addressed in this RC. Left a cluster of 10 nodes running for a day or two, pretty heavily taxed, and ran into no issues. Thanks -Mark > On Oct 30, 2019, at 3:20 PM, Matt Gilman wrote: > > +1 (binding) > > Verified signatures, hashes, build, etc. Tested both standalone and > clustered and secure and unsecured. > > Thanks for RMing Joe! > > Matt > > On Wed, Oct 30, 2019 at 1:51 PM Adam Taft wrote: > >> +1 (binding) >> >> Signatures verified. >> Hashes verified. >> Tests pass, source builds cleanly. >> I used both Java 11 & Java 8 to build. >> >> I did run into a problem compiling with Java 11 and running with Java 8. I >> don't believe this was a goal of the Java 11 compatibility changes, so >> nothing unexpected about this. But it's possibly worth discussion in >> another thread (which I'll send out). The convenience binary was compiled >> with Java 8, so no problems with compatibility either way. >> >> On Tue, Oct 29, 2019 at 11:32 AM Joe Witt wrote: >> >>> Hello, >>> >>> I am pleased to be calling this vote for the source release of Apache >> NiFi >>> nifi-1.10.0. >>> >>> As they say 'third time's a charm'. >>> >>> The source zip, including signatures, digests, etc. can be found at: >>> https://repository.apache.org/content/repositories/orgapachenifi-1151 >>> >>> The source being voted upon and the convenience binaries can be found at: >>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.10.0/ >>> >>> The Git tag is nifi-1.10.0-RC3 >>> The Git commit ID is b217ae20ad6a04cac874b2b00d93b7f7514c0b88 >>> >>> >> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b217ae20ad6a04cac874b2b00d93b7f7514c0b88 >>> >>> Checksums of nifi-1.10.0-source-release.zip: >>> SHA256: e9b0a14b3029acd69c6693781b6b6487c14dda12676db8b4a015bce23b1029c1 >>> SHA512: >>> >>> >> b07258cbc21d2e529a1aa3098449917e2d059e6b45ffcfcb6df094931cf16caa8970576555164d3f2290cfe064b5780ba1a8bf63dad04d20100ed559a1cfe133 >>> >>> Release artifacts are signed with the following key: >>> https://people.apache.org/keys/committer/joewitt.asc >>> >>> KEYS file available here: >>> https://dist.apache.org/repos/dist/release/nifi/KEYS >>> >>> 384 issues were closed/resolved for this release: >>> >>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993 >>> >>> Release note highlights can be found here: >>> >>> >> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0 >>> >>> The vote will be open for 72 hours. >>> Please download the release candidate and evaluate the necessary items >>> including checking hashes, signatures, build >>> from source, and test. Then please vote: >>> >>> [ ] +1 Release this package as nifi-1.10.0 >>> [ ] +0 no opinion >>> [ ] -1 Do not release this package because... >>> >>
Re: PULL ProvenanceEvent
I like the idea of creating PULL as a type. In fact, I'd propose that there
are three scenarios here:
RECEIVE - Passively acquire in a sort of hand-off situation. Ex: Kafka
subscription
PULL - Direct operations to seek out and fetch something in a targeted
fashion. Ex. GetHttp
QUERY - Go looking for the data and take what matches your search. Ex.
JsonQueryElasticsearch, GetMongo, any SQL query processor, etc.
On Wed, Oct 30, 2019 at 1:31 PM Nissim Shiman
wrote:
> Joe,
>
>
> It is hard to say how much value transit URI would bring to clarify a
> RECEIVE.
> For example a RECEIVE with transit URI of https: could be either a
> GetHTTP (i.e. active) or ListenHTTP (i.e. passive)
>
> but your idea of "a metadata item specifying active vs passive" is a very
> clever way to make this work with mimimal disruptions.
>
> My understanding of this is that the current receive() calls in
> ProvenanceReporter [1] will remain the same, but news ones will be added
> with a boolean parameter reflecting if the receive is active or passive.
> This will allow the current list of Provenance Events [2] to remain the
> same. So third party/custom processors can continue working as is
>
> Does this sound like what you are thinking?
>
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceReporter.java#L46
>
> [2]
> https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/provenance/ProvenanceEventType.java
>
>
> Thanks,
>
> Nissim
> On Tuesday, October 29, 2019, 12:47:40 PM EDT, Joe Witt <
> joe.w...@gmail.com> wrote:
>
> Nissim
>
> I like the idea to introduce a more refined type of event for how data is
> brought into nifi (active - PULL, passive - RECEIVE).
>
> That said it might be sufficient to simply have this distinction be on the
> "RECEIVE" event as a metadata item specifying active vs passive. The
> protocol utilized as mentioned in the transport URI should clarify this
> though.
>
> In short - i think there is a way here that is all opt-in for existing
> users and components.
>
> Thanks
>
> On Tue, Oct 29, 2019 at 9:41 AM Nissim Shiman
> wrote:
>
> > Adam,
> > good points...
> > I missed a step in explaining the use case where Provenance Events is
> > incomplete...
> > Where the second nifi does a GetSFTP from the *filesytem* that the first
> > nifi is located on
> > So the second nifi currently sends a RECEIVE event, but there is no
> > corresponding SEND event from the first nifi (nor should there be)
> > If the second nifi sent a PULL event, it would be easier for a system
> > overseer to know that there should be no corresponding SEND event
> >
> > Currently send/receive works well when nifi 1 does a PostHTTP and nifi 2
> > does a ListenHTTP, but not in the case above.
> >
> > The ERROR case you mention is a nice point as well, although not my
> > specific issue at the moment.
> > Thanks,
> > Nissim
> >
> >
> >
> >
> >
> >On Monday, October 28, 2019, 11:52:57 PM EDT, Adam Taft <
> > a...@adamtaft.com> wrote:
> >
> > > But a flowfile that was PULLed by the second nifi (from the first
> nifi)
> > will not necessarily have any provenance event generated by the first
> nifi.
> >
> > Isn't this the fault of the first NiFi to fail to emit a SEND event in
> > response to the second NiFi's request? In this scenario, shouldn't the
> > send/receive pair be:
> > NiFi-1 [SEND] :: NIFI-2 [RECEIVE]?
> >
> > What you describe is an odd use case for NiFi. NiFi is usually not in
> the
> > business of acting as a file server daemon in order to "send" flowfiles
> to
> > other systems. As you mention, HandleHttpResponse may be a lone wolf
> > example processor which generates a SEND event whose input originates
> from
> > a "listener". [1] The other ListenXYZ processors generally issue RECEIVE
> > events because they are receiving bytes, not generating them.
> >
> > Are there other processors in question? Something custom? Or is this
> > related to site-to-site transfers?
> >
> > I still kind of question the motive of a provenance event pair that is
> > trying to establish "who called who first". Honestly just trying to
> > understand the use case where a matching SEND/RECEIVE pair doesn't give
> you
> > what you need.
> >
> > The only thing I could see would be a processor that asks for data, but
> > then doesn't receive it due to some error condition. In this case,
> adding
> > some sort of ERROR event might be useful. "I attempted to retrieve data
> > from ${uri}, but the transfer failed because of ${error condition}".
> That
> > way, GetXYZ processors could report an error to provenance instead of as
> a
> > bulletin.
> >
> > If the problem is related to a processor or the framework itself not
> > generating an event, can we just fix that function to emit SEND in the
> > scenario that you describe? Changing the provenance model itself (beyond
> > possibly adding an ERROR event) feels like it would be the last scenario
> to
Re: [VOTE] Release Apache NiFi 1.10.0 (rc3)
+1 (non-binding) Built and ran successfully and built custom processor with 1.10.0 dependency and successfully uploaded NAR to NiFi Registry 0.5.0. On Wed, Oct 30, 2019 at 4:06 PM Mark Payne wrote: > +1 (binding) > > Was able to build and verify that all of the Jiras that I raised last time > around have been addressed in this RC. Left a cluster of 10 nodes running > for a day or two, pretty heavily taxed, and ran into no issues. > > Thanks > -Mark > > > > On Oct 30, 2019, at 3:20 PM, Matt Gilman > wrote: > > > > +1 (binding) > > > > Verified signatures, hashes, build, etc. Tested both standalone and > > clustered and secure and unsecured. > > > > Thanks for RMing Joe! > > > > Matt > > > > On Wed, Oct 30, 2019 at 1:51 PM Adam Taft wrote: > > > >> +1 (binding) > >> > >> Signatures verified. > >> Hashes verified. > >> Tests pass, source builds cleanly. > >> I used both Java 11 & Java 8 to build. > >> > >> I did run into a problem compiling with Java 11 and running with Java > 8. I > >> don't believe this was a goal of the Java 11 compatibility changes, so > >> nothing unexpected about this. But it's possibly worth discussion in > >> another thread (which I'll send out). The convenience binary was > compiled > >> with Java 8, so no problems with compatibility either way. > >> > >> On Tue, Oct 29, 2019 at 11:32 AM Joe Witt wrote: > >> > >>> Hello, > >>> > >>> I am pleased to be calling this vote for the source release of Apache > >> NiFi > >>> nifi-1.10.0. > >>> > >>> As they say 'third time's a charm'. > >>> > >>> The source zip, including signatures, digests, etc. can be found at: > >>> https://repository.apache.org/content/repositories/orgapachenifi-1151 > >>> > >>> The source being voted upon and the convenience binaries can be found > at: > >>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.10.0/ > >>> > >>> The Git tag is nifi-1.10.0-RC3 > >>> The Git commit ID is b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > >>> > >>> > >> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > >>> > >>> Checksums of nifi-1.10.0-source-release.zip: > >>> SHA256: > e9b0a14b3029acd69c6693781b6b6487c14dda12676db8b4a015bce23b1029c1 > >>> SHA512: > >>> > >>> > >> > b07258cbc21d2e529a1aa3098449917e2d059e6b45ffcfcb6df094931cf16caa8970576555164d3f2290cfe064b5780ba1a8bf63dad04d20100ed559a1cfe133 > >>> > >>> Release artifacts are signed with the following key: > >>> https://people.apache.org/keys/committer/joewitt.asc > >>> > >>> KEYS file available here: > >>> https://dist.apache.org/repos/dist/release/nifi/KEYS > >>> > >>> 384 issues were closed/resolved for this release: > >>> > >>> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993 > >>> > >>> Release note highlights can be found here: > >>> > >>> > >> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0 > >>> > >>> The vote will be open for 72 hours. > >>> Please download the release candidate and evaluate the necessary items > >>> including checking hashes, signatures, build > >>> from source, and test. Then please vote: > >>> > >>> [ ] +1 Release this package as nifi-1.10.0 > >>> [ ] +0 no opinion > >>> [ ] -1 Do not release this package because... > >>> > >> > >
Re: [VOTE] Release Apache NiFi 1.10.0 (rc3)
+1 (non-binding) Deployed cluster in Azure and smoke tested a fairly complicated data flow. Verified the fixes for the Azure EventHub Processor that were pulled into RC3 are working as expected. Shayne On Wed, Oct 30, 2019 at 5:59 PM Jeff Zemerick wrote: > +1 (non-binding) > > Built and ran successfully and built custom processor with 1.10.0 > dependency and successfully uploaded NAR to NiFi Registry 0.5.0. > > On Wed, Oct 30, 2019 at 4:06 PM Mark Payne wrote: > > > +1 (binding) > > > > Was able to build and verify that all of the Jiras that I raised last > time > > around have been addressed in this RC. Left a cluster of 10 nodes running > > for a day or two, pretty heavily taxed, and ran into no issues. > > > > Thanks > > -Mark > > > > > > > On Oct 30, 2019, at 3:20 PM, Matt Gilman > > wrote: > > > > > > +1 (binding) > > > > > > Verified signatures, hashes, build, etc. Tested both standalone and > > > clustered and secure and unsecured. > > > > > > Thanks for RMing Joe! > > > > > > Matt > > > > > > On Wed, Oct 30, 2019 at 1:51 PM Adam Taft wrote: > > > > > >> +1 (binding) > > >> > > >> Signatures verified. > > >> Hashes verified. > > >> Tests pass, source builds cleanly. > > >> I used both Java 11 & Java 8 to build. > > >> > > >> I did run into a problem compiling with Java 11 and running with Java > > 8. I > > >> don't believe this was a goal of the Java 11 compatibility changes, so > > >> nothing unexpected about this. But it's possibly worth discussion in > > >> another thread (which I'll send out). The convenience binary was > > compiled > > >> with Java 8, so no problems with compatibility either way. > > >> > > >> On Tue, Oct 29, 2019 at 11:32 AM Joe Witt wrote: > > >> > > >>> Hello, > > >>> > > >>> I am pleased to be calling this vote for the source release of Apache > > >> NiFi > > >>> nifi-1.10.0. > > >>> > > >>> As they say 'third time's a charm'. > > >>> > > >>> The source zip, including signatures, digests, etc. can be found at: > > >>> > https://repository.apache.org/content/repositories/orgapachenifi-1151 > > >>> > > >>> The source being voted upon and the convenience binaries can be found > > at: > > >>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.10.0/ > > >>> > > >>> The Git tag is nifi-1.10.0-RC3 > > >>> The Git commit ID is b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > > >>> > > >>> > > >> > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b217ae20ad6a04cac874b2b00d93b7f7514c0b88 > > >>> > > >>> Checksums of nifi-1.10.0-source-release.zip: > > >>> SHA256: > > e9b0a14b3029acd69c6693781b6b6487c14dda12676db8b4a015bce23b1029c1 > > >>> SHA512: > > >>> > > >>> > > >> > > > b07258cbc21d2e529a1aa3098449917e2d059e6b45ffcfcb6df094931cf16caa8970576555164d3f2290cfe064b5780ba1a8bf63dad04d20100ed559a1cfe133 > > >>> > > >>> Release artifacts are signed with the following key: > > >>> https://people.apache.org/keys/committer/joewitt.asc > > >>> > > >>> KEYS file available here: > > >>> https://dist.apache.org/repos/dist/release/nifi/KEYS > > >>> > > >>> 384 issues were closed/resolved for this release: > > >>> > > >>> > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12344993 > > >>> > > >>> Release note highlights can be found here: > > >>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.10.0 > > >>> > > >>> The vote will be open for 72 hours. > > >>> Please download the release candidate and evaluate the necessary > items > > >>> including checking hashes, signatures, build > > >>> from source, and test. Then please vote: > > >>> > > >>> [ ] +1 Release this package as nifi-1.10.0 > > >>> [ ] +0 no opinion > > >>> [ ] -1 Do not release this package because... > > >>> > > >> > > > > >
