Re: [VOTE] Release Apache NiFi 1.19.0 (RC1)
+1 (non-binding) Verified checksums and signatures. Performed full build with -Pcontrib-check using OpenJDK 1.8.0_352, 11.0.17, 17.0.5 Ran several relatively simple flows in non-cluster mode only. No issues observed. I don't believe this is worth holding up the 1.19.0 release, but while everything worked as expected with Java 8 and 11, there were some issues with Java 17 in generating documentation for several of the "scripted" processors, controller services and a reporting task. The following is from the nifi-app.log. And, in the UI, the documentation was not full/complete for these processors. 2022-11-27 20:50:27,703 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.processors.script.InvokeScriptedProcessor] 2022-11-27 20:50:28,243 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.processors.script.ExecuteScript] 2022-11-27 20:50:28,311 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.record.sink.script.ScriptedRecordSink] 2022-11-27 20:50:28,313 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.record.script.ScriptedRecordSetWriter] 2022-11-27 20:50:28,315 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.lookup.script.ScriptedLookupService] 2022-11-27 20:50:28,323 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.lookup.script.SimpleScriptedLookupService] 2022-11-27 20:50:28,326 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.rules.handlers.script.ScriptedActionHandler] 2022-11-27 20:50:28,327 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.record.script.ScriptedReader] 2022-11-27 20:50:28,328 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.rules.engine.script.ScriptedRulesEngine] 2022-11-27 20:50:28,521 WARN [main] o.apache.nifi.documentation.DocGenerator Documentation generation failed: Component Class [class org.apache.nifi.reporting.script.ScriptedReportingTask] On Sun, Nov 27, 2022 at 4:35 PM Nandor Soma Abonyi wrote: > +1 (non-binding) > > I’ve found two minor issues. I believe none of them is a blocker, but we > should follow up on them. Marked them with ***. > > Went through the release helper guide on: > Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63) > Java version: 1.8.0_332, vendor: Temurin > Default locale: en_US, platform encoding: UTF-8 > OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac" > > Imported flows from NiFi registry for these issues and verified them: > - NIFI-10760 - Add Api key authentication option to > ElasticSearchClientServiceImpl > - *** A validation has been removed, which prevented using Basic and > Api Key properties together. > If I set Username/Password and hide them by selecting Api Key > Authorization Scheme, the client will still use those credentials. If I set > Api Key, the client will add both credentials as a request header. > Opened NIFI-10880 for further discussion. > - NIFI-10668 - Persist proxy password in remote process group > - NIFI-10656 - Log ignored event with info instead of warning in PutADLS > - NIFI-10650 - Fix demarcator is appended to the end of the FlowFile's > content by ConsumeMQTT > - NIFI-10644 - Add Message Demarcator-style processing in PublishMQTT > - NIFI-10543 - Support broker failover in MQTT processors > - NIFI-10491 - PutAzureBlobStorage_v12 should have Conflict Resolution > Strategy > - NIFI-10317 - NullPointerException if AMQP header value is null > > *** Initially, I wanted to build NiFi in a docker container but > experienced intermittent test failures. Most of the time, an 5 sec Test > timeout caused issues in TestStandardFlowFileQueue. > I don’t know why such a timeout was used, but I opened NIFI-10881 > for further discussion. > > Thanks for RM'ing Joe! > Regards, > Soma > > > On Nov 23, 2022, at 4:58 PM, Joe Witt wrote: > > > > Hello, > > > > I am pleased to be calling this vote for the source release of Apache > NiFi > > 1.19.0. > > > > The source zip, including signatures, digests, etc. can be found at: > > https://repository.apache.org/content/repositories/orgapachenifi-1216 > > > > The source being voted upon and the convenience binaries can be found at: > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.19.0/ > > > > A helpful reminder on how the release candidate verification process > works: > > >
Re: [VOTE] Release Apache NiFi 1.19.0 (RC1)
+1 (non-binding) I’ve found two minor issues. I believe none of them is a blocker, but we should follow up on them. Marked them with ***. Went through the release helper guide on: Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63) Java version: 1.8.0_332, vendor: Temurin Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac" Imported flows from NiFi registry for these issues and verified them: - NIFI-10760 - Add Api key authentication option to ElasticSearchClientServiceImpl - *** A validation has been removed, which prevented using Basic and Api Key properties together. If I set Username/Password and hide them by selecting Api Key Authorization Scheme, the client will still use those credentials. If I set Api Key, the client will add both credentials as a request header. Opened NIFI-10880 for further discussion. - NIFI-10668 - Persist proxy password in remote process group - NIFI-10656 - Log ignored event with info instead of warning in PutADLS - NIFI-10650 - Fix demarcator is appended to the end of the FlowFile's content by ConsumeMQTT - NIFI-10644 - Add Message Demarcator-style processing in PublishMQTT - NIFI-10543 - Support broker failover in MQTT processors - NIFI-10491 - PutAzureBlobStorage_v12 should have Conflict Resolution Strategy - NIFI-10317 - NullPointerException if AMQP header value is null *** Initially, I wanted to build NiFi in a docker container but experienced intermittent test failures. Most of the time, an 5 sec Test timeout caused issues in TestStandardFlowFileQueue. I don’t know why such a timeout was used, but I opened NIFI-10881 for further discussion. Thanks for RM'ing Joe! Regards, Soma > On Nov 23, 2022, at 4:58 PM, Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.19.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1216 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.19.0/ > > A helpful reminder on how the release candidate verification process works: > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.19.0-RC1 > The Git commit ID is ec87bf93add2f645d2ea426002e0c24db54614ff > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=ec87bf93add2f645d2ea426002e0c24db54614ff > > Checksums of nifi-1.19.0-source-release.zip: > SHA256: 909c4fce81b305af955540f83115474d139c132e886847df7b76e94efd3d4641 > SHA512: > b3ba95984841140aa2a9a2ea471dd446d6fbd71dba320f723839e0938a7873f847e415a88b7c38e59907ad051c896949f0c7093c6d5e6b5db549de929ec32c6e > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 221 issues were closed/resolved for this release: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150 > > Release note highlights can be found here: > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.19.0 > > The vote will be open for at least 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.19.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 1.19.0 (RC1)
+1 (non-binding) Ran through release verification helper (added references to wifi-registry, wifi-toolkit and minify build artefacts to the Confluence page). Verified checksums & signature. Build completed on Mac M1 (arm64) using Maven 3.8.6; Java 17.0.5, vendor: Eclipse Adoptium. Started NiFi, designed a couple of simple flows. Locally built Docker images and tested connectivity between NiFi, Registry and Toolkit with some simple Flows and customer test harness. Cheers, --- Chris Sampson IT Consultant chris.samp...@naimuri.com > On 23 Nov 2022, at 15:58, Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.19.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1216 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.19.0/ > > A helpful reminder on how the release candidate verification process works: > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.19.0-RC1 > The Git commit ID is ec87bf93add2f645d2ea426002e0c24db54614ff > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=ec87bf93add2f645d2ea426002e0c24db54614ff > > Checksums of nifi-1.19.0-source-release.zip: > SHA256: 909c4fce81b305af955540f83115474d139c132e886847df7b76e94efd3d4641 > SHA512: > b3ba95984841140aa2a9a2ea471dd446d6fbd71dba320f723839e0938a7873f847e415a88b7c38e59907ad051c896949f0c7093c6d5e6b5db549de929ec32c6e > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 221 issues were closed/resolved for this release: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150 > > Release note highlights can be found here: > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.19.0 > > The vote will be open for at least 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.19.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because...
Re: [VOTE] Release Apache NiFi 1.19.0 (RC1)
+1 (binding) Ran through release helper, tested a couple of flows including one with the new UpdateDatabaseTable processor. Thanks for RM'ing Joe! On Wed, Nov 23, 2022 at 10:58 AM Joe Witt wrote: > > Hello, > > I am pleased to be calling this vote for the source release of Apache NiFi > 1.19.0. > > The source zip, including signatures, digests, etc. can be found at: > https://repository.apache.org/content/repositories/orgapachenifi-1216 > > The source being voted upon and the convenience binaries can be found at: > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.19.0/ > > A helpful reminder on how the release candidate verification process works: > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > The Git tag is nifi-1.19.0-RC1 > The Git commit ID is ec87bf93add2f645d2ea426002e0c24db54614ff > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=ec87bf93add2f645d2ea426002e0c24db54614ff > > Checksums of nifi-1.19.0-source-release.zip: > SHA256: 909c4fce81b305af955540f83115474d139c132e886847df7b76e94efd3d4641 > SHA512: > b3ba95984841140aa2a9a2ea471dd446d6fbd71dba320f723839e0938a7873f847e415a88b7c38e59907ad051c896949f0c7093c6d5e6b5db549de929ec32c6e > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/joewitt.asc > > KEYS file available here: > https://dist.apache.org/repos/dist/release/nifi/KEYS > > 221 issues were closed/resolved for this release: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150 > > Release note highlights can be found here: > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.19.0 > > The vote will be open for at least 72 hours. > Please download the release candidate and evaluate the necessary items > including checking hashes, signatures, build > from source, and test. Then please vote: > > [ ] +1 Release this package as nifi-1.19.0 > [ ] +0 no opinion > [ ] -1 Do not release this package because...
Re: Compression as Controller Service
Hello I think the comments around CompressContent are valid and fixable and your ideas there make a lot of sense. As you noted in the second part various components have their own built in compression. These are likely a bridge too far as their config/options/etc are well embedded into those libraries. Thanks On Thu, Nov 24, 2022 at 6:36 PM Matthew Hawkins wrote: > Hi team, > > Whilst giving a first go at adding brotli and zstd to CompressContent I > noticed a few things that I think we should address. > > 1. compression levels are hard-coded as 0-9 in the UI, this prevents > accessing every level possible under certain compressors, e.g. zstd has 22 > levels, brotli has 15. I hacked it, it's ugly, we should do the "right" > thing instead. > - I think this should be addressed by having a compressor register itself > along with metadata like how many levels it supports to a factory & the UI > utilise that property in a more dynamic fashion > - we could also use metadata to let a compressor flag known high cpu / > high mem situations so the flow Admin can avoid silly mistakes in config or > the JVM can be hinted at. > > 2. Some other processors such as PublishKafka, InvokeHTTP, > JsonRecordSetWriter, and others implement some compression algorithms > directly. > - This made me realise it would IMO be better offered as a Service and > each of these use cases can have the simple API of using the controller > service rather than duplicating compression algorithm specific code each > time the functionality is required. > > Am I on the right track? Java isn't my forte and I'm new to contributing to > NiFi so keen to hear what experienced devs think. > > -- > Kind regards, > Matthew Hawkins >
Re: Optional build profiles with Category X licenses
Mike There are some threads in Apache land how such a thing can happen for binary dependencies and only for purely optional modules someone else might choose to build. However, the cleanest option given we are talking about a designed extension point is to likely just build it and make it available on github and such. Thanks On Sun, Nov 27, 2022 at 8:22 AM Mike Thomsen wrote: > I know the official binaries cannot contain Category X license code, > but is it allowable to have optional build profiles that integrate > Category X licensed code? An example scenario that lead me to wonder > if this is doable is I would like to try to add optional support for > Elastic's Painless as script engine, but it's covered under a Category > X license. > > Thanks, > > Mike >
Input and output ports
Good day I'm current working on porting one of our nifi flows to minifi, I noticed that the documentation says the following about ports Note: Currently MiNiFi does not support external Input Ports or Output Ports. Any templates that contain these will fail transformation. Running the toolkit converter doesn't seem to break the port conversion but simply throws a warning that the config.yml needs to be edited. I'm still fairly new to this would like the a pointer in the right direction, I'm looking for documentation on the config.yml with examples on how to manual add our own Ports Regards
Optional build profiles with Category X licenses
I know the official binaries cannot contain Category X license code, but is it allowable to have optional build profiles that integrate Category X licensed code? An example scenario that lead me to wonder if this is doable is I would like to try to add optional support for Elastic's Painless as script engine, but it's covered under a Category X license. Thanks, Mike
Re: [VOTE] Release Apache NiFi 1.19.0 (RC1)
+1 binding Verified checksums, build, signature, started NiFi, designed a simple flow. Thanks, Arpad On Sat, Nov 26, 2022 at 6:41 PM Marton Szasz wrote: > +1 (binding) > > Followed the usual steps and tested a use case of collecting system > logs with minifi c++ and collecting them in nifi using ListenUDP. > > Thanks, > Marton > > On Sat, 26 Nov 2022 at 06:25, David Handermann > wrote: > > > > +1 (binding) > > > > - Verified signatures and hashes > > - Ran build using Maven 3.8.6 > > - Ran build on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-345 AMD64 > > - Ran build on macOS 13.0 with Azul Zulu JDK 17.0.4 AArch64 > > - Ran stateless and system tests on macOS 13.0 with Azul Zulu JDK 17.0.4 > > AArch64 > > > > - Ran NiFi on macOS 13.0 with Azul Zulu JDK 17.0.4 AArch64 > > - Ran NiFi on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-345 AMD64 > > - Configured SAML 2 authentication and reverse proxy access > > > > - Ran NiFi Registry on Ubuntu 22.04 with Azul Zulu JDK 1.8.0-345 AMD64 > > - Ran NiFi Registry on Ubuntu 22.04 with Azul Zulu JDK 17.0.4 AMD64 > > - Created Buckets > > - Verified Flow Version Control > > > > - NIFI-7392 Verified ValidateJson Processor with each relationship > > - NIFI-10577 Verified visibility of sensitive checkbox on Firefox in > Fetch > > Parameters dialog > > - NIFI-10506 Verified relocation of Repository Encryption to > > Administrator's Guide > > - NIFI-10569 Verified Maximum Threads property on HandleHttpRequest > > - NIFI-10594 Verified visibility of multiple blank lines in Labels > > - NIFI-10604 Verified Idle Timeout property handling in ListenTCP > > - NIFI-10614 Verified Bouncy Castle 1.71 and jdk18on libraries across > > modules > > - NIFI-10624 Verified removal of sensitive properties key warning from > > component documentation > > - NIFI-10625 Verified HTTP/2 support in NiFi Registry > > - NIFI-10628 Verified default Run Schedule on multiple Processors > > - NIFI-10647 Verified updated URL in bundled README instructions > > - NIFI-10648 Verified Apache Commons Text 1.10.0 upgraded across bundled > > modules > > - NIFI-10667 Verified Standard Private Key Service with valid and invalid > > PKCS8 keys > > - NIFI-10709 Verified presence of log in link on logout success > > - NIFI-10713 Verified deprecation warning logged for PGP in > EncryptContent > > Processor > > - NIFI-10798 Verified deprecation warning logged when running on Java 8 > > - NIFI-10819 Verified sasl.jaas.config with JndiLoginModule is invalid > for > > Kafka 2.6 Processors > > - NIFI-10842 Verified optional HTTP Protocols property in OAuth2 Token > > Provider > > - NIFI-10861 Verified deprecation warning logged for legacy Azure Blob > > Storage Processors > > > > Thanks Joe! > > > > Regards, > > David Handermann > > > > On Wed, Nov 23, 2022 at 9:58 AM Joe Witt wrote: > > > > > Hello, > > > > > > I am pleased to be calling this vote for the source release of Apache > NiFi > > > 1.19.0. > > > > > > The source zip, including signatures, digests, etc. can be found at: > > > https://repository.apache.org/content/repositories/orgapachenifi-1216 > > > > > > The source being voted upon and the convenience binaries can be found > at: > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.19.0/ > > > > > > A helpful reminder on how the release candidate verification process > works: > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate > > > > > > The Git tag is nifi-1.19.0-RC1 > > > The Git commit ID is ec87bf93add2f645d2ea426002e0c24db54614ff > > > > > > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=ec87bf93add2f645d2ea426002e0c24db54614ff > > > > > > Checksums of nifi-1.19.0-source-release.zip: > > > SHA256: > 909c4fce81b305af955540f83115474d139c132e886847df7b76e94efd3d4641 > > > SHA512: > > > > > > > b3ba95984841140aa2a9a2ea471dd446d6fbd71dba320f723839e0938a7873f847e415a88b7c38e59907ad051c896949f0c7093c6d5e6b5db549de929ec32c6e > > > > > > Release artifacts are signed with the following key: > > > https://people.apache.org/keys/committer/joewitt.asc > > > > > > KEYS file available here: > > > https://dist.apache.org/repos/dist/release/nifi/KEYS > > > > > > 221 issues were closed/resolved for this release: > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12352150 > > > > > > Release note highlights can be found here: > > > > > > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.19.0 > > > > > > The vote will be open for at least 72 hours. > > > Please download the release candidate and evaluate the necessary items > > > including checking hashes, signatures, build > > > from source, and test. Then please vote: > > > > > > [ ] +1 Release this package as nifi-1.19.0 > > > [ ] +0 no opinion > > > [ ] -1 Do not release this package because... > > > >
