Re: Queries regarding Nifi custom processor for Azure Data Explorer

[Joey Frazee]

Hi Tanmaya,

There’s been some community interest in ADX processors in the past. I’m 
curious, is this something your team is interested in contributing back to 
Apache NiFi, or do you plan to release it as a Microsoft open source project?

Either is fine of course.

One thing to consider is that conceptually data sources in Spark are usually a 
little different and hide a lot of details. NiFi processors tend to do fewer 
things and get composed into a flow. You could stuff most of the things you’re 
doing onto a single processor but if you have a good simple processor for 
writing to ADX some of the other stuff you mention might just be logic in a 
flow.

-joey

> On Oct 3, 2022, at 5:56 AM, Mike Thomsen  wrote:
> 
> I think that is at least partly doable within NiFi (ex yes, you can
> restrict processors to the primary node in a cluster), but I would
> recommend you consider a different approach for NiFi. Unlike Spark,
> NiFi is perfectly content to stream in huge amounts of data
> continuously without any temporary storage besides its repositories
> (flowfile, content, etc). Therefore, I think a potentially easier
> solution would be for your team to explore creating a connector
> between NiFi and Azure Data Explorer that allows the latter to
> firehose the former with data as it comes in and let the chips fall
> where they may.
> 
> You might find some useful concepts in the Kafka processors for things
> like managing a continuous flow of record data from a stream and
> converting it to blocks of NiFi record data (see the Record API in our
> documentation for details).
> 
> FWIW, I manage a data set in AWS w/ NiFi that is over 50TB compressed,
> and a fairly generic 5 node NiFi cluster crushes that data on cheap
> EC2 instances without issue. So handling TBs of data is something
> fairly out of the box for NiFi if you're worried about that.
> 
>> On Sat, Oct 1, 2022 at 12:05 AM Tanmaya Panda
>>  wrote:
>> 
>> Hi Team,
>> 
>> We at Microsoft opensource, are developing a custom for Azure Data Explorer 
>> sink connector for Apache Nifi. What we want to achieve is transactionality 
>> with data ingestion. The source of the processor can be TBs of telemetry 
>> data as well as CDC logs. What that means is in case of any failure while 
>> ingesting data of particular partition to ADX, will delete/cleanup any other 
>> ingested data of other partitions. Since Azure Data Explorer is an append 
>> only database, unfortunately we cant perform delete on the ingested data of 
>> same partition or other partition. So to achieve this kind of 
>> transactionality of large ingested data, we are thinking to implement 
>> something similar we have done for Apache Spark ADX connector. We will be 
>> creating temporary tables inside Azure Data Explorer before ingesting to the 
>> actual tables. The worker nodes in apache spark will create these temporary 
>> tables and report the ingestion status to the driver node. The driver node 
>> on receiving the success status of all the worker nodes performs ingestion 
>> on the actual table or else the ingestion is aborted along with cleanup of 
>> temporary table. So basically we aggregate the worker node task status in 
>> the driver node in spark to take further decision on whether to ingest data 
>> into ADX table or not.
>> Question 1: Is this possible in Apache Nifi follows a zero master cluster 
>> strategy as opposed to master-slave architecture of apache spark?
>> Question 2: In our custom processor in Nifi, is it possible to run custom 
>> code of a particular partition say the cluster coordinator node. Also is it 
>> possible to get the details of the partition inside the processor?
>> Question 3: Is is to possible to get the details of tasks executed at the 
>> various partitions and take decisions based on the task status. Can all 
>> these be done inside the same processor.
>> 
>> Thanks,
>> Tanmaya

Re: [VOTE] Release Apache NiFi 1.17.0 (RC2)

[Joey Frazee]

+1 (binding)

- Verified signatures and checksums
- Ran full build and contrib check on Linux w/ Java 11
- Ran some simple flows and verified fix for NIFI-10219

Thanks all!

-joey

> On Jul 30, 2022, at 4:28 AM, Ferenc Erdei  wrote:
> 
> +1 (non-binding)
> 
> - Went through the helper guide
> - Verified signatures and hashes
> - Built on OSX 12.5, with (AdoptOpenJDK)(build 1.8.0_292-b10), Maven 3.8.3
> - Started nifi and minifi and ran some simple flow
> 
> Regards,
> Ferenc
> 
>> On Fri, Jul 29, 2022 at 8:53 PM Bryan Bende  wrote:
>> 
>> +1 (binding)
>> 
>> - Verified everything in release helper
>> - Ran basic flows
>> - Saved/imported flows with NiFi Registry
>> 
>>> On Fri, Jul 29, 2022 at 2:50 PM Joe Skora  wrote:
>>> 
>>> +1 (binding) for release
>>> 
>>> * helper guide review and build: hashes, certs, files checked out.
>>> * tested canvas and backend to create a test flow without any problems.
>> 

Re: [VOTE] Release Apache NiFi 1.16.0 (rc3)

[Joey Frazee]

+1 (binding)

- Verified checksums and signatures
- Did full build in Java 11 on Fedora
- Ran build with rpm profile and tested rpm install
- Ran some basic flows

> On Mar 23, 2022, at 2:40 PM, Andrew Lim  wrote:
> 
> +1 (binding)
> 
> -Ran full clean install on OS X (Catalina 10.15.7, OpenJDK version 1.8.0_252)
> -Tested secure NiFi with secure NiFi Registry
> -Ran basic flows successfully; tested basic versioned flow functionality
> -Reviewed/tested Core UI fixes and documentation/website updates
> 
> Drew
> 
>> On Mar 21, 2022, at 5:41 PM, Joe Witt  wrote:
>> 
>> Hello,
>> 
>> I am pleased to be calling this vote for the source release of Apache NiFi
>> 1.16.0.
>> 
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1198
>> 
>> The source being voted upon and the convenience binaries can be found at:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.16.0/
>> 
>> A helpful reminder on how the release candidate verification process works:
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>> 
>> The Git tag is nifi-1.16.0-RC3
>> The Git commit ID is b019a9191f1c83bc7f547dc02c1b679b8936acee
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=b019a9191f1c83bc7f547dc02c1b679b8936acee
>> 
>> Checksums of nifi-1.16.0-source-release.zip:
>> SHA256: 2f16cb94df404d1bcc9c32835ba98da8940005a5d7ea5504c155ee42021a221e
>> SHA512:
>> cbd95f15cec5ffe506fef204526267b18b77d7266f6dc3e1bbc3c7600aac12e119977f7d8cf93dbbbc86fbb0739ba88aaa11a5381d29a463ec9a0c9a18f4e9e6
>> 
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/joewitt.asc
>> 
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>> 
>> 401 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350741
>> 
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0
>> 
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build from source, and test. Then
>> please vote:
>> 
>> [ ] +1 Release this package as nifi-1.16.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...
> 

Re: [VOTE] Release Apache NiFi 1.15.2

[Joey Frazee]

+1 (binding)

- Verified signatures and checksums
- Ran full build on Linux and Java 11
- Ran some simple flows

Thanks to everyone who turned around these comforting changes so quickly.

-joey

> On Dec 22, 2021, at 6:46 AM, Tony Kurc  wrote:
> 
> +1 (binding)
> 
> Verified signatures and hashes, and built without issue.
> 
>> On Wed, Dec 22, 2021 at 9:39 AM Matt Gilman  wrote:
>> 
>> +1 (binding) Release this package as nifi-1.15.2
>> 
>> Ran through the release helper. Looks great. Thanks for RMing Joe!
>> 
>> Matt
>> 
>>> On Tue, Dec 21, 2021 at 5:52 PM Joe Witt  wrote:
>>> 
>>> Hello,
>>> 
>>> I am pleased to be calling this vote for the source release of Apache
>>> NiFi 1.15.2.
>>> 
>>> This vote is purely bug fix and security focused. This is a
>>> continuation of our efforts to promptly and thoroughly respond to
>>> log4shell and related concerns.
>>> 
>>> The source zip, including signatures, digests, etc. can be found at:
>>> https://repository.apache.org/content/repositories/orgapachenifi-1193
>>> 
>>> The source being voted upon and the convenience binaries can be found at:
>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.2/
>>> 
>>> A helpful reminder on how the release candidate verification process
>> works:
>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>>> 
>>> The Git tag is nifi-1.15.2-RC1
>>> The Git commit ID is 1ea460b8556b07057366abb74a5552ace8946e87
>>> 
>>> 
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=1ea460b8556b07057366abb74a5552ace8946e87
>>> 
>>> Checksums of nifi-1.15.2-source-release.zip:
>>> SHA256: 29fcc35c81de80e0fe3f59044e6fbf21bcf523e656aa64914e7546e1d7705e6b
>>> SHA512:
>>> 
>> cabd1f1ad4942a61df0400488d35521202598c217ad8da97dc2d5abe21136604d1f1bb3de9ceb63bb441943de2e29e3515f5cf63607080094e1418d79eb5216b
>>> 
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/joewitt.asc
>>> 
>>> KEYS file available here:
>>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>> 
>>> 8 issues were closed/resolved for this release:
>>> 
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12351132
>>> 
>>> Release note highlights can be found here:
>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.2
>>> 
>>> Given the nature of the vote and its limited scope
>>> the vote will be open for 24 hours or until we have sufficient
>>> votes (instead of the normal 72 hours).
>>> 
>>> Please download the release candidate and evaluate the necessary items
>>> including checking hashes, signatures, build from source, and test.
>>> Then please vote:
>>> 
>>> [ ] +1 Release this package as nifi-1.15.2
>>> [ ] +0 no opinion
>>> [ ] -1 Do not release this package because...
>>> 
>> 

Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)

[Joey Frazee]

+1 (binding)

- Verified signatures and checksums (except for prior comment on minifi-c2)
- Ran full build on Linux with Java 1.8.0_292
- Ran various flows and ITs
- Built and ran dockermaven Docker build

-joey

> On Nov 5, 2021, at 1:50 PM, Andrew Lim  wrote:
> 
> +1 (binding)
> 
> -Ran full clean install on OS X (Catalina 10.15.2, OpenJDK version 1.8.0_252)
> -Tested secure NiFi with secure NiFi Registry
> -Ran basic flows successfully; tested basic versioned flow functionality
> -Reviewed UI fixes and documentation updates
> 
> Drew
> 
>> On Nov 3, 2021, at 3:29 PM, Joe Witt  wrote:
>> 
>> Hello,
>> 
>> I am pleased to be calling this vote for the source release of Apache
>> NiFi 1.15.0.
>> 
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1186
>> 
>> The source being voted upon and the convenience binaries can be found at:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/
>> 
>> A helpful reminder on how the release candidate verification process works:
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>> 
>> The Git tag is nifi-1.15.0-RC3
>> The Git commit ID is 7fdc07cccdc0e23d4986557a9314e36859704a3b
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=7fdc07cccdc0e23d4986557a9314e36859704a3b
>> 
>> Checksums of nifi-1.15.0-source-release.zip:
>> SHA256: 56fe317248941fdbc6216ec973e6ce898d0f682a54e2d063edbabf8542f5288a
>> SHA512: 
>> 63f10d9127cf1613c29bf2306be3d7a5b931b31304920706e0df5ea2fe87b58c410efed6ac37afc38d12c65e69f14aec7afb926000fc90cc13f15c738cd15c7f
>> 
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/joewitt.asc
>> 
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>> 
>> 234 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350382
>> 
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0
>> 
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build from source, and test.
>> Then please vote:
>> 
>> [ ] +1 Release this package as nifi-1.15.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...
> 

Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)

[Joey Frazee]

JFYI, the .sha256 files for minifi-c2 are wrong/actually sha512.

One interpretation of the release policy is that those aren’t being voted on 
and just for verification, so provided it’s fixed before the release gets 
checked in it might be ok.

-joey

> On Nov 5, 2021, at 8:29 AM, Pierre Villard  
> wrote:
> 
> +1 (binding)
> 
> Went through the usual steps, deployed a secured cluster with NiFi Registry
> on GCP, both configured with OIDC and using Azul JDK 11. Tested some of the
> new features and bug fixes. Looking good, this contains a lot of great
> stuff!
> 
> Thanks for taking care of the release process Joe
> 
> Thanks,
> Pierre
> 
>> Le ven. 5 nov. 2021 à 13:07, Chris Sampson
>>  a écrit :
>> 
>> +1 (non-binding)
>> 
>> - Built and ran from source on Ubuntu 20.04 (Windows 11 WSL2)
>> with AdoptOpenJDK-11.0.11+9
>> - Built and ran Docker Image (dockerhub) using the Dev Artifacts (from
>> re-based NIFI-8779 branch)
>> 
>> - Verified Single User Authentication
>> 
>> - Verified a few recent PRs via import and execution of a Flow definition
>> (S3 => Elasticsearch with some data manipulation in the middle)
>> 
>> 
>> Thanks for RM'ing Joe.
>> 
>> ---
>> *Chris Sampson*
>> IT Consultant
>> chris.samp...@naimuri.com
>> 
>> 
>> On Fri, 5 Nov 2021 at 04:35, David Handermann >> 
>> wrote:
>> 
>>> +1 (binding)
>>> 
>>> - Built from source on Ubuntu 21.10 and macOS 11.6 with Azul Zulu 11.0.12
>>> and Azul Zulu 1.8.0.282
>>> - Ran NiFi on Azul Zulu 11.0.12 and 1.8.0.282
>>> - Verified Single User Authentication and OIDC Authentication with
>> Keycloak
>>> - Verified updates to JWT client storage and cookie handling
>>> - Verified basic operation of NiFi Regstriy
>>> 
>>> - NIFI-6617 Verified encrypted repository configuration using new
>>> properties with BCFKS and PKCS12 KeyStores
>>> - NIFI-7322 Verified SignContentPGP and VerifyContentPGP together with
>>> EncryptContentPGP and DecryptContentPGP using various properties
>>> - NIFI-8424 Verified absence of HtmlDocumentWriter warnings on startup
>>> - NIFI-8943 Built, launched, and restarted Docker container verifying
>>> persistence of random sensitive properties key
>>> - NIFI-9059 Ran simple data flow using NiFi Stateless on Java 11
>>> - NIFI-9223 Verified ListenSyslog listens on 0.0.0.0 using default
>>> properties
>>> - NIFI-9251 Verified absence of unused generated password in NiFi
>> Registry
>>> - NIFI-9266 Verified Azure Key Vault Secret Sensitive Property Provider
>>> configuration for nifi.properties
>>> - NIFI-9322 Verified corrected REST API documentation for OIDC and SAML
>>> resources
>>> - NIFI-9339 Verified JoltTransformJSON custom UI works as expected
>>> 
>>> Thanks for managing the release process Joe!
>>> 
>>> Regards,
>>> David Handermann
>>> 
>>> 
>>> 
>>> 
>>> On Thu, Nov 4, 2021 at 6:33 PM Matt Burgess 
>> wrote:
>>> 
 +1 Release this package as nifi-1.15.0
 
 Ran through the release helper and some tests to verify various Jiras
 were included. Thanks again for RM'ing Joe!
 
 On Wed, Nov 3, 2021 at 3:29 PM Joe Witt  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.15.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> 
>> https://repository.apache.org/content/repositories/orgapachenifi-1186
> 
> The source being voted upon and the convenience binaries can be found
>>> at:
> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.15.0/
> 
> A helpful reminder on how the release candidate verification process
 works:
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> 
> The Git tag is nifi-1.15.0-RC3
> The Git commit ID is 7fdc07cccdc0e23d4986557a9314e36859704a3b
> 
 
>>> 
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=7fdc07cccdc0e23d4986557a9314e36859704a3b
> 
> Checksums of nifi-1.15.0-source-release.zip:
> SHA256:
>>> 56fe317248941fdbc6216ec973e6ce898d0f682a54e2d063edbabf8542f5288a
> SHA512:
 
>>> 
>> 63f10d9127cf1613c29bf2306be3d7a5b931b31304920706e0df5ea2fe87b58c410efed6ac37afc38d12c65e69f14aec7afb926000fc90cc13f15c738cd15c7f
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 234 issues were closed/resolved for this release:
> 
 
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350382
> 
> Release note highlights can be found here:
> 
 
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.15.0
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary
>> items
> including checking hashes, signatures, build 

Re: Flood of JIRAs and presumably PRs to follow for junit-5 migration?

[Joey Frazee]

Maybe this is an exception to the single squashed commit guidance for the 
initial pull?

I assume the intent is to make incremental progress and not have a PR with a 
hundred files affected, but if the different module changes corresponded to a 
different commit, GH will make it easy enough to have a draft and review each 
commit in isolation.

Would that be a reasonable approach?

-joey

> On Aug 25, 2021, at 9:36 AM, Joe Witt  wrote:
> 
> Mike,
> 
> Seeing a pretty stunning flood of JIRAs for 'Refactor nifi-bla to use
> JUnit 5' and I'm guessing we'll see the same in terms of PRs.  This is
> a really high administrative overhead approach to this.
> 
> Why not break this into one or maybe a few JIRAs/PRs total?
> 
> Thanks

Re: [DISCUSS] NiFi 2.0 Release Goals

[Joey Frazee]

I quite like this idea. It seemed like the first 2.0 release had been being 
held out for some bigger innovations (nar registry?), but that has also pushed 
out making nice breaking changes.

What would be the mechanics here? Just starting feeding all the candidates into 
JIRA? Listing out candidates in a wiki page first?

-joey

> On Jul 23, 2021, at 7:57 AM, Joe Witt  wrote:
> 
> Jon
> 
> You're right we have to be careful and you're right there are still
> significant Java 8 users out there.  But we also have to be careful
> about security and sustainability of the codebase.  If we had talked
> about this last year when that article came out I'd have agreed it is
> too early.  Interestingly that link seems to get updated and I tried
> [1] and found more recent data (not sure how recent).  Anyway it
> suggests Java 8 is still the top dog but we see good growth on 11.  In
> my $dayjob this aligns to what I'm seeing too.  Customers didn't seem
> to care about Java 11 until later half last year and now suddenly it
> is all over the place.
> 
> I think once we put out a NiFi 2.0 release we'd see rapid decrease in
> work on the 1.x line just being blunt.  We did this many years ago
> with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
> so) but it was purely bug fix/security related bits.  We would need to
> do something similar.  But feature work would almost certainly go to
> the 2.x line.  Maybe there are other workable models but my instinct
> suggests this is likely to follow a similar path.
> 
> ...anyway I agree it isn't that easy of a call to dump Java 8.  We
> need to make the call in both the interests of the user base and the
> contributor base of the community.
> 
> [1] https://www.jetbrains.com/lp/devecosystem-2021/java/
> 
> 
> Thanks
> Joe
> 
>> On Fri, Jul 23, 2021 at 7:46 AM Joe Witt  wrote:
>> 
>> Russ
>> 
>> Yeah the flow registry is a key part of it.  But also now you can
>> download the flow definition in JSON (upload i think is there now
>> too).  Templates offered a series of challenges such as we store them
>> in the flow definition which has made flows massive in an unintended
>> way which isn't fun for cluster behavior.
>> 
>> We have a couple cases where we headed down a particular concept and
>> came up with better approaches later.  We need to reconcile these with
>> the benefit of hindsight, and while being careful to be not overly
>> disruptive to existing users, to reduce the codebase/maintenance
>> burden and allow continued evolution of the project.
>> 
>> Thanks
>> 
>>> On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman  
>>> wrote:
>>> 
>>> Joe,
>>> 
>>> I apologize for the off-topic intrusion, but what replaces templates?
>>> The Registry? Templates rocked and we have used them since 0.5.x.
>>> 
>>> Russ
>>> 
>>> On 7/23/21 8:31 AM, Joe Witt wrote:
 David,
 
 I think this is a highly reasonable approach and such a focus will
 greatly help make a 2.0 release far more approachable to knock out.
 Not only that but tech debt reduction would help make work towards
 major features we'd think about in a 'major release' sense more
 approachable.
 
 We should remove all deprecated things (as well as verify we have the
 right list).  We should remove/consider removal of deprecated concepts
 like templates.  We should consider whether we can resolve the various
 ways we've handled what are now parameters down to one clean approach.
 We should remove options in the nifi.properties which turn out to
 never be used quite right (if there are).  There is quite a bit we can
 do purely in the name of tech debt reduction.
 
 Lots to consider here but I think this is the right discussion.
 
 Than ks
 
 On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende  wrote:
> I'm a +1 for this... Not sure if this falls under "Removing Deprecated
> Components", but I think we should also look at anything that has been
> marked as deprecated throughout the code base as a candidate for
> removal. There are quite a few classes, methods, properties, etc that
> have been waiting for a chance to be removed.
> 
> On Fri, Jul 23, 2021 at 10:13 AM David Handermann
>  wrote:
>> Team,
>> 
>> With all of the excellent work that many have contributed to NiFi over 
>> the
>> years, the code base has also accumulated some amount of technical debt. 
>> A
>> handful of components have been marked as deprecated, and some components
>> remain in the code base to support integration with old versions of 
>> various
>> products. Following the principles of semantic versioning, introducing a
>> major release would provide the opportunity to remove these deprecated 
>> and
>> unsupported components.
>> 
>> Rather than focusing the next major release on new features, what do you
>> think about focusing on technical debt removal? This approach 

Re: [DISCUSS] NiFi Registry -> NiFi consensus

[Joey Frazee]

+1

-joey

> On Jul 16, 2021, at 11:54 AM, Bryan Bende  wrote:
> 
> +1
> 
>> On Fri, Jul 16, 2021 at 2:31 PM Kevin Doran  wrote:
>> 
>> +1.
>> 
 On Jul 16, 2021, at 2:28 PM, Joe Witt  wrote:
>>> 
>>> Yep definitely. Thought this was sorted via the JIRA.
>>> 
>>> +1
>>> 
>>> On Fri, Jul 16, 2021 at 11:26 AM David Handermann
>>>  wrote:
 
 Thanks for handling this, Matt. As one of the reviewers on the PR, I vote
 +1 (non-binding).
 
 Regards,
 David Handermann
 
 On Fri, Jul 16, 2021 at 1:20 PM Matt Burgess  wrote:
 
> All,
> 
> We've been asked to record our consensus for the move of NiFi Registry
> to the NiFi codebase in a mailing list thread for posterity. Most
> discussion happened on the PR but INFRA would like a link to this
> thread showing consensus from PMC members, committers, and the
> community.
> 
> I didn't put my +1 on the PR but I am in favor of moving the NiFi
> Registry codebase into NiFi :) Please feel free to share your thoughts
> as well.
> 
> Thank you,
> Matt
> 
>> 

Re: [VOTE] Release Apache NiFi 1.14.0 (rc2)

[Joey Frazee]

+1 (binding)

- Verified signatures and checksums
- Completed build with contrib-check on OpenJDK with 1.8.0_292 and 11.0.11+9 on 
Arch Linux
- Ran flows standalone and in cluster with Java 11 Docker images built from 
source
- Ran the registry
- Ran Azure storage ITs
- Tested new decommission functionality

I noticed a couple random things. It looks like there isn't a zip file version 
of the stateless assembly. And there are a couple of new Docker quirks -- the 
README doesn't include anything about setting NIFI_WEB_HTTP_PORT to use it sans 
TLS (potentially intentional?) and NIFI_WEB_PROXY_HOST doesn't seem to be 
working with the default single user auth. Will open some issues.

-joey

> On Jul 13, 2021, at 11:35 AM, Marton Szasz  wrote:
> 
> +1 (binding)
> 
> Followed the release helper guide. Verified the binaries using a simple flow.
> 
> OS: Gentoo Linux amd64
> Maven 3.8.1
> JDKs:
> - IcedTea JDK 3.16.0 (openjdk version "1.8.0_252")
> - OpenJDK 8.292_p10 (custom build, bootstrapped with IcedTea)
> - OpenJDK 11.0.11_p9-r1 (custom build, bootstrapped with AdoptOpenJDK 
> binaries)
> - and checked the released convenience binaries
> 
> The nifi binaries worked with all of the above. I had to skip tests
> during build with icedtea, because it threw errors, most likely due to
> the old java version.
> 
>> On Tue, 13 Jul 2021 at 16:43, Mark Bean  wrote:
>> 
>> +1 (non-binding)
>> 
>> Checked out tag nifi-1.14.0-RC2, confirmed commit hash
>> Built from source code using contrib-check profile
>>  Linux, Java 8, Maven 3.8.1
>>  MacOS, Java 11, Maven 3.6.3
>> Installed secure 2-node cluster using embedded Zookeeper (ZK not secured
>> with TLS)
>> Installed nifi-registry
>> Ran sample flow simply for basic functionality; included network activity
>> with Post/ListenHTTP
>> Version controlled a process group to the 1.14.0 NiFi Registry
>> Checked out same flow from 1.14.0 NiFi Registry
>> Verified updates to NiFi Registry specific to this release functioned as
>> expected
>> 
>> 
>> On Tue, Jul 13, 2021 at 12:00 PM Chris Sampson
>>  wrote:
>> 
>>> +1 (non-binding)
>>> 
>>> - Built from source + test, checked hashes
>>> - Built Docker Image using available dev convenience artifacts (note that
>>> this led me to raise NIFI-8779 [1] and PR#5213 [2] to make building the
>>> Docker Image from non-default URLs possible with the provided DockerBuild
>>> script)
>>> - Ran single secure (by default) node (Docker)
>>> - Tested a few simple Flows
>>> - Tested simple ConsumeKinesis Flow
>>> 
>>> - Unit Tests failed for nifi-registry-web-api when running from the
>>> repository root, but running the tests again separately afterwards was
>>> successful as was running all tests under nifi-registry-core together
>>> 
>>> Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
>>> Java version: 11.0.11, vendor: AdoptOpenJDK
>>> Default locale: en_GB, platform encoding: UTF-8
>>> OS name: "linux", version: "5.11.0-22-generic", arch: "amd64", family:
>>> "unix"
>>> 
>>> O/S: Ubuntu 21.04
>>> 
>>> ---
>>> *Chris Sampson*
>>> IT Consultant
>>> chris.samp...@naimuri.com
>>> 
>>> 
>>> 
 On Tue, 13 Jul 2021 at 15:33, Nathan Gough  wrote:
>>> 
 +1 (non-binding)
 
 - Built from source + test, checked hashes
 - Ran with a secure three node cluster and X509 authentication, tested
>>> S2S
 and some other processors
 - Ran single secure node with OIDC authentication (G Suite)
 - Ran single secure node with SAML authentication (G Suite)
 
 java -version
 
 openjdk version "1.8.0_292"
 
 OpenJDK Runtime Environment (Zulu 8.54.0.21-CA-macosx) (build
 1.8.0_292-b10)
 
 OpenJDK 64-Bit Server VM (Zulu 8.54.0.21-CA-macosx) (build 25.292-b10,
 mixed mode)
 
 mvn -version
 Apache Maven 3.6.3
 
 
 Thanks for running the RC process again Joe!
 
 
 On Tue, Jul 13, 2021 at 9:33 AM Otto Fowler 
 wrote:
 
> +1 ( non binding )
> Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
> Maven home: /usr/local/Cellar/maven/3.8.1/libexec
> Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime:
> /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"
> 
> 
> 
> From: Joe Witt  
> Reply: dev@nifi.apache.org  
> Date: July 10, 2021 at 18:40:26
> To: dev@nifi.apache.org  
> Subject:  [VOTE] Release Apache NiFi 1.14.0 (rc2)
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache
> NiFi 1.14.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1183
> 
> The source being voted upon and the convenience binaries can be found
>>> at:
> 

Re: Dropping support for Java 8

[Joey Frazee]

Kevin, it’s recently possible to specify the underlying openjdk image tag as a 
property in the Maven build, e.g., -Pdocker  -Ddocker.image.tag=11-jre so it 
should be easier to start publishing those now too if it’s decided it’s a good 
idea.

The default remains 8 for the sorts of concerns being discussed, but this 
provides the flexibility for people to use an official source release to build 
11-based images.

-joey

> On Mar 26, 2021, at 8:13 AM, Joe Witt  wrote:
> 
> Good thread.  I'd say when a NiFi 2 line happens Java 8 would be gone
> completely and we would/should consider only supporting the latest LTS
> line perhaps.
> 
>> On Fri, Mar 26, 2021 at 8:05 AM Kevin Doran  wrote:
>> 
>> Hi JL,
>> 
>> It’s worth discussing/considering changes such as this periodically, so 
>> thanks for bringing it up.
>> 
>> Personally, I would be hesitant to make such a large change. While it would 
>> likely be a net-positive for the NiFi image itself, I think it would impact 
>> a number of community members that have Dockerfile’s that use our image as a 
>> starting point.
>> 
>> A GitHub code search [1] seems to confirm this, showing >100 Dockerfiles 
>> that contain “FROM apache/nifi*”
>> 
>> For NiFi 1.x, I think the best we could do is leverage tagging to offer 
>> image variants that differ in layers we build upon, for example OS or 
>> JDK/JRE variants. This seems to be a popular method, for example, Apache 
>> Tomcat offers a multiple of combinations of version, JDK, and OS [2].
>> 
>> So if it would be beneficial, we could add official images for other jdk 
>> versions indicated by tags, for example apache/nifi:1.13.2, 
>> apache/nifi:1.13.2-jre11, etc.
>> 
>> I believe this was part of the plan for the (empty) apache/nifi-container 
>> code repository [3]. I think the intention was always to build out a richer 
>> set of diverse container images based on files in this repository, which 
>> could be maintained/released decoupled from the NiFi source code itself. 
>> With so many in the community running containerized NiFi, perhaps it's worth 
>> reviving that discussion to see what, if anything, would be most valuable to 
>> add to our container offerings.
>> 
>> For NiFi 2 we can and should definitely consider what changes we want to 
>> make to our “default” base image, including which JRE.
>> 
>> [1] 
>> https://github.com/search?l==%22FROM+apache%2Fnifi%22+language%3ADockerfile=code
>>  
>> 
>> [2] https://hub.docker.com/_/tomcat?tab=description 
>> 
>> [3] https://github.com/apache/nifi-container 
>> 
>> 
>> Thanks!
>> Kevin
>> 
 On Mar 26, 2021, at 07:49, José Luis Pedrosa  wrote:
>>> 
>>> Hi All
>>> 
>>> I see that the docker images generated are based "openjdk:8-jre" should we
>>> (I volunteer) to update them to "11-jre"? as both versions are supported (8
>>> and 11) I don't see any reason why not, and will be more future proof.
>>> 
>>> Any opinions?
>>> 
>>> JL
>>> 
>>> 
>>> 
 On Thu, Mar 18, 2021 at 1:59 PM Joe Witt  wrote:
>>> 
 Mark
 
 That we can do with a NiFi 2 release for sure. Before then it isnt great.
 
 Oracles JVM is not what I see mostly in the wild any longer and we do see a
 ton of Java 8 usage still.
 
 We can and should drop Java 8 but itll be important to do it when we cut a
 lot of crud out as well.
 
 Thanks
 
 On Thu, Mar 18, 2021 at 6:03 AM Mark Bean  wrote:
 
> I'd like to discuss migrating to Java 11 as the minimum required Java
> version for NiFi. We've been supporting both Java 8 and Java 11 for some
> time now. There is increased overhead in verifying builds with two
> different versions. There are some features and syntax available in Java
 11
> which cannot be used in order for NiFi to remain compatible with both
> versions. Java 8 premier support (Oracle) runs out in one year. Java 17 -
> the next LTS version - is due out later this year.
> 
> There should be plenty of lead time for users to prepare for the
> transition. So I wanted to start the discussion well in advance of when
 we
> discontinue Java 8 support. And, logistically how do we best inform the
> community of upcoming changes like this?
> 
> Thanks,
> Mark
> 
 
>> 

Re: [ANNOUNCE] New NiFi PMC Member Joey Frazee

[Joey Frazee]

I want to make sure to say thanks for the recognition! I’m thrilled to be a 
part of this. The help and collaboration from the project members and 
contributors over the past years has made this a ton of fun and I’m very happy 
to be continuing with more of it.

-joey

> On Mar 25, 2021, at 11:57 AM, Otto Fowler  wrote:
> 
> Congratulations!
> 
>> On Thu, Mar 25, 2021 at 2:54 PM Joe Witt  wrote:
>> 
>> NiFi Community,
>> 
>> On behalf of the Apache NiFi PMC, I am pleased to announce that Joey Frazee
>> has accepted the PMC's invitation to join the Apache NiFi PMC.
>> 
>> Joey has been a contributor and committer of Apache NiFi for several
>> years and has been involved across the whole spectrum of code reviews,
>> JIRAs, code contributions, maintained a great 'awesome nifi' github
>> site, and more.
>> 
>> Please join us in congratulating and welcoming Joey to the Apache NiFi PMC.
>> 
>> Congratulations Joey!
>> 

Re: [VOTE] Release Apache NiFi 1.13.2

[Joey Frazee]

+1 (non-binding)

- Verified checksums and signatures and ran full build/contrib check on Java 11 
and 1.8 on Linux and macOS
- Validated regression and fix for NIFI-8337 using flow similar to the one in 
JIRA
- Checked docs update in NIFI-8324
- Did a quick check of license and notice

-joey

> On Mar 18, 2021, at 10:37 AM, Mark Payne  wrote:
> 
> +1 (binding)
> 
> Was able to build, and successfully run all system tests, including those 
> added for NIFI-8337. Also started up and manually verified that NIFI-8337 has 
> been addressed.
> 
> Thanks
> -Mark
> 
>> On Mar 18, 2021, at 1:09 PM, Matt Burgess  wrote:
>> 
>> +1 Release this package as nifi-1.13.2
>> 
>> Ran through release helper, verified fixes for NIFI-8297 and
>> NIFI-8334/8342 on an unsecured standalone instance. Thanks for RM'ing
>> and for the quick turnaround Joe!
>> 
>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>> Maven home: /Users/mburgess/.sdkman/candidates/maven/current
>> Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime:
>> /Users/mburgess/.sdkman/candidates/java/8.0.282.hs-adpt/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"
>> 
>>> On Thu, Mar 18, 2021 at 1:29 AM Joe Witt  wrote:
>>> 
>>> Hello,
>>> 
>>> Please note this has a shorter than normal vote period as this
>>> corrects an important regression introduced in 1.13.1 and is purely a
>>> focused bug fix release.
>>> 
>>> I am pleased to be calling this vote for the source release of Apache
>>> NiFi 1.13.2.
>>> 
>>> The source zip, including signatures, digests, etc. can be found at:
>>> https://repository.apache.org/content/repositories/orgapachenifi-1180
>>> 
>>> The source being voted upon and the convenience binaries can be found at:
>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.2/
>>> 
>>> A helpful reminder on how the release candidate verification process works:
>>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>>> 
>>> The Git tag is nifi-1.13.2-RC1
>>> The Git commit ID is 174938e5e3bfe36951d5607d0d53e78604b0e07b
>>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=174938e5e3bfe36951d5607d0d53e78604b0e07b
>>> 
>>> Checksums of nifi-1.13.2-source-release.zip:
>>> SHA256: db18f57b4629367f21f70ec9a332294ec6b6c090a661c974b0fa7a80b09a79be
>>> SHA512: 
>>> 279c5701fd7b6d76206f5e79bdf1340d432a8e4834b8e786559095781c77c96a4594e1da040c6b404d1bab615acf5aa9dc60d289e460e6d7261f454f6210c66b
>>> 
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/joewitt.asc
>>> 
>>> KEYS file available here:
>>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>> 
>>> 5 issues were closed/resolved for this release:
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350008
>>> 
>>> Release note highlights can be found here:
>>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.2
>>> 
>>> The vote will be open for 36 hours.
>>> Please download the release candidate and evaluate the necessary items
>>> including checking hashes, signatures, build from source, and test.
>>> Then please vote:
>>> 
>>> [ ] +1 Release this package as nifi-1.13.2
>>> [ ] +0 no opinion
>>> [ ] -1 Do not release this package because...
> 

Re: [VOTE] Release Apache NiFi 1.13.1

[Joey Frazee]

+1 (non-binding)

- Verified checksums and signatures
- Ran full build on Java 1.8 and 11 on Linux
- Verified NIFI-3383, NIFI-8231, and NIFI-8200

-joey

> On Mar 12, 2021, at 10:28 AM, Bryan Bende  wrote:
> 
> +1 (binding)
> 
> - Verified everything in the standard release helper
> - Setup secure standalone instance with SAML authentication
> 
>> On Fri, Mar 12, 2021 at 10:17 AM Marton Szasz  wrote:
>> 
>> +1 (non-binding)
>> 
>> Followed the release helper guide, tested the binary with a simple flow.
>> 
>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>> Maven home: /usr/share/maven-bin-3.6
>> Java version: 1.8.0_252, vendor: IcedTea, runtime: 
>> /opt/icedtea-bin-3.16.0/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "linux", version: "5.10.12-gentoo-x86_64", arch: "amd64",
>> family: "unix"
>> 
>> 
>>> On Fri, 12 Mar 2021 at 13:35, Otto Fowler  wrote:
>>> 
>>> +1
>>> 
>>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>>> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
>>> Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
>>> /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"
>>> 
>>> 
 On Mar 11, 2021, at 11:29, Joe Witt  wrote:
 
 Hello,
 
 I am pleased to be calling this vote for the source release of Apache NiFi
 1.13.1.
 
 The source zip, including signatures, digests, etc. can be found at:
 https://repository.apache.org/content/repositories/orgapachenifi-1179
 
 The source being voted upon and the convenience binaries can be found at:
 https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.1/
 
 A helpful reminder on how the release candidate verification process works:
 https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
 
 The Git tag is nifi-1.13.1-RC1
 The Git commit ID is acbc217cb7002d7489421f4d346b995a43b6ea01
 https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=acbc217cb7002d7489421f4d346b995a43b6ea01
 
 Checksums of nifi-1.13.1-source-release.zip:
 SHA256: 0a397df640e579720e26699e38a3738c5be05af01aad8aaeefc04eb58591faac
 SHA512:
 7f8df759d4345ccd6e75c169bd0aab1b7f4f64bf5a8b11b45bc1d7c8163acd0035922dcdbef232392279f4ea0710d4a97c55d480281bfe1d50b6401295633d48
 
 Release artifacts are signed with the following key:
 https://people.apache.org/keys/committer/joewitt.asc
 
 KEYS file available here:
 https://dist.apache.org/repos/dist/release/nifi/KEYS
 
 48 issues were closed/resolved for this release:
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
 
 Release note highlights can be found here:
 https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1
 
 The vote will be open for 72 hours.
 Please download the release candidate and evaluate the necessary items
 including checking hashes, signatures, build from source, and test. Then
 please vote:
 
 [ ] +1 Release this package as nifi-1.13.1
 [ ] +0 no opinion
 [ ] -1 Do not release this package because...
>>> 

Re: [VOTE] Release Apache NiFi 1.13.0 (rc4)

[Joey Frazee]

+1 (non-binding)

- Verified checksums, signatures, and commit id 
- Ran builds with Java 1.8 and 11 on Linux and macOS, and validated RPM build 
profile 
- Tested cluster coordination and state management with both embedded and 
external ZooKeepers with and without TLS
- Verified fix for PutAzureBlobStorage OOME and tested Blob and Queue storage 
with Azurite emulator
- Tested new --wait-for-init on several OSes

Thanks all!

-joey

> On Feb 11, 2021, at 9:29 AM, Matt Burgess  wrote:
> 
> +1 (binding)
> 
> Ran through release helper and tested various flows, some related to
> changes that have gone into 1.13.0 and specifically RC4. Thanks for
> RM'ing Joe!
> 
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime:
> /Users/mburgess/.sdkman/candidates/java/8.0.282.hs-adpt/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"
> 
>> On Wed, Feb 10, 2021 at 11:37 PM Joe Witt  wrote:
>> 
>> Hello,
>> 
>> I am pleased to be calling this vote for the source release of Apache NiFi
>> 1.13.0.
>> 
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1178
>> 
>> The source being voted upon and the convenience binaries can be found at:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
>> 
>> A helpful reminder on how the release candidate verification process works:
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>> 
>> The Git tag is nifi-1.13.0-RC4
>> The Git commit ID is 3bc6a122091214b33eee17a270163d7ca26e2a0c
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=3bc6a122091214b33eee17a270163d7ca26e2a0c
>> 
>> Checksums of nifi-1.13.0-source-release.zip:
>> SHA256: 95efe5db38e973c9f4062a7b2c95fdc5dad31d7c5e1d36ce1776b9b227c89b9c
>> SHA512:
>> d7dd9e851341ebd605784142a7861935f6f814bc612499013456a15713bc9119e426df8f26445c260bdb25cbfc21822cf0d44985bf372a065c9d8597953a3c4a
>> 
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/joewitt.asc
>> 
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>> 
>> 260 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
>> 
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0
>> 
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build from source, and test. Then
>> please vote:
>> 
>> [ ] +1 Release this package as nifi-1.13.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...

Re: [discuss] we need to enable secure by default...

[Joey Frazee]

This is probably a very important and overdue change.

I think it’s important to recognize that there’s a lot of different ways to 
unintentionally end up with a publicly accessible application and it can’t 
always be pinned to one person or role. Routing, firewall rules, OS admin, NiFi 
admin aren’t always the same people.

I like the idea of the localhost change.

I’d also be in favor of a making build profiles for opting in to some of the 
more dangerous restricted processors, or delivering the binaries for those 
separately.

This exploit allows people to run a reverse shell and it probably doesn’t stop 
on the NiFi host, so good Internet citizenship sort of says something must be 
done.

-joey

> On Feb 10, 2021, at 10:14 AM, Joe Witt  wrote:
> 
> Nathan just commented on the Jira about an obvious first step being to
> restrict HTTP to localhost only by default.  This is an easy and
> immediately doable first step.  I am planning to wait for the RC4 creation
> for that PR to land and get merged.
> 
> Thanks
> 
>> On Wed, Feb 10, 2021 at 10:24 AM Nathan Gough  wrote:
>> 
>> I 100% agree that something needs to be done. We cannot allow NiFi to build
>> a reputation that it is 'insecure' by allowing its default installation to
>> start up without any security. Especially considering how much work we put
>> in to make sure it IS a secure product that integrates with many
>> applications in a secure way. Security reputation is very important for
>> software. If some major exploitation of NiFi were to happen in the future,
>> we should be able to confidently say that we did our absolute best to
>> create a secure application. We shouldn't point at new users and say 'they
>> didn't configure it right'.
>> 
>> Personally, I am in favor of providing automatically generated certificates
>> and requiring the user to insert the client certificate in their browser,
>> and providing instructions and perhaps a YouTube video on how to do that.
>> Yes, X509 certificate errors are confusing and can be difficult for
>> beginners to troubleshoot. But those beginner users will also be the most
>> likely to use NiFi insecurely, connect it to the internet, and become part
>> of a user base who got burnt by NiFi being 'insecure'. I acknowledge this
>> is increasing the barrier for entry. If we intend to use a
>> username/password + server cert for HTTPs but no client cert, as stated
>> above we could automatically generate the password and provide this to the
>> user in a log or file.
>> 
>> 
>> On Wed, Feb 10, 2021 at 12:21 PM David Handermann <
>> exceptionfact...@gmail.com> wrote:
>> 
>>> Integrating an option to use Let's Encrypt would be a great improvement
>>> over self-signed certificates, but it is difficult to support that for
>>> instances of NiFi running on servers without Internet access.  Even when
>>> running NiFi for local development purposes, the development system is
>> not
>>> likely to have a publicly addressable DNS name, so Let's Encrypt is not
>> an
>>> option in those cases.
>>> 
>>> Regards,
>>> David Handermann
>>> 
 On Wed, Feb 10, 2021 at 11:09 AM Joe Witt  wrote:
>>> 
 Otto
 
 Installers like you mention are inherently brutal for portability so
>> very
 difficult for us in the community.  From the vendor world we of course
>> do
 things like that.  I think in apache nifi land we need a default 'even
>>> for
 devs' which is not wide open.
 
 James
 
 I do think adding such a warning is good.  But it doesn't help avoid
>>> these
 wildly abusive cases.  We need a secure by default path.  We can
>> probably
 do better than the self signed path if we add a 'before running' step
>> in
 the CLI for the user.  Not sure.
 
 Thanks
 
 On Wed, Feb 10, 2021 at 10:05 AM James Srinivasan <
 james.sriniva...@gmail.com> wrote:
 
> Would a suitably large warning on the http ui be a good starting
>> point?
> 
> Browsers are getting increasingly wary of self signed certs and we
 probably
> don't want to be encouraging people to ignore them.
> 
> What about easier acme+certbot support? (notwithstanding all the non
 public
> deployments)
> 
> On Wed, 10 Feb 2021, 15:25 Otto Fowler, 
>>> wrote:
> 
>> Aren’t most of these applications installed by an installer?
>> Maybe we can have one or more installers that setup a secure
>>> instance,
> and
>> those installers
>> could be the *production* nifi, and keep the zip distribution open
>>> for
>> developers?
>> 
>> 
>>> On Feb 10, 2021, at 10:04, David Handermann <
> exceptionfact...@gmail.com>
>> wrote:
>>> 
>>> I agree that a generated password is the way to go, and the
>>> challenge
> is
>>> making it available to the user.  Depending on how it is stored
>> for
 the
>>> identity provider, having a command line tool to read and print
>> it
> could
>> be
>>> a 

Re: [VOTE] Release Apache NiFi 1.13.0 (rc3)

[Joey Frazee]

Hey, I found a problem in nifi.sh introduced in NIFI-8123.

nifi.sh points to /bin/sh, the above changes added some uses of the declare 
builtin which isn’t available in all shells (ash, dash), so those commands fail 
creating some unexpected output at startup, and the new wait functionality 
won’t work.

Yeah, bash is usually available but the script points to /bin/sh and that’s not 
always one those — and isn’t on some Ubuntus.

I’m inclined to give this a -1 (non-binding) unless I’ve gotten this wrong.

-joey

> On Feb 7, 2021, at 8:38 PM, Otto Fowler  wrote:
> 
> +1
> 
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
> Java version: 1.8.0_282, vendor: AdoptOpenJDK, runtime: 
> /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.16", arch: "x86_64", family: “mac"
> 
> 
>> On Feb 7, 2021, at 01:09, Joe Witt  wrote:
>> 
>> Hello,
>> 
>> I am pleased to be calling this vote for the source release of Apache NiFi
>> 1.13.0.
>> 
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1177
>> 
>> The source being voted upon and the convenience binaries can be found at:
>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
>> 
>> A helpful reminder on how the release candidate verification process works:
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>> 
>> The Git tag is nifi-1.13.0-RC3
>> The Git commit ID is 547127c07aa9da2698b8b6f65e990b986065ef18
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=547127c07aa9da2698b8b6f65e990b986065ef18
>> 
>> Checksums of nifi-1.13.0-source-release.zip:
>> SHA256: 670f6c97a5a2ea488b3006cd6267fa9f02aeeb0d5d011b02f43f76c51ee787bc
>> SHA512:
>> a9ed5d8b5380bedbb18c94c22ba577901392141ba5f07c0c812e548d88df76a365d99ab7cb890d1803acb2e0f0215c1d6ed9dc4704af1da7e0854ef96fa649ad
>> 
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/joewitt.asc
>> 
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>> 
>> 253 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
>> 
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0
>> 
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build from source, and test. Then
>> please vote:
>> 
>> [ ] +1 Release this package as nifi-1.13.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...
> 

Re: [VOTE] Release Apache NiFi 1.13.0 (rc2)

[Joey Frazee]

+1 (non-binding)

- Verified checksums, signatures, and commit id 
- Ran builds with Java 1.8 and 11 on Linux and macOS, and validated RPM build 
profile 
- Tested cluster coordination and state management with both embedded and 
external ZooKeepers with TLS enabled and disabled
- Verified fix for PutAzureBlobStorage OOME and tested Blob and Queue storage 
with Azurite emulator

-joey

> On Feb 2, 2021, at 3:16 PM, Sushil Kumar  wrote:
> 
> +1 (non-binding) Release this package as nifi-1.13.0
> 
> Deployed this via helm chart(https://github.com/sushilkm/nifi-chart) on
> kubernetes.
> Thank you to all the contributors and reviewers.
> 
> 
>> On Tue, Feb 2, 2021 at 11:02 AM Matt Burgess  wrote:
>> 
>> +1 (binding) Release this package as nifi-1.13.0
>> 
>> Ran through release helper and tried various flows using some of the
>> new features and capabilities added in 1.13.0.  Thanks for RM'ing Joe!
>> 
>>> On Mon, Feb 1, 2021 at 8:10 PM Joe Witt  wrote:
>>> 
>>> Hello,
>>> 
>>> I am pleased to be calling this vote for the source release of Apache
>> NiFi
>>> 1.13.0.
>>> 
>>> The source zip, including signatures, digests, etc. can be found at:
>>> https://repository.apache.org/content/repositories/orgapachenifi-1176
>>> 
>>> The source being voted upon and the convenience binaries can be found at:
>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.13.0/
>>> 
>>> A helpful reminder on how the release candidate verification process
>> works:
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>>> 
>>> The Git tag is nifi-1.13.0-RC2
>>> The Git commit ID is c27e59fc679a2e982102a75b8b8df2b0f062af23
>>> 
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=c27e59fc679a2e982102a75b8b8df2b0f062af23
>>> 
>>> Checksums of nifi-1.13.0-source-release.zip:
>>> SHA256: 4913dd3d943afac710d1a277bf31beebf7a6207a20e1148849d69511f44fc97b
>>> SHA512:
>>> 
>> dc9935f0eb8692cd8493f5863bc8ae2ef0b52653fa69ff8b9a7e8db7dbd9ec6561f6ffdca4a1b55e43b289d04f5671f5ab4de30999838c5fca5c282c00a7c7f8
>>> 
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/joewitt.asc
>>> 
>>> KEYS file available here:
>>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>> 
>>> 238 issues were closed/resolved for this release:
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348700
>>> 
>>> Release note highlights can be found here:
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.0
>>> 
>>> The vote will be open for 72 hours.
>>> Please download the release candidate and evaluate the necessary items
>>> including checking hashes, signatures, build from source, and test. Then
>>> please vote:
>>> 
>>> [ ] +1 Release this package as nifi-1.13.0
>>> [ ] +0 no opinion
>>> [ ] -1 Do not release this package because...
>> 

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Joey Frazee]

You’ve fixed it now by removing it, but I took a look at the history and that 
module just wasn’t a dependency of anything anymore and the release plugin 
doesn’t automatically update the version in that case.

I think the RM guidance might just be “check that  was correctly 
updated” to catch this when it might matter.

-joey

> On Feb 1, 2021, at 1:33 PM, Joe Witt  wrote:
> 
> Team
> 
> starting RC2.  Joey Frazee noted "I noticed that
> nifi-docker/dockermaven-stateless/pom.xml
> and nifi-docker/docker-compose/docker-compose.yml in the source package and
> tag are still using 1.13.0-SNAPSHOT" while looking at RC1.  I have no
> instructions/guidance on what to do with those as an RM and I've never
> modified them before.  If we're going to keep these in the build, and I
> would advocate actually that we remove them, then someone needs to
> determine the whole lifecycle of how to manage/maintain them and
> document the steps for a RelMgr.  The dockerhub module we do have guidance
> for and they're currently correct.
> 
> Thanks
> 
>> On Mon, Feb 1, 2021 at 10:14 AM Joe Witt  wrote:
>> 
>> Team,
>> 
>> Will put RC2 for 1.13 back up today hopefully.  The fix is in for the
>> issue that sank the release.
>> 
>> Thanks
>> Joe
>> 
>>> On Thu, Jan 28, 2021 at 8:18 AM Joe Witt  wrote:
>>> 
>>> Otto
>>> 
>>> Truth is we dont know.  We cannot readily test with the amazing matrix
>>> that exists.  So we are leaning toward ensuring recent versions work as
>>> expected.
>>> 
>>> The goal is java 8 and 11 broadly right now. But these internal changes
>>> of exception types being used are brutal.
>>> 
>>> Thanks
>>> 
>>> On Thu, Jan 28, 2021 at 8:15 AM Otto Fowler 
>>> wrote:
>>> 
>>>> Thanks Joe,
>>>> 
>>>> I’m started down that road right after I sent this.  I’ll update when I
>>>> know.
>>>> I _have_ been building nifi with this jdk all along though.
>>>> 
>>>> If there are jdk minimum requirements, should they be documented ( or
>>>> are they and I missed it)?
>>>> 
>>>>> On Jan 28, 2021, at 09:35, Joe Witt  wrote:
>>>>> 
>>>>> Otto
>>>>> 
>>>>> I dont know we can keep all variants of Java 8 happy.  That one is
>>>> nearly
>>>>> three years old.  I would say try with something more recent.  It has
>>>> been
>>>>> pretty shocking how many api changes we have seen across these builds.
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> On Thu, Jan 28, 2021 at 6:12 AM Otto Fowler 
>>>> wrote:
>>>>> 
>>>>>> I’m trying to validate RC-1 and I’m seeing this exception building /
>>>>>> running tests
>>>>>> 
>>>>>> [INFO] Running org.apache.nifi.processors.standard.TestListenTCP
>>>>>> [ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time
>>>> elapsed:
>>>>>> 4.499 s <<< FAILURE! - in
>>>> org.apache.nifi.processors.standard.TestListenTCP
>>>>>> [ERROR]
>>>>>> 
>>>> testTLSClientAuthRequiredAndClientCertNotProvided(org.apache.nifi.processors.standard.TestListenTCP)
>>>>>> Time elapsed: 0.127 s  <<< FAILURE!
>>>>>> java.lang.AssertionError: unexpected exception type thrown;
>>>>>> expected: but
>>>> was:
>>>>>>   at
>>>>>> 
>>>> org.apache.nifi.processors.standard.TestListenTCP.testTLSClientAuthRequiredAndClientCertNotProvided(TestListenTCP.java:159)
>>>>>> Caused by: java.net.SocketException: Connection reset
>>>>>>   at
>>>>>> 
>>>> org.apache.nifi.processors.standard.TestListenTCP.runTCP(TestListenTCP.java:209)
>>>>>>   at
>>>>>> 
>>>> org.apache.nifi.processors.standard.TestListenTCP.lambda$testTLSClientAuthRequiredAndClientCertNotProvided$0(TestListenTCP.java:160)
>>>>>>   at
>>>>>> 
>>>> org.apache.nifi.processors.standard.TestListenTCP.testTLSClientAuthRequiredAndClientCertNotProvided(TestListenTCP.java:159)
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
>>>>>> Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
>&

Re: [VOTE] Release Apache NiFi 1.13.0

[Joey Frazee]

I noticed that nifi-docker/dockermaven-stateless/pom.xml and 
nifi-docker/docker-compose/docker-compose.yml in the source package and tag are 
still using 1.13.0-SNAPSHOT.

Is that a problem?

The former is experimental, and the latter is for convenience, so it’s not 
deadly but previous source archives had changed the version at the same time.

-joey

> On Jan 29, 2021, at 7:06 AM, John McGinn  wrote:
> 
>  Just a note that the "helpful reminder" Confluence page stills lists the 
> artifact names as 1.12.0.
> 
> 
> On Wednesday, January 27, 2021, 11:15:56 PM EST, Joe Witt 
>  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache NiFi
> 1.13.0.
> 
> (cut)
> 
> A helpful reminder on how the release candidate verification process works:
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>   

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Joey Frazee]

I have https://github.com/apache/nifi/pull/4632 which fixes an OOME in 
PutAzureBlobStorage reported in 
https://lists.apache.org/x/thread.html/rdef82be24828277b85bdc94dc57a8fb9df6f73552daeda289c941a51%40%3Cusers.nifi.apache.org%3E

It’s a pretty small change.

-joey

> On Jan 5, 2021, at 3:14 PM, Matt Burgess  wrote:
> 
> I am reviewing a PR at the moment but intend to review the graph stuff right 
> afterwards.
> 
>> On Jan 5, 2021, at 5:35 PM, Mike Thomsen  wrote:
>> 
>> I have a PR for graph that we need to close out because part of the graph
>> bundle will be broken without it.
>> 
 On Tue, Jan 5, 2021, 11:50 Mark Bean  wrote:
>>> 
>>> I'd like to see this PR completed as well
>>> https://github.com/apache/nifi/pull/4225
>>> 
>>> There's been discussion on it, and I don't see any further objections.
>>> 
>>> Thanks,
>>> Mark
>>> 
>>> On Thu, Nov 26, 2020 at 4:55 AM Pierre Villard <
>>> pierre.villard...@gmail.com>
>>> wrote:
>>> 
 Hi community,
 
 Starting a discussion thread around the release of NiFi 1.13.0. We added
 quite a lot of significant improvements and features since the release of
 1.12.1 - I see 143 JIRAs with 1.13.0 as the fix version. I think it makes
 sense to consider a new release.
 
 Please share here the JIRAs with opened pull requests that we would need
>>> to
 look at to make this release happen.
 
 Thanks,
 Pierre
 
>>> 

Re: [VOTE] Release Apache NiFi 1.12.1 (rc2)

[Joey Frazee]

+1 (non-binding)
Verified checksums, hashes, signatures
Ran full build w/ Java 1.8 and 11
Ran build w/ -Prpm profile
Created secure cluster using tls-toolkit and multiple SANs
Tested fixes for NIFI-7768 and NIFI-7794

-joey
On Sep 24, 2020, 1:05 PM -0700, Andrew Lim , wrote:
> +1 (binding)
>
> -Ran full clean install on OS X (Catalina 10.15.2)
> -Tested secure NiFi with secure NiFi Registry 0.7.0
> -Ran basic flows successfully; tested basic versioned flow functionality
> -Reviewed UI fixes and documentation updates
>
> Drew
>
> > On Sep 23, 2020, at 6:10 PM, Joe Witt  wrote:
> >
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache NiFi
> > 1.12.1.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1170
> >
> > The source being voted upon and the convenience binaries can be found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.12.1/
> >
> > A helpful reminder on how the release candidate verification process works:
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-1.12.1-RC2
> > The Git commit ID is accfaa3034fa5c3ef55d6402ac31e500247100f9
> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=accfaa3034fa5c3ef55d6402ac31e500247100f9
> >
> > Checksums of nifi-1.12.1-source-release.zip:
> > SHA256: 7735c632c6795bb0d299631454bd81006db33d51192cacc1404a5c9779607802
> > SHA512:
> > e7f06afc7617df7e3325bce8e34d1d78c4d130c40135661e59ae0eabef50df888759a125dae3ab9556fc940d621c695a50e674ebad8c3066716bfb5fbd27c1c4
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 39 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348757
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.1
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-1.12.1
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
>

Re: [VOTE] Release Apache NiFi 1.12.1

[Joey Frazee]

+1 (non-binding)
Verified checksums and signatures
Ran full build with contrib check on Linux and Zulu OpenJDK 11.0.7
Tested TLS toolkit and secure cluster with multiple SANs

-joey
On Sep 9, 2020, 8:51 AM -0700, Kevin Doran , wrote:
> +1 (binding)
>
> - Checked hashes/sigs
> - L is present
> - Built from source and built docker images from resulting artifacts
> - Ran a docker compose environment with NiFi Registry 0.7.0 and NiFi 1.12.1 
> RC1
> - Tested flow authoring and Registry integration in the test environment
>
> One thing to note is that running the docker profile from the decompressed 
> source distribution still results in some build errors due to shell scripts 
> we use for our docker container tests not being executable. At some point we 
> should look into improving our release process to try to preserve those 
> permissions, our making our build plugins robust enough that those 
> permissions are not required at build time.
>
> Kevin
>
> > On Sep 9, 2020, at 10:48, Bryan Bende  wrote:
> >
> > +1 (binding)
> >
> > Ran through the release helper and everything looks good, used 1.12.1
> > toolkit to generate certs for a secure instance.
> >
> > On Tue, Sep 8, 2020 at 11:36 PM Andrew Lim 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > -Ran full clean install on OS X (Catalina 10.15.2)
> > > -Tested secure NiFi with secure NiFi Registry 0.7.0
> > > -Ran basic flows successfully; tested basic versioned flow functionality
> > > -Reviewed documentation updates
> > >
> > > Drew
> > >
> > > > On Sep 8, 2020, at 12:59 PM, Joe Witt  wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi
> > > > 1.12.1.
> > > >
> > > > The source zip, including signatures, digests, etc. can be found at:
> > > > https://repository.apache.org/content/repositories/orgapachenifi-1167
> > > >
> > > > The source being voted upon and the convenience binaries can be found 
> > > > at:
> > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.12.1/
> > > >
> > > > A helpful reminder on how the release candidate verification process
> > > works:
> > > >
> > > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > > >
> > > > The Git tag is nifi-1.12.1-RC1
> > > > The Git commit ID is e8f708b3945183b7ab1b356c00d0fcbd929d6163
> > > >
> > > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=e8f708b3945183b7ab1b356c00d0fcbd929d6163
> > > >
> > > > Checksums of nifi-1.12.1-source-release.zip:
> > > > SHA256: 6fa7e4389ff33957133292e9749876affe1258e6f30d4e60b53e8c2d01d032cb
> > > > SHA512:
> > > >
> > > fee615ee885b05a290a4d6c7399ecf0b70a93cc1d5d9b019c7840ceb89d3279a540da4de75765c589c301fff751e62c9e15cc33d078b5fad03bbbd28e74f8ea9
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/joewitt.asc
> > > >
> > > > KEYS file available here:
> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > 19 issues were closed/resolved for this release:
> > > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12348757
> > > >
> > > > Release note highlights can be found here:
> > > >
> > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.1
> > > >
> > > > The vote will be open for 72 hours.
> > > > Please download the release candidate and evaluate the necessary items
> > > > including checking hashes, signatures, build
> > > > from source, and test. Then please vote:
> > > >
> > > > [ ] +1 Release this package as nifi-1.12.1
> > > > [ ] +0 no opinion
> > > > [ ] -1 Do not release this package because...
> > >
> > >
>

Re: [VOTE] Release Apache NiFi 1.12.0

[Joey Frazee]

+1 (non-binding)

Verified commit id, checksums, and signatures
Successfully built and ran tests on Linux with Java 1.8.0_252 and Java 11.0.7 
2020-04-14 and on macOS with Java 11.0.6 2020-01-14 LTS
Ran data flows on standalone and cluster
Verified JMS improvements, InvokeHTTP changes, and Azure integration in 
commercial and gov regions

-joey
On Aug 14, 2020, 1:31 PM -0700, Robert Fellows , wrote:
> + 1 (non-binding)
>
> Ran through the release helper, verified checksums and sigs.
> Full build with java 11
> Verified basic app usage/functionality.
>
>
>
> On Fri, Aug 14, 2020 at 4:04 PM Muazma Zahid  wrote:
>
> > +1 (non-binding)
> > Ran through the release helper. Deployed a 10 node cluster on Azure and
> > verified the functionality of new ADLS processors with a standard flow.
> > Looks good to me.
> >
> > Thanks,
> > Muazma
> >
> > On Fri, Aug 14, 2020 at 12:45 PM Andrew Lim 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > -Ran full clean install on OS X (Catalina 10.15.2)
> > > -Tested secure NiFi with secure NiFi Registry 0.7.0
> > > -Ran basic flows successfully; tested basic versioned flow functionality
> > > -Reviewed documentation updates. Lots of a great additions in this
> > release!
> > > -Reviewed core UI fixes and improvements. Verified that controller
> > > services are searchable [1], but filed a Jira for the navigation to the
> > > controller service being broken [2].
> > >
> > > Drew
> > >
> > > [1] https://issues.apache.org/jira/browse/NIFI-5925 <
> > > https://issues.apache.org/jira/browse/NIFI-5925>
> > > [2] https://issues.apache.org/jira/browse/NIFI-7742 <
> > > https://issues.apache.org/jira/browse/NIFI-7742>
> > >
> > > > On Aug 13, 2020, at 4:31 PM, Joe Witt  wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi
> > > > 1.12.0.
> > > >
> > > > The source zip, including signatures, digests, etc. can be found at:
> > > > https://repository.apache.org/content/repositories/orgapachenifi-1165
> > > >
> > > > The source being voted upon and the convenience binaries can be found
> > at:
> > > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.12.0/
> > > >
> > > > A helpful reminder on how the release candidate verification process
> > > works:
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > > >
> > > > The Git tag is nifi-1.12.0-RC1
> > > > The Git commit ID is 4d940bb151eb8d250b0319318b96d23c4a9819ae
> > > >
> > >
> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=4d940bb151eb8d250b0319318b96d23c4a9819ae
> > > >
> > > > Checksums of nifi-1.12.0-source-release.zip:
> > > > SHA256:
> > a67ecb34f32cc1f070ebb006b6f05456a2ac663b12f708eadac67754194a6c63
> > > > SHA512:
> > > >
> > >
> > 2e04235c4d49a76319af7756289ce11554a412bf5f7dcb6dc3915fc931df9f067142820c297e83bc36cb1079fb8384794ef457a20dd00568761eed6621701953
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/joewitt.asc
> > > >
> > > > KEYS file available here:
> > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > 335 issues were closed/resolved for this release:
> > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12346778
> > > >
> > > > Release note highlights can be found here:
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.12.0
> > > >
> > > > The vote will be open for 72 hours.
> > > > Please download the release candidate and evaluate the necessary items
> > > > including checking hashes, signatures, build
> > > > from source, and test. Then please vote:
> > > >
> > > > [ ] +1 Release this package as nifi-1.12.0
> > > > [ ] +0 no opinion
> > > > [ ] -1 Do not release this package because...
> > >
> > >
> >
>
>
> --
> ---
> Rob Fellows

Re: [discuss] 1.12.0 or 1.11.5...

[Joey Frazee]

I need to double check whether it needs rebased but there’s a docs PR for how 
to enable client TLS for ZooKeeper that I think would be good to include until 
the more integrated work is done:

https://github.com/apache/nifi/pull/4092

Would anyone have time to look?

-joey
On Aug 3, 2020, 11:47 AM -0700, Joe Witt , wrote:
> ok cool. will keep an eye there once the other items land.
>
> Thanks
>
> On Mon, Aug 3, 2020 at 11:26 AM Bryan Bende  wrote:
>
> > I was doing some testing and noticed some framework level classes like
> > prioritizers and authorizers were getting class not found exceptions.
> >
> > I believe it is related to the module/classpath changes we made as part of
> > NIFI-7592 [1], so we should hold on the RC until we can resolve that.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-7592
> >
> > On Mon, Aug 3, 2020 at 12:51 PM Joe Witt  wrote:
> >
> > > Mike,
> > >
> > > 7526 merged this weekend.
> > > 7605 sounds like LoPresto is going to look at.
> > >
> > > There are a couple others (kafka 2.5 and smb stuff i'd like to get in
> > > before I kick out the RC). Hopefully I can start that today.
> > >
> > > It is definitely time for 1.12!
> > >
> > > Thanks
> > >
> > > On Thu, Jul 30, 2020 at 3:13 PM Mike Thomsen 
> > > wrote:
> > >
> > > > 7605 is the PR for the user agent discussion and 7526 is a small SSL
> > > > refactor for the OAuth2TokenProvider that needs to be reviewed (as
> > > > OAuth2TokenProvider is already part of master).
> > > >
> > > > Thanks,
> > > >
> > > > Mike
> > > >
> > > > On Thu, Jul 30, 2020 at 12:14 PM Joe Witt  wrote:
> > > > >
> > > > > ...ok looking like NiFi 1.12 is pretty well in hand. I plan early
> > next
> > > > > week to start pulling the release together.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Wed, Jul 15, 2020 at 6:39 AM Pierre Villard <
> > > > pierre.villard...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Sounds good to me. Happy to take care of the RC duties for another
> > > > release
> > > > > > once the OIDC work is available.
> > > > > >
> > > > > > Le mar. 14 juil. 2020 à 21:31, Bryan Bende  a
> > > écrit
> > > > :
> > > > > >
> > > > > > > Spoke with Nathan again and there is still more work to do for
> > OIDC
> > > > in
> > > > > > > registry.
> > > > > > >
> > > > > > > If no one objects, I'll work on kicking out an RC for 0.7.0
> > > tomorrow
> > > > > > > morning, and we can land OIDC for the next release, likely an
> > > 0.8.0.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jul 7, 2020 at 8:48 AM Bryan Bende 
> > > wrote:
> > > > > > >
> > > > > > > > I can be RM for an 0.7.0 registry release.
> > > > > > > >
> > > > > > > > I spoke with Nathan and he has been making progress on the OIDC
> > > > > > support,
> > > > > > > > so I will wait a little bit to see if we can get that in, but
> > if
> > > it
> > > > > > > starts
> > > > > > > > taking longer then I'll proceed with getting an RC out.
> > > > > > > >
> > > > > > > > On Mon, Jul 6, 2020 at 5:41 AM Pierre Villard <
> > > > > > > pierre.villard...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Regarding the NiFi Registry 0.7.0 release - if we can have the
> > > > OIDC
> > > > > > > > > support
> > > > > > > > > in it, that would be great. I know Nathan is working on it
> > > (based
> > > > on
> > > > > > the
> > > > > > > > > JIRA) not sure if this is something close to being ready.
> > > > > > > > >
> > > > > > > > > Le sam. 4 juil. 2020 à 21:15, Mike Thomsen <
> > > > mikerthom...@gmail.com> a
> > > > > > > > > écrit :
> > > > > > > > >
> > > > > > > > > > FYI, if we add in https://github.com/apache/nifi/pull/4364
> > > and
> > > > a PR
> > > > > > > to
> > > > > > > > > > deprecate the elasticsearch v5 bundle we can free up about
> > > 60MB
> > > > in
> > > > > > the
> > > > > > > > > > convenience binary.
> > > > > > > > > >
> > > > > > > > > > On Thu, Jul 2, 2020 at 10:43 PM Joe Witt <
> > joe.w...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > ...and just saw Bryans note. Do we need more work before
> > > > kicking
> > > > > > > out
> > > > > > > > > reg
> > > > > > > > > > > release?
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jul 2, 2020 at 7:42 PM Joe Witt <
> > joe.w...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > 100% same page as Andy. Mark said later this week he
> > will
> > > > have
> > > > > > > some
> > > > > > > > > > key
> > > > > > > > > > > > bits wrapped. Lets see where we are then.
> > > > > > > > > > > >
> > > > > > > > > > > > thanks
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jul 2, 2020 at 7:22 PM Andy LoPresto <
> > > > > > > alopre...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Martin,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I understand everyone is anxious to get their hands on
> > > the
> > > > next
> > > > > > > > > > release,
> > > > > > > > > > > > > but this thread 

Re: [nifi] branch main created (now 239a2e8)

[Joey Frazee]

@alopresto Do you want everyone individually pushing to both master and main or 
are you intending to keep it in sync until it’s made the default?

-joey
On Jul 2, 2020, 6:39 PM -0500, alopre...@apache.org, wrote:
> This is an automated email from the ASF dual-hosted git repository.
>
> alopresto pushed a change to branch main
> in repository https://gitbox.apache.org/repos/asf/nifi.git.
>
>
> at 239a2e8 NIFI-7513 Added custom DNS resolution steps to walkthrough (#4359)
>
> No new revisions were added by this update.
>

Re: Reporting task can be set to RUNNING from REST API when invalid

[Joey Frazee]

One of those looks like it’s just a transient failure. The other on macOS was a 
flakey test that was fixed recently. If you want it to show green you can 
rebase.

If you re-run the jobs I’d expect to see the Ubuntu/1.8 pass but the macOS one 
probably won’t until you rebase your branch.

-joey
On Jun 25, 2020, 9:39 AM -0500, Vasily Makarov 
, wrote:
> Hi all!
> Investigated and resolved bug with https://github.com/apache/nifi/pull/4334 , 
> but there are build errors it seems not related to PR. Should I try to redo 
> my PR, or what to do?
> Also can someone do the code review if possible. (It is my second PR with 
> Apache NiFi but seems that I fixed the bug).

Re: [DISCUSS] rename master branch, look through code for other related issues

[Joey Frazee]

+1

I’m repeating this from elsewhere but I was on a team 7 years ago where a 
teammate asked us to stop using master and slave terminology, even master 
alone, because it made them uncomfortable. I can’t estimate how common that 
feeling is but this isn’t a theoretical exercise. As teammates and friends, it 
was an easy change, even if code was involved. And I assume much easier than 
having the courage to ask for it.

I’d say it’s also important to note that “but that’s not the original intended 
word sense” doesn’t alleviate that alienating experience. While potentially a 
matter of fact of the intent for some uses, “I want to use that word” is pretty 
unfriendly stacked against “that makes me feel unwelcome”.

Two guidelines from the code of conduct seem particularly apropos:

- Be empathetic, welcoming, friendly, and patient

- Be careful in the words that we choose

AFAICT there’s not an escape hatch for code, tools, or effort.

-joey

On Jun 18, 2020, 10:05 AM -0500, Edward Armes , wrote:
> I agree with this, and maybe that is the potential the step forward here
> is: issue a statement is issued saying something like this is a complex
> issue and instead of making changes that could cause further division
> within the community we are looking for those that are interested to help
> form a constructive working group that will help influence and resolve all
> of these issues in a positive way for all not only for project but also
> within the wider group of apache projects.
>
> Edward
>
>
>
> On Thu, 18 Jun 2020, 11:17 u...@moosheimer.com,  wrote:
>
> > Language is always changing and the meaning of words is changing,
> > sometimes positively and sometimes negatively.
> > I think that now is time for change again and we should discuss the use
> > of phrases and meanings.
> >
> > Of course we should change "Master Branch" to "Main Branch".
> > But I also think that we shouldn't just make quick changes because it's
> > opportune and hastily change a few words.
> >
> > An example: We could change Master/Slave to Leader/Follower. This may be
> > a perfect choice for most people in the world.
> > In German Leader is the English word for "Führer". And it is precisely
> > this word that we in Germany do not actually want to use for it.
> >
> > What I mean is that every country and every group (e.g. religion etc.)
> > has its own history and certain words or phrases are just not a perfect
> > choice.
> > We should try to go the ethically correct way worldwide.
> >
> > This concerns the adaptation of current words and phrases with a view to
> > all: in English, Indian, Chinese, German etc. but also for indigenous
> > peoples, different religions etc.
> > And cultural differences should also be taken into account.
> >
> > What I would wish for:
> > Apache.org should set up an "Ethics Board". A group of people of
> > different genders, all colors, religions and from different countries
> > and cultures all over our world.
> > This Ethics Board should find good and for no one discriminating words
> > or phrases for all the areas that stand out today as offensive.
> >
> > And it would be nice if not only computer scientists participated, but
> > also ethicists, philosophers, engineers, various religious people,
> > chemists, biologists, physicists, sociologists, etc.
> >
> > And this Council should set binding targets for all projects.
> >
> > Am 18.06.2020 um 09:36 schrieb Pierre Villard:
> > > > In my perspective this should be an issue for the entire community.
> > Being
> > > > able to identify an issue that directly affects another person but not
> > > > one’s self is the definition of privilege. If I can look at how the use
> > of
> > > > these words in someone’s daily life or career impacts them negatively,
> > > when
> > > > the change would not harm me at all, I see that as a failure on my
> > part. I
> > > > understand the desire to hear from the silent majority, but active
> > > > participation and discussion on the mailing list is the exact measure
> > > > described by the Apache process for participation in the community.
> > Those
> > > > who speak here are the ones who will have a voice.
> > > I could not agree more with the above.
> > >
> > > Le jeu. 18 juin 2020 à 04:29, Tony Kurc  a écrit :
> > >
> > > > I suppose I was a bit remiss in not unwinding and/or summarizing some of
> > > > what was in that yetus thread to prime the discussion, but a some of
> > what
> > > > Andy is mentioning is expanded on a bit in this ietf document [1],
> > which is
> > > > linked in one of the articles.
> > > >
> > > > 1. https://tools.ietf.org/id/draft-knodel-terminology-00.html
> > > >
> > > >
> > > > On Wed, Jun 17, 2020, 10:02 PM Andy LoPresto 
> > wrote:
> > > >
> > > > > Hi Edward, thanks for sharing your thoughts. I’ll reply inline.
> > > > >
> > > > > > - Some of the terms proposed are not industry standard and may
> > > > > potentially
> > > > > > cause significant issue for non-english speakers.
> > > > 

Re: [VOTE] Release Apache NiFi 1.11.4

[Joey Frazee]

+1 (non-binding)

Verified checksums, signatures, and commit hash
Ran build and all tests on Linux and OS X
Ran flows on standalone and cluster and verified some of the improvements/fixes
Built RPM profile and installed and ran

-joey
On Mar 20, 2020, 12:23 PM -0500, Marc Parisi , wrote:
> +1 binding
>
> * Ran through release helper guide.
> * Ran build and tested in clustered mode.
> *Tested site to site with minifi java without any issues or perceived
> leakage.
>
> Thanks,
> Marc
>
> On Fri, Mar 20, 2020 at 10:21 AM Andrew Lim 
> wrote:
>
> > +1 (non-binding)
> >
> > -Ran full clean install on OS X (Catalina 10.15.2)
> > -Tested secure NiFi with secure NiFi Registry 0.5.0; used NiFi Toolkit
> > 1.11.4 to generate configuration/cert files
> > -Ran basic flows successfully; tested basic versioned flow functionality
> > -Reviewed documentation updates
> >
> > Drew
> >
> > > On Mar 18, 2020, at 1:15 PM, Joe Witt  wrote:
> > >
> > > Hello,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> > NiFi
> > > 1.11.4.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1159
> > >
> > > The source being voted upon and the convenience binaries can be found at:
> > > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.11.4/
> > >
> > > A helpful reminder on how the release candidate verification process
> > works:
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > >
> > > The Git tag is nifi-1.11.4-RC1
> > > The Git commit ID is 4d940bb151eb8d250b0319318b96d23c4a9819ae
> > >
> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=4d940bb151eb8d250b0319318b96d23c4a9819ae
> > >
> > > Checksums of nifi-1.11.4-source-release.zip:
> > > SHA256: a67ecb34f32cc1f070ebb006b6f05456a2ac663b12f708eadac67754194a6c63
> > > SHA512:
> > >
> > 2e04235c4d49a76319af7756289ce11554a412bf5f7dcb6dc3915fc931df9f067142820c297e83bc36cb1079fb8384794ef457a20dd00568761eed6621701953
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/joewitt.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 45 issues were closed/resolved for this release:
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12347022
> > >
> > > Release note highlights can be found here:
> > >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.4
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build
> > > from source, and test. Then please vote:
> > >
> > > [ ] +1 Release this package as nifi-1.11.4
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> >
> >

Re: [VOTE] Release Apache NiFi 1.11.3

[Joey Frazee]

+1 (non-binding)

Verified checksums, signatures, and commit hash
Ran build and all tests on OpenJDK 1.8.0_242 on Linux and OS X
Created clusters on VMs and docker and ran flows

-joey
On Feb 24, 2020, 11:33 AM -0800, Matt Gilman , wrote:
> +1 binding
>
> Ran through release validation steps. Looks great!
>
> Thanks for RMing Joe!
>
> On Fri, Feb 21, 2020 at 10:21 PM Joe Witt  wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache NiFi
> > 1.11.3.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1158
> >
> > The source being voted upon and the convenience binaries can be found at:
> > https://dist.apache.org/repos/dist/dev/nifi/nifi-1.11.3/
> >
> > A helpful reminder on how the release candidate verification process works:
> >
> > https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> >
> > The Git tag is nifi-1.11.3-RC1
> > The Git commit ID is 10b509cf5bb88b12b0dd497543291f3c49a9903f
> >
> > https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=10b509cf5bb88b12b0dd497543291f3c49a9903f
> >
> > Checksums of nifi-1.11.3-source-release.zip:
> > SHA256: b2544719a8de02892b3436f1e6084b9b6011bf706a183fa2f059a9961634a7bd
> > SHA512:
> >
> > 353671e06dd60e186d2a9a29eb9b7c2d183757f6a0f76801b4ecb5d87aab99e701b9ebde932ee414d4b304afd2bd1d0a0e574906a385cfafc8de36cc47c2d0a3
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/joewitt.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 14 issues were closed/resolved for this release:
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12347022
> >
> > Release note highlights can be found here:
> >
> > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.3
> >
> > The vote will be open for 72 hours.
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build
> > from source, and test. Then please vote:
> >
> > [ ] +1 Release this package as nifi-1.11.3
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >

Re: [VOTE] Release Apache NiFi 1.11.1 (rc1)

[Joey Frazee]

+1 (non-binding)
Verified checksums, signatures, and commit hash
Ran build and tests on OpenJDK 1.8.0_242 on Linux
Ran rpm build target and tested install
Created small cluster and ran some flows

Note: I ran into intermittent TestListFile failures on OS X (there are existing 
JIRAs for these)

-joey

> On Feb 3, 2020, at 5:28 AM, Marc Parisi  wrote:
> 
> +1 binding
> 
> Verified sigs, hashes, etc per the typical release helper process.
> 
>  Since the list of resolved issues were so small I verified most, with the
> exception of 7051, 7080, and 7069. All others appear to be resolved.
>  Only built on PopOS and Centos due to superbowl -- but everything looks
> good.
> 
> 
>> On Mon, Feb 3, 2020 at 4:04 AM Yoshiaki Takahashi 
>> wrote:
>> 
>> +1 (non-binding)
>> 
>> - [x] Non-RPM (usual) build with Java 8.
>> - [x] Deploy on 3 node secured cluster in CentOS 7.7.0 and run on Java 8.
>> - [x] Non-RPM (usual) build with Java 11.
>> - [x] Deploy on 1 node non secured cluster in CentOS 7.7.0 and run on Java
>> 11.
>> 
>> Thanks for working for the release!
>> 
>>> On 2020/01/31 20:11:00, Joe Witt  wrote:
>>> Hello,
>>> 
>>> I am pleased to be calling this vote for the source release of Apache
>> NiFi
>>> 1.11.1.
>>> 
>>> The source zip, including signatures, digests, etc. can be found at:
>>> https://repository.apache.org/content/repositories/orgapachenifi-1156
>>> 
>>> The source being voted upon and the convenience binaries can be found at:
>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-1.11.1/
>>> 
>>> A helpful reminder on how the release candidate verification process
>> works:
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
>>> 
>>> The Git tag is nifi-1.11.1-RC1
>>> The Git commit ID is d22858d045fb3e5343a87d362855810963aa8556
>>> 
>> https://gitbox.apache.org/repos/asf?p=nifi.git;a=commit;h=d22858d045fb3e5343a87d362855810963aa8556
>>> 
>>> Checksums of nifi-1.11.1-source-release.zip:
>>> SHA256: 41878981689e08c51ed7bb921c46f6328c1e6b6c94460a2382069bd042ab7112
>>> SHA512:
>>> 
>> d309342732196225ffdbac3eaeed3c9b949bcee00509fb6d073618d869c6db56b426f5867849a8ae752d7acc8c6dd18a14cb377655774d86f91d736350eb2041
>>> 
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/joewitt.asc
>>> 
>>> KEYS file available here:
>>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>> 
>>> 9 issues were closed/resolved for this release:
>>> 
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12346906
>>> 
>>> Release note highlights can be found here:
>>> 
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.11.1
>>> 
>>> The vote will be open for 72 hours.
>>> Please download the release candidate and evaluate the necessary items
>>> including checking hashes, signatures, build
>>> from source, and test. Then please vote:
>>> 
>>> [ ] +1 Release this package as nifi-1.11.1
>>> [ ] +0 no opinion
>>> [ ] -1 Do not release this package because...
>>> 
>> 

Re: Processor for Delta Lake / Real Time Data Warehouse

[Joey Frazee]

Delta tables can be registered in the metastore and you can enable Delta SQL 
commands with Spark.

And there are integrations with other engines for the read path.

This isn’t to say a native integration isn’t needed; it is. But there are some 
options today.

-joey
On Feb 1, 2020, 7:13 PM -0600, Mike Thomsen , wrote:
> > But the specific use case you described can be done today using Nifi’s
> out of the box SQL processors and JDBC/ODBC with the all of mentioned
> databases/engines.
>
> I'm not aware of any direct JDBC route that would allow NiFi to use the SQL
> processors to hit Delta Lake. Would like to be proved wrong here (heck, it
> would make some of our use cases easier!) but AFAIK you have to have NiFi
> build Parquet and push to S3 or HDFS so Spark can do the transformation.
>
> On Sat, Feb 1, 2020 at 4:14 PM Joey Frazee 
> wrote:
>
> > Martin, I’ve been thinking about this one for a while but I think it needs
> > to be considered in the context of transactional table formats in general;
> > i.e., the incubating Apache Hudi and Apache Iceberg too.
> >
> > There are things that are inconvenient for NiFi to do with these table
> > formats.
> >
> > But the specific use case you described can be done today using Nifi’s out
> > of the box SQL processors and JDBC/ODBC with the all of mentioned
> > databases/engines.
> >
> > -joey
> > On Feb 1, 2020, 6:57 AM -0600, Martin Ebert , wrote:
> > > Hi community,
> > > how can we drive this topic forward? Jira is created:
> > > https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6976
> > >
> > > "A table in Delta Lake is both a batch table, as well as a streaming
> > source
> > > and sink. Streaming data ingest, batch historic backfill, and interactive
> > > queries all just work out of the box." (Delta.io)
> > >
> > > This is the decisive argument for me. A very impressive technological
> > > milestone that is just crying out to be implemented in Nifi. You find all
> > > details in the video here: https://youtu.be/VLd_qOrKrTI
> > >
> > > Delta Lake is related to Databricks, Athena and Presto. In our case it
> > > would be great to extract data from a database or any other source (can
> > be
> > > streaming) and send this data or stream to our Databricks cluster.
> > >
> > > I imagine it just like in the video. You have a Delta Lake processor,
> > where
> > > you can define to which Databricks cluster the data should go and which
> > > Delta Lake operation (upsert, merge, delete, ...) should happen with the
> > > data. That means Databricks is only the executing component and I don't
> > > have to code in Databricks in notebooks anymore. I also find the
> > > possibility to request an extra cluster with the processor cool.
> > >
> > > Being able to to the same with Athena and Presto would be a dream!
> >

Re: Processor for Delta Lake / Real Time Data Warehouse

[Joey Frazee]

Martin, I’ve been thinking about this one for a while but I think it needs to 
be considered in the context of transactional table formats in general; i.e., 
the incubating Apache Hudi and Apache Iceberg too.

There are things that are inconvenient for NiFi to do with these table formats.

But the specific use case you described can be done today using Nifi’s out of 
the box SQL processors and JDBC/ODBC with the all of mentioned 
databases/engines.

-joey
On Feb 1, 2020, 6:57 AM -0600, Martin Ebert , wrote:
> Hi community,
> how can we drive this topic forward? Jira is created:
> https://issues.apache.org/jira/projects/NIFI/issues/NIFI-6976
>
> "A table in Delta Lake is both a batch table, as well as a streaming source
> and sink. Streaming data ingest, batch historic backfill, and interactive
> queries all just work out of the box." (Delta.io)
>
> This is the decisive argument for me. A very impressive technological
> milestone that is just crying out to be implemented in Nifi. You find all
> details in the video here: https://youtu.be/VLd_qOrKrTI
>
> Delta Lake is related to Databricks, Athena and Presto. In our case it
> would be great to extract data from a database or any other source (can be
> streaming) and send this data or stream to our Databricks cluster.
>
> I imagine it just like in the video. You have a Delta Lake processor, where
> you can define to which Databricks cluster the data should go and which
> Delta Lake operation (upsert, merge, delete, ...) should happen with the
> data. That means Databricks is only the executing component and I don't
> have to code in Databricks in notebooks anymore. I also find the
> possibility to request an extra cluster with the processor cool.
>
> Being able to to the same with Athena and Presto would be a dream!

Re: [VOTE] Release Apache NiFi 1.6.0 (RC3)

[Joey Frazee]

Like slowpoke...

+1 (non-binding)

- Verified checksums and signature
- Successfully built and ran unit tests and flows on OSX (Oracle 1.8.0_152), 
Amazon Linux (Oracle 1.8.0_161)

FWIW, something isn’t working with the RPM build the way it was. Don’t have 
anything specific to report though and doesn’t affect the released binary so 
will dig into it. Checked list and issues.a.o and nothing turned up.

-joey

On Apr 6, 2018, 1:25 PM -0500, Kevin Doran , wrote:
> +1 (non-binding) Release this package as nifi-1.6.0
>
> Comments:
> - sig and hashes look good
> - ran a simple flow, saving/loading from Registry
> - full build w/tests passed, but it took two attempts. First time failed @ 
> nifi-ambari-reporting-task due to a test failure:
>
> [ERROR] Errors:
> [ERROR] 
> EncryptedWriteAheadProvenanceRepositoryTest.testShouldRegisterAndGetEvent:277 
> » FileNotFound
>
> I've never seen this failure before. Second build worked without issue. Just 
> mentioning it as something to keep an eye on regarding overall build 
> stability.
>
> Platform info:
>
> Apache Maven 3.5.0
> Java version: 1.8.0_131, vendor: Oracle Corporation
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac"
>
> Thanks,
> Kevin
>
> On 4/3/18, 19:49, "Joe Witt"  wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1124
>
> The Git tag is nifi-1.6.0-RC3
> The Git commit ID is f8466cb16d6723ddc3bf5f0e7f8ce8a47d27cbe5
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=f8466cb16d6723ddc3bf5f0e7f8ce8a47d27cbe5
>
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: d1e1c24f9af809bf812982962b61d07df4f1131e
> SHA256: 1e5028d594bb402aa36460f1b826d4e8a664ad6f0538deed20286cbf3c621fb8
> SHA512: 
> 8cb10cbafa6feeed712dbc0cf076496d6bc014276aab71383ff3481d8ea719cf1f39766abc76c75ba58ffca747df3bd6d9bac82e410de1c78673dcd16a5ddfee
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 162 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. The please vote:
>
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
>
>

Re: ListSFTP incoming relationship

[Joey Frazee]

I worked on this at one point to make it "easier" (haha...) to process a very 
deep directory tree with 100k+ files -- idea was to break it up into subtrees 
for concurrency, etc. by looping the outgoing relationship back to input.

It ended up being being painful. Looking at the diff again the most annoying 
things were:

- The list processors extend AbstractListProcessor which made the change 
invasive. It felt dirty to alter the already existing incoming attrs for the 
purpose of leaving performListing alone so it required interface and 
implementation changes across other processors.

- The state problem already mentioned was a downer. The size of the state store 
wasn't a problem in practice, assuming a trusted client and some vaguely small 
number of dirs. What was more annoying is you have to move to directory name 
keys and have a state migration hook to preserve compability. Right now there's 
two values stored so it's N*2 (at the time I convinced myself those were 
redundant but there's a lot of edge cases that have been patched over time so 
dunno for sure).

Having tried it I am on team ScanX for a second stateless processor.

If state or a last modified guard of some kind is needed, it can be implemented 
at the flow level in a number of ways (DMC, LookupService, DetectDupe, etc.). 
This isn't possible with ListX because it doesn't take input so embedding the 
last modified filter in those is way more of a necessity.

-joey

On Apr 1, 2018, 12:06 PM -0500, Pierre Villard , 
wrote:
> Hi Scott,
>
> In my opinion, based on the discussion here, I'd suggest you to implement
> the solution that you seem best to answer your needs and also taking in
> consideration all the feedback the community provided. Once you have
> something, best is to submit a pull request so that review and discussion
> can move forward on the implementation itself. I'd also recommend to file a
> JIRA with as much details as possible on what is the need, what are the
> options on the table and what is the implementation you want to propose
> (the more technical details you give, the sooner you'll get feedback for
> your code).
>
> Pierre
>
>
>
> 2018-04-01 18:40 GMT+02:00 scott :
>
> > Okay. I guess I didn't realize how Nifi dev felt about risk tolerance. I
> > think I can work around it by adding duplicate filtering or implement some
> > other state management solution.
> > So, what's the next step?
> >
> > Scott
> >
> > On Thu, Mar 29, 2018, 10:46 AM Bryan Bende  wrote:
> >
> > > Scott,
> > >
> > > You are correct that the overall discussion is about allowing incoming
> > > flow files to ListSFTP.
> > >
> > > However, the previous discussion on this thread highlighted that the
> > > main reason ListSFTP currently doesn't allow incoming flow files is
> > > because of challenges when storing state.
> > >
> > > This led to the proposal of a new processor that allowed incoming flow
> > > files, but did not store state, thus avoiding the challenges mentioned
> > > above. If we were going to store state in this new processor, then
> > > we'd be back to the exact same challenges.
> > >
> > > Providing an option to turn on state also doesn't really help, because
> > > if there is an option provided to users,then the option will be used,
> > > and it needs to work when it is used.
> > >
> > > If we can come up with something that stores state and works well for
> > > all scenarios, then we aren't against it, we just need to handle the
> > > challenges highlighted by Joe's original email.
> > >
> > > Regarding some of the other ideas...
> > >
> > > The current output of ListSFTP already includes flow file attributes
> > > for each listing that include the full path, filename, last update
> > > time, owner, group, permissions, and file size were you thinking
> > > of something different than that?
> > >
> > > See the "Writes Attributes" section here:
> > >
> > > https://nifi.apache.org/docs/nifi-docs/components/org.
> > apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.
> > processors.standard.ListSFTP/index.html
> > >
> > > Thanks,
> > >
> > > Bryan
> > >
> > >
> > >
> > > On Thu, Mar 29, 2018 at 12:43 PM, Andy LoPresto  > > wrote:
> > > > Scott,
> > > >
> > > > I think there are two conversations going on here. You are finding the
> > > > requirements for your specific use case, and that’s great. But I echo
> > > > Bryan’s point that a community processor for this scenario should not
> > > store
> > > > state at all. Sivaprasanna’s point that given dynamic directory input,
> > > > storing state based on that can cause massive data ingestion problems
> > > still
> > > > stands.
> > > >
> > > > For your specific use case, you can prototype (or possibly even get to
> > a
> > > > stable and robust-enough point) using ExecuteScript to model the
> > behavior
> > > > you need.
> > > >
> > > > In regards to the desired output format, I would 

Re: [CANCEL][VOTE] Release Apache NiFi 1.6.0 RC1

[Joey Frazee]

Joe, yes, referring to what Bryan discovered.

On Mar 25, 2018, 9:23 AM -0500, Joe Witt <joe.w...@gmail.com>, wrote:
> Team
>
> RC1 vote is cancelled to correct findings of vote process.
>
> Joey:
> I pushed the RC1 branch
> https://github.com/apache/nifi/tree/NIFI-4995-RC1 as per release
> guide. We will push the actual tag once we get to a release point.
>
> Can you clarify what fingerprint issue you are referring to? Just
> want to make sure this is what BryanB pointed out and not something
> else.
>
> Thanks
> Joe
>
> On Sun, Mar 25, 2018 at 10:12 AM, Joey Frazee <joey.fra...@icloud.com> wrote:
> > -1
> >
> > Ran through the usual release helper stuff, but it seems like the 
> > fingerprint issue is going to cause problems, so not sure how useful 
> > putting 1.6.0 out there will be if 1.6.1 will have to be turned around 
> > immediately.
> >
> > Did you mean to say there's a nifi-1.6.0 -RC tag? It doesn't look like the 
> > tag got pushed.
> >
> > -joey
> >
> > On Mar 24, 2018, 12:38 AM -0500, Pierre Villard 
> > <pierre.villard...@gmail.com>, wrote:
> > > -1 (binding)
> > >
> > > I confirm the issue mentioned by Bryan. That's actually what Matt and I
> > > experienced when trying the PR about the S2S Metrics Reporting task [1]. I
> > > thought it was due to my change but it appears it's not the case.
> > >
> > > [1] https://github.com/apache/nifi/pull/2575
> > >
> > > 2018-03-23 22:53 GMT+01:00 Bryan Bende <bbe...@gmail.com>:
> > >
> > > > After voting I happened to be using the RC to test something else and
> > > > came across a bug that I think warrants changing my vote to a -1.
> > > >
> > > > I created a simple two node cluster and made a standard convert record
> > > > flow. When I ran the flow I got a schema not found exception, so I
> > > > used the debugger which showed AvroSchemaRegistry had no schemas, even
> > > > though there was one in the UI.
> > > >
> > > > I then used the debugger to make sure the onPropertyModified was
> > > > getting when a schema was added, and it was which meant some after
> > > > adding the schema but before running the flow, it was being removed.
> > > >
> > > > As far as I can tell, the issue is related to changes introduced in
> > > > NIFI-4864... the intent here was for components with property
> > > > descriptors that have "dynamically modifies classpath" to be able to
> > > > smartly reload when they are started based on knowing if more
> > > > classpath resources were added.
> > > >
> > > > The issue is that for components that don't have any property
> > > > descriptors like this, they have a null fingerprint, and before
> > > > starting it compares null to the fingerprint of empty string, and
> > > > decides to reload [2].
> > > >
> > > > I think the fix should be fairly easy to just short-circuit at the
> > > > beginning of that method and return immediately if
> > > > additionalResourcesFingerprint is null, but will have to do some
> > > > testing.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-4864
> > > > [2] https://github.com/apache/nifi/blob/master/nifi-nar-
> > > > bundles/nifi-framework-bundle/nifi-framework/nifi-framework-
> > > > core-api/src/main/java/org/apache/nifi/controller/
> > > > AbstractConfiguredComponent.java#L313-L314
> > > >
> > > >
> > > > On Fri, Mar 23, 2018 at 4:20 PM, Matt Gilman <matt.c.gil...@gmail.com
> > > > wrote:
> > > > > +1 (binding) Release this package as nifi-1.6.0
> > > > >
> > > > > Executed the release helper and verified new granular restrictions 
> > > > > with
> > > > > regards to flow versioning.
> > > > >
> > > > > Thanks for RMing Joe!
> > > > >
> > > > > Matt
> > > > >
> > > > > On Fri, Mar 23, 2018 at 4:12 PM, Michael Moser <moser...@gmail.com
> > > > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Ran through release helper to verify the release and run NiFi on 
> > > > > > Ubuntu
> > > > > > 16.04. It worked as expected with no new comments to add.
> > > > > >
> &

Re: [VOTE] Release Apache NiFi 1.6.0

[Joey Frazee]

-1

Ran through the usual release helper stuff, but it seems like the fingerprint 
issue is going to cause problems, so not sure how useful putting 1.6.0 out 
there will be if 1.6.1 will have to be turned around immediately.

Did you mean to say there's a nifi-1.6.0 -RC tag? It doesn't look like the tag 
got pushed.

-joey

On Mar 24, 2018, 12:38 AM -0500, Pierre Villard , 
wrote:
> -1 (binding)
>
> I confirm the issue mentioned by Bryan. That's actually what Matt and I
> experienced when trying the PR about the S2S Metrics Reporting task [1]. I
> thought it was due to my change but it appears it's not the case.
>
> [1] https://github.com/apache/nifi/pull/2575
>
> 2018-03-23 22:53 GMT+01:00 Bryan Bende :
>
> > After voting I happened to be using the RC to test something else and
> > came across a bug that I think warrants changing my vote to a -1.
> >
> > I created a simple two node cluster and made a standard convert record
> > flow. When I ran the flow I got a schema not found exception, so I
> > used the debugger which showed AvroSchemaRegistry had no schemas, even
> > though there was one in the UI.
> >
> > I then used the debugger to make sure the onPropertyModified was
> > getting when a schema was added, and it was which meant some after
> > adding the schema but before running the flow, it was being removed.
> >
> > As far as I can tell, the issue is related to changes introduced in
> > NIFI-4864... the intent here was for components with property
> > descriptors that have "dynamically modifies classpath" to be able to
> > smartly reload when they are started based on knowing if more
> > classpath resources were added.
> >
> > The issue is that for components that don't have any property
> > descriptors like this, they have a null fingerprint, and before
> > starting it compares null to the fingerprint of empty string, and
> > decides to reload [2].
> >
> > I think the fix should be fairly easy to just short-circuit at the
> > beginning of that method and return immediately if
> > additionalResourcesFingerprint is null, but will have to do some
> > testing.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-4864
> > [2] https://github.com/apache/nifi/blob/master/nifi-nar-
> > bundles/nifi-framework-bundle/nifi-framework/nifi-framework-
> > core-api/src/main/java/org/apache/nifi/controller/
> > AbstractConfiguredComponent.java#L313-L314
> >
> >
> > On Fri, Mar 23, 2018 at 4:20 PM, Matt Gilman  > wrote:
> > > +1 (binding) Release this package as nifi-1.6.0
> > >
> > > Executed the release helper and verified new granular restrictions with
> > > regards to flow versioning.
> > >
> > > Thanks for RMing Joe!
> > >
> > > Matt
> > >
> > > On Fri, Mar 23, 2018 at 4:12 PM, Michael Moser  > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Ran through release helper to verify the release and run NiFi on Ubuntu
> > > > 16.04. It worked as expected with no new comments to add.
> > > >
> > > > -- Mike
> > > >
> > > >
> > > > On Fri, Mar 23, 2018 at 4:02 PM, Scott Aslan  > > > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > - Ran through release helper
> > > > > - Setup secure NiFi and verified a test flow
> > > > >
> > > > > On Fri, Mar 23, 2018 at 3:29 PM, Bryan Bende  > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > - Ran through release helper and everything checked out
> > > > > > - Verified some test flows with the restricted components + keytab
> > CS
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Mar 23, 2018 at 2:42 PM, Mark Payne  > > > > wrote:
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > Was able to verify hashes, build with contrib-check, and start up
> > > > > > application. Performed some basic functionality tests and all
> > worked as
> > > > > > expected.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > -Mark
> > > > > > >
> > > > > > >
> > > > > > > > On Mar 23, 2018, at 6:02 AM, Takanobu Asanuma <
> > > > tasan...@yahoo-corp.jp
> > > > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Thanks for all your efforts, Joe.
> > > > > > > >
> > > > > > > > I have one question. The version of the generated package is
> > > > > > nifi-1.7.0-SNAPSHOT. Is this correct at this stage? If it's ok,
> > > > > > +1(non-binding).
> > > > > > > >
> > > > > > > > - Succeeded "mvn -T 2.0C clean install -DskipTests -Prpm"
> > > > > > > > - Started secure cluster mode
> > > > > > > > - Verified sample dataflows work fine
> > > > > > > > - Verified the whitelist feature(NIFI-4761) works fine
> > > > > > > >
> > > > > > > > -Takanobu Asanuma
> > > > > > > >
> > > > > > > > -Original Message-
> > > > > > > > From: Joe Witt [mailto:joew...@apache.org]
> > > > > > > > Sent: Friday, March 23, 2018 1:12 AM
> > > > > > > > To: dev@nifi.apache.org
> > > > > > > > Subject: [VOTE] Release Apache NiFi 

Re: Will you accept contributions in Scala?

[Joey Frazee]

This probably necessitates a vote, yeah?

Frankly, I’m usually happier writing Scala, and I’ve not encountered any 
problems using processors written in Scala, but I think it’ll be important to 
tread lightly.

There’s a few things that pop into my head:

- Maintainability and reviewability. A very very good Java developer need not, 
by definition, either know how to write or identify good Scala or spot problems 
and bugs.
- Every Scala processor would either end up with a 5MB scala-lang.jar packaged 
into the .nar or we’d have to start including it in the core somewhere, if it’s 
not. It’s possible it might have already gotten pulled up from other 
dependencies.
- Style. There’s a tremendous amount of variation in Scala style because of its 
type system, implicits, macros, and functional nature. There are very good 
people out there that can write good Scala that isn’t readable by the 99%.
- Binary compatibility. Scala tends to be a little more brazen about breaking 
binary compatibility in major releases and those happen a bit more often than 
with Java. That’s not a problem for any potential source code in the project, 
but it could present some dependency issues someday.
- Testing. There’s N > 1 test frameworks and testing styles within those, so 
there’s a lot of options for introducing more variability into the tests.
- NiFi uses a lot of statics in setting up properties and relationships and the 
like, and idiomatic Scala approaches that stuff a bit differently, so it’ll be 
necessary to impose some style guidelines so there isn’t too much variation.

That said, there are some things that won’t be problematic:

- As mentioned, processors written in Scala do just work.
- The scala-maven-plugin works just fine allowing mixed Java-Scala projects 
(btw, it’d probably be not super great to do mixed Java-Scala and mixed 
Maven-SBT though).
- A lot of the above concerns could be addressed by having clear style 
guidelines.

Another thing: most of the projects I see deliver separate jars for Scala 
components are delivering idiomatic APIs wrapping Java (or vice versa). I think 
publishing   a separate set of jars/nars for stuff written in Scala would be 
odd since here it’d mostly be processors with new functionality and not 
functionality for using Scala. I could imagine a lib of implicits, traits, 
classes that could make the Scala development more enjoyable. That probably 
would make sense to deliver that way.

-joey

On Feb 10, 2018, 10:33 AM -0600, Bryan Bende , wrote:
> I agree more with Andy about sticking with Java. The more varying languages
> used, the more challenging it is to maintain. Once the code is part of the
> Apache NiFi git repo, it is now the responsibility of the committers and
> PMC members to maintain it.
>
> I’d even say I am somewhat against the groovy/Spock test code that Andy
> mentioned. I have frequently spent hours trying to fix a Spock test that
> broke from something I was working on. Every committer is familiar with
> JUnit, but only a couple know Spock. Just using this as an example that
> every committer knows Java, but only a couple probably know Scala, Clojure,
> etc.
>
> On Sat, Feb 10, 2018 at 10:25 AM Jeff  wrote:
>
> > +1 to Joe's response. If you can develop a component in Groovy or Scala
> > (or Clojure!) more quickly/comfortably, or if allowing components written
> > in other languages would encourage people to contribute more, I'm all for
> > it.
> >
> > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt  wrote:
> >
> > > i personally would be ok with it for an extension/processor provided it
> > > integrates well with the build.
> > >
> > > i would agree with andys view for core framework stuff but for
> > extensions i
> > > think we can do it like mikethomsen suggested.
> > >
> > > others?
> > >
> > > thanks
> > > joe
> > >
> > > On Feb 10, 2018 7:30 AM, "Mike Thomsen"  wrote:
> > >
> > > > I'm just a community contributor, so take that FWIW, but a compromise
> > > might
> > > > be to publish the Scala code as separate maven modules to maven central
> > > and
> > > > then submit a thoroughly tested processor written in Java. As long as
> > you
> > > > have enough unit and integration tests to give strong coverage, I
> > > wouldn't
> > > > imagine anyone here would have issues reviewing it. If the tests fail
> > > > because of code issues in the external dependencies, the obvious answer
> > > is
> > > > to just hold the PR until the tests pass.
> > > >
> > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
> > adam.we...@perkinelmer.com
> > > > wrote:
> > > >
> > > > > Devs,
> > > > >
> > > > > I have some interns starting with my team and we use Scala internally
> > > for
> > > > > our work.
> > > > > If I wanted to have them work to contribute some new processors,
> > would
> > > > > they have to be Java to be included with the base distribution or
> > could
> > > > > they use Scala as 

Re: [DISCUSS] Apache NiFi distribution has grown too large

[Joey Frazee]

I tend to have feelings similar to Michael about a multi-repo approach. I’ve 
rarely seen it help and more often seen it hurt — it’s confusing (especially to 
newcomers), stuff gets neglected because it’s easier to ignore, you need 
another master project or some such to do an entire build.

Maybe git submodules could help mitigate this, but creating independent 
assemblies or using different build profiles to enable building and packaging 
the binaries in different ways would satisfy everything except disentangling 
the releases.

-joey

On Jan 13, 2018, 12:40 PM -0600, Brandon DeVries , wrote:
> I agree... Long term extension registry, short term one repo with different
> assemblies (e.g. standard, slim, analytic, etc...).
>
> Brandon
>
> On Sat, Jan 13, 2018 at 1:35 PM Pierre Villard  wrote:
>
> > Option #3 also has my preference. But it's probably a good idea to only
> > keep one git repo and play with the assembly and Maven profiles for the
> > releases, no? It'd be certainly easier for release management process. But
> > this decision could also depend on how the option #3 is going to be
> > implemented I guess.
> >
> > 2018-01-13 6:36 GMT-07:00 Joe Witt :
> >
> > > thanks tony!
> > >
> > > On Jan 12, 2018 10:48 PM, "Tony Kurc"  wrote:
> > >
> > > > I put some of the data I was working with on the wiki -
> > > >
> > > > https://cwiki.apache.org/confluence/display/NIFI/NiFi+1.5.0+nar+files
> > > >
> > > > On Fri, Jan 12, 2018 at 10:28 PM, Jeremy Dyer  > wrote:
> > > >
> > > > > So my favorite option is Bryan’s option number “three” of using the
> > > > > extension registry. Now my thought is do we really need to add
> > > complexity
> > > > > and do anything in the mean time or just focus on that? Meaning we
> > have
> > > > > roughly 500mb of available capacity today so why don’t we spend those
> > > man
> > > > > hours we would spend on getting the second repo up on the extension
> > > > > registry instead?
> > > > >
> > > > > @Bryan do you have thoughts about the deployment of those bars in the
> > > > > extension registry? Since we won’t be able to build the release
> > binary
> > > > > anymore would we still need to create separate repos for the nars or
> > > > no?? I
> > > > > have used the registry a little but I’m not 100% sure on your vision
> > > for
> > > > > the nars
> > > > >
> > > > > - Jeremy Dyer
> > > > >
> > > > > Sent from my iPhone
> > > > >
> > > > > > On Jan 12, 2018, at 10:18 PM, Tony Kurc  wrote:
> > > > > >
> > > > > > I was looking at nar sizes, and thought some data may be helpful. I
> > > > used
> > > > > my recent RC1 verification as a basis for getting file sizes, and
> > just
> > > > got
> > > > > the file size for each file in the assembly named "*.nar". I don't
> > know
> > > > > whether the images I pasted in will go through, but I made some
> > > graphs.b
> > > > > The first is a histogram of nar file size in buckets of 10MB. The
> > > second
> > > > > basically is similar to a cumulative distribution, the x axis is the
> > > > "rank"
> > > > > of the nar (smallest to largest), and the y-axis is how what fraction
> > > of
> > > > > the all the sizes of the nars together are that rank or lower. In
> > other
> > > > > words, on the graph, the dot at 60 and ~27 means that the smallest 60
> > > > nars
> > > > > contribute only ~27% of the total. Of note, the standard and
> > framework
> > > > nars
> > > > > are at 83 and 84.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Fri, Jan 12, 2018 at 5:04 PM, Michael Moser <
> > moser...@gmail.com
> > > > > wrote:
> > > > > > > And of course, as I hit  I thought of one more thing.
> > > > > > >
> > > > > > > We could keep all of the code in 1 git repo (1 project) but the
> > > > > > > nifi-assembly part of the build could be broken up to build core
> > > NiFi
> > > > > > > separately from the tar/zip functional grouping of other NARs.
> > > > > > >
> > > > > > > On Fri, Jan 12, 2018 at 5:01 PM, Michael Moser <
> > moser...@gmail.com
> > > > > wrote:
> > > > > > >
> > > > > > > > Long term I would also like to see #3 be the solution. I think
> > > what
> > > > > > > > Joseph N described could be part of the capabilities of #3.
> > > > > > > >
> > > > > > > > I would like to add a note of caution with respect to
> > reorganizing
> > > > and
> > > > > > > > releasing extension bundles separately:
> > > > > > > >
> > > > > > > > - the burden on release manager expands because many more
> > > > projects
> > > > > > > > have to be released; probably not all on each release cycle
> > but
> > > > it
> > > > > could
> > > > > > > > still be many
> > > > > > > > - the chance of accidentally forgetting to release a project
> > > in a
> > > > > > > > release cycle becomes non-zero
> > > > > > > > - sharing code between projects gets a bit harder because you
> > > > have
> > > > > to
> > > > > > 

Re: [VOTE] Release Apache NiFi 1.5.0 (RC1)

[Joey Frazee]

+1

- Verified checksums, signature and commit ID
- Ran build w/ contrib-check with OpenJDK 1.8.0_131 on Amazon Linux 2017.03
- Tested version control and flow registry, CountText, FlattenJson and some 
other flows
- Tested RPM build and install on Amazon Linux 2017.03
- Skimmed L
- Gave it lots of <3

-joey

On Jan 10, 2018, 9:14 AM -0700, Chris Herrera , 
wrote:
> +1 (non-binding)
>
> Built with contrib-check
> Verified keys and hashes
>
> Ran through several flows in an oidc secured environment including several 
> custom processors. All seems working.
>
> Looks like a great release. Thanks guys!
>
> Regards,
> Chris
>
> On Jan 10, 2018, 9:37 AM -0600, Jeff Zemerick , wrote:
> > +1 non-binding
> >
> > Built with contrib-check.
> > Stepped through release guide.
> >
> > Ran into a minor issue with the TestListenSMTP unit tests on OSX but it
> > might be pretty specific to my computer.
> >
> > https://issues.apache.org/jira/browse/NIFI-4760
> >
> >
> > On Wed, Jan 10, 2018 at 10:26 AM, Pierre Villard <
> > pierre.villard...@gmail.com> wrote:
> >
> > > +1 (binding)
> > >
> > > Went through release helper guide and all looks good to me.
> > > Tested integration with a running registry instance.
> > > Ran some test workflows in a secured env and tested recent fixed about
> > > authorizers.
> > >
> > > Thanks for taking care of the release Joe!
> > > Great job from all the community around this release! Integration with the
> > > registry is neat.
> > >
> > > Pierre
> > >
> > >
> > >
> > > 2018-01-10 16:16 GMT+01:00 Bryan Bende :
> > >
> > > > +1 (binding)
> > > >
> > > > - Ran through release helper with no issues
> > > > - Ran into a minor issue related to component versioning when using
> > > > the registry and created this JIRA [1], would be more of an issue for
> > > > next release
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-4763
> > > >
> > > >
> > > > On Wed, Jan 10, 2018 at 10:05 AM, Matt Burgess  > > > wrote:
> > > > > +1 (binding), ran through release guide with no issues (verified sigs
> > > > > & sums), ran various flows using record processors, the new Jackson
> > > > > CSV parser, provenance reporting task with new filtering capability
> > > > > and output fields, etc.
> > > > >
> > > > > On Tue, Jan 9, 2018 at 5:19 AM, Joe Witt  wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I am pleased to be calling this vote for the source release of 
> > > > > > Apache
> > > > > > NiFi nifi-1.5.0.
> > > > > >
> > > > > > The source zip, including signatures, digests, etc. can be found at:
> > > > > > https://repository.apache.org/content/repositories/orgapachenifi-1116
> > > > > >
> > > > > > The Git tag is nifi-1.5.0-RC1
> > > > > > The Git commit ID is 46d30c7e92f0ad034d9b35bf1d05c350ab5547ed
> > > > > > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > > > 46d30c7e92f0ad034d9b35bf1d05c350ab5547ed
> > > > > >
> > > > > > Checksums of nifi-1.5.0-source-release.zip:
> > > > > > MD5: 046f2dde4af592dd8c05e55c2bbb3c4f
> > > > > > SHA1: 63b9a68b9f89200fd31f5561956a15b45b1b9c8c
> > > > > > SHA256: 40b155c4911414907835f2eb0d5a4da798935f27f1e5134218d904fe6c94
> > > > 2d13
> > > > > >
> > > > > > Release artifacts are signed with the following key:
> > > > > > https://people.apache.org/keys/committer/joewitt.asc
> > > > > >
> > > > > > KEYS file available here:
> > > > > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > > > >
> > > > > > 195 issues were closed/resolved for this release:
> > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > projectId=12316020=12341668
> > > > > >
> > > > > > Release note highlights can be found here:
> > > > > > https://cwiki.apache.org/confluence/display/NIFI/
> > > > Release+Notes#ReleaseNotes-Version1.5.0
> > > > > >
> > > > > > The vote will be open for 72 hours.
> > > > > > Please download the release candidate and evaluate the necessary 
> > > > > > items
> > > > > > including checking hashes, signatures, build
> > > > > > from source, and test. The please vote:
> > > > > >
> > > > > > [ ] +1 Release this package as nifi-1.5.0
> > > > > > [ ] +0 no opinion
> > > > > > [ ] -1 Do not release this package because...
> > > >
> > >

Re: Enrichment plugin for adding attributes from SQL

[Joey Frazee]

Brett, I think it’s great that you brought this up and made some specific 
suggestions, because it’s easy for people to overlook and hard to know how to 
do the right thing without that kind of feedback.

-joey

On Jan 7, 2018, 5:51 AM -0600, Brett Ryan , wrote:
>
> Probably my biggest suggestion overall for NiFi is for accessibility support, 
> it's almost non-existant and makes it really hard for folk like me who are 
> quite visually impaired. NiFi is almost exclusively a mouse driven tool, 
> which really is hard for the visually impaired.

Re: Enrichment plugin for adding attributes from SQL

[Joey Frazee]

nt, and whatever makes your flow work is fine. Just
> > > explaining why you likely won’t see a solution like that in the NiFi
> > > bundled components.
> > >
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > On Jan 4, 2018, at 4:07 PM, Brett Ryan <brett.r...@gmail.com> wrote:
> > >
> > > Thanks Andy, how would update attribute be able to get the value from sql?
> > >
> > > Consider a flow where a piece of information needs to be obtained from a
> > > DB but i do not want the contents of the current FF to be altered, using
> > > ExecuteSQL anywhere prior would not be possible due to replacing the FF
> > > contents.
> > >
> > > What I had was two seperate flows, one that updates an oauth key in a db
> > > keeping it fresh, the main flow would then read the db just before an
> > > invokehttp.
> > >
> > > I was originally using a distributed map cache but had concerns that it
> > > might not be secure and was also advised the cache server has been known 
> > > to
> > > go down.
> > >
> > > On 5 Jan 2018, at 04:01, Joey Frazee <joey.fra...@icloud.com> wrote:
> > >
> > > Andy, Brett,
> > >
> > > Taking a quick glance at the code it looks like it's enriching attributes
> > > from a database according to a query.
> > >
> > > If that's correct, there's a LookupAttribute processor that delegates
> > > lookups to a "LookupService" and adds attributes without altering content.
> > > There are a variety of these LookupServices included. I think what you
> > > implemented would make sense as a LookupService and then you could just
> > > configure the processor to use that. It could also be used with
> > > LookupRecord then too so there'd be a double payoff.
> > >
> > > -joey
> > >
> > > On Jan 4, 2018, 8:44 AM -0800, Andy LoPresto <alopre...@apache.org>,
> > > wrote:
> > > Hi Brett,
> > >
> > > It’s great that you found it easy to write a new processor for Apache
> > > NiFi. It is probably an indicator that we need to improve
> > > education/evangelism/documentation, however, that you did not find
> > > UpdateAttribute [1], which should do exactly what you were looking for.
> > >
> > > [1]
> > > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
> > >
> > > Andy LoPresto
> > > alopre...@apache.org
> > > alopresto.apa...@gmail.com
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> > > On Jan 4, 2018, at 7:03 AM, Brett Ryan <brett.r...@gmail.com> wrote:
> > >
> > > Hi all, having used NiFi for a couple days I wanted to add some attributes
> > > to a FlowFile while not altering the contents of that FlowFile.
> > >
> > > I had suggestions to use a script processor but that just sounded like a
> > > hack which could become a nuisance to replicate.
> > >
> > > Anyway, I figured I'd write a processor to do this, anyone interested you
> > > can find it here.
> > >
> > > https://github.com/brettryan/nifi-drunken-bundle
> > >
> > > Feel free to do with it what you please.
> > >
> > > I've published to maven central but it will take a day to appear in the
> > > search.
> > >
> > > 
> > > com.drunkendev
> > > nifi-drunken-nar
> > > 1.0.0
> > > nar
> > > 
> > > 
> > > com.drunkendev
> > > nifi-drunken-processors
> > > 1.0.0
> > > 
> > > 
> > > com.drunkendev
> > > nifi-drunken-bundle
> > > 1.0.0
> > > pom
> > > 
>

Re: Enrichment plugin for adding attributes from SQL

[Joey Frazee]

Andy, Brett,

Taking a quick glance at the code it looks like it's enriching attributes from 
a database according to a query.

If that's correct, there's a LookupAttribute processor that delegates lookups 
to a "LookupService" and adds attributes without altering content. There are a 
variety of these LookupServices included. I think what you implemented would 
make sense as a LookupService and then you could just configure the processor 
to use that. It could also be used with LookupRecord then too so there'd be a 
double payoff.

-joey

On Jan 4, 2018, 8:44 AM -0800, Andy LoPresto , wrote:
> Hi Brett,
>
> It’s great that you found it easy to write a new processor for Apache NiFi. 
> It is probably an indicator that we need to improve 
> education/evangelism/documentation, however, that you did not find 
> UpdateAttribute [1], which should do exactly what you were looking for.
>
> [1] 
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.4.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Jan 4, 2018, at 7:03 AM, Brett Ryan  wrote:
> >
> > Hi all, having used NiFi for a couple days I wanted to add some attributes 
> > to a FlowFile while not altering the contents of that FlowFile.
> >
> > I had suggestions to use a script processor but that just sounded like a 
> > hack which could become a nuisance to replicate.
> >
> > Anyway, I figured I'd write a processor to do this, anyone interested you 
> > can find it here.
> >
> > https://github.com/brettryan/nifi-drunken-bundle
> >
> > Feel free to do with it what you please.
> >
> > I've published to maven central but it will take a day to appear in the 
> > search.
> >
> > 
> >   com.drunkendev
> >   nifi-drunken-nar
> >   1.0.0
> >   nar
> > 
> > 
> >   com.drunkendev
> >   nifi-drunken-processors
> >   1.0.0
> > 
> > 
> >   com.drunkendev
> >   nifi-drunken-bundle
> >   1.0.0
> >   pom
> > 
> >
>

Re: [VOTE] Release Apache NiFi Registry 0.1.0

[Joey Frazee]

+1

- Verified checksums and signatures
- Successfully built and ran contrib-check w/ Oracle JDK 1.8.0_152
- Setup the registry w/ NIFI-4436
- Had some fun versioning PGs and rolling back and forth

Note: I ran into some trouble rolling back flows that include new funnels. It 
always results in an IllegalStateException with 'Funnel  is the destination 
of another component'. There aren't any problems creating a new PG via Import 
or rolling forward a flow that newly includes a funnel. Will open an issue.

On Dec 29, 2017, 1:56 PM -0600, Yolanda Davis , 
wrote:
> +1, binding
>
> Have been looking forward to this contribution; it is a great addition to
> the community!
>
> Following the release helper I verified signatures, hashes as well as
> the README,
> NOTICE, and LICENSE. Successfully built the project per instructions and
> built the NiFi project using the referenced PR. From there I tested in
> unsecured mode the basic test suggested in the helper guide. Also tried
> minor things such as deleting a flow from registry that was associated to a
> PG to ensure NiFi updated as expected.
>
> One thing that would be helpful is additional documentation on the NiFi
> side on how to integrate with Registry. I also noticed on my test to delete
> a flow where it seemed there was a pretty big lag when NiFi recognized the
> change, even beyond my attempt to refresh. Not sure if this is
> configurable or just odd behavior on my end.
>
> Another minor issue is when someone adds a process group, enters a name for
> it and then selects import to use a flow in registry. The name gets
> overwritten by the flow (which makes sense) but wondering if there's a way
> to either allow user to keep new name or at least let them know it will be
> overwritten by version in registry?
>
> Again great job!
>
> -yolanda
>
>
>
> On Thu, Dec 28, 2017 at 1:09 PM, Bryan Bende  wrote:
>
> > Hello,
> >
> > I am pleased to be calling this vote for the source release of Apache
> > NiFi Registry 0.1.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1115/
> >
> > The Git tag is nifi-registry-0.1.0-RC1
> > The Git commit ID is 81b99e7b04491eabb72ddf30754053ca12d0fcca
> > https://git-wip-us.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=
> > 81b99e7b04491eabb72ddf30754053ca12d0fcca
> >
> > Checksums of nifi-registry-0.1.0-source-release.zip:
> > MD5: 56244c3c296cdc9c3fcc6d22590b80d1
> > SHA1: 6354e91f868f40d6656ec2467bde307260ad63ca
> > SHA256: 2c680e441e6c4bfa2381bf004e9b19a6a79401a6a83e04597d0a714a95efd301
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/bbende.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/release/nifi/KEYS
> >
> > 65 issues were closed/resolved for this release:
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > projectId=12320920=12340217
> >
> > Release note highlights can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/
> > Release+Notes#ReleaseNotes-NiFiRegistry0.1.0
> >
> > The vote will be open for 96 hours.
> >
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.
> >
> > The please vote:
> >
> > [ ] +1 Release this package as nifi-registry-0.1.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because...
> >
>
>
>
> --
> --
> yolanda.m.da...@gmail.com
> @YolandaMDavis

[DISCUSS] CI / Travis / Jenkins

[Joey Frazee]

I’m sure everyone has noticed that Travis CI fails, incorrectly, more than it 
succeeds, often due to timeouts and not b/c of the incorrectness of a commit or 
PR.

This has been discussed previously, but it’s carried on, and become a low 
information signal about the PRs, which has two big impacts: (1) it’s ignored 
by experienced contributors and reviewers, and (2) it’s confusing or misleading 
to new contributors.

So, we really need to find a solution. I can think of a few:

1. Continue to push on INFRA to setup Jenkins for NiFi and its sub-projects.

2. Implement some kind of quick-test profile and shell script that checks the 
most important things along with the subdirectories affected by the PR, and 
continue to use Travis CI.

3. Use some other service like Circle CI or Codeship, which probably isn’t 
quite what ASF wants but it might make the CI more useful (it also might not).

4. Find a sponsor to support a more premium tier of Travis CI (or equiv.) so 
the build has enough resources to to succeed. This too probably isn’t 
preferable but I’m sure we can find a precedent.

I’m partial to pursuing (1) and (2) together because (1) would give us a long 
term solution and (2) would have some value for local builds (no need to run 
the full build) as well as making Travis CI tell us something. The first should 
be pretty low effort. The second will be labor intensive I think — to identify 
what counts as quick and change the poms — so it can’t be the answer on its own 
unless we want to wait longer to see Travis CI become informative.

What do the rest of you think?

-joey

Re: nifi: How to update and trnaform xml file inside ExecuteScript processor

[Joey Frazee]

Sally, there are better ways to approach this:

- First, the default for transforming XML should be to use TransformXml [1] 
with an XSLT. It kinda looks like you’re also trying to extract some elements 
into attributes so I’d look at EvaluateXPath [2] too. Your flow would then look 
like [_] --> [TransformXml] --> [EvaluateXPath] --> [_].

- If you must use ExecuteScript, since you’re using Groovy, you don’t have to 
use the native Java XML APIs. Please have a look at Groovy’s documentation on 
processing XML [3] in particular XmlSlurper, DOMBuilder and the GPath 
expressions if you need them. I can’t really tell what you’re doing with the 
file locks, but I doubt they’re necessary.

Generally speaking, I think it’d be helpful to look at ExecuteScript as a last 
resort Swiss army knife for those situations where an existing processor 
doesn’t do what you need to do. We’re always adding more processors and 
functionality so you’d be surprised at how many things you can cover without 
having to drop down to ExecuteScript.

The time and effort to pore through every processor doc is often less than 
writing (and debugging) the code for ExecuteScript.

1. 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.TransformXml/index.html
2. 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.4.0/org.apache.nifi.processors.standard.EvaluateXPath/index.html
3. http://groovy-lang.org/processing-xml.html

On Oct 14, 2017, 1:12 PM -0500, sally , wrote:
>
> Question by sally sally 12 secs ago nifi-processorjavaapache-nifigroovy
> I want to read xml file from folder and i want to update it and make new
> flowfile(my code will be placed in nifi processor) which will containt this
> renewated data but i can't parse xml document properly i got error like
> this:org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot
> cast object 'com.sun.org.apache.xerces.internal.dom.DeepNodeListImpl@181ae3f
>
> 1.what should i change?( except changing doc = builder.parse(inputStream)
> with doc=builder.parse(file) because i need to use string data for later
> processing)
>
> Another reason why i can't fulfill my task is that i can't make flowfile in
> which i would have placed all renewated xml data, for some reason i got
> empty flowfile even if i use this doc=builder.parse(file) i means use config
> file phisically( File file=new File('path')), what should i change to
> improve this case.
> if i use string content, should i use any alternative way to get xml tag
> data except using NodeList( i mean is it possible to use any better use
> case)? here is my groovy code:
>
> import org.apache.commons.io.IOUtils;
>
> import java.nio.charset.StandardCharsets;
>
> import java.io.RandomAccessFile;
>
> import java.nio.channels.FileLock;
>
> import java.io.BufferedReader;
>
> import org.apache.commons.io.IOUtils;
>
> import java.nio.charset.StandardCharsets;
>
> File file =newFile("C://Users//user//Desktop//2//conf.xml");
>
> String content ='';RandomAccessFile ini =newRandomAccessFile(file,"rws");
>
> FileLock lock = ini.getChannel().lock();BufferedReader s;
>
> DocumentBuilder dBuilder =null;Document doc=null;String
> start,startDate,endDate,runAs,makeVersion,patch;
>
> try{String sCurrentLine;
>
> s=newBufferedReader(Channels.newReader(ini.getChannel(),"UTF-8"));while((sCurrentLine
> = s.readLine())!=null){
> content+=sCurrentLine;}//make xml
> parsingDocumentBuilderFactory factory
> =DocumentBuilderFactory.newInstance();DocumentBuilder builder =
> factory.newDocumentBuilder();InputStream inputStream
> =newByteArrayInputStream(content.getBytes());
> doc = builder.parse(inputStream);NodeList nList =
> doc.getElementsByTagName("localAttributes");
>
> for(int temp =0; temp < nList.getLength(); temp++){Node nNode =
> nList.item(temp);
>
> if(nNode.getNodeType()==Node.ELEMENT_NODE){Element eElement =(Element)
> nNode;
> start =
> eElement.getElementsByTagName("start").item(0).getTextContent();
> startDate =
> eElement.getElementsByTagName("startDate").item(0).getTextContent();
> endDate =
> eElement.getElementsByTagName("endDate").item(0).getTextContent();
> patch =
> eElement.getElementsByTagName("patch").item(0).getTextContent();
> runAs =
> eElement.getElementsByTagName("runAs").item(0).getTextContent();
> makeVersion =
> eElement.getElementsByTagName("makeVersion").item(0).getTextContent();}}
> flowFile1 = session.putAttribute(flowFile1,"start", start);
> flowFile1 = session.putAttribute(flowFile1,"startDate",
> startDate);
> flowFile1 = session.putAttribute(flowFile1,"endDate", endDate);
> flowFile1 = session.putAttribute(flowFile1,"runAs", runAs);
> flowFile1 = session.putAttribute(flowFile1,"patch", patch);
> flowFile1 = session.putAttribute(flowFile1,"makeVersion",
> makeVersion);
>
> flowFile1 =
> session.putAttribute(flowFile1,"filename","conf.xml");
>
> DocumentBuilderFactory 

Re: [VOTE] Release Apache NiFi 1.4.0 (RC2)

[Joey Frazee]

+1 (non-binding)

- Verified checksums and signature
- Successfully built and ran tests on OSX (Oracle 1.8.0_131), Amazon Linux 
(Oracle 1.8.0_131), and Docker maven:latest (OpenJDK 1.8.0_141)
- Built RPM with `mvn -T 2.0C clean install -Prpm,generateArchives -DskipTests` 
and tested install
- Tried out DMC w/ Redis :)

Note: Inconsistently seeing timeouts and test failures with TestListenSMTP on 
OSX (probably a me problem though)

-joey

On Oct 1, 2017, 12:05 PM -0500, Joe Gresock , wrote:
> +1 (non-binding)
>
> Built on CentOS 7, upgraded an existing 1.3.0 cluster, ran a complex flow
> with self-RPGs, reporting tasks, and controller services. All looks good!
> The Avro viewer is especially nice, and the double-click feature already
> feels natural. Great work!
>
> On Sun, Oct 1, 2017 at 4:41 PM, Joe Percivall  wrote:
>
> > +1 (binding)
> >
> > Built on OSX 10.12.5 and Windows 8.1, and ran very simple flows on OSX,
> > Windows 8.1 and Windows 10. When building on Windows 8.1, ran into previous
> > test failures and a couple new ones found here[1]. As stated before, not
> > important enough to downvote the release.
> >
> > Also one thing to comment, this vote thread doesn't have the SHA256 listed
> > but it is included with the other artifacts. It is:
> > 852dcda482342d9ae60e05e137a025cbf17d948e804716e3c185992a88e8cd8a
> >
> > Thanks for everyone's hard work!
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-3840
> >
> > On Sun, Oct 1, 2017 at 10:44 AM, Joe Witt  wrote:
> >
> > > +1 (binding)
> > >
> > > Did all the normal release validation/L/sigs/etc..
> > >
> > > Did a bunch of different tests largely focused on high performance and
> > > stability. Things look really good.
> > >
> > > I did run into a test failure which I've narrowed down to the latest
> > > JRE/JDK version. Details
> > > https://issues.apache.org/jira/browse/NIFI-4445
> > >
> > > Thanks and great job Jeff on RM and to the community this is an
> > > awesome release. Now we need to focus on the growing list of PRs
> > > which reflect a wide range of new contributors!
> > >
> > > On Sat, Sep 30, 2017 at 11:27 AM, Gerdan Rezende dos Santos
> > >  wrote:
> > > > Verified hash, local build was successful on OS X, confirmed!
> > > > Good Version!
> > > >
> > > > On Sat, 30 Sep 2017 at 09:32 Tony Kurc  wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Built on Ubuntu 14.04 with Java 1.8.0 and Maven 3.5.0. Verified hashes
> > > and
> > > > > signature. Tested some simple flows. Tested some tls toolkit
> > operations.
> > > > > Did not see any issues with LICENSE or NOTICE files.
> > > > >
> > > > > On Sat, Sep 30, 2017 at 4:17 AM, Koji Kawamura <
> > ijokaruma...@gmail.com
> > > > > wrote:
> > > > >
> > > > > > +1 (binding) Release this package as nifi-1.4.0
> > > > > >
> > > > > > Verified hashes, local build was successful on OS X, confirmed S2S
> > > > > > communication with older versions.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Sep 30, 2017 at 9:27 AM, Andy LoPresto <
> > alopre...@apache.org
> > > > > > wrote:
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > Build environment: Mac OS X 10.11.6, Java 1.8.0_101, Maven 3.3.9,
> > > JCE
> > > > > > > Unlimited Strength Cryptographic Jurisdiction Policies installed
> > > > > > >
> > > > > > > * verified GPG signature is valid and SHA512 digest
> > > > > > > * verified all checksums
> > > > > > > * verified all tests
> > > > > > > * verified checkstyle
> > > > > > > * verified Knox properties present in default nifi.properties
> > > > > > > * verified normal flow
> > > > > > > * verified ListenHTTP and HandleHTTPRequest only accept restricted
> > > > > SSLCS
> > > > > > > * verified bad authorizers.xml (copied from 1.2.0 -- missing
> > > > > > > managedAuthorizer) causes startup fail
> > > > > > > * verified good authorizers.xml works
> > > > > > > * verified secure instance works with client cert auth
> > > > > > > * verified secure instance works with Knox SSO
> > > > > > > * verified encrypted flow value migration works without Jasypt
> > > > > > >
> > > > > > > Andy LoPresto
> > > > > > > alopre...@apache.org
> > > > > > > alopresto.apa...@gmail.com
> > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> > EF69
> > > > > > >
> > > > > > > On Sep 29, 2017, at 1:54 PM, Andrew Lim <
> > andrewlim.apa...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > -Ran full clean install on OS X (10.11.4)
> > > > > > > -Tested UI changes including Variable Registry UI
> > > > > > > -Tested flows using Record reader/writer processors and controller
> > > > > > services,
> > > > > > > working as expected
> > > > > > >
> > > > > > >
> > > > > > > On Sep 29, 2017, at 4:50 PM, Michael Moser  > > wrote:
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > 

Re: [VOTE] Release Apache NiFi 1.4.0

[Joey Frazee]

I think there could be an issue with the deps in the nifi-mongodb-services-nar. 
It includes nifi-lookup-services which should either be unnecessary or should 
just be provided scope (just need the services API dependency). So it’s 
possible that all the impls in nifi-lookup-services are indeed included twice.

Does that jive with what you’re seeing? I.e., for LookupService properties do 
you see double of everything?

-joey

On Sep 25, 2017, 10:30 AM -0500, Richard St. John , wrote:
> Joe,
>
> The issue I encountered was related to, I believe, the packaging of the 
> mongodb lookup service.  I am using the XMLlookup service and have a 
> processor with a reference to the XML lookup service.  When I upgraded from 
> 1.3 to 1.4, the processor became invalid due to “incompatible type” of 
> service.  The lookup attribute processor appeared to be attempting to use the 
> mongodb lookup service.  I re-added the xml lookup service, being careful to 
> use the one in the nifi-lookup-services-nar and not the one packaged in the 
> nifi-mongodb-services-nar.  After do that, the lookup attribute processor was 
> valid and able to link to the xml lookup service.
>
> Rick.
>
> --
> Richard St. John, PhD
> Asymmetrik
> 141 National Business Pkwy, Suite 110
> Annapolis Junction, MD 20701
>
> On Sep 25, 2017, 10:55 AM -0400, Joe Witt , wrote:
> > -1 (binding) based on what Rick ran into.
> >
> > Otherwise though the release is looking good. I'm running through a
> > series of tests now and things going well.
> >
> > Rick,
> > I agree there are duplicate controller services and sourced to the
> > mongo system. And we must fix/remove those.
> >
> > However, the issue for upgrading is one I'd like to better understand.
> > What is the problem you're seeing? It is not required that controller
> > services have unique class names. The requirement is that the
> > artifact/coordinate is unique across the class name/extension
> > bundle/version. So lets figure out why this is actually breaking you.
> >
> > Thanks
> > Joe
> >
> > On Mon, Sep 25, 2017 at 10:47 AM, Richard St. John  
> > wrote:
> > > -1 non-binding.
> > >
> > > There are duplicate lookup services registered and it’s causing issues
> > > upgrading from 1.3.0 to 1.4.0. It seems to be related to the mongo lookup
> > > service.
> > >
> > > Rick.
> > >
> > > --
> > > Richard St. John, PhD
> > > Asymmetrik
> > > 141 National Business Pkwy, Suite 110
> > > Annapolis Junction, MD 20701
> > >
> > > On Sep 24, 2017, 9:15 PM -0400, Jeff , wrote:
> > >
> > > There is an error in my previous email. 192 issues were closed and
> > > resolved for this release.
> > >
> > > On Sun, Sep 24, 2017 at 9:11 PM Jeff  wrote:
> > >
> > > Hello,
> > >
> > > I am pleased to be calling this vote for the source release of Apache NiFi
> > > nifi-1.4.0.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1110
> > >
> > > The Git tag is nifi-1.4.0-RC1
> > > The Git commit ID is 466931665caab96df1c2c6b62d4b3c6cffeb3539
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=466931665caab96df1c2c6b62d4b3c6cffeb3539
> > >
> > > Checksums of nifi-1.4.0-source-release.zip:
> > > MD5: 9862f59ad1bfa12f2ce041e4c69b1c93
> > > SHA1: 3e2d0dcf0a83df5336a2962fbf6f10c3ae61c588
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/jstorck.asc
> > >
> > > KEYS file available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 8 issues were closed/resolved for this release:
> > >
> > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12340589
> > >
> > > Release note highlights can be found here:
> > >
> > > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version-1.4.0
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build
> > > from source, and test. The please vote:
> > >
> > > [ ] +1 Release this package as nifi-1.4.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> > >
> > >

Re: Binaries in embedded redis jar files

[Joey Frazee]

In addition to only being for testing, these binaries are only used for an 
integration test, so we don't even need to worry about whether they're going to 
work on everyone's OS/cause test failures if the binaries don't work for you.

Now it is packaging up the entire Redis server binary, though, which is a 
little different than a class file or dll, so I think a different question is 
whether we want ITs that bring their own server in in this way?

A mock server might be sufficient or I've seen projects using docker-maven to 
provide all the server resources needed for ITs. This could be helpful for us 
for the Elastic, C*, etc. ITs too if we did it.

So binaries = ok and no warnings needed, but it might still be a valid question 
about whether we want to use this redis-embedded lib; but it doesn't strike me 
as a very urgent thing.

> On Jul 19, 2017, at 1:54 PM, Tony Kurc  wrote:
> 
> Mike -
> Is there something about these binaries versus all of the other myriad
> binaries (jars, .class files, dlls and .so files) included in the
> convenience jar? I wouldn't think so, these are just more binary artifacts.
> I honestly can't think of another apache project that provides similar
> convenience binaries that have warnings such as those you mentioned.
> 
> Tony
> 
>> On Wed, Jul 19, 2017 at 2:19 PM, Michael Moser  wrote:
>> 
>> Whoa, this causes serious concerns for me.  While I can understand
>> that the source code is the official ASF artifact, the nifi.apache.org
>> site provides convenience binaries which I'm sure many people use.
>> When NiFi 1.4.0 releases, we should write very conspicuous warnings on
>> the Downloads page to advise users about this.
>> 
>> -- Mike
>> 
>> 
>>> On Wed, Jul 19, 2017 at 2:04 PM, Joe Witt  wrote:
>>> It is not a problem.  It would be an issue if those were part of our
>>> source release which is the official apache release artifact.
>>> 
>>> 
>>> 
 On Wed, Jul 19, 2017 at 1:58 PM, Joe Skora  wrote:
 I noticed a new jar dependency for Redis includes binaries.
 
 $ jar -xf
> ~/.m2/repository/com/github/kstyrc/embedded-redis/0.6/
>> embedded-redis-0.6.jar
> $ ls -l
> total 6292
> drwxr-xr-x. 3 user1 users  36 Apr 12  2015 META-INF/
> drwxr-xr-x. 3 user1 users  21 Apr 12  2015 redis/
> -rw-r--r--. 1 user1 users 4236300 Apr 12  2015 redis-server-2.8.19
> -rw-r--r--. 1 user1 users  806944 Apr 12  2015 redis-server-2.8.19.app
> -rw-r--r--. 1 user1 users 1392640 Apr 12  2015 redis-server-2.8.19.exe
> $ file redis-server*
> redis-server-2.8.19: ELF 64-bit LSB executable, x86-64, version 1
> (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24,
> BuildID[sha1]=c59ab61e1d5b94e26d079b04e9e593b6be3f0230, not stripped
> redis-server-2.8.19.app: Mach-O 64-bit executable
> redis-server-2.8.19.exe: PE32+ executable (console) x86-64, for MS
>> Windows
> 
 
 So, installing NiFi will include externally built binaries, not just
 scripts but full ELF, MacOS, and Windows PE binaries.
 
 Is this a problem for Apache?  Does this cause concern for anyone else?
 
 Regards,
 Joe
>> 

Re: FTPS support in FTPTransfer (List/Fetch/Get/PutFTP) in NiFi 1.3

[Joey Frazee]

Jeremy, I started on this at one point (see 
https://github.com/jfrazee/nifi/commits/NIFI-2188). It’s lacking tests and 
there’s some changes needed to make it more explicit when you’ve misconfigured 
it, but it might be worth a go if you want to try building from the branch.

I somehow knew this would be for Box.com.

Thanks!

-joey

On Jul 18, 2017, 10:49 AM -0700, Jeremy Farbota , wrote:
> Moving this comment to dev per Andy's suggestion.
>
> Currently NiFi does not support FTPS.
>
> Box.com does not use SFTP (only FTP over SSL or FTPS). It'd be great to
> move my ftp stuff to NiFi, but I need FTPS. AFAIK the FTPTransfer util
> would need to include FTPSClient library and reference it. While I could
> tape this together with an execute script or invoke script, I'd prefer it
> built-in so I can utilize all the functionality of ListFtp, FetchFtp, etc.
>
> Let me know if anyone would be interested to work on this, and I'd be glad
> to test and give feedback.
>
> Kindly,
>
>
> [image: Payoff, Inc.]
> *Jeremy Farbota*
> Software Engineer, Data
> Payoff, Inc.
>
> jfarb...@payoff.com
> (217) 898-8110 <+2178988110
>
> On Mon, Jul 17, 2017 at 5:53 PM, Andy LoPresto  wrote:
>
> > Jeremy,
> >
> > I believe you are correct that substituting the FTPSClient class (explicit
> > by default) would work [1]. If your workflow is pretty consistent in the
> > necessary operations, you may not even need to extend the entire processor,
> > and could instead script something quickly via ExecuteScript/
> > InvokeScriptedProcessor.
> >
> > I can’t speak to an explicit reason FTPS was not implemented. It is likely
> > that the user requirements at the time found (S)FTP to be sufficient for
> > the documented use cases. In general, existing tickets which target
> > non-latest releases, have a low priority, are unassigned, and have not been
> > updated in many months are unlikely to be included in future releases
> > without revitalized attention. If this is something you’re interested in,
> > I’m sure there would be support on the lists (probably more on
> > dev@nifi.apache.org) to help you pursue it. If you’re not prepared to
> > tackle it, you may still have better luck finding another community member
> > willing to help there.
> >
> > [1] https://stackoverflow.com/a/36309570/70465
> >
> >
> > Andy LoPresto
> > alopre...@apache.org
> > *alopresto.apa...@gmail.com *
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > On Jul 17, 2017, at 4:47 PM, Jeremy Farbota  wrote:
> >
> > Our secure file storage provider does not support SFTP. I'm hoping to
> > migrate our folder sync processes and some other operational stuff to NiFi.
> > Unfortunately it seems NiFi does not support FTPS.
> >
> >
> > After a little research[1] it seems that one could extend FTPTransfer[2]
> > to work with FTPS[3][4] (possibly using an implicit = true/false parameter
> > in the processor).
> >
> >
> > Is there a reason why FTPS support was left out? Does anyone have a custom
> > processor that handles FTPS already?
> >
> >
> > I noticed this ticket which is listed under 0.7 low priority[5]. Is this
> > something that might come in a later version?
> >
> >
> > Thanks in advance.
> >
> >
> > [1] https://community.hortonworks.com/questions/33314/can-nifi-connect-to-
> > ftps-protocol.html
> > [2] https://github.com/apache/nifi/blob/master/nifi-nar-
> > bundles/nifi-standard-bundle/nifi-standard-processors/src/
> > main/java/org/apache/nifi/processors/standard/util/FTPTransfer.java
> > [3] https://commons.apache.org/proper/commons-net/
> > apidocs/org/apache/commons/net/ftp/FTPSClient.html
> > [4] https://commons.apache.org/proper/commons-net/
> > examples/ftp/FTPClientExample.java
> > [5] https://issues.apache.org/jira/browse/NIFI-2278
> >
> > [image: Payoff, Inc.]
> > *Jeremy Farbota*
> > Software Engineer, Data
> > Payoff, Inc.
> >
> > jfarb...@payoff.com
> >
> >
> >

Re: RTC clarification

[Joey Frazee]

Joe(s), as you mentioned, even if we have a non-committer review, it can’t be 
merged until a committer decides whether to accept whatever decision was 
provided. So, the burden is still on committers as to whether it’s really a +1 
or not. And presumably this should only happen if there’s some evidence that 
the right things happened.

So, I think the big question is how much to encourage non-committers to review. 
And, I think there’s probably more to gain from trying to be very encouraging 
than not. It builds community, it can be a teaching moment, and it can offload 
work where there isn’t enough time or expertise.

This would be really easy if the project required a double +1 to merge because 
there wouldn’t be any confusion about whether any one reviewer had sole 
authority, but I think if the developer guide and README was very encouraging 
and very explicit about how non-committer reviews would be treated, it’ll be 
easy to roll with.

On Jul 7, 2017, 9:10 AM -0500, Joe Witt , wrote:
> We're looking at a possible set of scenarios comprised of a
> 'submitter' and a 'reviewer'. There can be one or more reviewers. A
> submitter or reviewer is either a committer or is not a committer. If
> there is at least one reviewer that is a committer it is a committer
> reviewed scenario. So with that in mind...
>
> A) Committer submitted, Non-Committer reviewed = This can be merged in
> the event of a positive review outcome.
>
> B) Committer submitted, Committer reviewed = This can be merged in the
> event of a positive review outcome.
>
> C) Non-Committer submitted, Non-Committer reviewed = This cannot be
> merged until there is at least one positive committer review.
> However, that review is made easier for the committer as someone who
> is still building merit in the community has already done work here
> and further this is a great chance to work with/grow those folks in
> the community.
>
> D) Non-committer submitted, Committer reviewed = This can be merged in
> the event of a positive review outcome.
>
> Scenario B and D is easy and probably there is no discussion needed.
> Same for scenario C as there has to be at least one committer involved
> for the code to get pushed in the first place. I suspect it is only
> scenario A where we have some 'yeah but...' sort of thoughts going on.
>
> I am favorable to us allowing scenario A to result in a merge because
> at the end of the day there is a committer involved, heavily involved
> in fact, and they've already been voted by the community to be a
> committer. That is a high bar as it should be.
>
> The risk of bad commits getting through needs to be balanced against
> the risk of community growth being hampered by slow review cycles.
> Let's be honest, we need the help with review bandwidth. It is really
> hard to keep up and to my mind one of the best ways to annoy/deter
> contributors from advancing is to make the time between contribution
> to feedback take a long time.
>
>
> On Fri, Jul 7, 2017 at 9:58 AM, Joe Skora  wrote:
> > I agree that non-committer review make more sense if the reviewer has
> > established some level of credibility and involvement in the community.
> >
> > Ultimately, the final committer has to be a community member and they will
> > have final responsibility for the code being Apache compliant and NiFi
> > ready, so even reviews by less well known members of the community should
> > still be low risk.
> >
> > On Fri, Jul 7, 2017 at 9:24 AM, Bryan Bende  wrote:
> >
> > > I agree with encouraging reviews from everyone, but I lean towards
> > > "binding" reviews coming from committers.
> > >
> > > If we allow any review to be binding, there could be completely
> > > different levels of review that occur...
> > >
> > > There could be someone who isn't a committer yet, but has been
> > > contributing already and will probably do a very thorough review of
> > > someone's contribution, and there could be someone else who we never
> > > interacted with us before and writes "+1 LGTM" on a PR and we have no
> > > idea if they know what they are talking about or if they even tried
> > > the contribution to see if it works. Obviously a committer can also
> > > write "+1 LGTM", but I think when that comes from a committer it holds
> > > more weight.
> > >
> > > I think we may also want to clarify if we are only talking about
> > > "submitted by committer, reviewed by non-committer" or also talking
> > > about "submitted by non-committer, reviewed by non-committer".
> > >
> > > For the first case I can see the argument that since the contribution
> > > is from a committer who is already trusted, they can make the right
> > > judgement call based on the review. Where as in the second case, just
> > > because a community member submitted something and another community
> > > member says it looks good, doesn't necessarily mean a committer should
> > > come along and automatically merge it 

Re: NiFi and MapR streams?

[Joey Frazee]

Konstantin, this is a bit of a guess, but this might work with a NiFi that was 
built against MapR’s kafka-clients. So you’d need to change the kafka9.version 
property when building NiFi. For example:

mvn clean install -DskipTests -Pmapr -Dhadoop.version=2.7.0-mapr-1703 
-Dkafka9.version=0.9.0.0-mapr-1703

The versions need to be whatever version of the MapR client libs you have 
installed. I haven’t tested this though so admittedly it’s a shot in the dark.

-joey

> On Jun 29, 2017, at 8:35 AM, KonstantinB  wrote:
> 
> Hi guys,
> 
> is there a way NiFi to consume MapR Streams? I try ConsumeKafka, which are
> using the 0.9 client - MapR Streams are compatible with Kafka 0.9, but it
> seems not working. Probably because MapR streams not used Zookeeper for
> stream detections, but а CLDB.
> Any help will be appreciated. Thanks.
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-and-MapR-streams-tp16306.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Expression Language For AMQP Processors

[Joey Frazee]

Sometimes it’s just an historical oversight that EL support isn’t available for 
certain properties, but in this case the connection is initialized only once, 
so the connection specific properties like hostname, port, username and 
password cannot be evaluated against FlowFile attributes in any meaningful way.

Now, since this processor was first contributed, the variable registry was 
added so there is now a scenario where you could use EL and properties from the 
variable registry. This would be a reasonable enhancement, though there’s a 
risk of confusion since it won’t behave correctly if the expressions are in 
terms of FlowFile attributes.

-joey

> On Jun 27, 2017, at 8:33 AM, hemantvsn  wrote:
> 
> Hi,
> 
> For publishing/consuming messages on Rabbit queue, NIFI provides support via
> PublishAMQP and ConsumeAMQP processors.
> 
> However, only limited properties support EL while others don't.
> 
> Properties supporting EL
> Exchange Name
> Routing Key
> 
> Thus we can dynamically control these.
> 
> However remaining properties like
> hostname
> port
> username
> password
> 
> aren't controlled by EL
> 
> Any idea why this was done and things to do to make them configurable
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Expression-Language-For-AMQP-Processors-tp16269.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: MiNiFi Java as Windows Service

[Joey Frazee]

Jefff, there was a related thread about this a few months back: 
https://lists.apache.org/thread.html/05abafa804b0bb774211ef602d5fc2ec3aa8bdf5c584f2aab3014b42@%3Cdev.nifi.apache.org%3E
 


In that, Andre suggested an approach using https://github.com/kohsuke/winsw 
 and maybe he has a branch already, but 
anything with a compatible license will certainly be a great contribution.

I’m sure me and him, and others are very interested in this.

-joey

> On Jun 26, 2017, at 1:17 PM, Aldrin Piri  wrote:
> 
> Hi Jeff,
> 
> A PR would most certainly be welcomed and (likely could be an easy win for
> NiFi as well).  The only caveat to be mindful of is that any
> frameworks/tools/utilities should be friendly with ALv2 terms as per
> http://www.apache.org/legal/resolved.html.   If you have any particular
> items in mind and are unsure, please feel free to follow up on here and we
> can work though what licensing looks like.
> 
> On Mon, Jun 26, 2017 at 2:12 PM, Jeff Zemerick  wrote:
> 
>> Hi,
>> 
>> I see a ticket to make the C++ version run as a Windows service
>> (MINIFI-89). Is there a recommended method of running the Java version as a
>> Windows service? If not, would there be any interest in a pull request to
>> add that functionality?
>> 
>> Thanks,
>> Jeff
>> 

Re: AvroSchemaRegistry doesn't enjoy copy and paste?

[Joey Frazee]

Andre, if there are any line breaks in the schema and leading spaces on those 
new lines, then this occurs. So if you minify the avsc or remove the leading 
spaces, all should be good.

Will open a JIRA on this since, including myself, I think you’re the third to 
see this. Any chance you’re using Safari?

> On Jun 22, 2017, at 8:53 AM, Andrew Grande  wrote:
> 
> Definitely something is auto replacing quotes, I can confirm pasting worked
> fine before from a programmer's editor.
> 
> Andrew
> 
> On Thu, Jun 22, 2017, 9:06 AM Mark Payne  wrote:
> 
>> Andre,
>> 
>> I've not seen this personally. I just clicked on the link you sent, copied
>> the schema,
>> and pasted it in, and it did not have any problems. What application are
>> you copying
>> the text from? I've certainly seen that some applications (specifically
>> Microsoft Outlook
>> and Office) love to take double-quotes and change them into other
>> characters so that
>> they look nicer. But if you then copy that and paste it, it is not pasting
>> a double-quote but
>> some other unicode character.
>> 
>> Would recommend you open the below link in Chrome and copy from there and
>> see if
>> that works?
>> 
>> Thanks
>> -Mark
>> 
>> 
>> 
>>> On Jun 22, 2017, at 8:56 AM, Andre  wrote:
>>> 
>>> All,
>>> 
>>> I was playing with the AvroSchemaRegistry and noticed it seems to not
>> play
>>> ball when the DFM pastes the schema into the dynamic property value.
>>> 
>>> To test it I basically copied the demo schema from Mark's blog post[1]
>> and
>>> pasted into a NiFi 1.3.0 instance. To my surprise the controller would
>> not
>>> validate, instead it displayed:
>>> 
>>> "was expecting double-quote to start field name"
>>> 
>>> I also faced similar errors using the following schema:
>>> 
>>> 
>> https://github.com/fluenda/SecuritySchemas/blob/master/CEFRev23/cefRev23_nifi.avsc
>>> 
>>> Has anyone else seen this?
>>> 
>>> Cheers
>> 
>> 

Re: LookupAttribute + CSV Lookup handling of non-matches

[Joey Frazee]

Andre, there's a flag for "Include Empty Values". If set to false then the 
effect is that non-matches result in a transfer to unmatched. The default is 
true so I'd expect to see what you're seeing.

The description for that property isn't good enough though. I don't know how 
you would have known this now that I'm looking at the docs again.

-joey

> On Jun 21, 2017, at 11:36 PM, Andre  wrote:
> 
> Hi all,
> 
> I have been playing with the lookup attribute and noticed a behavior that
> seems counter intuitive to me:
> 
> When using the SimpleCSVLookupService I noticed that whenever the service
> fails to find a match to a particular value, the LookupAttribute processor
> still populated the respective dynamic property with "null" and routed the
> flowfile to match instead of leaving the field untouched and transferring
> the flow to unmatched.
> 
> Is this expected behavior? If so, what is the use of unmatched?
> 
> Kind regards

Re: [VOTE] Release Apache NiFi 1.3.0

[Joey Frazee]

+1 (non-binding)

- Verified checksums, signatures and commit ID
- Successfully built and ran tests on OSX, Amazon Linux, and Docker maven:latest
- Built RPM with `mvn -T 2.0C clean install -Prpm,generateArchives -DskipTests` 
and tested install
- Tested data flows with ConvertRecord and LookupAttribute processors

Great job y'all!

> On Jun 6, 2017, at 10:08 PM, Aldrin Piri  wrote:
> 
> +1, binding
> 
> Hashes and signature looks good
> Build and contrib check was clean
> README, L look good
> Verified some simple flows
> 
> 
> Environment:
> Apache Maven 3.3.1 (cab6659f9874fa96462afef40fcf6bc033d58c1c;
> 2015-03-13T16:10:27-04:00)
> Maven home: /usr/local/maven
> Java version: 1.8.0_121, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.18.17-13.el7.x86_64", arch: "amd64", family:
> "unix"
> CentOS Linux release 7.3.1611 (Core)
> 
> On Tue, Jun 6, 2017 at 9:37 PM, Koji Kawamura 
> wrote:
> 
>> +1 (binding)
>> 
>> - ran through release helper
>> - ran clean build, test passed without issue
>> - ran simple flow using standalone/non-secure/secure/clustered NiFi
>> - confirmed few JIRAs data flow those had been addressed within 1.3.0
>> 
>> 
>> On Wed, Jun 7, 2017 at 2:15 AM, Pierre Villard
>>  wrote:
>>> +1 (binding)
>>> 
>>> - verified signatures, checksums, license, notice and readme.
>>> - full build on OSX, Windows 10, CentOS 7 (faced some test errors while
>>> building on Windows but the failing tests have already opened JIRAs to
>>> track them)
>>> - checked UIs changes
>>> - tested some workflows for Record based processors and SiteToSite
>>> reporting tasks
>>> 
>>> Release this package as nifi-1.3.0
>>> 
>>> 
>>> 2017-06-06 18:55 GMT+02:00 Yolanda Davis :
>>> 
 +1 (binding)
 
 -ran through release helper, verified hashes, signatures, checked
>> README,
 LICENSE, NOTICE
 -ran build and tested with flows using QueryRecord
>> (RecordReaders/Writers
 etc), worked as expected.
 
 On Tue, Jun 6, 2017 at 12:44 PM, Joe Skora  wrote:
 
> +1 (binding)
> 
> * Verified GPG signature.
> * Verified hashes.
> * Built source with contrib-check profile.
> * Verified README, NOTICE, and LICENSE.
> * Verified commit ID.
> * Verified RC was branched from master.
> * Ran binary and tested basic functionality.
> 
> 
> On Tue, Jun 6, 2017 at 10:27 AM, Bryan Bende 
>> wrote:
> 
>> +1 (binding) Release this package as nifi-1.3.0
>> 
>> - Ran through release helper and everything checked out
>> - Ran some test flows with HDFS processors and verified they worked
>> correctly
>> 
>> On Tue, Jun 6, 2017 at 12:42 AM, James Wing 
>> wrote:
>>> +1 (binding).  I ran through the release helper -- verified
>> signature
> and
>>> hashes, ran the full build, checked readme, license, and notice in
> source
>>> and output.  I ran a simple flow and verified that it still works.
>>> 
>>> Thanks for putting this release together, Matt.
>>> 
>>> James
>>> 
>>> 
>>> On Mon, Jun 5, 2017 at 10:54 AM, Matt Gilman >> 
>> wrote:
>>> 
 Hello,
 
 
 I am pleased to be calling this vote for the source release of
 Apache
>> NiFi
 nifi-1.3.0.
 
 
 The source zip, including signatures, digests, etc. can be found
>> at:
 
 https://repository.apache.org/content/repositories/
 orgapachenifi-1108
 
 
 The Git tag is nifi-1.3.0-RC1
 
 The Git commit ID is ddb73612bd1512d8b2151b81f9aa40811bca2aaa
 
 https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
 ddb73612bd1512d8b2151b81f9aa40811bca2aaa
 
 
 Checksums of nifi-1.3.0-source-release.zip:
 
 MD5: 8b115682ac392342b9edff3bf0658ecb
 
 SHA1: f11cdebbabdc0d8f1f0dd4c5b880ded39d17f234
 
 SHA256: 9ba5565729d98c472c31a1fdbc44e9
 dc1eee87a2cf5184e8428743f75314
>> 5b7f
 
 
 Release artifacts are signed with the following key:
 
 https://people.apache.org/keys/committer/mcgilman.asc
 
 
 KEYS file available here:
 
 https://dist.apache.org/repos/dist/release/nifi/KEYS
 
 
 110 issues were closed/resolved for this release:
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?
 projectId=12316020=12340498
 
 
 Release note highlights can be found here:
 
 https://cwiki.apache.org/confluence/display/NIFI/
 

Re: [VOTE] Release Apache NiFi MiNiFI C++ 0.2.0 (RC2)

[Joey Frazee]

+1 (non-binding)

- Verified checksums, signatures and commit id
- Successfully built and ran tests with Apple LLVM version 8.1.0 
(clang-802.0.42)
- Successfully ran make package and used binaries from generated archive
- Ran example data flow from README.md
- Checked for L

> On May 9, 2017, at 4:06 PM, Bryan Rosander  wrote:
> 
> +1 non-binding
> 
> Verified signature, checksums
> Built on Ubuntu 16.04, OSx bare metal, Ubuntu 16.04, Fedora 25, Centos 7
> Docker containers -
> https://github.com/brosander/minifi-cpp-tooling/tree/master/Dockerfiles
> Ran flow using secure site to site from the above docker containers into a
> 3 node NiFi 1.2.0 cluster
> 
> Notes: README.md doesn't list boost as a runtime dependency.  Fedora and
> Centos both required its installation for MiNiFi to work, possible that
> Ubuntu already has it.
> 
> On Tue, May 9, 2017 at 10:03 AM, Matt Gilman 
> wrote:
> 
>> +1 (binding)
>> 
>> - Verified signature, hashes, build, etc
>> - Ran through sample flows
>> 
>> Looks Good!
>> 
>> Matt
>> 
>> On Mon, May 8, 2017 at 7:10 PM, Marc  wrote:
>> 
>>> +1 non binding
>>>   * sigs and hashes verified
>>>   * build with ubuntu 16.04 and osx.
>>>   * ran flows with all supported processors sans ListenHTTP.
>>>   * Ran into same test issues but we've created some tickets (
>> MINIFI-304
>>> ) -- for which I have a fix and we'll be introducing in subsequent
>>> versions.
>>> 
>>> On Mon, May 8, 2017 at 7:05 PM, Tony Kurc  wrote:
>>> 
 Resend from my apache email (ignore previous):
 
 +1 (binding)
 
 - verified hashes and signature
 - checked over the README, LICENSE and NOTICE
 - build without issue on ubuntu 16.06 (x86_64)
 - ran a simple flow without problems
 
 On Mon, May 8, 2017 at 6:04 PM, Tony Kurc  wrote:
 
> +1 (binding)
> 
> - verified hashes and signature
> - checked over the README, LICENSE and NOTICE
> - build without issue on ubuntu 16.06 (x86_64)
> - ran a simple flow without problems
> 
> On Mon, May 8, 2017 at 10:57 AM, Kevin Doran <
>> kdoran.apa...@gmail.com>
> wrote:
> 
>> +1 (non-binding), despite one minor bug found, for which I opened
>> MINIFI-303 [1].
>> 
>> - Verified signature, hashes, git commit
>> - Built successfully (Mac OS 10.12.4)
>> - Verified tests (Mac OS 10.12.4)
>> - Verified linting
>> - Reviewed README, NOTICE, and LICENSE, both in source and in build
>> output.
>> - Verified application works as expected with a few variants of flow
>> config file
>> 
>> Potential future improvements (have not created JIRAs or searched
>> for
>> existing JIRAs):
>> 
>> - nifi.security.need.ClientAuth property not working when set to
>> false
>> (MINIFI-303 opened) [1]
>> - Fix or note expected CMake Dev Warning [2]
>> - Fix or note expected for compile warnings for civetweb, built
>> under
 the
>> thirdparty/ directory as part of the minifi build. [3]
>> 
>> [1] https://issues.apache.org/jira/browse/MINIFI-303
>> 
>> [2] CMake Warning output:
>> 
>> CMake Warning (dev) at libminifi/CMakeLists.txt:22 (project):
>>  Policy CMP0048 is not set: project() command manages VERSION
 variables.
>>  Run "cmake --help-policy CMP0048" for policy details.  Use the
>> cmake_policy
>>  command to set the policy and suppress this warning.
>> 
>>  The following variable(s) would be set to empty:
>> 
>>PROJECT_VERSION_MAJOR
>>PROJECT_VERSION_MINOR
>>PROJECT_VERSION_PATCH
>> This warning is for project developers.  Use -Wno-dev to suppress
>> it.
>> 
>> 
>> [3] Example of civetweb warning:
>> 
>> nifi-minifi-cpp-0.2.0-source/thirdparty/civetweb-1.9.1/src/
 civetweb.c:14680:56:
>> warning: expansion of date or time macro is not reproducible
>>  [-Wdate-time]
>>NULL, NULL, block, sizeof(block), "Build: %s%s",
>> __DATE__, eol);
>> 
>> 
>> 
>> On 5/8/17, 10:33, "Bryan Bende"  wrote:
>> 
>>+1 (binding) Release this package as nifi-minifi-cpp-0.2.0
>> 
>>- Verified signature and hashes
>>- Built on OSX
>>- Successfully ran binary using provided sample config for s2s
>> 
>> 
>> 
>>On Mon, May 8, 2017 at 2:26 AM, Koji Kawamura <
 ijokaruma...@gmail.com>
>> wrote:
>>> +1 (non-binding)
>>> 
>>> Full build and test finished successfully without any issue on
>>> OS
 X.
>>> 
>>> Here are the things that I look forward in future improvements
>> (didn't
>>> check existing JIRAs):
>>> 
>>> 
>> --
>>> 1. minifi.sh restart does not working?
>>> 

Re: [VOTE] Release Apache NiFi 1.2.0 (RC2)

[Joey Frazee]

Andy, 

I opened a JIRA [1] and posted a link to the test log output [2]. I’m running 
Docker for Mac 17.03.1-ce-mac5 (16048) and launching the maven:latest container 
with `docker run -it maven:latest /bin/bash`.

Event with a plain clean install (no parallelism or contrib-check) I 
consistently get:

Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.65 sec <<< 
FAILURE! - in org.apache.nifi.provenance.CryptoUtilsTest
testShouldNotValidateUnreadableFileBasedKeyProvider(org.apache.nifi.provenance.CryptoUtilsTest)
  Time elapsed: 0.111 sec  <<< FAILURE!
org.codehaus.groovy.runtime.powerassert.PowerAssertionError: assert 
!unreadableKeyProviderIsValid
   ||
   |true
   false
at 
org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:402)
at 
org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:650)
at 
org.apache.nifi.provenance.CryptoUtilsTest.testShouldNotValidateUnreadableFileBasedKeyProvider(CryptoUtilsTest.groovy:214)

Running org.apache.nifi.provenance.AESProvenanceEventEncryptorTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.285 sec - in 
org.apache.nifi.provenance.AESProvenanceEventEncryptorTest

Results :

Failed tests: 
  CryptoUtilsTest.testShouldNotValidateUnreadableFileBasedKeyProvider:214 
assert !unreadableKeyProviderIsValid
   ||
   |true
   false

1. https://issues.apache.org/jira/browse/NIFI-3836
2. https://gist.github.com/jfrazee/cf097440e0fd882612ba60acb6301134

> On May 8, 2017, at 1:12 AM, Andy LoPresto <alopresto.apa...@gmail.com> wrote:
> 
> Joey,
> 
> I am curious about the test failure. That is the test I refactored due to the 
> POSIX attribute issue failing on Windows right before the RC. It passed when 
> I built on Linux so if you can open a Jira or at least provide me with a 
> stack trace and any steps to make it reproducible, I would appreciate it. 
> 
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On May 7, 2017, at 22:17, Joey Frazee <joey.fra...@icloud.com> wrote:
>> 
>> +1 (non-binding)
>> 
>> - Verified signature, checksums and commit id
>> - Successfully ran `mvn clean install -Pcontrib-check` on OS X 10.12.4 with 
>> Oracle JDK 1.8.0_121 and Amazon Linux with OpenJDK 1.8.0_121
>> - Successfully ran `mvn clean install -DskipTests` on Docker maven:latest 
>> with OpenJDK 1.8.0_121 but 
>> CryptoUtilsTest.testShouldNotValidateUnreadableFileBasedKeyProvider always 
>> failed with !unreadableKeyProviderIsValid
>> - Built RPM with `mvn -T 4.0C clean install -Prpm,generateArchives 
>> -DskipTests` and tested RPM install
>> - Tested GenerateTableFetch and QueryDatabaseTable with MS SQL 2008 
>> [NIFI-3585], among other flows.
>> 
>>> On May 5, 2017, at 9:07 PM, Bryan Bende <bbe...@apache.org> wrote:
>>> 
>>> Hello,
>>> 
>>> I am pleased to be calling this vote for the source release of Apache
>>> NiFi nifi-1.2.0 (RC2).
>>> 
>>> The source zip, including signatures, digests, etc. can be found at:
>>> https://repository.apache.org/content/repositories/orgapachenifi-1104
>>> 
>>> The Git tag is nifi-1.2.0-RC2
>>> The Git commit ID is 3a605af8e0ac024fb0ba67262d49dab2727b2576
>>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=3a605af8e0ac024fb0ba67262d49dab2727b2576
>>> 
>>> Checksums of nifi-1.2.0-source-release.zip:
>>> MD5: 90e298a9e23a9dab65358daddd8b5990
>>> SHA1: e138941f576bdb1dff17df6674c19ffae3ef6719
>>> SHA256: c18398800c435dabff9f45ec55a450ca78c3c1aec222aa295c0e2057912feeb6
>>> 
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/bbende.asc
>>> 
>>> KEYS file available here:
>>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>> 
>>> 381 issues were closed/resolved for this release:
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12338432
>>> 
>>> Release note highlights can be found here:
>>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.2.0
>>> 
>>> The vote will be open for 72 hours.
>>> Please download the release candidate and evaluate the necessary items
>>> including checking hashes, signatures, build
>>> from source, and test.  The please vote:
>>> 
>>> [ ] +1 Release this package as nifi-1.2.0
>>> [ ] +0 no opinion
>>> [ ] -1 Do not release this package because because...
>> 

Re: [VOTE] Release Apache NiFi 1.2.0 (RC2)

[Joey Frazee]

+1 (non-binding)

- Verified signature, checksums and commit id
- Successfully ran `mvn clean install -Pcontrib-check` on OS X 10.12.4 with 
Oracle JDK 1.8.0_121 and Amazon Linux with OpenJDK 1.8.0_121
- Successfully ran `mvn clean install -DskipTests` on Docker maven:latest with 
OpenJDK 1.8.0_121 but 
CryptoUtilsTest.testShouldNotValidateUnreadableFileBasedKeyProvider always 
failed with !unreadableKeyProviderIsValid
- Built RPM with `mvn -T 4.0C clean install -Prpm,generateArchives -DskipTests` 
and tested RPM install
- Tested GenerateTableFetch and QueryDatabaseTable with MS SQL 2008 
[NIFI-3585], among other flows.

> On May 5, 2017, at 9:07 PM, Bryan Bende  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.2.0 (RC2).
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1104
> 
> The Git tag is nifi-1.2.0-RC2
> The Git commit ID is 3a605af8e0ac024fb0ba67262d49dab2727b2576
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=3a605af8e0ac024fb0ba67262d49dab2727b2576
> 
> Checksums of nifi-1.2.0-source-release.zip:
> MD5: 90e298a9e23a9dab65358daddd8b5990
> SHA1: e138941f576bdb1dff17df6674c19ffae3ef6719
> SHA256: c18398800c435dabff9f45ec55a450ca78c3c1aec222aa295c0e2057912feeb6
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/bbende.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 381 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12338432
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.2.0
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
> 
> [ ] +1 Release this package as nifi-1.2.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because because...

Re: Commit d90cf84 seems to break build?

[Joey Frazee]

Andre I think that solution strikes the right balance without being a lot of 
effort.

As mentioned Travis-ci's docs say that caching stuff like .m2 doesn't help but 
in my experience you usually still get a little bump so it's worth it as long 
as the ci isn't lying about the results. Removing ~/.m2/org/apache/nifi should 
minimize that, so good.

> On Mar 25, 2017, at 6:48 PM, Andre  wrote:
> 
> Aldrin,
> 
> While we may not get much granularity we can certainly hack our way by "rm
> -rf ~/.m2/repository/whatever_we_want_to_delete" prior to calling maven ?
> 
> I appreciate caching is sometime a pain, but given our build currently
> takes almost 40 minutes and travis-ci.org jobs will timeout at 50 minutes,
> I suspect the time saved by not having to download dependencies comes very
> handy.
> 
> I would suggest that we keep caching on (the thing that happened with
> jB(C|c)rypt is not that usual after all...) but remove what we are testing
> (i.e. ~/.m2/repository/org/nifi ) prior to build, what do you think?
> 
> Cheers
> 
> 
>> On Sun, Mar 26, 2017 at 3:38 AM, Aldrin Piri  wrote:
>> 
>> Awesome, thanks for fixing it up, Bryan.
>> 
>> I don't think we can get that kind of granularity with Travis,
>> unfortunately.  However, the last time was because an artifact changed its
>> name (or more specifically, casing).
>> 
>> Not sure removing caching is the best option, but seems like the the
>> optimization may not provide as much value in build speedup as the
>> consistent, cleanroom environment for builds.  Just a thought.
>> 
>>> On Sat, Mar 25, 2017 at 11:18 AM, Bryan Bende  wrote:
>>> 
>>> Thanks Aldrin, I pushed the commit.
>>> 
>>> As far as travis, I am not familiar with how it works, but can you
>>> specify what to cache?
>>> 
>>> In this case we didn't need a completely clean .m2, we just needed a
>>> clean .m2/org/apache/nifi.
>>> 
>>> On Sat, Mar 25, 2017 at 11:10 AM, Aldrin Piri 
>>> wrote:
 Please just push to correct. Simple fixes are fine in my book.
 
 Does it make sense to potentially scrap caching in Travis?
 
 This is another time we have missed something like this that no caching
 would have prevented. Additionally, given the large footprint of the
 repository download it seems as though its benefit may be marginal as
>> per
 https://docs.travis-ci.com/user/caching/#How-does-caching-work%3F
> On Sat, Mar 25, 2017 at 11:04 Bryan Bende  wrote:
> 
> Andre,
> 
> Thanks for bringing this up.
> 
> The standard prioritizers moved from the standard bundle to the
> framework bundle, and sure enough the parent was still set as standard
> bundle. We had never built with a clean repo and were getting lucky,
> so I am glad you found this.
> 
> In
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-standard-
>>> prioritizers/pom.xml
> 
> The parent should be nifi-framework and not
> nifi-standard-bundle.
> 
> If no one objects I can push up this change, not sure if we need a
> formal PR for a one line change for something thats already broken??
> 
> Thanks,
> 
> Bryan
> 
>> On Sat, Mar 25, 2017 at 9:33 AM, Andre  wrote:
>> dev,
>> 
>> Is anyone else having issues building master from "clean" (i.e. rm
>> -rf
>> ~/.m2/repositories/org/apache/nifi) after commit  d90cf84 ?
>> 
>> My attempts to build currently yield:
>> 
>> $ mvn -T2.0C -DskipTests=true -Pdir-only clean install
>> [INFO] Scanning for projects...
>> [ERROR] [ERROR] Some problems were encountered while processing the
>>> POMs:
>> [WARNING] 'parent.relativePath' of POM
>> org.apache.nifi:nifi-standard-prioritizers:[unknown-version]
>> 
> (/home/afucs/nifi/nifi-nar-bundles/nifi-framework-bundle/
>>> nifi-framework/nifi-standard-prioritizers/pom.xml)
>> points at org.apache.nifi:nifi-framework instead of
>> org.apache.nifi:nifi-standard-bundle, please verify your project
> structure
>> @ line 17, column 13
>> [FATAL] Non-resolvable parent POM for
>> org.apache.nifi:nifi-standard-prioritizers:[unknown-version]: Could
>>> not
>> find artifact org.apache.nifi:nifi-standard-
>> bundle:pom:1.2.0-SNAPSHOT
>>> and
>> 'parent.relativePath' points at wrong local POM @ line 17, column 13
>> @
>> [ERROR] The build could not read 1 project -> [Help 1]
>> [ERROR]
>> [ERROR]   The project
>> org.apache.nifi:nifi-standard-prioritizers:[unknown-version]
>> 
> (/home/afucs/nifi/nifi-nar-bundles/nifi-framework-bundle/
>>> nifi-framework/nifi-standard-prioritizers/pom.xml)
>> has 1 error
>> [ERROR] Non-resolvable parent POM for
>> org.apache.nifi:nifi-standard-prioritizers:[unknown-version]: Could
>>> not
>> find artifact org.apache.nifi:nifi-standard-
>> 

Re: SQL to CSV?

[Joey Frazee]

Michael, I think you’re right to call this out. I frequently find myself 
stringing together flows with ExecuteScripts (which you should be able to use 
to pull a schema out by creating an Avro DataFileStream from the InputStream 
and then calling getSchema().toString()) or conversions to/from JSON and Avro 
to handle all the scenarios.

I think the heart of the solution shouldn’t just be the addition of an output 
attribute including the schema, but something generic like you mention in your 
(4), especially considering that there are at least 7 issues [1-7] open for 
variations on this. Instead of a just a converter processor, though, it’d 
probably be smart to make it some kind of controller service so it can be 
easily exposed to other processors like ExecuteSQL and QueryDatabaseTable.

-joey

1. https://issues.apache.org/jira/browse/NIFI-2743
2. https://issues.apache.org/jira/browse/NIFI-1623
3. https://issues.apache.org/jira/browse/NIFI-1623
4. https://issues.apache.org/jira/browse/NIFI-1702
5. https://issues.apache.org/jira/browse/NIFI-1704
6. https://issues.apache.org/jira/browse/NIFI-1398
7. https://issues.apache.org/jira/browse/NIFI-2725

> On Dec 28, 2016, at 3:18 PM, Knapp, Michael  
> wrote:
> 
> Nifi Devs,
> 
> I noticed you have two processors (ExecuteSQL and QueryDatabaseTable) that 
> perform SQL select statements and put the results into a flow file.  While I 
> am not sure what their difference is, I did notice that they both produce 
> avro, and the schema is inferred from the result set.  While the schema is 
> included in the output file’s contents, I am not sure of any easy way to get 
> that from a *StreamCallback.  So I am wondering,
> 
> 
> 1.   Could we update the processor to support multiple output formats?  I 
> think CSV should definitely be supported.  Parquet might also be useful for 
> me.  JSON is an option but since you already have a ConvertAvroToJSON 
> processor that is not a big deal for me.
> 
> 2.   Could we update the processor to include the schema as one of the 
> output flow file attributes?
> 
> 3.   Is there any utility to get an avro schema from the input stream 
> callback?
> 
> 4.   Has anybody thought about writing a processor to convert Avro to 
> CSV?  Or even something more generic than that, a generic format conversion 
> processor?  It could support CSV, JSON, Avro, Parquet, XML, and possibly 
> others.
> 
> Please let me know,
> 
> Michael Knapp
> Capital One
> 
> 
> The information contained in this e-mail is confidential and/or proprietary 
> to Capital One and/or its affiliates and may only be used solely in 
> performance of work or services for Capital One. The information transmitted 
> herewith is intended only for use by the individual or entity to which it is 
> addressed. If the reader of this message is not the intended recipient, you 
> are hereby notified that any review, retransmission, dissemination, 
> distribution, copying or other use of, or taking of any action in reliance 
> upon this information is strictly prohibited. If you have received this 
> communication in error, please contact the sender and delete the material 
> from your computer.

Re: [DISCUSS] Official Apache NiFi Docker Image

[Joey Frazee]

+1

I think this is a great idea because there are at least half a dozen or more 
Dockerfiles and published images floating around. Having something that is 
endorsed and reviewed by the project should help ensure quality.

One question though: Will the images target a single instance NiFi or a 
cluster, e.g., using compose? Or both?

-joey

> On Dec 28, 2016, at 8:55 AM, Jeremy Dyer  wrote:
> 
> Team,
> 
> I wanted to discuss getting an official Apache NiFi Docker image similar to
> other Apache projects like storm [1], httpd [2], thrift [3], etc.
> 
> Official Docker images are hosted at http://www.dockerhub.com and made
> available to the Docker runtime of end users without them having to build
> the images themselves. The process of making a Docker image "official",
> meaning that it is validated and reviewed by a community of Docker folks
> for security flaws, best practices, etc, works very closely to how our
> standard contribution process to NiFi works today. We as a community would
> create our Dockerfile(s) and review them just like we review any JIRA today
> and then commit that against our codebase.
> 
> There is an additional step from there in that once we have a commit
> against our codebase we would need an "ambassador" (I happily volunteer to
> handle this if there are no objections) who would open a Github Pull
> Request against the official docker image repo [4]. Once that PR has
> successfully been reviewed by the official repo folks it would be hosted on
> Dockerhub and readily available to end users.
> 
> In my mind the steps required to reach this goal would be.
> 1. Create NiFi, MiNiFi, MiNiFi-CPP JIRAs for creating the initial folder
> structure and baseline Dockerfiles in each repo. I also volunteer myself to
> take this on as well.
> 2. Once JIRA is completed, reviewed, and community thumbs up is given I
> will request the Dockerhub repo handle of "library/apachenifi" with the
> maintainer of that repos contact email as 
> 2a). I suggest we follow the naming structure like
> "library/apachenifi:nifi-1.1.0", "library/apachenifi:minifi-0.1.0",
> "libraryapachenifi:minifi-cpp-0.1.0". This makes our official image much
> more clean than having 3 separate official images for each subproject.
> 3) I will open a PR against [4] with our community Dockerfiles
> 4) After each release I will continue to open pull requests against [4] to
> ensure the latest releases are present.
> 
> Please let me know your thoughts.
> 
> [1] - https://hub.docker.com/r/library/storm/
> [2] - https://hub.docker.com/_/httpd/
> [3] - https://hub.docker.com/_/thrift/
> [4] - https://github.com/docker-library/official-images
> 
> Thanks,
> Jeremy Dyer

Re: [VOTE] Release Apache NiFi 1.1.1 (RC1)

[Joey Frazee]

+1 (non-binding)

- Verified commit hash, checksums and GPG signature
- Checked root LICENSE and NOTICE
- Checked version in pom files
- Ran `mvn -T 2.0C clean install -Pcontrib-check`
- Tested with PutElasticsearchHttp with/without connection failure (NIFI-3194)
- Tested with ValidateCsv (NIFI-3175)
- Tested CSV to Hive data flow with Avro, ORC conversions and PutHiveQL

> On Dec 20, 2016, at 12:29 PM, James Wing  wrote:
> 
> +1 (non-binding). I ran through the release helper -- verified hashes,
> license/notice/readme files, full build, and tested the resulting binary
> without issues on JDK 1.8.0_101, Amazon Linux.
> 
> Thanks,
> 
> James
> 
> On Mon, Dec 19, 2016 at 2:35 PM, Joe Percivall <
> joeperciv...@yahoo.com.invalid> wrote:
> 
>> Hello Apache NiFi Community,
>> 
>> I am pleased to be calling this vote for the source release of Apache
>> NiFi, nifi-1.1.1.
>> 
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1097
>> 
>> The Git tag is nifi-1.1.1-RC1
>> The Git commit hash is a92f2e36ed6be695e4dc6f624f6b3a96e6d1a57c
>> * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
>> a92f2e36ed6be695e4dc6f624f6b3a96e6d1a57c
>> * https://github.com/apache/nifi/commit/a92f2e36ed6be695e4dc6f624f6b3a
>> 96e6d1a57c
>> 
>> Checksums of nifi-1.1.1-source-release.zip:
>> MD5: 74955060d8ee295d77a23607ac644a6e
>> SHA1: 82efc0dc3141d0fad0205b33539e5928da87ad17
>> SHA256: 25fab8d7abfecf4c0ccef1ed9cd5f0849c829c0741142ed4074bc8dd0781f7d0
>> 
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/jpercivall
>> 
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>> 
>> 16 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> projectId=12316020=12338797
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/
>> Release+Notes#ReleaseNotes-Version1.1.1
>> 
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build from source, and test. Then
>> please vote:
>> 
>> [ ] +1 Release this package as nifi-1.1.1
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because...
>> 
>> Thanks!

Re: NATS Apache NiFi Processor for NiFi docs page

[Joey Frazee]

Brian, I went ahead and added it to https://github.com/jfrazee/awesome-nifi, 
which, while not an official Apache project page of any sort, I maintain to 
keep track of community work like this.

-joey

> On Nov 28, 2016, at 11:37 AM, Brian Flannery  wrote:
> 
> Hi all -- great job on NiFi. The number of Processors etc speaks for itself
> and the docs are pretty detailed.
> 
> I am the Community and Ecosystem lead for NATS.io -- you may already be
> aware of it; a very lightweight, simple alternative to messaging systems
> like RabbitMQ, JMS etc.
> 
> There is a community developed NATS/NiFi library -
> https://github.com/mring33621/nats-messaging-for-nifi
> 
> Can you list this on your documentation page where you list the various
> processors? https://nifi.apache.org/docs.html
> 
> If I need to do anything like a PR just let me know, but figured I could
> just send you the GH link?
> 
> All the best,
> 
> B
> 
> *Brian Flannery* | Community and Ecosystem, NATS
> br...@apcera.com | 805.570.8410
> 
> [image: www.nats.io] 
> 
> *Cloud Native Infrastructure from Apcera: NATS.io  *www.nats.io

Re: [VOTE] Release Apache NiFi 1.1.0 (RC1)

[Joey Frazee]

-1 (non-binding)

While verifying the LICENSE and NOTICE it occurred to me that some test data 
that was included (by me sadly) in TestExtractHL7Attributes is MPL (category B) 
licensed, which while ok for binary dependencies is not permitted for source 
dependencies.

I'll PR and remove these ASAP, but I think a second RC is going to have to 
happen :(

-joey

> On Nov 25, 2016, at 3:58 PM, Tony Kurc  wrote:
> 
> It's not a blocker for me, as it is clearly a config issue versus an actual
> license issue
> 
>> On Nov 25, 2016 4:18 PM, "Koji Kawamura"  wrote:
>> 
>> Tony, Joe,
>> 
>> Sorry about the nifi-websocket-bundle Rat check issue, I should have
>> added apache-rat-plugin exclude configuration in its pom.xml like
>> other projects such as nifi-toolkit-tls does.
>> 
>> Created a JIRA for that:
>> https://issues.apache.org/jira/browse/NIFI-3103
>> 
>> I'll send a PR immediately. I hope it doesn't affect voting and
>> releasing process.
>> 
>> Thanks,
>> Koji
>> 
>>> On Sat, Nov 26, 2016 at 1:41 AM, Joe Witt  wrote:
>>> Tony
>>> 
>>> I don't believe I ram contrib-check on Windows or Linux.  I did that on
>> osx.
>>> 
>>> My win environment is win10 home.  Very recent Java 8 amd maven 3.3.9.
>>> 
>>> Thanks
>>> Joe
>>> 
 On Nov 25, 2016 11:36 AM, "Tony Kurc"  wrote:
 
 Joe Witt,
 I'm not able to build on Windows 10, I'm failing a rat check in
 nifi-websocket-services-jetty. Any clue what might be in your
>> environment
 that may make it work for you and not me? (maven 3.3.3, java 1.8.0_91)
 
 From maven build:
 [INFO] --- apache-rat-plugin:0.11:check (default) @
 nifi-websocket-services-jetty ---
 [INFO] 51 implicit excludes (use -debug for more details).
 [INFO] Exclude: nb-configuration.xml
 [INFO] Exclude: nbactions.xml
 [INFO] Exclude: DEPENDENCIES
 [INFO] Exclude: .github/PULL_REQUEST_TEMPLATE.md
 [INFO] 17 resources included (use -debug for more details)
 [INFO] Rat check: Summary of files. Unapproved: 1 unknown: 1 generated:
>> 0
 approved: 14 licence.
 
 From rat.txt
 
 *
 Summary
 ---
 Generated at: 2016-11-25T11:27:37-05:00
 Notes: 0
 Binaries: 2
 Archives: 0
 Standards: 15
 
 Apache Licensed: 14
 Generated Documents: 0
 
 JavaDocs are generated and so license header is optional
 Generated files do not required license headers
 
 1 Unknown Licenses
 
 ***
 
 Unapproved licenses:
 
 
 C:/development/nifi-1.1.0/nifi-nar-bundles/nifi-websocket-bundle/nifi-
 websocket-services-jetty/src/test/resources/certs/localhost.crt
 
 
 
 
> On Fri, Nov 25, 2016 at 10:44 AM, Joe Witt  wrote:
> 
> No problem.  Thanks
> 
> On Nov 25, 2016 10:42 AM, "Andre"  wrote:
> 
> Joe,
> 
> The non-binding was more in the sense it is not a show stopper (as I
 don't
> foresee too many people upgrading that way) but I guess I should have
 made
> it more explicit. :-)
> 
> Regarding SNAPSHOT, my bad... good news is that
> 
> nifi.build.revision=1b2b9f1
> 
> which happens to be Andy's last commit before the version change.
> 
> Testing again with the source packages...
> 
> 
> 
>> On Sat, Nov 26, 2016 at 2:25 AM, Joe Witt  wrote:
>> 
>> Andre
>> 
>> BTW as a member of the PMC your votes are binding.
>> 
>> I am not quite sure the state of your snapshot version relative to
>> the
>> release version.  Definitely worth filing a JIRA and doing further
>> evaluation.
>> 
>> Thanks
>> Joe
>> 
>> On Fri, Nov 25, 2016 at 10:13 AM, Andre 
>> wrote:
>>> Joe,
>>> 
>>> -0 (non-binding)
>>> 
>>> When testing "rolling upgrade" I noticed that as nodes restarted
>> they
>> were
>>> given new Node Ids (I suspect the NodeId is related to the version
 they
>>> run?). This results on a cluster with 50% of nodes showing up as
>>> disconnected.
>>> 
>>> 
>>> Not sure if this is particular to my test environment but would be
> great
>> if
>>> someone try to reproduce:
>>> 
>>> 
>>> 
>>> On a working 3 node secure cluster with embedded zookeeper and
>> "easy
>>> upgrades directory structure":
>>> 
>>> - Untar nifi snapshot
>>> - mv the output to under /path/to/nifi/
>>> 
>>> you should now have
>>> 
>>> /path/to/nifi/config
>>> /path/to/nifi/nifi-1.0.0
>>> /path/to/nifi/nifi-1.1.0-SNAPSHOT
>>> 
>>> - ensure permissions of nifi-1.1.0-SNAPSHOT are correct (i.e.
>> chown,
>> chmod,
>>> etc)
>>> - cd nifi-1.1.0-SNAPSHOT
>>> - 

Re: NiFi Spark Processor

[Joey Frazee]

Shanka, 

It's hard to tell without more details, which you can probably find in the 
logs, but the pom.xml in that project has not been updated for any of the 
recent (< 1y) releases of NiFi (0.6, 0.7, 1.0), so you're probably going to 
have to change the version properties and, if using NiFi 1.0, change 
ProcessorLog to ComponentLog in the actual processor implementation.

If you want to check the logs for relevant messages you can find these in 
./logs/nifi-app.log and ./logs/nifi-bootstrap.log.

I'll add that, from a stylistic point of view, you might be better off looking 
into one the Spark job servers (spark-jobserver, livy) and using NiFi's HTTP 
processors to submit jobs that way. It should be more manageable and then you 
won't need this custom processor.

-joey

> On Nov 16, 2016, at 2:16 AM, shankhamajumdar  
> wrote:
> 
> Hi,
> 
> I am trying to use custom Spark Nifi processor. I have downloaded the
> processor from https://github.com/qiansl127/nifi-spark-bundle and then
> executed the maven build. After the maven build I have copied
> nifi-spark-nar-0.3.0.nar inside NiFi lib folder and restarted the NiFi
> server which is not working. 
> 
> Please advice.
> 
> Regards,
> Shankha
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Spark-Processor-tp13899.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: NiFi processor to fetch attribute value dynamically from file or table

[Joey Frazee]

There's definitely more than one way to do this, so I'll throw out some of the 
approaches I've taken:

- Wrap the config store in a web service and use InvokeHttp with Put Response 
Body in Attribute set to true, followed by an ExecuteScript processor that 
evaluates the web service response and merges the attributes onto the FlowFile.

- Use ExecuteScript to read a file, e.g., YAML, and merge the values into 
attributes.

- Use multiple ScanAttributes with routing rules. ScanAttribute lets you setup 
a reloadable dictionary, so you can treat each dictionary as the result of the 
lookup key. This will only make since if there's a small number of values.

- Use FetchDistributedMapCache with Put Cache Value in Attribute set to true.

- Put the config store in a database and use ExtractText and ReplaceText to 
first move the content to an attribute, get the config, put the config in an 
attribute and return the original contents to the FlowFile.

- Use the variable registry. One caveat however: for now, the variable registry 
cannot be updated dynamically so you'd have to restart NiFi if you need to 
change something.

- Last, I've been doing some work on a LookupAttribute processor [1] that uses 
controller services to abstract what the config store is. It either fetches all 
keys or specific keys by name if you use dynamic properties. Right now it has a 
reloadable properties file implementation and I've started on a JSON REST 
interface but haven't pushed the code yet. As the variable registry progresses, 
however, this could very likely become obsolete.
-joey

1. https://github.com/jfrazee/nifi-lookup-service/tree/file-based-lookup-service

On Nov 15, 2016, at 12:44 PM, Mothilal M  wrote:

Hey Hi,

I want NiFi processor to fetch attribute value on run time. Example - if I
am filtering twitter feeds by specific keywords, i want to maintain the
list of keywords in a separate repository like file or table and not
confined as a text box value. In that case, how will NiFi processor fetch
that values from external file or table to attribute value.

Warm Regards,
M. Mothilal

Re: [VOTE] Release Apache NiFi 1.0.0 (RC1)

[Joey Frazee]

+1 (non-binding)

- Verified checksums, signature, and commit hash
- Ran build on OS X with all tests and contrib-check
- Tested recent ExtractHL7Attributes (NIFI-2564) and TransformXml (NIFI-2142) 
improvements, and template import/export on an unsecured cluster with external 
ZK
On Aug 26, 2016, at 11:29 AM, Joe Percivall  
wrote:

Hello Apache NiFi Community,

I am pleased to be calling this vote for the source release of Apache NiFi,
nifi-1.0.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1090/

Note: There is a second binary being distributed now, the NiFi Toolkit. It
can be used to facilitate securing a NiFi instance.

The Git tag is nifi-1.0.0-RC1
The Git commit hash is 74d5224783dfdc513f6b3ad5ed96671d3c581707
* 
https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=74d5224783dfdc513f6b3ad5ed96671d3c581707
* https://github.com/apache/nifi/commit/74d5224783dfdc513f6b3ad5ed96671d3c581707

Checksums of nifi-1.0.0-source-release.zip:
MD5: 8bdba49a73b94d036fad6c63b0ebe39d
SHA1: 504c58f9b2fb305c41598a17f5b78f68f2b2fa3d
SHA256: 22167ede5127683ca8de6dbd2fb9112cb1de650b7cfff7e640c905521447af92

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/jpercivall

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

595 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12332640

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.0.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build from source, and test. Then
please vote:

[ ] +1 Release this package as nifi-1.0.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Thanks!

Convention for handling trailing spaces in property values

[Joey Frazee]

Hey guys, I was recently testing a PR and was seeing inconsistent FileNotFound 
exceptions using a path specified in a processor property. It ended up being 
because one of the flows had a trailing space in the value and the other didn't.

I'm curious about whether there's any thoughts on what the convention should be 
here? I.e., should the processor trim the property value when pulling it out of 
the context?

This is a trivial matter in some sense, but (ignoring whether it's good) it is 
possible to create filesystem objects with trailing spaces, so it could be the 
data flow developer's real intention. On the other hand, it's a bad user 
experience. Having to just remember to trim the property kinda sucks too 
because then every processor implementation will need to do it and that'll 
probably be inconsistent. I started thinking maybe it'd be worth adding an 
asPath() to StandardPropertyValue which could enforce an opinion on this.

Thoughts?

-joey

Re: Plan to add community links about nifi to website - lazy consensus

[Joey Frazee]

I think both are important.

The bar should potentially be higher for including direct links to individual 
processors or content on the wiki since it could be viewed as a tacit 
endorsement of those specific items by the project. A link to the github 
resource on the other hand is more an endorsement of the overall thing, which 
is a lesser commitment I think.

I have a very obvious bias on this point, but I don't think the Apache project 
wiki pages always have a good level of community involvement and often aren't 
in view. There's a network effect to these sorts of github pages (excuse the 
silly naming by the way, the community set that standard) which makes them more 
accessible.
-joey

On Apr 26, 2016, at 10:47 AM, Joe Witt <joe.w...@gmail.com> wrote:

It was so clear in my head - so wrong once written.

I was thinking just a straight link to Joey's site but perhaps the
wiki page with references to sites like Joey's makes sense. What do
you think?

Thanks
Joe

On Tue, Apr 26, 2016 at 11:45 AM, Tony Kurc <trk...@gmail.com> wrote:
I little confused about your proposal for "Externally Maintained
References" in the menu - will it be a link to a page on the wiki? Or would
it be another page CM'ed along with rest of the website?




On Tue, Apr 26, 2016 at 11:39 AM, Joe Witt <joe.w...@gmail.com> wrote:

Team,

Bryan Bende gave a great talk on Apache NiFi at the Hadoop Summit in
Ireland recently that I'd like to add to the apache nifi website
videos section.
https://www.youtube.com/watch?v=V77M-8ABrdE

Joey Frazee is helping maintain a great github site for general
ideas/approaches to using and extending NiFi. Seems a bit like a cool
incubator concept for NiFi ideas.
https://github.com/jfrazee/awesome-nifi

I'd like to add a pointer on our apache nifi website under the
documentation dropdown to point at 'Externally Maintained References'
and it would like to Joey's to the 'awesome-nifi' page.

I'll wait a few days (72 hours) and unless there is objection will
assume lazy consensus.

Thanks
Joe

Re: [DRAFT][REPORT] Apache NiFi - April 2016

[Joey Frazee]

Joe,

There was a meetup talk in Seattle too. Trifling addition compared to 
everything you have below but hopefully a good addition nonetheless.

> On Apr 7, 2016, at 2:13 PM, Joe Witt  wrote:
> 
> Aldrin input: Fixed date.
> 
> Tony input:
> - Yes I think the increasing committer ranks suggests a strong PMC
> pipeline.  I can add that as a clarifying point.
> - For the language change i'm good with it.  New draft below.
> 
> +
> Report from the Apache NiFi committee [Joe Witt]
> 
> ## Description:
> - Apache NiFi is an easy to use, powerful, and reliable system to process
> and distribute data.
> 
> ## Issues:
> - There are no issues requiring board attention at this time.
> 
> ## Activity:
> - Released NiFi 0.5 with considerable new core features and extensions.
> - Released NiFi 0.6 adding features to set stage for 1.0 release.
> - Key features and milestones for 1.0 release have been put together by the
> community in a public roadmap and are in active development. Our default
> version control branch (master) is now dedicated to 1.0 development.
> - The community has put together a plan and guidelines for
> contributors to develop features for both 1.0 and 0.X lines concurrently
> - Meetups featuring NiFi talks have occurred in the reporting period in
> Santa Clara, Chicago, WashingtonDC, New York City, Baltimore,
> Toronto, Dublin, Denmark, Montreal
> - Established child project known as 'MiNiFi' to provide an
> alternative command and control and implementation model tailored
> toward the first mile challenges of data acquisition. JIRA and Git
> infrastructure setup and contributions underway.
> 
> ## Health report:
> - Activity on mailing lists, JIRA, Git continues trending well.
> - During the last review cycle several new committers have joined.
> - Based on growth of active committers the pipeline for
> PMC progression is strong. We need to better formulate how to progress
> to PMC.
> 
> ## PMC changes:
> 
> - Currently 13 PMC members.
> - No new PMC members in the last three months.
> - Last PMC addition was Sean Busbey on Wed Nov 25 2015
> 
> ## Committer base changes:
> 
> - Currently 21 committers.
> - New commmitters:
>- Andy LoPresto was added as a committer on Sat Apr 02 2016
>- Oleg Zhurakousky was added as a committer on Wed Mar 16 2016
>- Matt Burgess was added as a committer on Fri Mar 04 2016
>- Joe Skora was added as a committer on Mon Feb 15 2016
> 
> ## Releases:
> 
> - nifi-0.6.0 was released on Fri Mar 26 2016
> - nifi-0.5.1 was released on Thu Feb 25 2016
> - nifi-0.5.0 was released on Mon Feb 15 2016
> 
> ## Mailing list activity:
> 
> - us...@nifi.apache.org:
>- 262 subscribers (up 56 in the last 3 months):
>- 832 emails sent to list (709 in previous quarter)
> 
> - dev@nifi.apache.org:
>- 231 subscribers (up 23 in the last 3 months):
>- 2610 emails sent to list (1389 in previous quarter)
> 
> ## JIRA activity:
> 
> - 388 JIRA tickets created in the last 3 months
> - 237 JIRA tickets closed/resolved in the last 3 months
> 
> 
>> On Thu, Apr 7, 2016 at 2:32 PM, Tony Kurc  wrote:
>> Joe - hopefully constructive comments:
>> To me, based on the lack of new PMC members since November could be
>> considered unsubstantiated - are you basing this on number of new
>> committers?
>> - Pipeline for PMC progression is very strong.
>> 
>> As an aside I will take as an action to put together a DISCUSS thread on
>> the dev mailing list expectations from the PMC on what we're looking for in
>> new committer and PMC candidates.
>> 
>> On this one, are you up for a language change?
>> - Substantial progress made toward the 1.0 release line which is now being
>> built on master.
>> 
>> If so, I'd recommend to the extent of:
>> "Key features and milestones for 1.0 release have been put together by the
>> community in a public roadmap and are in active development. Our default
>> version control branch (master) is now dedicated to 1.0 development."
>> 
>> And maybe "The community has put together a plan and guidelines for
>> contributors to develop features for both 1.0 and 0.X lines concurrently"
>> 
>> 
>> 
>>> On Wed, Apr 6, 2016 at 11:37 PM, Matt Burgess  wrote:
>>> 
>>> +1, impressed with community pace, productivity, engagement, and
>>> enthusiasm, the report/numbers definitely reflect this.
>>> 
 On Apr 6, 2016, at 11:20 PM, Aldrin Piri  wrote:
 
 Joe, looks great overall.  Minor point is that the vote for 0.6.0 wrapped
 up Saturday, 26 March.
 
 Awesome growth for the community.  Certainly felt like that was the case,
 nice to see actual numbers.
 
> On Wed, Apr 6, 2016 at 10:55 PM, Joe Witt  wrote:
> 
> NiFi Community,
> 
> Please review the included April 2016 report for the Apache board.
> If anything seems inaccurate or missing please advise.
> 
> Thanks
> Joe
> 
> --