Re: Queries regarding Nifi custom processor for Azure Data Explorer

Hi Tanmaya, There’s been some community interest in ADX processors in the past. I’m curious, is this something your team is interested in contributing back to Apache NiFi, or do you plan to release it as a Microsoft open source project? Either is fine of course. One thing to consider is that

Re: [VOTE] Release Apache NiFi 1.17.0 (RC2)

+1 (binding) - Verified signatures and checksums - Ran full build and contrib check on Linux w/ Java 11 - Ran some simple flows and verified fix for NIFI-10219 Thanks all! -joey > On Jul 30, 2022, at 4:28 AM, Ferenc Erdei wrote: > > +1 (non-binding) > > - Went through the helper guide > -

Re: [VOTE] Release Apache NiFi 1.16.0 (rc3)

+1 (binding) - Verified checksums and signatures - Did full build in Java 11 on Fedora - Ran build with rpm profile and tested rpm install - Ran some basic flows > On Mar 23, 2022, at 2:40 PM, Andrew Lim wrote: > > +1 (binding) > > -Ran full clean install on OS X (Catalina 10.15.7, OpenJDK

Re: [VOTE] Release Apache NiFi 1.15.2

+1 (binding) - Verified signatures and checksums - Ran full build on Linux and Java 11 - Ran some simple flows Thanks to everyone who turned around these comforting changes so quickly. -joey > On Dec 22, 2021, at 6:46 AM, Tony Kurc wrote: > > +1 (binding) > > Verified signatures and

Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)

+1 (binding) - Verified signatures and checksums (except for prior comment on minifi-c2) - Ran full build on Linux with Java 1.8.0_292 - Ran various flows and ITs - Built and ran dockermaven Docker build -joey > On Nov 5, 2021, at 1:50 PM, Andrew Lim wrote: > > +1 (binding) > > -Ran full

Re: [VOTE] Release Apache NiFi 1.15.0 (rc3)

JFYI, the .sha256 files for minifi-c2 are wrong/actually sha512. One interpretation of the release policy is that those aren’t being voted on and just for verification, so provided it’s fixed before the release gets checked in it might be ok. -joey > On Nov 5, 2021, at 8:29 AM, Pierre Villard

Re: Flood of JIRAs and presumably PRs to follow for junit-5 migration?

Maybe this is an exception to the single squashed commit guidance for the initial pull? I assume the intent is to make incremental progress and not have a PR with a hundred files affected, but if the different module changes corresponded to a different commit, GH will make it easy enough to

Re: [DISCUSS] NiFi 2.0 Release Goals

I quite like this idea. It seemed like the first 2.0 release had been being held out for some bigger innovations (nar registry?), but that has also pushed out making nice breaking changes. What would be the mechanics here? Just starting feeding all the candidates into JIRA? Listing out

Re: [DISCUSS] NiFi Registry -> NiFi consensus

+1 -joey > On Jul 16, 2021, at 11:54 AM, Bryan Bende wrote: > > +1 > >> On Fri, Jul 16, 2021 at 2:31 PM Kevin Doran wrote: >> >> +1. >> On Jul 16, 2021, at 2:28 PM, Joe Witt wrote: >>> >>> Yep definitely. Thought this was sorted via the JIRA. >>> >>> +1 >>> >>> On Fri, Jul 16,

Re: [VOTE] Release Apache NiFi 1.14.0 (rc2)

+1 (binding) - Verified signatures and checksums - Completed build with contrib-check on OpenJDK with 1.8.0_292 and 11.0.11+9 on Arch Linux - Ran flows standalone and in cluster with Java 11 Docker images built from source - Ran the registry - Ran Azure storage ITs - Tested new decommission

Re: Dropping support for Java 8

Kevin, it’s recently possible to specify the underlying openjdk image tag as a property in the Maven build, e.g., -Pdocker -Ddocker.image.tag=11-jre so it should be easier to start publishing those now too if it’s decided it’s a good idea. The default remains 8 for the sorts of concerns being

Re: [ANNOUNCE] New NiFi PMC Member Joey Frazee

:57 AM, Otto Fowler wrote: > > Congratulations! > >> On Thu, Mar 25, 2021 at 2:54 PM Joe Witt wrote: >> >> NiFi Community, >> >> On behalf of the Apache NiFi PMC, I am pleased to announce that Joey Frazee >> has accepted the PMC's invitation t

Re: [VOTE] Release Apache NiFi 1.13.2

+1 (non-binding) - Verified checksums and signatures and ran full build/contrib check on Java 11 and 1.8 on Linux and macOS - Validated regression and fix for NIFI-8337 using flow similar to the one in JIRA - Checked docs update in NIFI-8324 - Did a quick check of license and notice -joey >

Re: [VOTE] Release Apache NiFi 1.13.1

+1 (non-binding) - Verified checksums and signatures - Ran full build on Java 1.8 and 11 on Linux - Verified NIFI-3383, NIFI-8231, and NIFI-8200 -joey > On Mar 12, 2021, at 10:28 AM, Bryan Bende wrote: > > +1 (binding) > > - Verified everything in the standard release helper > - Setup

Re: [VOTE] Release Apache NiFi 1.13.0 (rc4)

+1 (non-binding) - Verified checksums, signatures, and commit id - Ran builds with Java 1.8 and 11 on Linux and macOS, and validated RPM build profile - Tested cluster coordination and state management with both embedded and external ZooKeepers with and without TLS - Verified fix for

Re: [discuss] we need to enable secure by default...

This is probably a very important and overdue change. I think it’s important to recognize that there’s a lot of different ways to unintentionally end up with a publicly accessible application and it can’t always be pinned to one person or role. Routing, firewall rules, OS admin, NiFi admin

Re: [VOTE] Release Apache NiFi 1.13.0 (rc3)

Hey, I found a problem in nifi.sh introduced in NIFI-8123. nifi.sh points to /bin/sh, the above changes added some uses of the declare builtin which isn’t available in all shells (ash, dash), so those commands fail creating some unexpected output at startup, and the new wait functionality

Re: [VOTE] Release Apache NiFi 1.13.0 (rc2)

+1 (non-binding) - Verified checksums, signatures, and commit id - Ran builds with Java 1.8 and 11 on Linux and macOS, and validated RPM build profile - Tested cluster coordination and state management with both embedded and external ZooKeepers with TLS enabled and disabled - Verified fix

Re: [DISCUSS] Release of Apache NiFi 1.13.0

this when it might matter. -joey > On Feb 1, 2021, at 1:33 PM, Joe Witt wrote: > > Team > > starting RC2. Joey Frazee noted "I noticed that > nifi-docker/dockermaven-stateless/pom.xml > and nifi-docker/docker-compose/docker-compose.yml in the source package and &g

Re: [VOTE] Release Apache NiFi 1.13.0

I noticed that nifi-docker/dockermaven-stateless/pom.xml and nifi-docker/docker-compose/docker-compose.yml in the source package and tag are still using 1.13.0-SNAPSHOT. Is that a problem? The former is experimental, and the latter is for convenience, so it’s not deadly but previous source

Re: [DISCUSS] Release of Apache NiFi 1.13.0

I have https://github.com/apache/nifi/pull/4632 which fixes an OOME in PutAzureBlobStorage reported in https://lists.apache.org/x/thread.html/rdef82be24828277b85bdc94dc57a8fb9df6f73552daeda289c941a51%40%3Cusers.nifi.apache.org%3E It’s a pretty small change. -joey > On Jan 5, 2021, at 3:14 PM,

Re: [VOTE] Release Apache NiFi 1.12.1 (rc2)

+1 (non-binding) Verified checksums, hashes, signatures Ran full build w/ Java 1.8 and 11 Ran build w/ -Prpm profile Created secure cluster using tls-toolkit and multiple SANs Tested fixes for NIFI-7768 and NIFI-7794 -joey On Sep 24, 2020, 1:05 PM -0700, Andrew Lim , wrote: > +1 (binding) > >

Re: [VOTE] Release Apache NiFi 1.12.1

+1 (non-binding) Verified checksums and signatures Ran full build with contrib check on Linux and Zulu OpenJDK 11.0.7 Tested TLS toolkit and secure cluster with multiple SANs -joey On Sep 9, 2020, 8:51 AM -0700, Kevin Doran , wrote: > +1 (binding) > > - Checked hashes/sigs > - L is present > -

Re: [VOTE] Release Apache NiFi 1.12.0

+1 (non-binding) Verified commit id, checksums, and signatures Successfully built and ran tests on Linux with Java 1.8.0_252 and Java 11.0.7 2020-04-14 and on macOS with Java 11.0.6 2020-01-14 LTS Ran data flows on standalone and cluster Verified JMS improvements, InvokeHTTP changes, and Azure

Re: [discuss] 1.12.0 or 1.11.5...

I need to double check whether it needs rebased but there’s a docs PR for how to enable client TLS for ZooKeeper that I think would be good to include until the more integrated work is done: https://github.com/apache/nifi/pull/4092 Would anyone have time to look? -joey On Aug 3, 2020, 11:47

Re: [nifi] branch main created (now 239a2e8)

@alopresto Do you want everyone individually pushing to both master and main or are you intending to keep it in sync until it’s made the default? -joey On Jul 2, 2020, 6:39 PM -0500, alopre...@apache.org, wrote: > This is an automated email from the ASF dual-hosted git repository. > > alopresto

Re: Reporting task can be set to RUNNING from REST API when invalid

One of those looks like it’s just a transient failure. The other on macOS was a flakey test that was fixed recently. If you want it to show green you can rebase. If you re-run the jobs I’d expect to see the Ubuntu/1.8 pass but the macOS one probably won’t until you rebase your branch. -joey

Re: [DISCUSS] rename master branch, look through code for other related issues

+1 I’m repeating this from elsewhere but I was on a team 7 years ago where a teammate asked us to stop using master and slave terminology, even master alone, because it made them uncomfortable. I can’t estimate how common that feeling is but this isn’t a theoretical exercise. As teammates and

Re: [VOTE] Release Apache NiFi 1.11.4

+1 (non-binding) Verified checksums, signatures, and commit hash Ran build and all tests on Linux and OS X Ran flows on standalone and cluster and verified some of the improvements/fixes Built RPM profile and installed and ran -joey On Mar 20, 2020, 12:23 PM -0500, Marc Parisi , wrote: > +1

Re: [VOTE] Release Apache NiFi 1.11.3

+1 (non-binding) Verified checksums, signatures, and commit hash Ran build and all tests on OpenJDK 1.8.0_242 on Linux and OS X Created clusters on VMs and docker and ran flows -joey On Feb 24, 2020, 11:33 AM -0800, Matt Gilman , wrote: > +1 binding > > Ran through release validation steps.

Re: [VOTE] Release Apache NiFi 1.11.1 (rc1)

+1 (non-binding) Verified checksums, signatures, and commit hash Ran build and tests on OpenJDK 1.8.0_242 on Linux Ran rpm build target and tested install Created small cluster and ran some flows Note: I ran into intermittent TestListFile failures on OS X (there are existing JIRAs for these)

Re: Processor for Delta Lake / Real Time Data Warehouse

ocessors to hit Delta Lake. Would like to be proved wrong here (heck, it > would make some of our use cases easier!) but AFAIK you have to have NiFi > build Parquet and push to S3 or HDFS so Spark can do the transformation. > > On Sat, Feb 1, 2020 at 4:14 PM Joey Frazee > wrote: &g

Re: Processor for Delta Lake / Real Time Data Warehouse

Martin, I’ve been thinking about this one for a while but I think it needs to be considered in the context of transactional table formats in general; i.e., the incubating Apache Hudi and Apache Iceberg too. There are things that are inconvenient for NiFi to do with these table formats. But the

Re: [VOTE] Release Apache NiFi 1.6.0 (RC3)

Like slowpoke... +1 (non-binding) - Verified checksums and signature - Successfully built and ran unit tests and flows on OSX (Oracle 1.8.0_152), Amazon Linux (Oracle 1.8.0_161) FWIW, something isn’t working with the RPM build the way it was. Don’t have anything specific to report though and

Re: ListSFTP incoming relationship

I worked on this at one point to make it "easier" (haha...) to process a very deep directory tree with 100k+ files -- idea was to break it up into subtrees for concurrency, etc. by looping the outgoing relationship back to input. It ended up being being painful. Looking at the diff again the

Re: [CANCEL][VOTE] Release Apache NiFi 1.6.0 RC1

-RC1 as per release > guide. We will push the actual tag once we get to a release point. > > Can you clarify what fingerprint issue you are referring to? Just > want to make sure this is what BryanB pointed out and not something > else. > > Thanks > Joe > > On Sun, Mar

Re: [VOTE] Release Apache NiFi 1.6.0

-1 Ran through the usual release helper stuff, but it seems like the fingerprint issue is going to cause problems, so not sure how useful putting 1.6.0 out there will be if 1.6.1 will have to be turned around immediately. Did you mean to say there's a nifi-1.6.0 -RC tag? It doesn't look like

Re: Will you accept contributions in Scala?

This probably necessitates a vote, yeah? Frankly, I’m usually happier writing Scala, and I’ve not encountered any problems using processors written in Scala, but I think it’ll be important to tread lightly. There’s a few things that pop into my head: - Maintainability and reviewability. A

Re: [DISCUSS] Apache NiFi distribution has grown too large

I tend to have feelings similar to Michael about a multi-repo approach. I’ve rarely seen it help and more often seen it hurt — it’s confusing (especially to newcomers), stuff gets neglected because it’s easier to ignore, you need another master project or some such to do an entire build. Maybe

Re: [VOTE] Release Apache NiFi 1.5.0 (RC1)

+1 - Verified checksums, signature and commit ID - Ran build w/ contrib-check with OpenJDK 1.8.0_131 on Amazon Linux 2017.03 - Tested version control and flow registry, CountText, FlattenJson and some other flows - Tested RPM build and install on Amazon Linux 2017.03 - Skimmed L - Gave it lots

Re: Enrichment plugin for adding attributes from SQL

Brett, I think it’s great that you brought this up and made some specific suggestions, because it’s easy for people to overlook and hard to know how to do the right thing without that kind of feedback. -joey On Jan 7, 2018, 5:51 AM -0600, Brett Ryan , wrote: > > Probably

Re: Enrichment plugin for adding attributes from SQL

6E F65B 2F7D EF69 > > > > > > On Jan 4, 2018, at 4:07 PM, Brett Ryan <brett.r...@gmail.com> wrote: > > > > > > Thanks Andy, how would update attribute be able to get the value from sql? > > > > > > Consider a flow where a piece of infor

Re: Enrichment plugin for adding attributes from SQL

Andy, Brett, Taking a quick glance at the code it looks like it's enriching attributes from a database according to a query. If that's correct, there's a LookupAttribute processor that delegates lookups to a "LookupService" and adds attributes without altering content. There are a variety of

Re: [VOTE] Release Apache NiFi Registry 0.1.0

+1 - Verified checksums and signatures - Successfully built and ran contrib-check w/ Oracle JDK 1.8.0_152 - Setup the registry w/ NIFI-4436 - Had some fun versioning PGs and rolling back and forth Note: I ran into some trouble rolling back flows that include new funnels. It always results in an

[DISCUSS] CI / Travis / Jenkins

I’m sure everyone has noticed that Travis CI fails, incorrectly, more than it succeeds, often due to timeouts and not b/c of the incorrectness of a commit or PR. This has been discussed previously, but it’s carried on, and become a low information signal about the PRs, which has two big

Re: nifi: How to update and trnaform xml file inside ExecuteScript processor

Sally, there are better ways to approach this: - First, the default for transforming XML should be to use TransformXml [1] with an XSLT. It kinda looks like you’re also trying to extract some elements into attributes so I’d look at EvaluateXPath [2] too. Your flow would then look like [_] -->

Re: [VOTE] Release Apache NiFi 1.4.0 (RC2)

+1 (non-binding) - Verified checksums and signature - Successfully built and ran tests on OSX (Oracle 1.8.0_131), Amazon Linux (Oracle 1.8.0_131), and Docker maven:latest (OpenJDK 1.8.0_141) - Built RPM with `mvn -T 2.0C clean install -Prpm,generateArchives -DskipTests` and tested install -

Re: [VOTE] Release Apache NiFi 1.4.0

I think there could be an issue with the deps in the nifi-mongodb-services-nar. It includes nifi-lookup-services which should either be unnecessary or should just be provided scope (just need the services API dependency). So it’s possible that all the impls in nifi-lookup-services are indeed

Re: Binaries in embedded redis jar files

In addition to only being for testing, these binaries are only used for an integration test, so we don't even need to worry about whether they're going to work on everyone's OS/cause test failures if the binaries don't work for you. Now it is packaging up the entire Redis server binary, though,

Re: FTPS support in FTPTransfer (List/Fetch/Get/PutFTP) in NiFi 1.3

Jeremy, I started on this at one point (see  https://github.com/jfrazee/nifi/commits/NIFI-2188). It’s lacking tests and there’s some changes needed to make it more explicit when you’ve misconfigured it, but it might be worth a go if you want to try building from the branch. I somehow knew this

Re: RTC clarification

Joe(s), as you mentioned, even if we have a non-committer review, it can’t be merged until a committer decides whether to accept whatever decision was provided. So, the burden is still on committers as to whether it’s really a +1 or not. And presumably this should only happen if there’s some

Re: NiFi and MapR streams?

Konstantin, this is a bit of a guess, but this might work with a NiFi that was built against MapR’s kafka-clients. So you’d need to change the kafka9.version property when building NiFi. For example: mvn clean install -DskipTests -Pmapr -Dhadoop.version=2.7.0-mapr-1703

Re: Expression Language For AMQP Processors

Sometimes it’s just an historical oversight that EL support isn’t available for certain properties, but in this case the connection is initialized only once, so the connection specific properties like hostname, port, username and password cannot be evaluated against FlowFile attributes in any

Re: MiNiFi Java as Windows Service

Jefff, there was a related thread about this a few months back: https://lists.apache.org/thread.html/05abafa804b0bb774211ef602d5fc2ec3aa8bdf5c584f2aab3014b42@%3Cdev.nifi.apache.org%3E

Re: AvroSchemaRegistry doesn't enjoy copy and paste?

Andre, if there are any line breaks in the schema and leading spaces on those new lines, then this occurs. So if you minify the avsc or remove the leading spaces, all should be good. Will open a JIRA on this since, including myself, I think you’re the third to see this. Any chance you’re using

Re: LookupAttribute + CSV Lookup handling of non-matches

Andre, there's a flag for "Include Empty Values". If set to false then the effect is that non-matches result in a transfer to unmatched. The default is true so I'd expect to see what you're seeing. The description for that property isn't good enough though. I don't know how you would have

Re: [VOTE] Release Apache NiFi 1.3.0

+1 (non-binding) - Verified checksums, signatures and commit ID - Successfully built and ran tests on OSX, Amazon Linux, and Docker maven:latest - Built RPM with `mvn -T 2.0C clean install -Prpm,generateArchives -DskipTests` and tested install - Tested data flows with ConvertRecord and

Re: [VOTE] Release Apache NiFi MiNiFI C++ 0.2.0 (RC2)

+1 (non-binding) - Verified checksums, signatures and commit id - Successfully built and ran tests with Apple LLVM version 8.1.0 (clang-802.0.42) - Successfully ran make package and used binaries from generated archive - Ran example data flow from README.md - Checked for L > On May 9, 2017, at

Re: [VOTE] Release Apache NiFi 1.2.0 (RC2)

Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On May 7, 2017, at 22:17, Joey Frazee <joey.fra...@icloud.com> wrote: >> >> +1 (non-binding) >> >> - Verified signature, checksums and commit id >> - Successfully ran `mvn clean in

Re: [VOTE] Release Apache NiFi 1.2.0 (RC2)

+1 (non-binding) - Verified signature, checksums and commit id - Successfully ran `mvn clean install -Pcontrib-check` on OS X 10.12.4 with Oracle JDK 1.8.0_121 and Amazon Linux with OpenJDK 1.8.0_121 - Successfully ran `mvn clean install -DskipTests` on Docker maven:latest with OpenJDK

Re: Commit d90cf84 seems to break build?

Andre I think that solution strikes the right balance without being a lot of effort. As mentioned Travis-ci's docs say that caching stuff like .m2 doesn't help but in my experience you usually still get a little bump so it's worth it as long as the ci isn't lying about the results. Removing

Re: SQL to CSV?

Michael, I think you’re right to call this out. I frequently find myself stringing together flows with ExecuteScripts (which you should be able to use to pull a schema out by creating an Avro DataFileStream from the InputStream and then calling getSchema().toString()) or conversions to/from

Re: [DISCUSS] Official Apache NiFi Docker Image

+1 I think this is a great idea because there are at least half a dozen or more Dockerfiles and published images floating around. Having something that is endorsed and reviewed by the project should help ensure quality. One question though: Will the images target a single instance NiFi or a

Re: [VOTE] Release Apache NiFi 1.1.1 (RC1)

+1 (non-binding) - Verified commit hash, checksums and GPG signature - Checked root LICENSE and NOTICE - Checked version in pom files - Ran `mvn -T 2.0C clean install -Pcontrib-check` - Tested with PutElasticsearchHttp with/without connection failure (NIFI-3194) - Tested with ValidateCsv

Re: NATS Apache NiFi Processor for NiFi docs page

Brian, I went ahead and added it to https://github.com/jfrazee/awesome-nifi, which, while not an official Apache project page of any sort, I maintain to keep track of community work like this. -joey > On Nov 28, 2016, at 11:37 AM, Brian Flannery wrote: > > Hi all -- great

Re: [VOTE] Release Apache NiFi 1.1.0 (RC1)

-1 (non-binding) While verifying the LICENSE and NOTICE it occurred to me that some test data that was included (by me sadly) in TestExtractHL7Attributes is MPL (category B) licensed, which while ok for binary dependencies is not permitted for source dependencies. I'll PR and remove these

Re: NiFi Spark Processor

Shanka, It's hard to tell without more details, which you can probably find in the logs, but the pom.xml in that project has not been updated for any of the recent (< 1y) releases of NiFi (0.6, 0.7, 1.0), so you're probably going to have to change the version properties and, if using NiFi

Re: NiFi processor to fetch attribute value dynamically from file or table

There's definitely more than one way to do this, so I'll throw out some of the approaches I've taken: - Wrap the config store in a web service and use InvokeHttp with Put Response Body in Attribute set to true, followed by an ExecuteScript processor that evaluates the web service response and

Re: [VOTE] Release Apache NiFi 1.0.0 (RC1)

+1 (non-binding) - Verified checksums, signature, and commit hash - Ran build on OS X with all tests and contrib-check - Tested recent ExtractHL7Attributes (NIFI-2564) and TransformXml (NIFI-2142) improvements, and template import/export on an unsecured cluster with external ZK On Aug 26,

Convention for handling trailing spaces in property values

Hey guys, I was recently testing a PR and was seeing inconsistent FileNotFound exceptions using a path specified in a processor property. It ended up being because one of the flows had a trailing space in the value and the other didn't. I'm curious about whether there's any thoughts on what

Re: Plan to add community links about nifi to website - lazy consensus

ebsite videos section. https://www.youtube.com/watch?v=V77M-8ABrdE Joey Frazee is helping maintain a great github site for general ideas/approaches to using and extending NiFi. Seems a bit like a cool incubator concept for NiFi ideas. https://github.com/jfrazee/awesome-nifi I'd like to add a pointer on o

Re: [DRAFT][REPORT] Apache NiFi - April 2016

Joe, There was a meetup talk in Seattle too. Trifling addition compared to everything you have below but hopefully a good addition nonetheless. > On Apr 7, 2016, at 2:13 PM, Joe Witt wrote: > > Aldrin input: Fixed date. > > Tony input: > - Yes I think the increasing