Re: UpdateAttribute Failure Relationship

t; >>>> > >>>> On Wed, Feb 7, 2024 at 8:11 PM Phillip Lord > >>>> wrote: > >>>> > >>>>> IMO... UpdateAttribute has been around since the beginning of time, I > >>>> can't > >>>>> see adding a fa

Re: UpdateAttribute Failure Relationship

Or better, the failure relationship just doesn't even exist until the property "Has Failure Relationship" is set to True. This involves updating UpdateAttribute to have dynamic relationships (the failure relationships appearing on true), which isn't hard to do in processor code. This has the

Re: [discuss] Time for a NiFi 2.0 M1 release?

I'm also hoping that both 1.x and 2.x lines can receive the PackageFlowFile processor that Mike Moser recently proposed. That way, the M1 release and the most recent 1.x release will have a simple (or logical) replacement for PostHTTP. In general, it would be nice to have 1.x lined up with 2.0-M1

Re: new PackageFlowFile processor

llows. > > > > For now, something focused narrowly on FlowFile Version 3 encoding > > seems like the best approach. > > > > I recommend referencing this discussion in a new Jira issue and > > outlining the general design goals. > > > > Regards, > > D

Re: new PackageFlowFile processor

transition while PostHTTP is still available on their canvas. Wishful thinking that we can make the entire journey from 1.x to 2.x as smooth as possible, but this could potentially help some. On Fri, Sep 8, 2023 at 10:55 AM Adam Taft wrote: > +1 on this as well. It's something I've kind of

Re: new PackageFlowFile processor

+1 on this as well. It's something I've kind of griped about before (with the loss of PostHTTP). I don't think it would be horrible (as per Joe's concern) to offer a N:1 "bundling" property. It would just have to be stupid simple. No "groups", timeouts, correlation attributes, minimum entries,

Re: [discuss] nifi 2.0 and Java 21…

Yes, please. +1 Exactly what Mark said. Virtual threads have potential to be extremely impactful to applications like NiFi. /Adam On Wed, Sep 6, 2023 at 7:26 AM Mark Payne wrote: > Thanks for bringing his up, Joe. > > I would definitely be a +1. I think the new virtual thread concept would >

Re: ValidateXml Processor - validatexml.invalid.error = Validation failed (NiFi 1.19.1)

o take this. > > On Thu, Feb 9, 2023 at 2:04 AM Bilal Bektaş .invalid> > wrote: > > > Hi Adam and Dan, > > > > Thank you for your quick response. > > > > Ticket number: NIFI-11156 > > > > Best wishes, > > > > --Bilal > > &

Re: PostHTTP Deprecation Concerns

on under its name "nifi-flow-over-tcp" both on > GitHub and on Maven Central. > githubDOTcom/EndzeitBegins/nifi-flow-over-tcp > > > Maybe this can be helpful to you as well in the aforementioned cases you > previously made use of the PostHTTP processor. > > > Best regards

Re: ValidateXml Processor - validatexml.invalid.error = Validation failed (NiFi 1.19.1)

Bilal, And I will be more than happy to help fix this. This is plaguing myself as well. Please let me know what your ticket number is, and I will help in whatever way is needed to get this fixed. /Adam On Wed, Feb 8, 2023 at 10:13 AM Dan S wrote: > Bilal, > Please create a bug ticket for

Re: PostHTTP Deprecation Concerns

the response. /Adam On Wed, Jan 11, 2023 at 9:27 PM Adam Taft wrote: > Hi Mathew, > > > It's quite remarkable you're advocating against standard practice > presumably > > for your own convenience. > > Wow, absolutely not stated nor implied in my message. And even b

Re: PostHTTP Deprecation Concerns

le on your production flow and metrics > analysis, just make sure your provenance db has sufficient space. > > Kind regards, > > On Thu, Jan 12, 2023, 10:09 Adam Taft wrote: > > > Just wanted to note a concern on the deprecation (and presumed removal) > of > > the

PostHTTP Deprecation Concerns

Just wanted to note a concern on the deprecation (and presumed removal) of the PostHTTP processor in the upcoming 2.0 release. While yes, for traditional client interactions with an external HTTP service, utilizing InvokeHTTP for your POST operation is probably sensible. The concern is that there

Re: [discuss] NiFi 1.20 and NiFi 2.0

This is really insightful and spot on ... Kevin wrote: > Good migration tooling will take a while to develop and test, and the core > contributors to that effort may not have sufficient variety of flows to > evaluate when the migration tools are "done" for the majority of the > community to have

Re: [discuss] NiFi 1.20 and NiFi 2.0

> [1] > https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals > > > > On Mon, Jan 9, 2023 at 6:08 PM Adam Taft wrote: > > > I think this sentence is capturing some of my question ... > > > > David wrote: > > > I think i

Re: [discuss] NiFi 1.20 and NiFi 2.0

mental fix releases > for the initial 2.0.0 release train, but I do not expect delaying a 2.0.0 > release for new features, as that is not part of the release goals. > > I think it would be helpful to see some traction on the 2.0 release goals > before attempting to sketch out a potential timeline.

Re: [discuss] NiFi 1.20 and NiFi 2.0

Joe / team, Question on this. I think it would be helpful to understand the desired timelines for the first 2.0.0 release. I know it's not strictly predictable, but having a sense of what the timing looks like is important to help understand the implications of a "maintenance only" 1.x line. The

Re: Cobol/EBCDIC Source

Hi Frank, NiFi does not currently have native support for Cobol formats, such as copybook or EBCDIC. However, NiFi is built to be very extensible/pluggable, and as such, support can be added for any type of data conversion that you can imagine. For example, a new conversion processor from EBCDIC

Re: setup TLS configuration

Elvis, I found this document which might help give you clues to convert between IBM MQ's "kdb" format and the traditional Java "jks" format. In principle, it looks like you will need to export your client certificates, etc. out from your kdb store:

Re: proposal to extend several component key properties to use Expression Lng instead of VarRegs only (based off https://issues.apache.org/jira/browse/NIFI-8214)

Definitely appreciate having as much "control" as possible afforded to each flowfile. The use cases described here are spot on and I've hit this myself previously. Any endpoint definition would ideally be configurable from the flowfile itself via expression language. It's easy enough to hard code

Re: Using nifi in separated networks

architecture because one thread > cannot receive, decode, and write a gigabit per second. I used the > disruptor library. Receive a packet in one thread, decode it in another > thread. A third thread gets the packet and write the content in the right > order to a flow. > > > >

Re: Using nifi in separated networks

Marc, How would this differ from a more generic use of the existing processors, PutTCP/ListentTCP and PutUDP/ListenUDP? I'm not sure what value is being added above these existing processors, but I'm sure I'm missing something. There's already an ability to serialize flowfiles via MergeContent.

Re: [DISCUSS] NiFi 2.0 Release Goals

ot > make it a pain for users. We got away with things at 0.x to 1.0 that we > cannot get away with on 2.0 > > Thanks > > On Fri, Jul 30, 2021 at 3:41 PM Adam Taft wrote: > > > I'm not seeing the side thread that was going to discuss deprecation of > > PostHTTP. Ha

Re: [DISCUSS] NiFi 2.0 Release Goals

I'm not seeing the side thread that was going to discuss deprecation of PostHTTP. Has that thread started and I just don't see it? One (significant?) concern with PostHTTP is the smooth integration of NiFi-to-NiFi communication that is very transparently enabled with the ListenHTTP and PostHTTP

Re: invokeHttp routing of exceptions like ConnectException and IOException to failure instead of retry

Hi David, *> "What is the reasoning for routing them to failure instead of retry?"* Good question ... HTTP status codes give good hints as to what a client should do for retry/no-retry operations. Generally 400 error codes do not get retried, 500 codes get retried, etc. It doesn't, however,

Re: Java 11 Compilation

latest stable Java at that > time (11, 13) > > thanks > > On Wed, Oct 30, 2019 at 12:51 PM Adam Taft wrote: > > > While building 1.10.0-rc3, I wanted to experiment with the compilation > and > > runtime variants using Java 8 and Java 11. The summary of this > experiment &g

Java 11 Compilation

While building 1.10.0-rc3, I wanted to experiment with the compilation and runtime variants using Java 8 and Java 11. The summary of this experiment was: Comp: Java 8 Run: Java 8 => SUCCESS Comp: Java 8 Run: Java 11 => SUCCESS Comp: Java 11 Run: Java 8 => FAILURE Comp: Java 11 Run:

Re: [VOTE] Release Apache NiFi 1.10.0 (rc3)

+1 (binding) Signatures verified. Hashes verified. Tests pass, source builds cleanly. I used both Java 11 & Java 8 to build. I did run into a problem compiling with Java 11 and running with Java 8. I don't believe this was a goal of the Java 11 compatibility changes, so nothing unexpected about

Re: PULL ProvenanceEvent

; Nissim > > On Friday, October 11, 2019, 11:30:19 AM EDT, Nissim Shiman > wrote: > > Adam, > "Yes" to your first question and the four processor examples you listed. > > I will need to get back to you regarding your other points. > > Thanks, > Nissim > &

Re: Release vote helper docker setups

In general I like this idea. I'd like to even suggest a possibly broader vision that aims towards a more stable build environment that would be the "reference" environment for building NiFi. I have been kicking around and looking at a Docker based build environment for NiFi. The idea is that you

Re: PULL ProvenanceEvent

Nissim, Just to be clear, you are trying to distinguish between processors which are actively "pulling" data (GetXYZ) vs. processors which just "listen" for data (ListenXYZ)? Is that your basic vision? GetFile => PULL GetHTTP => PULL ListenHTTP => RECEIVE ListenTCP => RECEIVE Could you clarify

Re: Maven Build Error - nifi-properties-loader sub-project test failures

ion to *1.8.0_222* and Maven to *3.6.2* > did indeed *fix my build issue*! > > Aram S. Openden > aram.open...@gmail.com > > > > On Thu, Oct 10, 2019 at 1:10 AM Adam Taft wrote: > > > Aram, > > > > Just to rule out the obvious ... Can you update your Maven

Re: Maven Build Error - nifi-properties-loader sub-project test failures

ere: > http://nifi.apache.org/quickstart.html > > It seems like it is reading material from files (test files) and they don't > contain what is expected so I wonder about git settings. > > Thanks > Joe > > On Thu, Oct 10, 2019 at 1:10 AM Adam Taft wrote: > > > Ar

Re: Maven Build Error - nifi-properties-loader sub-project test failures

Aram, Just to rule out the obvious ... Can you update your Maven and Java versions, which would include: - Maven 3.6.2 - Java 1.8.0_222 Also, are you including a MAVEN_OPTS environment to increase your JVM memory in Maven? $> export MAVEN_OPTS="-Xms1g -Xmx3g" Thanks, Adam On Wed, Oct 9, 2019

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

influences behavior in > all three modules and requires one or more reviewers with comprehensive > knowledge over all aspects of the project. > > > Andy LoPresto > alopre...@apache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

> problem completely (in fact, to your point, it may uncover some > > unfortunate tight-coupling that needs to be reworked on the current > > master before the split can happen), but I do think it will encourage > > developers to more faithfully build to APIs and a

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

nce binaries resulting from a single source release > artifact though. > > Thanks > > On Fri, Jul 12, 2019 at 12:26 PM Adam Taft wrote: > > > I think the concerns around user management are valid, are they not? > > Overhead in JIRA goes up (assigning rights to us

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

hat. > > in any event im not making a statement of whether to do many repos or not. > just correcting some potentially misleading claims. > > thanks > > On Fri, Jul 12, 2019, 12:01 PM Adam Taft wrote: > > > Just as a point of discussion, I'm not entirely sure that sp

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Just as a point of discussion, I'm not entirely sure that splitting into multiple physical git repositories is actually adding any value. I think it's worth consideration that all the (good) changes being proposed are done under a single mono-repository model. If we split into multiple

Re: [DISCUSS] Apache NiFi distribution has grown too large

I'd also vote for an OSGi backend (in the long term). It's something that has been on my mind (and mentioned) for years now. The Nar classloader ecosystem is trying to implement features of OSGi (and doing it somewhat poorly at that, if you are honest). Not saying that OSGi is the right

Re: About NIFI-3620: Multipart support in invokeHTTP.java

Multipart is just a set of related content types (multipart/form-data, multipart/mixed). InvokeHTTP doesn't care too much about content types, it just sends bytes verbatim from the flowfile payload. What should be considered is for an upstream processor to create the multipart payload in the

Re: Dockerfile and Docker Hub Management

Aldrin, +1 to separate repository (bullet #2). The basic premise that Docker releases should happen separate from the main distribution is spot on. I think a separate repository would help keep this separation. I tend to believe that the future of NiFi distributions will be via

Re: [EXT] Re: [DISCUSS} Closing in on a NiFi 1.4.0 release?

Here's another good link to try, maybe a little easier to read: https://issues.apache.org/jira/projects/NIFI/versions/12340589 On Wed, Sep 20, 2017 at 8:01 AM, Brandon DeVries wrote: > Mayank, > > Try this: > > https://issues.apache.org/jira/issues/?jql=project%20% >

Re: Does PostHTTP support Multipart/form-data ?

The multipart/form-data body would have to be preemptively created and stored in your flowfile payload. InvokeHTTP could then be used to POST the message body to the remote server (after having set the appropriate content-type). i.e. you have to manually construct the multipart form in the

Re: How to ingest files into HDFS via Apache NiFi from non-hadoop environment

This is a bit outside of the box, but I have actually implemented this solution previously. My scenario was very similar. NIFI was installed outside of the firewalled HDFS cluster. The only external access to the HDFS cluster was through SSH. Therefore, my solution was to use SSH to call a

Re: [VOTE] Release Apache NiFi 1.3.0

uot;:83 ? NullPointer StandardFlowSynchronizerSpec.scaling of #filename with encoding version "#flowEncodingVersion":83 ? NullPointer StandardFlowSynchronizerSpec.scaling of #filename with encoding version "#flowEncodingVersion":83 ? NullPointer StandardFlowSynchronizerSpec.scaling

Re: [VOTE] Release Apache NiFi 0.7.4

+1 (binding) Verified gpg signature. Verified all hashes on source zipfile. Performed mvn clean install -Pcontrib-checks Builds cleanly, all tests pass in docker container centos:latest w/ openjdk-1.8.0 and maven 3.5.0 LICENSE, NOTICE, README look good. Binary runs as expected with a simple

Re: [VOTE] Release Apache NiFi 1.3.0

lazy like me and just using a disposable Docker container). > > NIFI-3836 is open for it. > > If this is what it is, just build as a non root user. > > > On Jun 5, 2017, at 5:25 PM, Adam Taft <a...@adamtaft.com> wrote: > > > > I'm getting a t

Re: [VOTE] Release Apache NiFi 1.3.0

I'm getting a test failure for this RC. Here is the maven snippet. --- T E S T S --- Running org.apache.nifi.provenance.CryptoUtilsTest Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time

Re: [GitHub] nifi issue #272: NIFI-1620 Allow empty Content-Type in InvokeHTTP processor

I added a comment to the JIRA ticket associated with this pull request. I think there should be discussion / buy-in from others on the aestetics of introducing a new processor property for this edge case. Instead, I think the goals of this request could be fulfilled without strictly introducing

Re: Dynamic URLs using InvokeHttp from an array

You are probably missing the necessary change to the following file: META-INF/services/org.apache.nifi.processor.Processor ​If you haven't modified this file to include your processor, this would be the problem. Adam ​ On Fri, Apr 1, 2016 at 2:53 PM, kkang wrote: > I

Re: Dynamic URLs using InvokeHttp from an array

Yeah, these solutions won't work for thousands of iterations. Andy's suggestion for using ExecuteScript starts to sound very compelling, especially if you are algorithmically generating your term values. Another thought for you. Uwe Geercken was experimenting with a processor which could read

Re: Dynamic URLs using InvokeHttp from an array

you'd end up with 10 flowfiles that could be sent to InvokeHTTP. That might be a fun way to solve this. :) On Thu, Mar 31, 2016 at 9:55 AM, Adam Taft <a...@adamtaft.com> wrote: > One (possibly bad) idea would be to try and loop your flow around the > UpdateAttribute processor using RouteOnAt

Re: Dynamic URLs using InvokeHttp from an array

One (possibly bad) idea would be to try and loop your flow around the UpdateAttribute processor using RouteOnAttribute. UpdateAttribute has an "advanced" mode which would let you do logic something like: if $foo == "" then set $foo = "step 1"; if $foo == "step 1" then set $foo = "step 2"; if

Re: Re: Processor: User friendly vs system friendly design

elocity, FreeMarker, and others might be really nice. > > Extra bonus points for Markdown or Asciidoc transforms as well (but these > > might be too separate of a use case). > > > > Hope this helps. > > > > Adam > > > > [1] http://nifi.apache.org/de

Re: Processor: User friendly vs system friendly design

I'm probably on the far end of favoring composibility and processor reuse. In this case, I would even go one step further and suggest that you're talking about three separate operations: 1. Split a multi-line CSV input file into individual single line flowfiles. 2. Read columns from a single

Re: InvokeHTTP body

I think it makes total sense that POST/PUT requests read from the flowfile content. Therefore, the problem should be fixed further up in the flow design. For example, try these solutions: GenerateFlowFile -> ReplaceText -> InvokeHTTP (or) GetFile -> InvokeHTTP The problem you're describing

Re: [DISCUSS] git branching model

One of the harder things with gitflow is using it in combination with maven. It's ideal that the tags and releases are tracking closely with the maven pom.xml version. gitflow, on its own, doesn't keep the pom version updated with the git release names. Because of the general importance of

Re: Are we thinking about Penalization all wrong?

If we're willing to have a LoopFlowFile processor, why not consider a PenalizeFlowFile processor too? Just throwing it out for discussions sake, but penalization could ultimately be realized in multiple ways: a) by both the processor developer (and DFM via penalty duration), as it is done today;

Re: NiFi 0.4.1 InvokeHttp processor POST error issue

Joe, Just as a quick observation, this statement isn't completely accurate: > "... and can stream the contents instead of loading into memory" The original InvokeHTTP code (pre okhttp) explicitly set the content-length header, because it was known (the flowfile payload content length is always

Re: remote command execution via SSH?

Sumo, On Tue, Nov 24, 2015 at 10:27 PM, Sumanth Chinthagunta wrote: > I think you guys may have configured password less login for SSH (keys?) > ​Correct. I'm using SSH key exchange for authentication. It's usually done password-less, true, but it doesn't necessarily have

Re: Keep Files

cessor, configured with > /data/input_links/, and set Keep Source File to False. When the GetFile > processor picks up the file, it'll read the contents and create a flowfile > by following the symlink, delete the symlink, and the original file will > remain in /data/input_files. > > O

Re: Keep Files

Also, as a potential work-around, it's possible to use GetFile with "delete" mode and then somewhere in your flow, use PutFile to place the file back down into a "complete" directory. i.e. something like: /path/incoming <- use GetFile to pick up files here /path/complete <- use PutFile to

Re: ExecuteStreamCommand tests

git revert is your friend. https://git-scm.com/docs/git-revert It's not "rollback" -- it's another new commit with the changes reinstated. On Thu, Nov 12, 2015 at 5:45 PM, Joe Witt wrote: > ok - will undo the commit. I get to learn a new git trick? Or just > add them

Re: Incorporation of other Maven repositories

I'm concerned that not all networks will be able to connect with and use the JCenter repository. If it's not in Maven Central, we should likely avoid the dependency and instead find alternative approaches. Adam On Fri, Nov 6, 2015 at 11:31 AM, Joe Witt wrote: > joe

Re: Incorporation of other Maven repositories

oe Witt <joe.w...@gmail.com> wrote: > > > What are some examples of networks which can access maven central but > > > cannot access JCenter? > > > > > > Thanks > > > Joe > > > > > > On Fri, Nov 6, 2015 at 12:10 PM, Adam Taft <a...@adam

Re: LogAttribute - Sending that output to a custom logger?

This thread has forked into two different conversations: 1. improvements to LogAttribute processor; 2. improvements to processor documentation. 1) re: improvements to LogAttribute - we already have NIFI-67 [1] that suggests a number of improvements to LogAttribute. One of these is the use of a

Re: LogAttribute - Sending that output to a custom logger?

> > ----- Reply message - > From: "Adam Taft" <a...@adamtaft.com> > Date: Mon, Nov 2, 2015 19:23 > Subject: LogAttribute - Sending that output to a custom logger? > To: <dev@nifi.apache.org> > > This thread has forked into two different convers

Re: Source code for Version 0.3.0

; proper release tag, signed by the RM on passage of the release vote. I > >> don't recall off-hand what the phrasing was in the VOTE thread (if > >> any). > >> > >> On Mon, Sep 21, 2015 at 8:13 AM, Adam Taft <a...@adamtaft.com> wrote: > >>> > &

Re: [VOTE] Release Apache NiFi nifi-nar-maven-plugin 1.1.0

+1 - Validated source signature, hashes, licensing. Nar plugin compiles and builds nifi-0.3.0-SNAPSHOT without problem on Mac 10.10.4 with JDK 8u60, maven 3.3.3. Running -Dmode=tree against the standard-nar results in: [INFO] --- nifi-nar-maven-plugin:1.1.0:provided-nar-dependencies

Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

difficult RM task and as something that creates confusion. So for me, this is an easy discussion if we can clearly articulate value of the master/develop distinction. Thanks Joe On Thu, Aug 13, 2015 at 4:44 PM, Adam Taft a...@adamtaft.com wrote: The default branch is not a feature

Re: Route Original Flow File Base on InvokeHTTP Response

One option I think we kicked around at some point was to capture the response body as a flowfile attribute in the original flowfile. For reasonably sized response bodies, this would work OK. It would be a nice way to handle your situation, because then the response becomes an attribute of the

Re: nifi error

One possible option to help on this might be to commit a .gitattributes file, which would basically name the problematic test files and mark them to not modify their line endings. http://git-scm.com/docs/gitattributes I think the format would look something like: /path/to/problematic/test/file