Re: PostHTTP Deprecation Concerns
Hi Lucas,
Cheers, and thanks for sharing.
One of the solutions discussed even recently on the user's mailing list was
effectively putting MergeContent in front of InvokeHTTP. MergeContent would
be configured with the "FlowFile Stream v3" protocol, which effectively
bundles up flowfiles similar to PostHTTP with the send-as-flowfile
attribute enabled. ListenHTTP will receive the bundled flowfiles and unwind
the packaging upon reception.
This is probably the smallest solution for those trying to replace PostHTTP
in use today. As I mentioned in my original replies, I don't think it's a
100% drop-in replacement, but probably the simplest solution available
going forward, especially when NiFi 2.0 is released.
I'm going to check out your code sometime, thanks again for responding to
an old thread and offering a solution.
/Adam
On Sun, Feb 12, 2023 at 8:21 AM Lucas Ottersbach
wrote:
> Hi Adam,
>
> I know this thread has been opened over a month ago, but we recently had to
> move FlowFiles, including both attributes and content, from one NiFi
> cluster to another and could not built upon the built-in Site-to-Site
> transfer mechanisms due to network restrictions between the clusters.
>
> We've built upon an existing solution from a community member which has
> been dormant for some time. It uses a pair of two custom processors to
> transfer FlowFile content and attributes using raw TCP connections.
> You can find the solution under its name "nifi-flow-over-tcp" both on
> GitHub and on Maven Central.
> githubDOTcom/EndzeitBegins/nifi-flow-over-tcp
>
>
> Maybe this can be helpful to you as well in the aforementioned cases you
> previously made use of the PostHTTP processor.
>
>
> Best regards
>
> Adam Taft schrieb am Do., 12. Jan. 2023, 05:39:
>
> > David,
> >
> > Thank you for the reasonable response to my questions. Much appreciated.
> >
> > I'm not a huge fan of the MergeContent -> InvokeHTTP -> {} -> ListenHTTP
> ->
> > UnpackContent approach to provide the same functionality. But I do
> > acknowledge that's the most direct replacement option without PostHTTP.
> > It's adding extract processors to the chain for something that is
> > effectively a transport issue. NiFi-to-Nifi using PostHTTP was a simple
> > transport-oriented solution, and packing the data with MergeContent first
> > isn't quite the same level of fidelity. You also miss the two-phase
> commit
> > built into those extra bits. MergeContent is often a bit of a beast
> > in-and-of-itself too.
> >
> > Flowfile attributes conveyed as HTTP headers definitely don't work for
> > complex attribute values. But yes, I know that the functionality exists
> > (having some history with that processor myself).
> >
> > Thanks again for the response.
> >
> > /Adam
> >
> >
> >
> >
> > On Wed, Jan 11, 2023 at 9:27 PM Adam Taft wrote:
> >
> > > Hi Mathew,
> > >
> > > > It's quite remarkable you're advocating against standard practice
> > > presumably
> > > > for your own convenience.
> > >
> > > Wow, absolutely not stated nor implied in my message. And even
> borderline
> > > offensive.
> > >
> > > What I asked was simply, why remove it, if it's not hurting anything. I
> > > agree with your statement that there is a (very small) cost for
> > maintaining
> > > the component in the source tree. But PostHTTP is not in the same scope
> > as
> > > compared to a component that has a dependency on an abandoned,
> insecure,
> > or
> > > completely out of standards library (for example).
> > >
> > > PostHTTP has a reasonable use case (as I described) that is not
> directly
> > > matched with other processors. The two-phase commit protocol sitting
> > > between PostHTTP and ListenHTTP has demonstrated to bear good fruit
> over
> > > many hardened years of use. I think it's a reasonable reply to my
> > question
> > > to just simply suggest that the interaction between PostHTTP and
> > ListenHTTP
> > > is just not supported by NiFi going forward. But please don't tell me
> my
> > > question/concern is "out of convenience."
> > >
> > > There is lacking documentation as to the rationale behind the
> deprecation
> > > of PostHTTP. I might be missing it, can you please send me the link to
> > the
> > > rationale? That's what this thread is trying to address. It sounds
> like,
> > > from your answer, that the rationale is to reduce code footprint, which
> > > isn't the strongest argument for its removal given its established
> > > historical use. Seems like we'd want more than just reduced footprint
> for
> > > such a heavily used processor, no?
> > >
> > > /Adam
> > >
> > >
> > > On Wed, Jan 11, 2023 at 7:53 PM Matthew Hawkins
> > > wrote:
> > >
> > >> Hi Adam,
> > >>
> > >> PostHTTP was marked deprecated 3 years ago (aka six technology
> > lifetimes).
> > >> The successive technologies to replace it's functionality are well
> > >> documented and proven in production. The technical reason to remove it
> > is
> > >> that it is superfluous code that has a cost to maintain
Re: PostHTTP Deprecation Concerns
Hi Adam,
I know this thread has been opened over a month ago, but we recently had to
move FlowFiles, including both attributes and content, from one NiFi
cluster to another and could not built upon the built-in Site-to-Site
transfer mechanisms due to network restrictions between the clusters.
We've built upon an existing solution from a community member which has
been dormant for some time. It uses a pair of two custom processors to
transfer FlowFile content and attributes using raw TCP connections.
You can find the solution under its name "nifi-flow-over-tcp" both on
GitHub and on Maven Central.
githubDOTcom/EndzeitBegins/nifi-flow-over-tcp
Maybe this can be helpful to you as well in the aforementioned cases you
previously made use of the PostHTTP processor.
Best regards
Adam Taft schrieb am Do., 12. Jan. 2023, 05:39:
> David,
>
> Thank you for the reasonable response to my questions. Much appreciated.
>
> I'm not a huge fan of the MergeContent -> InvokeHTTP -> {} -> ListenHTTP ->
> UnpackContent approach to provide the same functionality. But I do
> acknowledge that's the most direct replacement option without PostHTTP.
> It's adding extract processors to the chain for something that is
> effectively a transport issue. NiFi-to-Nifi using PostHTTP was a simple
> transport-oriented solution, and packing the data with MergeContent first
> isn't quite the same level of fidelity. You also miss the two-phase commit
> built into those extra bits. MergeContent is often a bit of a beast
> in-and-of-itself too.
>
> Flowfile attributes conveyed as HTTP headers definitely don't work for
> complex attribute values. But yes, I know that the functionality exists
> (having some history with that processor myself).
>
> Thanks again for the response.
>
> /Adam
>
>
>
>
> On Wed, Jan 11, 2023 at 9:27 PM Adam Taft wrote:
>
> > Hi Mathew,
> >
> > > It's quite remarkable you're advocating against standard practice
> > presumably
> > > for your own convenience.
> >
> > Wow, absolutely not stated nor implied in my message. And even borderline
> > offensive.
> >
> > What I asked was simply, why remove it, if it's not hurting anything. I
> > agree with your statement that there is a (very small) cost for
> maintaining
> > the component in the source tree. But PostHTTP is not in the same scope
> as
> > compared to a component that has a dependency on an abandoned, insecure,
> or
> > completely out of standards library (for example).
> >
> > PostHTTP has a reasonable use case (as I described) that is not directly
> > matched with other processors. The two-phase commit protocol sitting
> > between PostHTTP and ListenHTTP has demonstrated to bear good fruit over
> > many hardened years of use. I think it's a reasonable reply to my
> question
> > to just simply suggest that the interaction between PostHTTP and
> ListenHTTP
> > is just not supported by NiFi going forward. But please don't tell me my
> > question/concern is "out of convenience."
> >
> > There is lacking documentation as to the rationale behind the deprecation
> > of PostHTTP. I might be missing it, can you please send me the link to
> the
> > rationale? That's what this thread is trying to address. It sounds like,
> > from your answer, that the rationale is to reduce code footprint, which
> > isn't the strongest argument for its removal given its established
> > historical use. Seems like we'd want more than just reduced footprint for
> > such a heavily used processor, no?
> >
> > /Adam
> >
> >
> > On Wed, Jan 11, 2023 at 7:53 PM Matthew Hawkins
> > wrote:
> >
> >> Hi Adam,
> >>
> >> PostHTTP was marked deprecated 3 years ago (aka six technology
> lifetimes).
> >> The successive technologies to replace it's functionality are well
> >> documented and proven in production. The technical reason to remove it
> is
> >> that it is superfluous code that has a cost to maintain and zero
> benefit.
> >> Backwards compatibility is never guaranteed for components marked
> >> deprecated for such a long length of time in any software product let
> >> alone
> >> nifi specifically.
> >>
> >> Your organisation is free to continue using the version of nifi it is on
> >> today and not take any further action. It is unhelpful to suggest every
> >> other organisation should be held back in progress because yours refuses
> >> to
> >> take the necessary flow maintenance action. One of the impetus for a
> major
> >> version upgrade is to specifically jettison deprecated components. It's
> >> quite remarkable you're advocating against standard practice presumably
> >> for
> >> your own convenience.
> >>
> >> Site to site connectivity is conducted with either raw sockets or http
> >> (which is https on secured nifi) so I'm highly skeptical there is any
> >> performance deprecation in InvokeHTTP or S2S over PostHTTP, given the
> >> former can take advantage of http/2 and the latter not. It's easy to
> >> monitor nifi and prove through metrics in any case. Sadly in enterprise
> >>
