Well that blows. Should I create a jira ticket to disable two phased commit?
On Tuesday, June 7, 2016, Mark Payne <[email protected]> wrote: > Edgardo, > > You'd run into a lot of problems trying to use that solution, as many > attributes contain > characters that are not valid in HTTP headers, and HTTP Headers are > delineated with > new-lines, so if you have an attribute with new-lines you'll get really > weird results. > > -Mark > > > > On Jun 7, 2016, at 3:52 PM, Edgardo Vega <[email protected] > <javascript:;>> wrote: > > > > Mark, > > > > Amazon only supports sticky session via cookies. > > > > Disabling the two-phase commit would be really nice > > > > What if you do a invokehttp with send all the attributes as Http headers > > and on the receive side on listenhttp do a .* to turn all the headers > back > > into attribute? Would that work? > > > > Cheers, > > > > Edgardo > > > > On Tue, Jun 7, 2016 at 3:19 PM, Mark Payne <[email protected] > <javascript:;>> wrote: > > > >> The idea behind the DELETE mechanism is that in some environments there > >> were timeouts > >> that would occur quite frequently between PostHTTP / ListenHTTP and this > >> resulted in quite > >> a lot of data duplication. By adding in the two-phase commit, we were > able > >> to drastically reduce > >> the amount of data duplication, as a timeout anywhere in the first > >> (typically MUCH longer) phase > >> would result in the data on the receiving side being dropped because the > >> receiving side would > >> not delete the hold that it placed on the FlowFiles. > >> > >> It would be reasonable to add an option for PostHTTP so that it requests > >> not to perform a two-phase > >> commit. Alternatively, you could use either PostHTTP with 'Send as > >> FlowFile' set to 'false' or you > >> could use InvokeHTTP. These do not send the attributes, though, so you > >> would need to precede this > >> with a MergeContent with Merge Format of "FlowFile Stream, v3". > >> Then, on the receiving side, you could use UnpackContent to unpack these > >> FlowFile Packages back > >> into their 'native' form. > >> > >> Or, a simpler option, if Amazon's ELB supports it, is to configure the > ELB > >> such that HTTP Requests that > >> contain the same value for the "x-nifi-transaction-id" header will go to > >> the same node. This > >> header was added specifically to allow for this functionality through > Load > >> Balancers, > >> but I don't know if ELB specifically supports this or not. > >> > >> Thanks > >> -Mark > >> > >> > >>> On Jun 7, 2016, at 2:16 PM, Aldrin Piri <[email protected] > <javascript:;>> wrote: > >>> > >>> InvokeHTTP may be the better option if the user is not interested in > >>> transmitting content _packaged as_ FlowFiles. Someone with a bit more > >>> history than myself can provide some additional context if I have > strayed > >>> off the path, but PostHTTP and ListenHTTP were precursors to Site to > >> Site. > >>> While they can transmit arbitrary content, were created for this > >>> inter-instance communication to aid in the guaranteed delivery > semantics. > >>> The listed hold, in this case, is part of that transaction occurring > >> where > >>> a response is returned to acknowledge receipt via ListenHTTP [1] and > the > >>> ContentAcknowledgementServlet [2]. > >>> > >>> [1] > >>> > >> > https://github.com/apache/nifi/blob/1bd2cf0d09a7111bcecffd0f473aa71c25a69845/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java > >>> [2] > >>> > >> > https://github.com/apache/nifi/blob/1bd2cf0d09a7111bcecffd0f473aa71c25a69845/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/servlets/ContentAcknowledgmentServlet.java > >>> > >>> On Tue, Jun 7, 2016 at 2:10 PM, Bryan Bende <[email protected] > <javascript:;>> wrote: > >>> > >>>> Looks like PostHttp interprets the response, and based on a series of > >>>> conditions can intentionally issue a delete. > >>>> > >>>> I can't fully understand what is happening, but the code is here: > >>>> > >>>> > >> > https://github.com/apache/nifi/blob/1bd2cf0d09a7111bcecffd0f473aa71c25a69845/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PostHTTP.java#L754 > >>>> > >>>> Unless someone understands what is happening there, maybe InvokeHttp > >> could > >>>> be used to make the post instead? > >>>> > >>>> -Bryan > >>>> > >>>> > >>>> On Tue, Jun 7, 2016 at 1:53 PM, Edgardo Vega <[email protected] > <javascript:;>> > >>>> wrote: > >>>> > >>>>> Joe, > >>>>> > >>>>> We were testing with another nifi machine outside the elb to post a > >>>>> flowfile in using the PostHttp processor. It seems that on each post > >>>> there > >>>>> is an immediate delete call. Behind the load balancer it goes > haywire. > >>>>> > >>>>> Cheers,, > >>>>> > >>>>> Edgardo > >>>>> > >>>>> On Tue, Jun 7, 2016 at 8:40 AM, Joe Witt <[email protected] > <javascript:;>> wrote: > >>>>> > >>>>>> Edgardo > >>>>>> > >>>>>> Are you saying the clients are posting and then calling delete? > >>>>>> > >>>>>> Also the more complex but flexible options are handle http request > and > >>>>>> response. > >>>>>> > >>>>>> Thanks > >>>>>> Joe > >>>>>> On Jun 7, 2016 7:04 AM, "Edgardo Vega" <[email protected] > <javascript:;>> > >> wrote: > >>>>>> > >>>>>>> I wanted to throw Nifi behind a AWS ELB. I then have the ELB > pointing > >>>>> at > >>>>>>> the nifi cluster. On the cluster I have a ListenHttp. I want to > allow > >>>>>>> people to post flow files to that url. > >>>>>>> > >>>>>>> When testing this setup, it seems I am getting an error due to the > >>>> fact > >>>>>> it > >>>>>>> seems like first there is a post and then a delete to confirm that > it > >>>>> was > >>>>>>> posted. How do I get Nifi to stop doing that? I just want nifi to > >>>> post > >>>>>> and > >>>>>>> if its get something larger than a 400 fail otherwise succeed. > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Cheers, > >>>>>>> > >>>>>>> Edgardo > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Cheers, > >>>>> > >>>>> Edgardo > >>>>> > >>>> > >> > >> > > > > > > -- > > Cheers, > > > > Edgardo > > -- Cheers, Edgardo Sent from Gmail Mobile
