Understood. Thanks for the help (really) !
On Wed, Jun 19, 2019 at 1:24 PM Andy LoPresto <[email protected]> wrote: > Wyllys, > > It may help to think of NiFi processors the same way you think about *nix > command line tools — each tool has a very specific and limited > responsibility and focus. Rather than continue to add generic features to > each in order to cover all use cases, you combine the right tools in the > necessary order to achieve your goal (i.e. cat, grep, sed, cut, etc.). No > individual tool cares where the input data is coming from or where the > output data is going; a critical idea which is part of the Flow Based > Programming [1] design philosophy that NiFi models. Adding features > increases complexity, configuration, and the opportunity for bugs or > unexpected behavior, especially in edge cases. > > The InvokeHTTP processor is concerned with transmitting flowfile > attributes & content to a remote HTTP endpoint. Rather than include > additional functionality to massage the content, we would recommend using > an additional preceding processor to form the content into the expected > format. > > I definitely understand the desire to add “just one thing” to make a > specific use case easier, but we have to balance that with the > maintainability and supportability of the framework as a whole moving > forward. Hope this helps. Thanks. > > [1] > https://nifi.apache.org/docs/nifi-docs/html/overview.html#the-core-concepts-of-nifi > > Andy LoPresto > [email protected] > *[email protected] <[email protected]>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Jun 19, 2019, at 7:23 AM, Wyllys Ingersoll < > [email protected]> wrote: > > OK, that helps. I put AttributesToJSON and a ReplaceText processor in > front of my invokeHTTP and was able to get it to work as I wanted. > > One of the confusing issues is that InvokeHTTP does not have any > configuration that allows me to modify the flowfile content, which forces > me to use those additional processors to make sure the content is correct. > It might be a little more convenient if invokeHTTP had a couple of > parameters to allow for setting the content format so that the extra > processors are not necessary. > > > > On Tue, Jun 18, 2019 at 11:01 PM Andy LoPresto <[email protected]> > wrote: > >> Wyllys, >> >> I am sorry but I am having a different experience when I try to reproduce >> the scenario you describe. >> >> I set up a sample flow which uses an InvokeHTTP processor to send a >> flowfile to a remote HTTP endpoint. That endpoint happens to be a disparate >> flow in NiFi because it was easy to configure on the fly, but it could be >> any remote web server. >> >> I can verify that when I send flowfile content, it is not transmitting >> the flowfile attributes unless I explicitly configure it to. Please see >> below an annotated excerpt from the nifi-app.log file and a link to the >> template I exported with this flow. >> >> I think the issue may be one of terminology? Attributes and content are >> separate in NiFi and flowfiles distinguish very cleanly between the two. >> Are you referring to elements of the JSON body as “attributes”? In that >> case, you would need to use a ReplaceText, EvaluateJsonPath, >> JoltTransformJson, or other processor as Jerry suggested to >> extract/manipulate the JSON in the flowfile content to the exact data you >> wish to send over HTTP before using InvokeHTTP. If you want to do this but >> not lose the other content, you can split the flow by dragging an >> additional “success” relationship from the preceding processor to handle >> the two different flow behaviors. >> >> Template for flow (tested on NiFi 1.9.2): >> https://gist.github.com/alopresto/0be1cf65ba70d5d36aeae1e6f2f60702 >> >> Log output from flow: >> https://gist.github.com/alopresto/b596372005032b9c80063a0652a955ea >> >> >> Andy LoPresto >> [email protected] >> *[email protected] <[email protected]>* >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> On Jun 18, 2019, at 7:47 PM, Jerry Vinokurov <[email protected]> >> wrote: >> >> Hi Wyllys, >> >> One way that I solve this problem in my work is to use the >> AttributesToJSON processor to form the body of the POST before sending it. >> Granted, that does override the body of the flowfile, so I'm not sure if >> that works for your specific case, but it does allow you to select which >> attributes you want to turn into JSON key/value pairs. For more complex >> formations I would suggest checking out the JOLT Transform processor, which >> can be quite powerful, if somewhat painful to work with. >> >> Jerry >> >> On Tue, Jun 18, 2019 at 9:49 PM Wyllys Ingersoll < >> [email protected]> wrote: >> >>> Andy - >>> Yes, Im referring to the PUT and POST methods, in which case the >>> processor just sends the entire flowfile as a JSON object in the message >>> body. I'd prefer to either have the option to exclude some of the flow >>> attributes or (even better) have the ability to craft my own message body. >>> There are lots of instances where the receiver of the PUT or POST expects a >>> particular structure that doesn't easily work with just a flat JSON-ified >>> set of flow attributes. >>> >>> One example: >>> We have a flow that has an authentication token as one of the >>> attributes and a bunch of other key/value pairs used for other purposes. >>> In the InvokeHTTP processor, I use the auth token attribute in an >>> Authorization header by creating a dynamic attribute with the correct >>> format ("Authorization: Bearer ${token}"). However, the recipient of the >>> PUT/POST also expects the body of the request to be formatted a specific >>> way and does not expect or want to see the auth token or some of the other >>> unrelated information that ends up getting transmitted as part of the >>> message body simply because they are flow attributes. So, if InvokeHTTP >>> were able to exclude certain fields from the message body AND also allow >>> the body of the message to be configurable into a structure other than just >>> a flat dictionary of flow attributes, it would be much more powerful and >>> useful. As it stands, I'm thinking I may have to develop a custom >>> processor to get past this issue, which is not ideal at all. >>> >>> Thanks! >>> Wyllys Ingersoll >>> >>> >>> >>> >>> On Tue, Jun 18, 2019 at 8:34 PM Andy LoPresto <[email protected]> >>> wrote: >>> >>>> Hi Wyllys, >>>> >>>> Sorry to hear you are having trouble with this processor. Can you >>>> please provide a more detailed example of an incoming flowfile and what >>>> your expected output is compared to what is currently happening? Based on >>>> my understanding, the flowfile attributes are only sent as request headers, >>>> and that can be controlled using the regular expression value of >>>> “Attributes to Send”. I believe only the flowfile content is sent as the >>>> request body when using PUT or POST. Thanks. >>>> >>>> >>>> Andy LoPresto >>>> [email protected] >>>> *[email protected] <[email protected]>* >>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >>>> >>>> On Jun 18, 2019, at 3:01 PM, Wyllys Ingersoll < >>>> [email protected]> wrote: >>>> >>>> >>>> it would be nice to be able to exclude attributes from the message body >>>> when doing a PUT or POST in the invokeHTTP processor. Currently, there's >>>> no way to selectively choose which attributes go into the message body >>>> without using a replaceText processor in front of it, but that completely >>>> removes those attributes from being used later which is not always the >>>> desirable. >>>> >>>> There are lots of situations where it would be nice to be able to use >>>> some flow attributes just in the request header or for other parts of the >>>> processor configuration without including them in the message body itself. >>>> For example, taking an authentication token that is carried along as a flow >>>> attribute so it can be used in an authorization header and NOT included in >>>> the body of the request. >>>> >>>> In general, invokeHTTP needs to allow for more control over the format >>>> and content of the message body being sent. >>>> >>>> -Wyllys Ingersoll >>>> >>>> >>>> >> >> -- >> http://www.google.com/profiles/grapesmoker >> >> >> >
