Mike,
To really do a good job with Enrichment from an http endpoint, we need two
transformations really. We need the ability to transform the input Record into
what the web service wants/needs. And we also need the ability to take the
response from that web service and join that response together with the input
Record.
Ideally there would be script-based ways to do each of these, as well as other
ways, such as using SQL to query for certain fields from the input record and
certain fields from the lookup service response. I have some ideas floating
around in my head but I’ve not really laid anything out very clearly. There are
really two different approaches that I think could work here.
The first approach would be to have a “request transformation” property that
would allow for taking an input record and transforming it into a request for
the web service. There would then be another “response transformation” property
that would merge the input and the response into a single Record. This is, in a
way, simple, because it it means that the LookupRecord processor is all that is
needed. But it would need a Controller Service for the Record Reader, one for
the Record Writer, and a third for the Lookup Service. It would likely also
require a Controller Service for the “request transformation” and another for
the “response transformation.” So it ends up being a lot more complex to
configure than it first seems. But perhaps this is okay...
The second approach would be to create a couple of different processors that
would allow for the implementation of the Claim Check Pattern [1]. Here, we
would have a “CheckFlowFile” processor, which could be used in conjunction with
any existing processor for transforming the incoming Record into the needed
request. The response would then be obtained, and then would need to be merged
back together with some sort of “Merge Records” processor. This approach is
probably cleaner in terms of flow design and easier to understand. It also
results in better code re-use. Unfortunately, though, it does come with the
complication of figuring out how to properly correlate the appropriate input
Record with the appropriate “enrichment response” Record….
Just some thoughts :)
[1]
https://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html
On Jun 29, 2020, at 8:22 PM, Mike Thomsen
mailto:mikerthom...@gmail.com>> wrote:
Matt,
Yeah, I was thinking about that, but there are a lot of variables that have
come up since I wrote that service. One of the big ones is how to take partial
responses and merge them? There's no transformer API for that service. Nothing
like a Groovy script, JOLT, etc.
What do you think? I think something JOLT-based similar to JoltTransformRecord
could be a starting point.
On Mon, Jun 29, 2020 at 5:14 PM Matt Burgess
mailto:mattyb...@apache.org>> wrote:
Mike,
I think you can use LookupRecord with a RestLookupService to do this.
If it's missing features or it otherwise doesn't work for your use
case, please let us know and/or write up whatever Jiras you feel are
appropriate.
Regards,
Matt
On Mon, Jun 29, 2020 at 4:56 PM Mike Thomsen
mailto:mikerthom...@gmail.com>> wrote:
>
> Does anyone know a good pattern using the Record API to enrich a data set
> record by record with a REST API?
>
> Thanks,
>
> Mike
