Jon, In 1.6, we added a new client service for ES. There's only one processor that uses it for now, JsonQueryElasticSearch, but I have more under way that I hope will make it into 1.7. If you take a look at the existing version of the client service implementation that uses the official ElasticSearch APIs. You might be able to quickly get this up and running with a new implementation that uses the AWS SDK instead of the official API.
FYI, JsonQueryElasticSearch is designed so you can build a query in Kibana and just drop it into NiFi and go. So any querying other than scrolls should work out of the box if you create a proper implementation of ElasticSearchClientService that uses the Amazon code. On Mon, Apr 9, 2018 at 8:29 PM, Otto Fowler <[email protected]> wrote: > The aws java sdk has a purpose built ElasticSearchClient class. > The way to do this and be consistent would be to add a new nifi-aws > processor for the ES client, > as there is for s3 and dynamoDB etc. > > https://github.com/apache/nifi/pull/2588 is my outstanding PR for > HttpInvoke for AWS Gateway Web APIs. > A similar approach being having version of the ElasticSearchHttp but in the > aws nar. > > > > On April 9, 2018 at 19:11:45, Jon Logan ([email protected]) wrote: > > Hi All, > > We are looking to use the built-in Nifi Elasticsearch Http processors with > signed AWS ES requests. I tried to extend them to do so, but ran into > issues with things being in-extensible (private-static-final in some > cases), and was wondering if this would be something that would be > considered to be merged into the baseline? > > There's two main ways I saw doing it -- either modifying the existing ES > code to allow for it, or making new AWS-specific extended adapters to do > it. In the former, it might require dependencies between the ES code and > the AWS code for optional CredentialProviders, etc., and am not sure how > isolated you all try to keep things. > > In either case, AWS essentially adds an HTTP-header signature to > authenticate the request against your ES instance. The easiest path to do > this seems to be writing a bridge between the ES OkHttpClient requests and > the AWS requests to generate the signature correctly. > > > Just wanted to get some feedback to be sure this wouldn't be a waste of > time. > > > Thanks! > Jon >
