Thanks, a few notes -- as far as not using AWS ES, we have been having issues with standard-ES, mainly the lack of encryption and authentication without the X-Pack (yes, I could proxy but that seems like a band-aid).. note that AWS VPCs are not encrypted internally, and any machine that you want to have updates must have a public-facing IP address (no VPC-only).
This could be done using the AWS ES SDK, but it seems like that's more trouble than it's worth...the AWS ES endpoints are identical, yet support the additional signing attribute. It would seem unnecessary to have two disjoint implementations, rather than just adding the signatures to the request...similar to this (I found this on Google, so no guarantees!) https://gist.github.com/missingfaktor/664b01b7989e91033657e2e703f14ed6 . I think the only piece of the AWS SDK this would require would be the things to get the credentials, and the signing part itself. The new ES client service looks much easier to use than the previous one. Is this going to eventually be used by the existing processors (ex. FetchEsHttp, PutEsHttp, etc)? I suppose this could be even have a TransportClient implementation and merge the FetchEs FetchEsHttp and FetchEs5 (and put, etc) processors? On Mon, Apr 9, 2018 at 9:47 PM, Mike Thomsen <mikerthom...@gmail.com> wrote: > Related, interesting take on AWS ES: > > https://code972.com/blog/2017/12/111-why-you-shouldnt-use- > aws-elasticsearch-service > > I saw some other critical posts that went into detail on other issues. > Considering how easy it is to stand up ES yourself, I think it would be > worthwhile to strongly consider going that route. > > On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > Sorry, just like the gateway api class, it is for managing ES not > calling. > > > > > > On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > > > The aws java sdk has a purpose built ElasticSearchClient class. > > The way to do this and be consistent would be to add a new nifi-aws > > processor for the ES client, > > as there is for s3 and dynamoDB etc. > > > > https://github.com/apache/nifi/pull/2588 is my outstanding PR for > > HttpInvoke for AWS Gateway Web APIs. > > A similar approach being having version of the ElasticSearchHttp but in > the > > aws nar. > > > > > > > > On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote: > > > > Hi All, > > > > We are looking to use the built-in Nifi Elasticsearch Http processors > with > > signed AWS ES requests. I tried to extend them to do so, but ran into > > issues with things being in-extensible (private-static-final in some > > cases), and was wondering if this would be something that would be > > considered to be merged into the baseline? > > > > There's two main ways I saw doing it -- either modifying the existing ES > > code to allow for it, or making new AWS-specific extended adapters to do > > it. In the former, it might require dependencies between the ES code and > > the AWS code for optional CredentialProviders, etc., and am not sure how > > isolated you all try to keep things. > > > > In either case, AWS essentially adds an HTTP-header signature to > > authenticate the request against your ES instance. The easiest path to do > > this seems to be writing a bridge between the ES OkHttpClient requests > and > > the AWS requests to generate the signature correctly. > > > > > > Just wanted to get some feedback to be sure this wouldn't be a waste of > > time. > > > > > > Thanks! > > Jon > > >
