The existing processors will not be refactored to use it. I'm writing new
ones.

On Tue, Apr 10, 2018 at 12:08 AM, Jon Logan <jmlo...@buffalo.edu> wrote:

> Thanks, a few notes -- as far as not using AWS ES, we have been having
> issues with standard-ES, mainly the lack of encryption and authentication
> without the X-Pack (yes, I could proxy but that seems like a band-aid)..
> note that AWS VPCs are not encrypted internally, and any machine that you
> want to have updates must have a public-facing IP address (no VPC-only).
>
> This could be done using the AWS ES SDK, but it seems like that's more
> trouble than it's worth...the AWS ES endpoints are identical, yet support
> the additional signing attribute. It would seem unnecessary to have two
> disjoint implementations, rather than just adding the signatures to the
> request...similar to this (I found this on Google, so no guarantees!)
> https://gist.github.com/missingfaktor/664b01b7989e91033657e2e703f14ed6 . I
> think the only piece of the AWS SDK this would require would be the things
> to get the credentials, and the signing part itself.
>
> The new ES client service looks much easier to use than the previous one.
> Is this going to eventually be used by the existing processors (ex.
> FetchEsHttp, PutEsHttp, etc)? I suppose this could be even have a
> TransportClient implementation and merge the FetchEs FetchEsHttp and
> FetchEs5 (and put, etc) processors?
>
> On Mon, Apr 9, 2018 at 9:47 PM, Mike Thomsen <mikerthom...@gmail.com>
> wrote:
>
> > Related, interesting take on AWS ES:
> >
> > https://code972.com/blog/2017/12/111-why-you-shouldnt-use-
> > aws-elasticsearch-service
> >
> > I saw some other critical posts that went into detail on other issues.
> > Considering how easy it is to stand up ES yourself, I think it would be
> > worthwhile to strongly consider going that route.
> >
> > On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler <ottobackwa...@gmail.com>
> > wrote:
> >
> > > Sorry, just like the gateway api class, it is for managing ES not
> > calling.
> > >
> > >
> > > On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> > >
> > > The aws java sdk has a purpose built ElasticSearchClient class.
> > > The way to do this and be consistent would be to add a new nifi-aws
> > > processor for the ES client,
> > > as there is for s3 and dynamoDB etc.
> > >
> > > https://github.com/apache/nifi/pull/2588 is my outstanding PR for
> > > HttpInvoke for AWS Gateway Web APIs.
> > > A similar approach being having version of the ElasticSearchHttp but in
> > the
> > > aws nar.
> > >
> > >
> > >
> > > On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote:
> > >
> > > Hi All,
> > >
> > > We are looking to use the built-in Nifi Elasticsearch Http processors
> > with
> > > signed AWS ES requests. I tried to extend them to do so, but ran into
> > > issues with things being in-extensible (private-static-final in some
> > > cases), and was wondering if this would be something that would be
> > > considered to be merged into the baseline?
> > >
> > > There's two main ways I saw doing it -- either modifying the existing
> ES
> > > code to allow for it, or making new AWS-specific extended adapters to
> do
> > > it. In the former, it might require dependencies between the ES code
> and
> > > the AWS code for optional CredentialProviders, etc., and am not sure
> how
> > > isolated you all try to keep things.
> > >
> > > In either case, AWS essentially adds an HTTP-header signature to
> > > authenticate the request against your ES instance. The easiest path to
> do
> > > this seems to be writing a bridge between the ES OkHttpClient requests
> > and
> > > the AWS requests to generate the signature correctly.
> > >
> > >
> > > Just wanted to get some feedback to be sure this wouldn't be a waste of
> > > time.
> > >
> > >
> > > Thanks!
> > > Jon
> > >
> >
>

Reply via email to