Re: Signing AWS Elasticsearch Requests?

[Mike Thomsen]

The existing processors will not be refactored to use it. I'm writing new
ones.

On Tue, Apr 10, 2018 at 12:08 AM, Jon Logan  wrote:

> Thanks, a few notes -- as far as not using AWS ES, we have been having
> issues with standard-ES, mainly the lack of encryption and authentication
> without the X-Pack (yes, I could proxy but that seems like a band-aid)..
> note that AWS VPCs are not encrypted internally, and any machine that you
> want to have updates must have a public-facing IP address (no VPC-only).
>
> This could be done using the AWS ES SDK, but it seems like that's more
> trouble than it's worth...the AWS ES endpoints are identical, yet support
> the additional signing attribute. It would seem unnecessary to have two
> disjoint implementations, rather than just adding the signatures to the
> request...similar to this (I found this on Google, so no guarantees!)
> https://gist.github.com/missingfaktor/664b01b7989e91033657e2e703f14ed6 . I
> think the only piece of the AWS SDK this would require would be the things
> to get the credentials, and the signing part itself.
>
> The new ES client service looks much easier to use than the previous one.
> Is this going to eventually be used by the existing processors (ex.
> FetchEsHttp, PutEsHttp, etc)? I suppose this could be even have a
> TransportClient implementation and merge the FetchEs FetchEsHttp and
> FetchEs5 (and put, etc) processors?
>
> On Mon, Apr 9, 2018 at 9:47 PM, Mike Thomsen 
> wrote:
>
> > Related, interesting take on AWS ES:
> >
> > https://code972.com/blog/2017/12/111-why-you-shouldnt-use-
> > aws-elasticsearch-service
> >
> > I saw some other critical posts that went into detail on other issues.
> > Considering how easy it is to stand up ES yourself, I think it would be
> > worthwhile to strongly consider going that route.
> >
> > On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler 
> > wrote:
> >
> > > Sorry, just like the gateway api class, it is for managing ES not
> > calling.
> > >
> > >
> > > On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> > >
> > > The aws java sdk has a purpose built ElasticSearchClient class.
> > > The way to do this and be consistent would be to add a new nifi-aws
> > > processor for the ES client,
> > > as there is for s3 and dynamoDB etc.
> > >
> > > https://github.com/apache/nifi/pull/2588 is my outstanding PR for
> > > HttpInvoke for AWS Gateway Web APIs.
> > > A similar approach being having version of the ElasticSearchHttp but in
> > the
> > > aws nar.
> > >
> > >
> > >
> > > On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote:
> > >
> > > Hi All,
> > >
> > > We are looking to use the built-in Nifi Elasticsearch Http processors
> > with
> > > signed AWS ES requests. I tried to extend them to do so, but ran into
> > > issues with things being in-extensible (private-static-final in some
> > > cases), and was wondering if this would be something that would be
> > > considered to be merged into the baseline?
> > >
> > > There's two main ways I saw doing it -- either modifying the existing
> ES
> > > code to allow for it, or making new AWS-specific extended adapters to
> do
> > > it. In the former, it might require dependencies between the ES code
> and
> > > the AWS code for optional CredentialProviders, etc., and am not sure
> how
> > > isolated you all try to keep things.
> > >
> > > In either case, AWS essentially adds an HTTP-header signature to
> > > authenticate the request against your ES instance. The easiest path to
> do
> > > this seems to be writing a bridge between the ES OkHttpClient requests
> > and
> > > the AWS requests to generate the signature correctly.
> > >
> > >
> > > Just wanted to get some feedback to be sure this wouldn't be a waste of
> > > time.
> > >
> > >
> > > Thanks!
> > > Jon
> > >
> >
>

Re: Signing AWS Elasticsearch Requests?

[Jon Logan]

Thanks, a few notes -- as far as not using AWS ES, we have been having
issues with standard-ES, mainly the lack of encryption and authentication
without the X-Pack (yes, I could proxy but that seems like a band-aid)..
note that AWS VPCs are not encrypted internally, and any machine that you
want to have updates must have a public-facing IP address (no VPC-only).

This could be done using the AWS ES SDK, but it seems like that's more
trouble than it's worth...the AWS ES endpoints are identical, yet support
the additional signing attribute. It would seem unnecessary to have two
disjoint implementations, rather than just adding the signatures to the
request...similar to this (I found this on Google, so no guarantees!)
https://gist.github.com/missingfaktor/664b01b7989e91033657e2e703f14ed6 . I
think the only piece of the AWS SDK this would require would be the things
to get the credentials, and the signing part itself.

The new ES client service looks much easier to use than the previous one.
Is this going to eventually be used by the existing processors (ex.
FetchEsHttp, PutEsHttp, etc)? I suppose this could be even have a
TransportClient implementation and merge the FetchEs FetchEsHttp and
FetchEs5 (and put, etc) processors?

On Mon, Apr 9, 2018 at 9:47 PM, Mike Thomsen  wrote:

> Related, interesting take on AWS ES:
>
> https://code972.com/blog/2017/12/111-why-you-shouldnt-use-
> aws-elasticsearch-service
>
> I saw some other critical posts that went into detail on other issues.
> Considering how easy it is to stand up ES yourself, I think it would be
> worthwhile to strongly consider going that route.
>
> On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler 
> wrote:
>
> > Sorry, just like the gateway api class, it is for managing ES not
> calling.
> >
> >
> > On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
> >
> > The aws java sdk has a purpose built ElasticSearchClient class.
> > The way to do this and be consistent would be to add a new nifi-aws
> > processor for the ES client,
> > as there is for s3 and dynamoDB etc.
> >
> > https://github.com/apache/nifi/pull/2588 is my outstanding PR for
> > HttpInvoke for AWS Gateway Web APIs.
> > A similar approach being having version of the ElasticSearchHttp but in
> the
> > aws nar.
> >
> >
> >
> > On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote:
> >
> > Hi All,
> >
> > We are looking to use the built-in Nifi Elasticsearch Http processors
> with
> > signed AWS ES requests. I tried to extend them to do so, but ran into
> > issues with things being in-extensible (private-static-final in some
> > cases), and was wondering if this would be something that would be
> > considered to be merged into the baseline?
> >
> > There's two main ways I saw doing it -- either modifying the existing ES
> > code to allow for it, or making new AWS-specific extended adapters to do
> > it. In the former, it might require dependencies between the ES code and
> > the AWS code for optional CredentialProviders, etc., and am not sure how
> > isolated you all try to keep things.
> >
> > In either case, AWS essentially adds an HTTP-header signature to
> > authenticate the request against your ES instance. The easiest path to do
> > this seems to be writing a bridge between the ES OkHttpClient requests
> and
> > the AWS requests to generate the signature correctly.
> >
> >
> > Just wanted to get some feedback to be sure this wouldn't be a waste of
> > time.
> >
> >
> > Thanks!
> > Jon
> >
>

Re: Signing AWS Elasticsearch Requests?

[Mike Thomsen]

Related, interesting take on AWS ES:

https://code972.com/blog/2017/12/111-why-you-shouldnt-use-aws-elasticsearch-service

I saw some other critical posts that went into detail on other issues.
Considering how easy it is to stand up ES yourself, I think it would be
worthwhile to strongly consider going that route.

On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler  wrote:

> Sorry, just like the gateway api class, it is for managing ES not calling.
>
>
> On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com) wrote:
>
> The aws java sdk has a purpose built ElasticSearchClient class.
> The way to do this and be consistent would be to add a new nifi-aws
> processor for the ES client,
> as there is for s3 and dynamoDB etc.
>
> https://github.com/apache/nifi/pull/2588 is my outstanding PR for
> HttpInvoke for AWS Gateway Web APIs.
> A similar approach being having version of the ElasticSearchHttp but in the
> aws nar.
>
>
>
> On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote:
>
> Hi All,
>
> We are looking to use the built-in Nifi Elasticsearch Http processors with
> signed AWS ES requests. I tried to extend them to do so, but ran into
> issues with things being in-extensible (private-static-final in some
> cases), and was wondering if this would be something that would be
> considered to be merged into the baseline?
>
> There's two main ways I saw doing it -- either modifying the existing ES
> code to allow for it, or making new AWS-specific extended adapters to do
> it. In the former, it might require dependencies between the ES code and
> the AWS code for optional CredentialProviders, etc., and am not sure how
> isolated you all try to keep things.
>
> In either case, AWS essentially adds an HTTP-header signature to
> authenticate the request against your ES instance. The easiest path to do
> this seems to be writing a bridge between the ES OkHttpClient requests and
> the AWS requests to generate the signature correctly.
>
>
> Just wanted to get some feedback to be sure this wouldn't be a waste of
> time.
>
>
> Thanks!
> Jon
>

Re: Signing AWS Elasticsearch Requests?

[Otto Fowler]

Sorry, just like the gateway api class, it is for managing ES not calling.


On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com) wrote:

The aws java sdk has a purpose built ElasticSearchClient class.
The way to do this and be consistent would be to add a new nifi-aws
processor for the ES client,
as there is for s3 and dynamoDB etc.

https://github.com/apache/nifi/pull/2588 is my outstanding PR for
HttpInvoke for AWS Gateway Web APIs.
A similar approach being having version of the ElasticSearchHttp but in the
aws nar.



On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote:

Hi All,

We are looking to use the built-in Nifi Elasticsearch Http processors with
signed AWS ES requests. I tried to extend them to do so, but ran into
issues with things being in-extensible (private-static-final in some
cases), and was wondering if this would be something that would be
considered to be merged into the baseline?

There's two main ways I saw doing it -- either modifying the existing ES
code to allow for it, or making new AWS-specific extended adapters to do
it. In the former, it might require dependencies between the ES code and
the AWS code for optional CredentialProviders, etc., and am not sure how
isolated you all try to keep things.

In either case, AWS essentially adds an HTTP-header signature to
authenticate the request against your ES instance. The easiest path to do
this seems to be writing a bridge between the ES OkHttpClient requests and
the AWS requests to generate the signature correctly.


Just wanted to get some feedback to be sure this wouldn't be a waste of
time.


Thanks!
Jon

Re: Signing AWS Elasticsearch Requests?

[Mike Thomsen]

Jon,

In 1.6, we added a new client service for ES. There's only one processor
that uses it for now, JsonQueryElasticSearch, but I have more under way
that I hope will make it into 1.7. If you take a look at the existing
version of the client service implementation that uses the official
ElasticSearch APIs. You might be able to quickly get this up and running
with a new implementation that uses the AWS SDK instead of the official API.

FYI, JsonQueryElasticSearch is designed so you can build a query in Kibana
and just drop it into NiFi and go. So any querying other than scrolls
should work out of the box if you create a proper implementation of
ElasticSearchClientService that uses the Amazon code.

On Mon, Apr 9, 2018 at 8:29 PM, Otto Fowler  wrote:

> The aws java sdk has a purpose built ElasticSearchClient class.
> The way to do this and be consistent would be to add a new nifi-aws
> processor for the ES client,
> as there is for s3 and dynamoDB etc.
>
> https://github.com/apache/nifi/pull/2588 is my outstanding PR for
> HttpInvoke for AWS Gateway Web APIs.
> A similar approach being having version of the ElasticSearchHttp but in the
> aws nar.
>
>
>
> On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote:
>
> Hi All,
>
> We are looking to use the built-in Nifi Elasticsearch Http processors with
> signed AWS ES requests. I tried to extend them to do so, but ran into
> issues with things being in-extensible (private-static-final in some
> cases), and was wondering if this would be something that would be
> considered to be merged into the baseline?
>
> There's two main ways I saw doing it -- either modifying the existing ES
> code to allow for it, or making new AWS-specific extended adapters to do
> it. In the former, it might require dependencies between the ES code and
> the AWS code for optional CredentialProviders, etc., and am not sure how
> isolated you all try to keep things.
>
> In either case, AWS essentially adds an HTTP-header signature to
> authenticate the request against your ES instance. The easiest path to do
> this seems to be writing a bridge between the ES OkHttpClient requests and
> the AWS requests to generate the signature correctly.
>
>
> Just wanted to get some feedback to be sure this wouldn't be a waste of
> time.
>
>
> Thanks!
> Jon
>

Re: Signing AWS Elasticsearch Requests?

[Otto Fowler]

The aws java sdk has a purpose built ElasticSearchClient class.
The way to do this and be consistent would be to add a new nifi-aws
processor for the ES client,
as there is for s3 and dynamoDB etc.

https://github.com/apache/nifi/pull/2588 is my outstanding PR for
HttpInvoke for AWS Gateway Web APIs.
A similar approach being having version of the ElasticSearchHttp but in the
aws nar.



On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote:

Hi All,

We are looking to use the built-in Nifi Elasticsearch Http processors with
signed AWS ES requests. I tried to extend them to do so, but ran into
issues with things being in-extensible (private-static-final in some
cases), and was wondering if this would be something that would be
considered to be merged into the baseline?

There's two main ways I saw doing it -- either modifying the existing ES
code to allow for it, or making new AWS-specific extended adapters to do
it. In the former, it might require dependencies between the ES code and
the AWS code for optional CredentialProviders, etc., and am not sure how
isolated you all try to keep things.

In either case, AWS essentially adds an HTTP-header signature to
authenticate the request against your ES instance. The easiest path to do
this seems to be writing a bridge between the ES OkHttpClient requests and
the AWS requests to generate the signature correctly.


Just wanted to get some feedback to be sure this wouldn't be a waste of
time.


Thanks!
Jon