Re: Signing AWS Elasticsearch Requests?
Thanks, a few notes -- as far as not using AWS ES, we have been having issues with standard-ES, mainly the lack of encryption and authentication without the X-Pack (yes, I could proxy but that seems like a band-aid).. note that AWS VPCs are not encrypted internally, and any machine that you want to have updates must have a public-facing IP address (no VPC-only). This could be done using the AWS ES SDK, but it seems like that's more trouble than it's worth...the AWS ES endpoints are identical, yet support the additional signing attribute. It would seem unnecessary to have two disjoint implementations, rather than just adding the signatures to the request...similar to this (I found this on Google, so no guarantees!) https://gist.github.com/missingfaktor/664b01b7989e91033657e2e703f14ed6 . I think the only piece of the AWS SDK this would require would be the things to get the credentials, and the signing part itself. The new ES client service looks much easier to use than the previous one. Is this going to eventually be used by the existing processors (ex. FetchEsHttp, PutEsHttp, etc)? I suppose this could be even have a TransportClient implementation and merge the FetchEs FetchEsHttp and FetchEs5 (and put, etc) processors? On Mon, Apr 9, 2018 at 9:47 PM, Mike Thomsen wrote: > Related, interesting take on AWS ES: > > https://code972.com/blog/2017/12/111-why-you-shouldnt-use- > aws-elasticsearch-service > > I saw some other critical posts that went into detail on other issues. > Considering how easy it is to stand up ES yourself, I think it would be > worthwhile to strongly consider going that route. > > On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler > wrote: > > > Sorry, just like the gateway api class, it is for managing ES not > calling. > > > > > > On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > > > The aws java sdk has a purpose built ElasticSearchClient class. > > The way to do this and be consistent would be to add a new nifi-aws > > processor for the ES client, > > as there is for s3 and dynamoDB etc. > > > > https://github.com/apache/nifi/pull/2588 is my outstanding PR for > > HttpInvoke for AWS Gateway Web APIs. > > A similar approach being having version of the ElasticSearchHttp but in > the > > aws nar. > > > > > > > > On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote: > > > > Hi All, > > > > We are looking to use the built-in Nifi Elasticsearch Http processors > with > > signed AWS ES requests. I tried to extend them to do so, but ran into > > issues with things being in-extensible (private-static-final in some > > cases), and was wondering if this would be something that would be > > considered to be merged into the baseline? > > > > There's two main ways I saw doing it -- either modifying the existing ES > > code to allow for it, or making new AWS-specific extended adapters to do > > it. In the former, it might require dependencies between the ES code and > > the AWS code for optional CredentialProviders, etc., and am not sure how > > isolated you all try to keep things. > > > > In either case, AWS essentially adds an HTTP-header signature to > > authenticate the request against your ES instance. The easiest path to do > > this seems to be writing a bridge between the ES OkHttpClient requests > and > > the AWS requests to generate the signature correctly. > > > > > > Just wanted to get some feedback to be sure this wouldn't be a waste of > > time. > > > > > > Thanks! > > Jon > > >
Re: Signing AWS Elasticsearch Requests?
Related, interesting take on AWS ES: https://code972.com/blog/2017/12/111-why-you-shouldnt-use-aws-elasticsearch-service I saw some other critical posts that went into detail on other issues. Considering how easy it is to stand up ES yourself, I think it would be worthwhile to strongly consider going that route. On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler wrote: > Sorry, just like the gateway api class, it is for managing ES not calling. > > > On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com) wrote: > > The aws java sdk has a purpose built ElasticSearchClient class. > The way to do this and be consistent would be to add a new nifi-aws > processor for the ES client, > as there is for s3 and dynamoDB etc. > > https://github.com/apache/nifi/pull/2588 is my outstanding PR for > HttpInvoke for AWS Gateway Web APIs. > A similar approach being having version of the ElasticSearchHttp but in the > aws nar. > > > > On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote: > > Hi All, > > We are looking to use the built-in Nifi Elasticsearch Http processors with > signed AWS ES requests. I tried to extend them to do so, but ran into > issues with things being in-extensible (private-static-final in some > cases), and was wondering if this would be something that would be > considered to be merged into the baseline? > > There's two main ways I saw doing it -- either modifying the existing ES > code to allow for it, or making new AWS-specific extended adapters to do > it. In the former, it might require dependencies between the ES code and > the AWS code for optional CredentialProviders, etc., and am not sure how > isolated you all try to keep things. > > In either case, AWS essentially adds an HTTP-header signature to > authenticate the request against your ES instance. The easiest path to do > this seems to be writing a bridge between the ES OkHttpClient requests and > the AWS requests to generate the signature correctly. > > > Just wanted to get some feedback to be sure this wouldn't be a waste of > time. > > > Thanks! > Jon >
Re: Signing AWS Elasticsearch Requests?
Sorry, just like the gateway api class, it is for managing ES not calling. On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com) wrote: The aws java sdk has a purpose built ElasticSearchClient class. The way to do this and be consistent would be to add a new nifi-aws processor for the ES client, as there is for s3 and dynamoDB etc. https://github.com/apache/nifi/pull/2588 is my outstanding PR for HttpInvoke for AWS Gateway Web APIs. A similar approach being having version of the ElasticSearchHttp but in the aws nar. On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote: Hi All, We are looking to use the built-in Nifi Elasticsearch Http processors with signed AWS ES requests. I tried to extend them to do so, but ran into issues with things being in-extensible (private-static-final in some cases), and was wondering if this would be something that would be considered to be merged into the baseline? There's two main ways I saw doing it -- either modifying the existing ES code to allow for it, or making new AWS-specific extended adapters to do it. In the former, it might require dependencies between the ES code and the AWS code for optional CredentialProviders, etc., and am not sure how isolated you all try to keep things. In either case, AWS essentially adds an HTTP-header signature to authenticate the request against your ES instance. The easiest path to do this seems to be writing a bridge between the ES OkHttpClient requests and the AWS requests to generate the signature correctly. Just wanted to get some feedback to be sure this wouldn't be a waste of time. Thanks! Jon
Re: Signing AWS Elasticsearch Requests?
Jon, In 1.6, we added a new client service for ES. There's only one processor that uses it for now, JsonQueryElasticSearch, but I have more under way that I hope will make it into 1.7. If you take a look at the existing version of the client service implementation that uses the official ElasticSearch APIs. You might be able to quickly get this up and running with a new implementation that uses the AWS SDK instead of the official API. FYI, JsonQueryElasticSearch is designed so you can build a query in Kibana and just drop it into NiFi and go. So any querying other than scrolls should work out of the box if you create a proper implementation of ElasticSearchClientService that uses the Amazon code. On Mon, Apr 9, 2018 at 8:29 PM, Otto Fowler wrote: > The aws java sdk has a purpose built ElasticSearchClient class. > The way to do this and be consistent would be to add a new nifi-aws > processor for the ES client, > as there is for s3 and dynamoDB etc. > > https://github.com/apache/nifi/pull/2588 is my outstanding PR for > HttpInvoke for AWS Gateway Web APIs. > A similar approach being having version of the ElasticSearchHttp but in the > aws nar. > > > > On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote: > > Hi All, > > We are looking to use the built-in Nifi Elasticsearch Http processors with > signed AWS ES requests. I tried to extend them to do so, but ran into > issues with things being in-extensible (private-static-final in some > cases), and was wondering if this would be something that would be > considered to be merged into the baseline? > > There's two main ways I saw doing it -- either modifying the existing ES > code to allow for it, or making new AWS-specific extended adapters to do > it. In the former, it might require dependencies between the ES code and > the AWS code for optional CredentialProviders, etc., and am not sure how > isolated you all try to keep things. > > In either case, AWS essentially adds an HTTP-header signature to > authenticate the request against your ES instance. The easiest path to do > this seems to be writing a bridge between the ES OkHttpClient requests and > the AWS requests to generate the signature correctly. > > > Just wanted to get some feedback to be sure this wouldn't be a waste of > time. > > > Thanks! > Jon >
Re: Signing AWS Elasticsearch Requests?
The aws java sdk has a purpose built ElasticSearchClient class. The way to do this and be consistent would be to add a new nifi-aws processor for the ES client, as there is for s3 and dynamoDB etc. https://github.com/apache/nifi/pull/2588 is my outstanding PR for HttpInvoke for AWS Gateway Web APIs. A similar approach being having version of the ElasticSearchHttp but in the aws nar. On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote: Hi All, We are looking to use the built-in Nifi Elasticsearch Http processors with signed AWS ES requests. I tried to extend them to do so, but ran into issues with things being in-extensible (private-static-final in some cases), and was wondering if this would be something that would be considered to be merged into the baseline? There's two main ways I saw doing it -- either modifying the existing ES code to allow for it, or making new AWS-specific extended adapters to do it. In the former, it might require dependencies between the ES code and the AWS code for optional CredentialProviders, etc., and am not sure how isolated you all try to keep things. In either case, AWS essentially adds an HTTP-header signature to authenticate the request against your ES instance. The easiest path to do this seems to be writing a bridge between the ES OkHttpClient requests and the AWS requests to generate the signature correctly. Just wanted to get some feedback to be sure this wouldn't be a waste of time. Thanks! Jon
Signing AWS Elasticsearch Requests?
Hi All, We are looking to use the built-in Nifi Elasticsearch Http processors with signed AWS ES requests. I tried to extend them to do so, but ran into issues with things being in-extensible (private-static-final in some cases), and was wondering if this would be something that would be considered to be merged into the baseline? There's two main ways I saw doing it -- either modifying the existing ES code to allow for it, or making new AWS-specific extended adapters to do it. In the former, it might require dependencies between the ES code and the AWS code for optional CredentialProviders, etc., and am not sure how isolated you all try to keep things. In either case, AWS essentially adds an HTTP-header signature to authenticate the request against your ES instance. The easiest path to do this seems to be writing a bridge between the ES OkHttpClient requests and the AWS requests to generate the signature correctly. Just wanted to get some feedback to be sure this wouldn't be a waste of time. Thanks! Jon
Re: Update docker tag 'latest' to nifi 1.6.0
Joe, A person would need to pull the image version: docker pull apache/nifi: Tag it as latest: docker tag apache/nifi: apache/nifi:latest and then push this tag to Docker Hub: docker push apache/nifi:latest Pushing to Docker Hub is only possible for those folks in the PMC who opted in and provided their Docker Hub usernames as part of the requested JIRA [1]. I have done it for 1.6.0 to ensure I had the correct sequence listed here. [1] https://issues.apache.org/jira/browse/INFRA-15526 On Mon, Apr 9, 2018 at 8:44 AM, Joe Witt wrote: > Aldrin > > As noted in the announce thread our docker tag for latest is not > updated. I see examples of how to do it but I want to make sure I > dont mess it up. Do you have some instructions to share? I can put > them in the release guide. > > Thanks >
Re: Programmatically building a nested schema
Add a RecordField to the RecordSchema where the DataType is a RecordDataType... a RecordDataType then has a child schema. May be helpful to look at the code that converts between RecordSceham and Avro schemas: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java#L195-L205 On Sun, Apr 8, 2018 at 6:05 PM, Mike Thomsen wrote: > Can someone tell me how to build a nested schema using RecordSchema and > RecordField objects? The use case here is to make MongoDBLookupService able > to roughly detect the schema from the returned bson Document. > > Thanks, > > Mike
Update docker tag 'latest' to nifi 1.6.0
Aldrin As noted in the announce thread our docker tag for latest is not updated. I see examples of how to do it but I want to make sure I dont mess it up. Do you have some instructions to share? I can put them in the release guide. Thanks
Re: [ANNOUNCE] Apache NiFi 1.6.0 release
I think the latest tag is not updated. I just switched from pull apache/nifi:latest to nifi:1.6.0 and got it. On Mon, Apr 9, 2018 at 8:15 AM, Pierre Villard wrote: > Mike, > > It's up on my side: https://hub.docker.com/r/apache/nifi/builds/ > > Pierre > > 2018-04-09 12:28 GMT+02:00 Mike Thomsen : > > > Any word on when the Docker repository is going to be updated? > > > > Thanks, > > > > Mike > > > > On Sun, Apr 8, 2018 at 10:46 PM, Joe Witt wrote: > > > > > Hello > > > > > > The Apache NiFi team would like to announce the release of Apache NiFi > > > 1.6.0. > > > > > > Apache NiFi is an easy to use, powerful, and reliable system to > > > process and distribute > > > data. Apache NiFi was made for dataflow. It supports highly > > > configurable directed graphs > > > of data routing, transformation, and system mediation logic. > > > > > > More details on Apache NiFi can be found here: > > > https://nifi.apache.org/ > > > > > > The release artifacts can be downloaded from here: > > > https://nifi.apache.org/download.html > > > > > > Release note highlights can be found here: > > > https://cwiki.apache.org/confluence/display/NIFI/ > > > Release+Notes#ReleaseNotes-Version1.6.0 > > > > > > Maven artifacts have been made available here: > > > https://repository.apache.org/content/repositories/releases/ > > > org/apache/nifi/ > > > > > > Issues closed/resolved for this list can be found here: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > projectId=12316020&version=12342422 > > > > > > Thank you > > > The Apache NiFi team > > > > > >
Re: [ANNOUNCE] Apache NiFi 1.6.0 release
Mike, It's up on my side: https://hub.docker.com/r/apache/nifi/builds/ Pierre 2018-04-09 12:28 GMT+02:00 Mike Thomsen : > Any word on when the Docker repository is going to be updated? > > Thanks, > > Mike > > On Sun, Apr 8, 2018 at 10:46 PM, Joe Witt wrote: > > > Hello > > > > The Apache NiFi team would like to announce the release of Apache NiFi > > 1.6.0. > > > > Apache NiFi is an easy to use, powerful, and reliable system to > > process and distribute > > data. Apache NiFi was made for dataflow. It supports highly > > configurable directed graphs > > of data routing, transformation, and system mediation logic. > > > > More details on Apache NiFi can be found here: > > https://nifi.apache.org/ > > > > The release artifacts can be downloaded from here: > > https://nifi.apache.org/download.html > > > > Release note highlights can be found here: > > https://cwiki.apache.org/confluence/display/NIFI/ > > Release+Notes#ReleaseNotes-Version1.6.0 > > > > Maven artifacts have been made available here: > > https://repository.apache.org/content/repositories/releases/ > > org/apache/nifi/ > > > > Issues closed/resolved for this list can be found here: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > projectId=12316020&version=12342422 > > > > Thank you > > The Apache NiFi team > > >
Re: Host a web service on Nifi to be able to consume by third party systems
Not really. The embedded Jetty container is meant to host NiFi web components, not third party apps. There's no real reason to deploy a WAR file there because you could just spin up Tomcat, drop the SOAP service there and have them both accessible through a reverse proxy. Thanks, Mike On Mon, Apr 9, 2018 at 7:59 AM, pksrinivasm wrote: > Hi, > > I would like to host SOAP web service in NIFI. My third party system > accepts > only SOAP endpoints to be consumed. Is there a way to achieve? > > Thanks, > Kalyan > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/ >
Hosting SOAP web service on NIFI
Hi, I would like to host a SOAP web service in NIFI. Is it possible to do? My third party system accepts only SOAP messages. Any help would be highly appreciated. Thanks, Kalyan -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Host a web service on Nifi to be able to consume by third party systems
Hi, I would like to host SOAP web service in NIFI. My third party system accepts only SOAP endpoints to be consumed. Is there a way to achieve? Thanks, Kalyan -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: [ANNOUNCE] Apache NiFi 1.6.0 release
Any word on when the Docker repository is going to be updated? Thanks, Mike On Sun, Apr 8, 2018 at 10:46 PM, Joe Witt wrote: > Hello > > The Apache NiFi team would like to announce the release of Apache NiFi > 1.6.0. > > Apache NiFi is an easy to use, powerful, and reliable system to > process and distribute > data. Apache NiFi was made for dataflow. It supports highly > configurable directed graphs > of data routing, transformation, and system mediation logic. > > More details on Apache NiFi can be found here: > https://nifi.apache.org/ > > The release artifacts can be downloaded from here: > https://nifi.apache.org/download.html > > Release note highlights can be found here: > https://cwiki.apache.org/confluence/display/NIFI/ > Release+Notes#ReleaseNotes-Version1.6.0 > > Maven artifacts have been made available here: > https://repository.apache.org/content/repositories/releases/ > org/apache/nifi/ > > Issues closed/resolved for this list can be found here: > https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12316020&version=12342422 > > Thank you > The Apache NiFi team >
