Hi, We have recently moved to AWS ES service, I am using the following code:
https://github.com/awslabs/flink-stream-processing-refarch/blob/master/flink-taxi-stream-processor/src/main/java/com/amazonaws/flink/refarch/utils/ElasticsearchJestSink.java (Note that this is not the inbuilt Flink ESSink) You can read this blog post: https://aws.amazon.com/blogs/big-data/build-a-real-time-stream-processing-pipeline-with-apache-flink-on-aws/ Regards, Vinay Patil On Sun, Aug 27, 2017 at 7:02 PM, ant burton [via Apache Flink User Mailing List archive.] <ml+s2336050n15173...@n4.nabble.com> wrote: > Thanks! I'll check later this evening. > > On Sun, 27 Aug 2017 at 07:44, arpit srivastava <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=15173&i=0>> wrote: > >> We also had same setup where ES cluster was behind a proxy server for >> which port 80 was used which redirected it to ES cluster 9200 port. >> >> For using Flink we got the actual ip address of the ES nodes and put that >> in ips below. >> >> transportAddresses.add(new >> InetSocketAddress(InetAddress.getByName("127.0.0.1"), >> 9300))transportAddresses.add(new >> InetSocketAddress(InetAddress.getByName("10.2.3.1"), 9300)) >> >> But this worked only because 9300 port was open on ES nodes in our setup >> and so accessible from our Flink cluster. >> >> Get your node list on your ES Cluster using >> >> curl -XGET 'http://<your aws es url>/_nodes' >> >> >> >> and then check whether you can telnet on that <es node ip> on port 9300 >> from your flink cluster nodes >> >> $ *telnet <es node ip> 9300* >> >> If this works then you can use above solution. >> >> >> On Sun, Aug 27, 2017 at 4:09 AM, ant burton <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=15173&i=1>> wrote: >> >>> Hi Ted, >>> >>> Changing the port from 9300 to 9200 in the example you provides causes >>> the error in the my original message >>> >>> my apologies for not providing context in the form of code in my >>> original message, to confirm I am using the example you provided in my >>> application and have it working using port 9300 in a docker environment >>> locally. >>> >>> Thanks, >>> >>> On 26 Aug 2017, at 23:24, Ted Yu <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=15173&i=2>> wrote: >>> >>> If port 9300 in the following example is replaced by 9200, would that >>> work ? >>> >>> https://ci.apache.org/projects/flink/flink-docs- >>> release-1.3/dev/connectors/elasticsearch.html >>> >>> Please use Flink 1.3.1+ >>> >>> On Sat, Aug 26, 2017 at 3:00 PM, ant burton <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=15173&i=3>> wrote: >>> >>>> Hello, >>>> >>>> Has anybody been able to use the Flink Elasticsearch connector to sink >>>> data to AWS ES. >>>> >>>> I don’t believe this is possible as AWS ES only allows access to port >>>> 9200 (via port 80) on the master node of the ES cluster, and not port 9300 >>>> used by the the Flink Elasticsearch connector. >>>> >>>> The error message that occurs when attempting to connect to AWS ES via >>>> port 80 (9200) with the Flink Elasticsearch connector is: >>>> >>>> Elasticsearch client is not connected to any Elasticsearch nodes! >>>> >>>> Could anybody confirm the above? and if possible provide an alternative >>>> solution? >>>> >>>> Thanks you, >>> >>> >>> >>> >> > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/Flink-Elastic-Sink-AWS-ES-tp15162p15173.html > To start a new topic under Apache Flink User Mailing List archive., email > ml+s2336050n1...@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Elastic-Sink-AWS-ES-tp15162p15174.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
