Re: Flink Elastic Sink

Hi, aj > I was confused before as I was thinking the sink builder is called only once > but it gets called for every batch request, correct me if my understanding is > wrong. You’re right that sink builder should be called only once rather than every batch requests, could you post some code

Re: Flink Elastic Sink

Thanks, It worked. I was confused before as I was thinking the sink builder is called only once but it gets called for every batch request, correct me if my understanding is wrong. On Fri, May 29, 2020 at 9:08 AM Leonard Xu wrote: > Hi，aj > > In the implementation of ElasticsearchSink,

Re: Flink Elastic Sink

Hi，aj In the implementation of ElasticsearchSink, ElasticsearchSink won't create index and only start a Elastic client for sending requests to the Elastic cluster. You can simply extract the index(date value in your case) from your timestamp field and then put it to an IndexRequest[2],

Re: Flink Elastic Sink

Hi, Anuj. >From my understanding, you could send IndexRequest to the indexer in `ElasticsearchSink`. It will create a document under the given index and type. So, it seems you only need to get the timestamp and concat the `date` to your index. Am I understanding that correctly? Or do you want to

Flink Elastic Sink

Hello All, I am getting many events in Kafka and I have written a link job that sinks that Avro records from Kafka to S3 in parquet format. Now, I want to sink these records into elastic search. but the only challenge is that I want to sink record on time indices. Basically, In Elastic, I want

Re: Connection leak with flink elastic Sink

/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java#L334 >> >> On 13 December 2018 at 5:59:34 PM, Chesnay Schepler (ches...@apache.org) >> wrote: >> >> Specifically which connector are you using, and which Flink version? >> >> On 12.12

Re: Connection leak with flink elastic Sink

Flink version? On 12.12.2018 13:31, Vijay Bhaskar wrote: > Hi > We are using flink elastic sink which streams at the rate of 1000 > events/sec, as described in > https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html. > We are observing connection

Re: Connection leak with flink elastic Sink

13 December 2018 at 5:59:34 PM, Chesnay Schepler (ches...@apache.org) > wrote: > > Specifically which connector are you using, and which Flink version? > > On 12.12.2018 13:31, Vijay Bhaskar wrote: > > Hi > > We are using flink elastic sink which streams at the rate of 1000

Re: Connection leak with flink elastic Sink

/ElasticsearchSinkBase.java#L334 On 13 December 2018 at 5:59:34 PM, Chesnay Schepler (ches...@apache.org) wrote: Specifically which connector are you using, and which Flink version? On 12.12.2018 13:31, Vijay Bhaskar wrote: > Hi > We are using flink elastic sink which streams at th

Re: Connection leak with flink elastic Sink

Specifically which connector are you using, and which Flink version? On 12.12.2018 13:31, Vijay Bhaskar wrote: Hi We are using flink elastic sink which streams at the rate of 1000 events/sec, as described in https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors

Re: Connection leak with flink elastic Sink

Hi Bhaskar, I think Gordon might help you, I am pulling him into the discussion. Best, Andrey > On 12 Dec 2018, at 13:31, Vijay Bhaskar wrote: > > Hi > We are using flink elastic sink which streams at the rate of 1000 events/sec, > as described in > https://ci.apache.

Connection leak with flink elastic Sink

Hi We are using flink elastic sink which streams at the rate of 1000 events/sec, as described in https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html . We are observing connection leak of elastic connections. After few minutes all the open connections

Re: Flink Elastic Sink AWS ES

It seems AWS ES setup is hiding the nodes ip. Then I think you can try @vinay patil's solution. Thanks, Arpit On Tue, Aug 29, 2017 at 3:56 AM, ant burton wrote: > Hey Arpit, > > _cat/nodes?v=ip,port > > > returns the following which I have not added the x’s they were

Re: Flink Elastic Sink AWS ES

Hey Arpit, > _cat/nodes?v=ip,port returns the following which I have not added the x’s they were returned on the response ipport x.x.x.x 9300 Thanks your for you help Anthony > On 28 Aug 2017, at 10:34, arpit srivastava wrote: > > Hi Ant, > > Can you try

Re: Flink Elastic Sink AWS ES

Hi Ant, Can you try this. curl -XGET 'http:///_cat/nodes?v=ip,port' This should give you ip and port On Mon, Aug 28, 2017 at 3:42 AM, ant burton wrote: > Hi Arpit, > > The response fromm _nodes doesn’t contain an ip address in my case. Is > this something that you

Re: Flink Elastic Sink AWS ES

Hi Arpit, The response fromm _nodes doesn’t contain an ip address in my case. Is this something that you experienced? > curl -XGET 'http:///_nodes' Thanks, > On 27 Aug 2017, at 14:32, ant burton wrote: > > Thanks! I'll check later this evening. > > On Sun, 27 Aug

Re: Flink Elastic Sink AWS ES

ES. >>>> >>>> I don’t believe this is possible as AWS ES only allows access to port >>>> 9200 (via port 80) on the master node of the ES cluster, and not port 9300 >>>> used by the the Flink Elasticsearch connector. >>>> >>>> The error message that occurs when attem

Re: Flink Elastic Sink AWS ES

We also had same setup where ES cluster was behind a proxy server for which port 80 was used which redirected it to ES cluster 9200 port. For using Flink we got the actual ip address of the ES nodes and put that in ips below. transportAddresses.add(new

Re: Flink Elastic Sink AWS ES

Hi Ted, Changing the port from 9300 to 9200 in the example you provides causes the error in the my original message my apologies for not providing context in the form of code in my original message, to confirm I am using the example you provided in my application and have it working using

Re: Flink Elastic Sink AWS ES

If port 9300 in the following example is replaced by 9200, would that work ? https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html Please use Flink 1.3.1+ On Sat, Aug 26, 2017 at 3:00 PM, ant burton wrote: > Hello, > > Has anybody

Flink Elastic Sink AWS ES

Hello, Has anybody been able to use the Flink Elasticsearch connector to sink data to AWS ES. I don’t believe this is possible as AWS ES only allows access to port 9200 (via port 80) on the master node of the ES cluster, and not port 9300 used by the the Flink Elasticsearch connector. The