Re: Elasticsearch connector support?

Hi HangThanks for the link. I will wait for 3.1 connector release and hope it will be included. Med venlig hilsen / Best regardsLasse NedergaardDen 27. nov. 2023 kl. 12.00 skrev Hang Ruan :Hi, Lasse.There is already a discussion about the connector releases for 1.18[1].Best,Hang[1] 

Re: Elasticsearch connector support?

Hi, Lasse. There is already a discussion about the connector releases for 1.18[1]. Best, Hang [1] https://lists.apache.org/thread/r31f988m57rtjy4s75030pzwrlqybpq2 Lasse Nedergaard 于2023年11月24日周五 22:57写道： > Hi > > From the documentation I can see there isn’t any ES support in Flink 1.18 >

Elasticsearch connector support?

Hi From the documentation I can see there isn’t any ES support in Flink 1.18 right now and Flink-26088 (ES 8 support) is still open. Does anyone has an idea when ES connector support will be available in 1.18 Please let me know. Med venlig hilsen / Best regards Lasse Nedergaard

Flink Streaming API ElasticSearch Connector 长连接

各位大佬好！ 我在使用Flink ES连接器的时候，有时候报以下错误： Caused by: java.io.IOException breakpoint : 远程主机强迫关闭了一个现有的连接 初步判断，应该是没有维持住长连接保活，所以如果一段时间不写入数据，连接就断了。 请问各位大佬，ElasticSearch Connector 有什么参数可以维持长连接吗？ ElasticSearch Connector 代码如下： jsonStringStream .sinkTo( new Elasticsearch7SinkBuilder

Re: Elasticsearch Connector 1.17 not on maven central.

>>> flink-connector-elasticsearch7 >>> 3.0.0-1.17 >>> >> >> >> It seems that currently the latest version for elasticsearch connector >> on maven central repository is 3.0.0-1.16( >> https://central.sonatype.com/artifact/org.apache.flink/flink-connector-elasticsearch7/3.0.0-1.16). >> I wonder if something was missed out. >> >> Thanks! >> >

Re: Elasticsearch Connector 1.17 not on maven central.

lease-1.17/docs/connectors/datastream/elasticsearch/, > maven can't find the required version: > >> >> org.apache.flink >> flink-connector-elasticsearch7 >> 3.0.0-1.17 >> > > > It seems that currently the latest version for elasticsearch co

Elasticsearch Connector 1.17 not on maven central.

7 > 3.0.0-1.17 > It seems that currently the latest version for elasticsearch connector on maven central repository is 3.0.0-1.16( https://central.sonatype.com/artifact/org.apache.flink/flink-connector-elasticsearch7/3.0.0-1.16). I wonder if something was missed out. Thanks!

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

nvolved! > > On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler > wrote: > >> The Apache Flink community is very happy to announce the release of >> Apache Flink Elasticsearch Connector 3.0.0. >> >> Apache Flink® is an open-source stream processing framewo

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

nvolved! > > On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler > wrote: > >> The Apache Flink community is very happy to announce the release of >> Apache Flink Elasticsearch Connector 3.0.0. >> >> Apache Flink® is an open-source stream processing framewo

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

lly happy with the first externalized connector for Flink. Thanks a lot to > all of you involved! > > On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink Elasticse

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

lly happy with the first externalized connector for Flink. Thanks a lot to > all of you involved! > > On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler <mailto:ches...@apache.org>> wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink Elasticse

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

Really happy with the first externalized connector for Flink. Thanks a lot to all of you involved! On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink Elasticsearch Connector 3.0.0. > >

Re: [ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

Really happy with the first externalized connector for Flink. Thanks a lot to all of you involved! On Thu, Nov 10, 2022 at 12:51 PM Chesnay Schepler wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink Elasticsearch Connector 3.0.0. > >

[ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Elasticsearch Connector 3.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available

[ACCOUNCE] Apache Flink Elasticsearch Connector 3.0.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Elasticsearch Connector 3.0.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available

Re: Flink Elasticsearch connector 7 with SSL

Hi Shishal, Please don't cross-post your question in multiple channels. As answered in Slack, SSL connections to Elasticsearch7 are currently not supported and are tracked under https://issues.apache.org/jira/browse/FLINK-27054 An Elasticsearch8 connector is being tracked under

Flink Elasticsearch connector 7 with SSL

Hi Flink community members, In my flink job, I am trying to use *elasticsearch7* connector. I could not find a way to communicate over SSL when using *Elasticsearch7SinkBuilder* Am I missing something? Also there is an deprecated class *ElasticsearchSink.Builder* where it has *RestClientFactory*

Re: Number of parallel connections for Elasticsearch Connector

-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBase.java#L204 > . > > Best, > Yangze Guo > > On Sun, Jan 17, 2021 at 11:33 AM Rex Fenley wrote: > > > > I found the following, indicating that there is no concurrency for th

Re: Number of parallel connections for Elasticsearch Connector

Guo On Sun, Jan 17, 2021 at 11:33 AM Rex Fenley wrote: > > I found the following, indicating that there is no concurrency for the > Elasticsearch Connector > https://github.com/apache/flink/blob/97bfd049951f8d52a2e0aed14265074c4255ead0/flink-connectors/flink-connector-elasticsearch-bas

Re: Number of parallel connections for Elasticsearch Connector

I found the following, indicating that there is no concurrency for the Elasticsearch Connector https://github.com/apache/flink/blob/97bfd049951f8d52a2e0aed14265074c4255ead0/flink-connectors/flink-connector-elasticsearch-base/src/main/java/org/apache/flink/streaming/connectors/elasticsearch

Number of parallel connections for Elasticsearch Connector

Hello, How many connections does the ES connector use to write to Elasticsearch? We have a single machine with 16 vCPUs and parallelism of 4 running our job, with -p 4 I'd expect there to be 4 parallel bulk request writers / connections to Elasticsearch. Is there a place in the code to confirm

Re: flink1.11 elasticsearch connector

020 at 3:37 PM 赵帅 wrote: > > > > elasticsearch7.6有账号认证，目前flink1.11 elasticsearch connector sql > api如何加入账号认证？ >

Re: flink1.11 elasticsearch connector

1.11版本中尚不支持username和password的设置，这两个配置在1.12中加入了新的es connector[1] [1] https://issues.apache.org/jira/browse/FLINK-18361 Best, Yangze Guo On Thu, Oct 29, 2020 at 3:37 PM 赵帅 wrote: > > elasticsearch7.6有账号认证，目前flink1.11 elasticsearch connector sql api如何加入账号认证？

flink1.11 elasticsearch connector

elasticsearch7.6有账号认证，目前flink1.11 elasticsearch connector sql api如何加入账号认证？

Re: ElasticSearch Connector

Wysakowicz > mailto:dwysakow...@apache.org>> wrote: > > Hi, > > I'm afraid you cannot write to different indices using the table > API ElasticSearch connector. Now I know why you wanted to go > through dat

Re: Secure TLS/SSL ElasticSearch connector for current and future connector

Hi Christophe, Thanks so much for the pointers. That helps. Looking at the latest update on https://issues.apache.org/jira/browse/FLINK-8101 , there was an issue related to HLR retry handling. If I read this correctly, there is a bug in ES/HLR

Re: Secure TLS/SSL ElasticSearch connector for current and future connector

Hi Fritz, I think the High Level Rest Client implementation in this PR: https://github.com/apache/flink/pull/5374 should work. If you don't get the certificate properly available in your Java certs, you might want to redefine the createClient method to do something along those lines to get the

Secure TLS/SSL ElasticSearch connector for current and future connector

Hi All, Anyone know if Flink has TLS/SSL support for the current ES connector ? If yes, any sample configuration/code ? If not, would TLS/SSL be support in the upcoming ES connector using Java High Level client ? Thanks, Fritz

Re: ElasticSearch Connector for version 6.x and scala 2.11

Actually, Flink's netty dependency (4.0.27) is shaded away into the "org.apache.flink.shaded.netty4.io.netty" package now (since version 1.4) and should thus not clash anymore. However, other netty versions may come into play from the job itself or from the integration of Hadoop's classpath (if

Re: ElasticSearch Connector for version 6.x and scala 2.11

Hi A workmate of mine tried to migrate the existing flink connector to ElasticSearch 6 but we had problems with netty dependencies that clashed (Flink uses 4.0.27 and ES is on 4.1). You can change the flink-connector-elasticsearch5 connector to ES 5.6.4, but then you have to do some adaptions to

Re: ElasticSearch Connector for version 6.x and scala 2.11

Hi Rahul, Flink does not provide a connector for ElasticSearch 6 yet. There is this JIRA issue to track the development progress [1]. Best, Fabian [1] https://issues.apache.org/jira/browse/FLINK-8101 2017-12-01 7:22 GMT+01:00 Rahul Raj : > Hi All, > > Is there a Flink

ElasticSearch Connector for version 6.x and scala 2.11

Hi All, Is there a Flink Elastic search connector for version 6.0 and scala 2.11? I couldn't find it listed here https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html . Regards, Rahul Raj

Re: Flink Elasticsearch Connector: Lucene Error message

all lucene and elasticsearch libs in your flink env and just use maven to manage dependencies, when working with the flink elasticsearch connector. so in the first place i deleted the libs in the folder to see if its working, but it did not. then we thought if maybe flink loads already the libs

Re: Flink Elasticsearch Connector: Lucene Error message

TL;DR: remove all lucene and elasticsearch libs in your flink env and just use maven to manage dependencies, when working with the flink elasticsearch connector. so in the first place i deleted the libs in the folder to see if its working, but it did not. then we thought if maybe flink loads

Re: Flink Elasticsearch Connector: Lucene Error message

> wrote: Hi Timo, Hi Gordon, thx for the reply! I checked the connection from both clusters to each other, and i can telnet to the 9300 port of flink, so i think the connection is not an issue here.  We are currently using in our live env a custom elasticsearch connector, which used s

Re: Flink Elasticsearch Connector: Lucene Error message

How are you running it on the cluster? Standalone, YARN? >> >> Best, >> Aljoscha >> >> On 13. Jul 2017, at 13:56, Fabian Wollert <fabian.woll...@zalando.de> >> wrote: >> >> Hi Timo, Hi Gordon, >> >> thx for the reply! I checked the connectio

Re: Flink Elasticsearch Connector: Lucene Error message

o:fabian.woll...@zalando.de>> wrote: >> >> Hi Timo, Hi Gordon, >> >> thx for the reply! I checked the connection from both clusters to each >> other, and i can telnet to the 9300 port of flink, so i think the connection >> is not an issue here. >&g

Re: Flink Elasticsearch Connector: Lucene Error message

t to the 9300 port of flink, so i think the > connection is not an issue here. > > We are currently using in our live env a custom elasticsearch connector, > which used some extra lib's deployed on the cluster. i found one lucene lib > and deleted it (since all dependencies should

Re: Flink Elasticsearch Connector: Lucene Error message

telnet to the 9300 port of flink, so i think the connection is not > an issue here. > > We are currently using in our live env a custom elasticsearch connector, > which used some extra lib's deployed on the cluster. i found one lucene lib > and deleted it (since all depe

Re: Flink Elasticsearch Connector: Lucene Error message

Hi Timo, Hi Gordon, thx for the reply! I checked the connection from both clusters to each other, and i can telnet to the 9300 port of flink, so i think the connection is not an issue here. We are currently using in our live env a custom elasticsearch connector, which used some extra lib's

Re: Flink Elasticsearch Connector: Lucene Error message

Hi Fabian, I loop in Gordon. Maybe he knows whats happening here. Regards, Timo Am 13.07.17 um 13:26 schrieb Fabian Wollert: Hi everyone, I'm trying to make use of the new Elasticsearch Connector <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.h

Re: Elasticsearch connector and number of shards

I've just added a JIRA improvement ticket for this ( https://issues.apache.org/jira/browse/FLINK-4491). Best, Flavio On Wed, Jul 20, 2016 at 4:21 PM, Maximilian Michels wrote: > The connector doesn't cover this use case. Through the API you need to > use the

Re: Elasticsearch connector and number of shards

The connector doesn't cover this use case. Through the API you need to use the IndicesAdminClient: https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html Otherwise Elasticsearch creates an index with shards automatically. We could add support for configuring

Re: Elasticsearch connector and number of shards

Indeed, we've tried with the parameter *index.number_of_shards* but it didn't work so I fear that this parameter is not handled by the current implementation..am I wrong? On Mon, Jul 18, 2016 at 11:37 AM, Ufuk Celebi wrote: > I've never used the Elasticsearch sink, but the docs

Re: Elasticsearch connector and number of shards

I've never used the Elasticsearch sink, but the docs say: "Note how a Map of Strings is used to configure the Sink. The configuration keys are documented in the Elasticsearch documentationhere. Especially important is the cluster.name parameter that must correspond to the name of your cluster."

Elasticsearch connector and number of shards

Hi to all, we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we wanted to set the number of shards when creating a new index. Is that possible at the moment? Best, Flavio

Re: Elasticsearch Connector

, Till ​ On Thu, Jun 16, 2016 at 5:14 PM, Eamon Kavanagh <kavanagh.c.ea...@gmail.com> wrote: > Hey Support, > > I'm trying to use Flink's Elasticsearch connector but I'm having trouble. > When I add the dependency seen here ( > https://ci.apache.org/projects/flink/flink-docs-

Elasticsearch Connector

Hey Support, I'm trying to use Flink's Elasticsearch connector but I'm having trouble. When I add the dependency seen here ( https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/connectors/elasticsearch2.html) to my pom file, IntelliJ can't find it. I also can't find

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

gt;> Glad to see it's developing. >>> Can I ask would the same feature (reconnect) be useful for Kafka >>> connector ? >>> For example, if the IP of broker changes. >>> >>> >>> >>> -- >>> View this message in context: >

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

> >> >> -- >> View this message in context: >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Any-way-for-Flink-Elasticsearch-connector-reflecting-IP-change-of-Elasticsearch-cluster-tp6597p6688.html >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> > >

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

ul for Kafka connector > ? > For example, if the IP of broker changes. > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Any-way-for-Flink-Elasticsearch-connector-reflecting-IP-change-of-Elasticsearch-cluster-tp6597p66

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

Glad to see it's developing. Can I ask would the same feature (reconnect) be useful for Kafka connector ? For example, if the IP of broker changes. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Any-way-for-Flink-Elasticsearch-connector

Re: Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

? 2016-05-02 9:38 GMT+02:00 Sendoh <unicorn.bana...@gmail.com>: > Hi, > > When using Elasticsearch connector, Is there a way to reflect IP change of > Elasticsearch cluster? > We use DNS of Elasticsearch in data sink, e.g. elasticsearch-dev.foo.de. > However, when we repla

Any way for Flink Elasticsearch connector reflecting IP change of Elasticsearch cluster?

Hi, When using Elasticsearch connector, Is there a way to reflect IP change of Elasticsearch cluster? We use DNS of Elasticsearch in data sink, e.g. elasticsearch-dev.foo.de. However, when we replace the old Elasticsearch cluster with a new one, the Elasticsearch connector cannot write

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Hi Madhu, Not yet. The API has changed slightly. We'll add one very soon. In the meantime I've created an issue to keep track of the status: https://issues.apache.org/jira/browse/FLINK-3115 Thanks, Max On Thu, Dec 3, 2015 at 10:50 PM, Madhukar Thota wrote: > is

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Hi Madhu, Great. Do you want to contribute it back via a GitHub pull request? If not that's also fine. We will try look into the 2.0 connector next week. Best, Max On Fri, Dec 4, 2015 at 4:16 PM, Madhukar Thota wrote: > i have created working connector for

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Sure. I can submit the pull request. On Fri, Dec 4, 2015 at 12:37 PM, Maximilian Michels wrote: > Hi Madhu, > > Great. Do you want to contribute it back via a GitHub pull request? If > not that's also fine. We will try look into the 2.0 connector next > week. > > Best, > Max >

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

shouldn't be better to have both connectors for ES?one for 1.x and another for 2.x? On 4 Dec 2015 20:55, "Madhukar Thota" wrote: > Sure. I can submit the pull request. > > On Fri, Dec 4, 2015 at 12:37 PM, Maximilian Michels > wrote: > >> Hi Madhu, >>

Re: Flink-Elasticsearch connector support for elasticsearch 2.0

Hi Madhu Would you be able to provide the use case here in ElasticSearch with Flink? Thanks Deepak On Sat, Dec 5, 2015 at 1:25 AM, Madhukar Thota wrote: > Sure. I can submit the pull request. > > On Fri, Dec 4, 2015 at 12:37 PM, Maximilian Michels >