[jira] [Created] (FLINK-35374) Flink 1.14 kafka connector Demo Error

2024-05-16 Thread hongxu han (Jira)
hongxu han created FLINK-35374: -- Summary: Flink 1.14 kafka connector Demo Error Key: FLINK-35374 URL: https://issues.apache.org/jira/browse/FLINK-35374 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-35109) Drop support for Flink 1.17 and 1.18 in Flink Kafka connector

2024-04-15 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-35109: -- Summary: Drop support for Flink 1.17 and 1.18 in Flink Kafka connector Key: FLINK-35109 URL: https://issues.apache.org/jira/browse/FLINK-35109 Project: Flink

[jira] [Created] (FLINK-35034) codegen compile error raised when use kafka connector and protobuf format

2024-04-07 Thread yufeng.sun (Jira)
yufeng.sun created FLINK-35034: -- Summary: codegen compile error raised when use kafka connector and protobuf format Key: FLINK-35034 URL: https://issues.apache.org/jira/browse/FLINK-35034 Project: Flink

[jira] [Created] (FLINK-35011) The change in visibility of MockDeserializationSchema cause compilation failure in kafka connector

2024-04-04 Thread Jiabao Sun (Jira)
Jiabao Sun created FLINK-35011: -- Summary: The change in visibility of MockDeserializationSchema cause compilation failure in kafka connector Key: FLINK-35011 URL: https://issues.apache.org/jira/browse/FLINK-35011

[jira] [Created] (FLINK-35008) Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.0 for Flink Kafka connector

2024-04-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-35008: -- Summary: Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.0 for Flink Kafka connector Key: FLINK-35008 URL: https://issues.apache.org/jira/browse/FLINK-35008

[jira] [Created] (FLINK-35007) Update Flink Kafka connector to support 1.19 and test 1.20-SNAPSHOT

2024-04-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-35007: -- Summary: Update Flink Kafka connector to support 1.19 and test 1.20-SNAPSHOT Key: FLINK-35007 URL: https://issues.apache.org/jira/browse/FLINK-35007 Project

[jira] [Created] (FLINK-34995) flink kafka connector source stuck when partition leader invalid

2024-04-03 Thread yansuopeng (Jira)
yansuopeng created FLINK-34995: -- Summary: flink kafka connector source stuck when partition leader invalid Key: FLINK-34995 URL: https://issues.apache.org/jira/browse/FLINK-34995 Project: Flink

[jira] [Created] (FLINK-34466) Implement Lineage Interface in Kafka Connector

2024-02-19 Thread Zhenqiu Huang (Jira)
Zhenqiu Huang created FLINK-34466: - Summary: Implement Lineage Interface in Kafka Connector Key: FLINK-34466 URL: https://issues.apache.org/jira/browse/FLINK-34466 Project: Flink Issue Type

[jira] [Created] (FLINK-34320) Flink Kafka connector tests time out

2024-01-31 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-34320: -- Summary: Flink Kafka connector tests time out Key: FLINK-34320 URL: https://issues.apache.org/jira/browse/FLINK-34320 Project: Flink Issue Type: Bug

Re: [DISCUSS] Release new version of Flink's Kafka connector

2024-01-26 Thread Hang Ruan
ersioning [1]. > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > < > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > > > > > > Best, > > Mason > &g

Re: [DISCUSS] Release new version of Flink's Kafka connector

2024-01-26 Thread Leonard Xu
ng [1]. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > > <https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development> > > Best, > Mason > > On Thu, Jan 25, 2024 at 2:16 PM Martijn Visser

Re: [DISCUSS] Release new version of Flink's Kafka connector

2024-01-26 Thread Martijn Visser
onnector versioning [1]. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development > > Best, > Mason > > On Thu, Jan 25, 2024 at 2:16 PM Martijn Visser > wrote: > > > Hi everyone, > > > > The latest version of the Flink

Re: [DISCUSS] Release new version of Flink's Kafka connector

2024-01-25 Thread Mason Chen
with what was agreed upon for external connector versioning [1]. [1] https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development Best, Mason On Thu, Jan 25, 2024 at 2:16 PM Martijn Visser wrote: > Hi everyone, > > The latest version of the Flink Kafka connect

Re: [DISCUSS] Release new version of Flink's Kafka connector

2024-01-25 Thread Hongshun Wang
t; > The latest version of the Flink Kafka connector that's available is > currently v3.0.2, which is compatible with both Flink 1.17 and Flink 1.18. > > I would like to propose to create a release which is either v3.1, or v4.0 > (see below), with compatibility for Flink 1.17 and Fl

[DISCUSS] Release new version of Flink's Kafka connector

2024-01-25 Thread Martijn Visser
Hi everyone, The latest version of the Flink Kafka connector that's available is currently v3.0.2, which is compatible with both Flink 1.17 and Flink 1.18. I would like to propose to create a release which is either v3.1, or v4.0 (see below), with compatibility for Flink 1.17 and Flink 1.18

[jira] [Created] (FLINK-34154) Bump org.apache.zookeeper:zookeeper from 3.5.9 to 3.7.2 for Kafka connector

2024-01-18 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-34154: -- Summary: Bump org.apache.zookeeper:zookeeper from 3.5.9 to 3.7.2 for Kafka connector Key: FLINK-34154 URL: https://issues.apache.org/jira/browse/FLINK-34154

[jira] [Created] (FLINK-34149) Flink Kafka connector can't compile against 1.19-SNAPSHOT

2024-01-18 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-34149: -- Summary: Flink Kafka connector can't compile against 1.19-SNAPSHOT Key: FLINK-34149 URL: https://issues.apache.org/jira/browse/FLINK-34149 Project: Flink

[jira] [Created] (FLINK-34127) Kafka connector repo runs a duplicate of `IntegrationTests` framework tests

2024-01-16 Thread Mason Chen (Jira)
Mason Chen created FLINK-34127: -- Summary: Kafka connector repo runs a duplicate of `IntegrationTests` framework tests Key: FLINK-34127 URL: https://issues.apache.org/jira/browse/FLINK-34127 Project

[jira] [Created] (FLINK-33512) Update download link in doc of Kafka connector

2023-11-09 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-33512: - Summary: Update download link in doc of Kafka connector Key: FLINK-33512 URL: https://issues.apache.org/jira/browse/FLINK-33512 Project: Flink Issue Type

[jira] [Created] (FLINK-33497) Update the Kafka connector to support DISTRIBUTED BY clause

2023-11-09 Thread Timo Walther (Jira)
Timo Walther created FLINK-33497: Summary: Update the Kafka connector to support DISTRIBUTED BY clause Key: FLINK-33497 URL: https://issues.apache.org/jira/browse/FLINK-33497 Project: Flink

[jira] [Created] (FLINK-33484) Flink Kafka Connector Offset Lag Issue with Transactional Data and Read Committed Isolation Level

2023-11-08 Thread Darcy Lin (Jira)
Darcy Lin created FLINK-33484: - Summary: Flink Kafka Connector Offset Lag Issue with Transactional Data and Read Committed Isolation Level Key: FLINK-33484 URL: https://issues.apache.org/jira/browse/FLINK-33484

[RESULT] [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Tzu-Li (Gordon) Tai
> > >> >> > > > >> >> > > > On Sun, 29 Oct 2023 at 08:02, Leonard Xu >> wrote: >> >> > > > >> >> > > > > +1 (binding) >> >> > > > > >> >> > > > > - Verified

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Tzu-Li (Gordon) Tai
> > > > >> > > > > >> > > > On Sun, 29 Oct 2023 at 08:02, Leonard Xu > wrote: > >> > > > > >> > > > > +1 (binding) > >> > > > > > >> > > > > - Verified signatures >

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Mason Chen
Verified signatures >> > > > > - Verified hashsums >> > > > > - Checked Github release tag >> > > > > - Built from source code succeeded >> > > > > - Checked release notes >> > > > > - Reviewed the web PR >&

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Mason Chen
elease notes > > > > > - Reviewed the web PR > > > > > > > > > > Best, > > > > > Leonard > > > > > > > > > > > > > > > > 2023年10月29日 上午11:34,mystic lama 写道: > > > > > > >

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Sergey Nuyanzin
> > > > > > > 2023年10月29日 上午11:34,mystic lama 写道: > > > > > > > > > > +1 (non-binding) > > > > > > > > > > - verified signatures > > > > > - build with Java 8 and Java 11 - build suc

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Tzu-Li (Gordon) Tai
; > > > > - verified signatures > > > > - build with Java 8 and Java 11 - build success > > > > > > > > Minor observation > > > > - RAT check flagged that README.md is missing ASL > > > > > > > > On

[jira] [Created] (FLINK-33401) Kafka connector has broken version

2023-10-30 Thread Pavel Khokhlov (Jira)
Pavel Khokhlov created FLINK-33401: -- Summary: Kafka connector has broken version Key: FLINK-33401 URL: https://issues.apache.org/jira/browse/FLINK-33401 Project: Flink Issue Type: Bug

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Samrat Deb
EADME.md is missing ASL > > > > > > On Fri, 27 Oct 2023 at 23:40, Xianxun Ye > > wrote: > > > > > >> +1(non-binding) > > >> > > >> - Started a local Flink 1.18 cluster, read and wrote with Kafka and > > Upsert > > >&

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Ahmed Hamdy
; - build with Java 8 and Java 11 - build success > > > > Minor observation > > - RAT check flagged that README.md is missing ASL > > > > On Fri, 27 Oct 2023 at 23:40, Xianxun Ye > wrote: > > > >> +1(non-binding) > >> > >> - Started

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-29 Thread Leonard Xu
build with Java 8 and Java 11 - build success > > Minor observation > - RAT check flagged that README.md is missing ASL > > On Fri, 27 Oct 2023 at 23:40, Xianxun Ye wrote: > >> +1(non-binding) >> >> - Started a local Flink 1.18 cluster, read and wrote with Kaf

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-28 Thread mystic lama
ith Kafka and Upsert > Kafka connector successfully to Kafka 2.2 cluster > > One minor question: should we update the dependency manual of these two > documentations[1][2]? > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#dependencies > [

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-28 Thread Xianxun Ye
+1(non-binding) - Started a local Flink 1.18 cluster, read and wrote with Kafka and Upsert Kafka connector successfully to Kafka 2.2 cluster One minor question: should we update the dependency manual of these two documentations[1][2]? [1] https://nightlies.apache.org/flink/flink-docs-master

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-26 Thread Martijn Visser
, successfully read and wrote with the Kafka connector to Confluent Cloud with AVRO and Schema Registry enabled On Thu, Oct 26, 2023 at 5:09 AM Qingsheng Ren wrote: > > +1 (binding) > > - Verified signature and checksum > - Verified that no binary exists in the source archive > - Built fro

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-25 Thread Qingsheng Ren
- Nothing suspicious in LICENSE and NOTICE file - Reviewed web PR Thanks for the effort, Gordon! Best, Qingsheng On Thu, Oct 26, 2023 at 5:13 AM Tzu-Li (Gordon) Tai wrote: > Hi everyone, > > Please review and vote on release candidate #1 for version 3.0.1 of the > Apache Flink Kaf

[VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-25 Thread Tzu-Li (Gordon) Tai
Hi everyone, Please review and vote on release candidate #1 for version 3.0.1 of the Apache Flink Kafka Connector, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This release contains important changes for the following: - Supports

[jira] [Created] (FLINK-33265) Support source parallelism setting for Kafka connector

2023-10-12 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-33265: - Summary: Support source parallelism setting for Kafka connector Key: FLINK-33265 URL: https://issues.apache.org/jira/browse/FLINK-33265 Project: Flink

Re: Kafka Connector

2023-10-12 Thread Martijn Visser
Hi David, I didn't see this message, but I did go over the Flink repo yesterday and closed off all PRs that were relevant to the Kafka connector. Removing the label won't help much, if a user classifies a Jira ticket as Connector/Kafka but opens a PR in the Flink repo, the label will be re-added

[jira] [Created] (FLINK-33219) Kafka connector has architecture test violation against Flink 1.18

2023-10-09 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-33219: - Summary: Kafka connector has architecture test violation against Flink 1.18 Key: FLINK-33219 URL: https://issues.apache.org/jira/browse/FLINK-33219 Project: Flink

[jira] [Created] (FLINK-33191) Kafka Connector should directly depend on 3rd-party libs instead of flink-shaded repo

2023-10-04 Thread Jing Ge (Jira)
Jing Ge created FLINK-33191: --- Summary: Kafka Connector should directly depend on 3rd-party libs instead of flink-shaded repo Key: FLINK-33191 URL: https://issues.apache.org/jira/browse/FLINK-33191 Project

Kafka Connector

2023-10-04 Thread David Radley
Hi, I was looking at the pr backlog in the Flink repository and realise that there are 51 hits on the search https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen+kafka-connector. And 25 hits on https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen+kafka-connector+label%3Acomponent

[jira] [Created] (FLINK-33124) Kafka Connector not working for table

2023-09-20 Thread Aarsh Shah (Jira)
Aarsh Shah created FLINK-33124: -- Summary: Kafka Connector not working for table Key: FLINK-33124 URL: https://issues.apache.org/jira/browse/FLINK-33124 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-33104) Nightly run for Flink Kafka connector fails

2023-09-18 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-33104: -- Summary: Nightly run for Flink Kafka connector fails Key: FLINK-33104 URL: https://issues.apache.org/jira/browse/FLINK-33104 Project: Flink Issue Type

[jira] [Created] (FLINK-33017) Nightly run for Flink Kafka connector fails

2023-09-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-33017: -- Summary: Nightly run for Flink Kafka connector fails Key: FLINK-33017 URL: https://issues.apache.org/jira/browse/FLINK-33017 Project: Flink Issue Type

[jira] [Created] (FLINK-32743) Flink kafka connector source can directly parse data collected from kafka-connect

2023-08-03 Thread Xiaojian Sun (Jira)
Xiaojian Sun created FLINK-32743: Summary: Flink kafka connector source can directly parse data collected from kafka-connect Key: FLINK-32743 URL: https://issues.apache.org/jira/browse/FLINK-32743

[jira] [Created] (FLINK-32582) Move TypeSerializerUpgradeTestBase from Kafka connector into flink-connector-common

2023-07-12 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-32582: - Summary: Move TypeSerializerUpgradeTestBase from Kafka connector into flink-connector-common Key: FLINK-32582 URL: https://issues.apache.org/jira/browse/FLINK-32582

Re: [ANNOUNCE] Kafka connector code removed from apache/master

2023-06-28 Thread Martijn Visser
Thank you for this! On Tue, Jun 27, 2023 at 8:57 PM Mason Chen wrote: > Hi all, > > I would like to inform you that we have removed the Kafka connector code > from the Flink main repo. This should reduce the developer confusion of > which repo to submit PRs. > > Regarding a

[ANNOUNCE] Kafka connector code removed from apache/master

2023-06-27 Thread Mason Chen
Hi all, I would like to inform you that we have removed the Kafka connector code from the Flink main repo. This should reduce the developer confusion of which repo to submit PRs. Regarding a few nuances, we have kept the Confluent avro format in the main repo. This is because the format

[jira] [Created] (FLINK-32451) Refactor Confluent Schema Registry E2E Tests to remove Kafka connector dependency

2023-06-27 Thread Mason Chen (Jira)
Mason Chen created FLINK-32451: -- Summary: Refactor Confluent Schema Registry E2E Tests to remove Kafka connector dependency Key: FLINK-32451 URL: https://issues.apache.org/jira/browse/FLINK-32451

[jira] [Created] (FLINK-32327) Python Kafka connector runs into strange NullPointerException

2023-06-13 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32327: Summary: Python Kafka connector runs into strange NullPointerException Key: FLINK-32327 URL: https://issues.apache.org/jira/browse/FLINK-32327 Project: Flink

Re: [DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-12 Thread Alexander Sorokoumov
Hi Jark, John, Thank you for the discussion! I will proceed with completing the patch that adds exactly-once to upsert-kafka connector. Best, Alexander On Wed, Apr 12, 2023 at 12:21 AM Jark Wu wrote: > Hi John, > > Thank you for your valuable input. It sounds reasonable to me.

Re: [DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-12 Thread Jark Wu
. > Exactly > >> > once > >> > can’t recognize duplicated records and drop duplications. That means > >> > duplicated > >> > records are written into topics even if exactly-once mode is enabled. > >> > > >> > > >> &g

Re: [DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-11 Thread John Roesler
ogether with idempotent >> producers prevent duplicated records[1], at least in the cases when >> upstream does not produce them intentionally and across checkpoints. >> >> Could you please elaborate or point me to the docs that explain the reason >> for duplicated rec

Re: [DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-08 Thread Jark Wu
ckpoints? I am relatively new > to Flink and not aware of it. According to the kafka connector > documentation, it does support exactly once semantics by configuring ' > sink.delivery-guarantee'='exactly-once'[2]. It is not clear to me why we > can't make upsert-kafka configurable i

Re: [DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-07 Thread Alexander Sorokoumov
for duplicated records upstream and across checkpoints? I am relatively new to Flink and not aware of it. According to the kafka connector documentation, it does support exactly once semantics by configuring ' sink.delivery-guarantee'='exactly-once'[2]. It is not clear to me why we can't make

Re: [DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-07 Thread Jark Wu
Flink community, > > I would like to discuss if it is worth adding EXACTLY_ONCE delivery > semantics to upsert-kafka connector. According to upsert-kafka docs[1] and > ReducingUpsertSink javadoc[2], the connector is correct even with duplicate > records under AT_LEAST_ONCE beca

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-04-06 Thread Chesnay Schepler
Do we have to move Debezium _now_? There is no hard dependency between debezium-json and the Kafka connector, nor does the format depend on Kafka afaict. So is this only about the e2e test that uses debezium-json + kafka connector? If so, then I would suggest to put debezium-json issue aside

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-04-05 Thread Mason Chen
; > talked with @PatrickRen <https://github.com/PatrickRen> offline, don't > > have a suitable way to fix it before. and we will solved it in this week > > > > Shammon FY 于2023年3月25日周六 13:13写道: > > > > > Thanks Jing and Gordon, I have closed the pr > >

[DISCUSS] EXACTLY_ONCE delivery semantics for upsert-kafka connector

2023-04-04 Thread Alexander Sorokoumov
Hello Flink community, I would like to discuss if it is worth adding EXACTLY_ONCE delivery semantics to upsert-kafka connector. According to upsert-kafka docs[1] and ReducingUpsertSink javadoc[2], the connector is correct even with duplicate records under AT_LEAST_ONCE because the records

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-27 Thread Tzu-Li (Gordon) Tai
l solved it in this week > > Shammon FY 于2023年3月25日周六 13:13写道: > > > Thanks Jing and Gordon, I have closed the pr > > https://github.com/apache/flink/pull/21965 and will open a new one for > > kafka connector > > > > > > Best, > > shammon FY > >

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-26 Thread Jacky Lau
don't have a suitable way to fix it before. and we will solved it in this week Shammon FY 于2023年3月25日周六 13:13写道: > Thanks Jing and Gordon, I have closed the pr > https://github.com/apache/flink/pull/21965 and will open a new one for > kafka connector > > > Best, > shammon FY

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-24 Thread Shammon FY
Thanks Jing and Gordon, I have closed the pr https://github.com/apache/flink/pull/21965 and will open a new one for kafka connector Best, shammon FY On Saturday, March 25, 2023, Ran Tao wrote: > Thank you Gordon and all the people who have worked on the externalized > kafka implemen

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-24 Thread Ran Tao
There is another PR related to Kafka > > connector: https://github.com/apache/flink/pull/21965 > > > > Best regards, > > Jing > > > > On Fri, Mar 24, 2023 at 4:06 PM Tzu-Li (Gordon) Tai > > > wrote: > > > > > Hi all, > > > > > &

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-24 Thread Tzu-Li (Gordon) Tai
Thanks Jing! I missed https://github.com/apache/flink/pull/21965 indeed. Please let us know if anything else was overlooked. On Fri, Mar 24, 2023 at 8:13 AM Jing Ge wrote: > Thanks Gordon for driving this! There is another PR related to Kafka > connector: https://github.com/apache/flin

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-24 Thread Jing Ge
Thanks Gordon for driving this! There is another PR related to Kafka connector: https://github.com/apache/flink/pull/21965 Best regards, Jing On Fri, Mar 24, 2023 at 4:06 PM Tzu-Li (Gordon) Tai wrote: > Hi all, > > Now that Flink 1.17 has been released, and given that we've alrea

[ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-24 Thread Tzu-Li (Gordon) Tai
Hi all, Now that Flink 1.17 has been released, and given that we've already synced the latest Kafka connector code up to Flink 1.17 to the apache/flink-connector-kafka repo (thanks to Mason and Martijn for most of the effort!), we're now in the final step of completely removing the Kafka

[jira] [Created] (FLINK-31483) Implement Split Deletion Support in Flink Kafka Connector

2023-03-16 Thread Ruibin Xing (Jira)
Ruibin Xing created FLINK-31483: --- Summary: Implement Split Deletion Support in Flink Kafka Connector Key: FLINK-31483 URL: https://issues.apache.org/jira/browse/FLINK-31483 Project: Flink

Re: Confluent Kafka connector for protobuf format

2022-12-19 Thread Martijn Visser
Hi Krish, Thanks for reaching out! I've assigned the ticket to you and also granted you access to create a FLIP. Best regards, Martijn On Sat, Dec 17, 2022 at 9:52 PM Krish Narukulla wrote: > Hi Team, > > I have written a connector for confluent kafka for protobuf and tested it >

Confluent Kafka connector for protobuf format

2022-12-17 Thread Krish Narukulla
Hi Team, I have written a connector for confluent kafka for protobuf and tested it internally. Could you please assign Jira: https://issues.apache.org/jira/browse/FLINK-29731 to me? Also give me access to create FLIP if needed. My Jira and confluence user name is : *krisnaru* Thanks Krish

Re: [DISCUSS] Externalizing Kafka connector

2022-12-12 Thread Konstantin Knauf
Hi Mason, great, thanks a lot for working on this. Will greatly speed up CI of the core repository once this is finalized. Cheers Konstantin Am Mi., 7. Dez. 2022 um 07:23 Uhr schrieb Mason Chen : > Hi all, > > I've finished the first pass on externalizing the Kafka connec

[DISCUSS] Externalizing Kafka connector

2022-12-06 Thread Mason Chen
Hi all, I've finished the first pass on externalizing the Kafka connector under the release-1.16 branch in this https://github.com/apache/flink-connector-kafka/pull/1. The docs, connectors, and e2e tests have been relocated and CI on my fork has been running green. Thanks for everyone else's work

[jira] [Created] (FLINK-30218) [Kafka Connector] java.lang.OutOfMemoryError: Metaspace

2022-11-25 Thread Lukas Mahl (Jira)
Lukas Mahl created FLINK-30218: -- Summary: [Kafka Connector] java.lang.OutOfMemoryError: Metaspace Key: FLINK-30218 URL: https://issues.apache.org/jira/browse/FLINK-30218 Project: Flink Issue

[jira] [Created] (FLINK-30052) Move existing Kafka connector code from Flink repo to dedicated Kafka repo

2022-11-17 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30052: -- Summary: Move existing Kafka connector code from Flink repo to dedicated Kafka repo Key: FLINK-30052 URL: https://issues.apache.org/jira/browse/FLINK-30052

[jira] [Created] (FLINK-30051) Create repository for Kafka connector

2022-11-17 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-30051: -- Summary: Create repository for Kafka connector Key: FLINK-30051 URL: https://issues.apache.org/jira/browse/FLINK-30051 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-29977) Kafka connector not compatible with kafka-clients 3.3x

2022-11-10 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-29977: Summary: Kafka connector not compatible with kafka-clients 3.3x Key: FLINK-29977 URL: https://issues.apache.org/jira/browse/FLINK-29977 Project: Flink

[jira] [Created] (FLINK-29920) Minor reformat Kafka connector documentation

2022-11-07 Thread Mingliang Liu (Jira)
Mingliang Liu created FLINK-29920: - Summary: Minor reformat Kafka connector documentation Key: FLINK-29920 URL: https://issues.apache.org/jira/browse/FLINK-29920 Project: Flink Issue Type

Re: Utilizing Kafka headers in Flink Kafka connector

2022-10-13 Thread Shengkai Fang
hi. You can use SQL API to parse or write the header in the Kafka record[1] if you are using Flink SQL. Best, Shengkai [1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#available-metadata Yaroslav Tkachenko 于2022年10月13日周四 02:21写道: > Hi, > > You can

Re: Utilizing Kafka headers in Flink Kafka connector

2022-10-12 Thread Yaroslav Tkachenko
Hi, You can implement a custom KafkaRecordDeserializationSchema (example https://docs.immerok.cloud/docs/cookbook/reading-apache-kafka-headers-with-apache-flink/#the-custom-deserializer) and just avoid emitting the record if the header value matches what you need. On Wed, Oct 12, 2022 at 11:04

Utilizing Kafka headers in Flink Kafka connector

2022-10-12 Thread Great Info
I have some flink applications that read streams from Kafka, now the producer side code has introduced some additional information in Kafka headers while producing records. Now I need to change my consumer-side logic to process the records if the header contains a specific value, if the header

Re: Need an help in understanding Kafka connector

2022-10-08 Thread Sriram Ganesh
Thanks!. On Sun, Oct 9, 2022, 08:45 Qingsheng Ren wrote: > Hi Sriram, > > A short answer: the interval of polling is adjusted “dynamically” (by > blocking the KafkaConsumer#poll call) according to the traffic of data. > > I think this line [1] is what you are looking for. > > Basically

Re: Need an help in understanding Kafka connector

2022-10-08 Thread Qingsheng Ren
Hi Sriram, A short answer: the interval of polling is adjusted “dynamically” (by blocking the KafkaConsumer#poll call) according to the traffic of data. I think this line [1] is what you are looking for. Basically KafkaSource fires KafkaPartitionSplitReader.fetch calls repeatedly in a loop,

Need an help in understanding Kafka connector

2022-10-07 Thread Sriram Ganesh
Hi Everyone, I am trying to understand how Flink works in realtime with Kafka. Since Kafka works on polling, what will be the minimal time for Flink to poll Kafka?. Any explanation or documentation will be helpful. Thanks, Sriram G

[jira] [Created] (FLINK-28622) Can't restore a flink job that uses Table API and Kafka connector with savepoint

2022-07-21 Thread Nathan (Jira)
Nathan created FLINK-28622: -- Summary: Can't restore a flink job that uses Table API and Kafka connector with savepoint Key: FLINK-28622 URL: https://issues.apache.org/jira/browse/FLINK-28622 Project: Flink

[jira] [Created] (FLINK-28475) kafka connector won't stop when the stopping offset is zero

2022-07-09 Thread Leo zhang (Jira)
Leo zhang created FLINK-28475: - Summary: kafka connector won't stop when the stopping offset is zero Key: FLINK-28475 URL: https://issues.apache.org/jira/browse/FLINK-28475 Project: Flink Issue

[jira] [Created] (FLINK-28266) Kafka connector fails: Invalid negative offset

2022-06-27 Thread Arseniy Tashoyan (Jira)
Arseniy Tashoyan created FLINK-28266: Summary: Kafka connector fails: Invalid negative offset Key: FLINK-28266 URL: https://issues.apache.org/jira/browse/FLINK-28266 Project: Flink Issue

[jira] [Created] (FLINK-28069) Cannot attach SSL JKS file for Kafka connector

2022-06-14 Thread SunShun (Jira)
SunShun created FLINK-28069: --- Summary: Cannot attach SSL JKS file for Kafka connector Key: FLINK-28069 URL: https://issues.apache.org/jira/browse/FLINK-28069 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-27730) Kafka connector document code sink has an error

2022-05-21 Thread liuwei (Jira)
liuwei created FLINK-27730: -- Summary: Kafka connector document code sink has an error Key: FLINK-27730 URL: https://issues.apache.org/jira/browse/FLINK-27730 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-27436) option `properties.group.id` is not effective in kafka connector for finksql

2022-04-28 Thread Spongebob (Jira)
Spongebob created FLINK-27436: - Summary: option `properties.group.id` is not effective in kafka connector for finksql Key: FLINK-27436 URL: https://issues.apache.org/jira/browse/FLINK-27436 Project

[jira] [Created] (FLINK-26928) Remove unnecessary Docker network creation in Kafka connector tests

2022-03-30 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-26928: - Summary: Remove unnecessary Docker network creation in Kafka connector tests Key: FLINK-26928 URL: https://issues.apache.org/jira/browse/FLINK-26928 Project: Flink

[jira] [Created] (FLINK-26409) Remove meaningless Kafka connector test case KafkaConsumerTestBase.runBrokerFailureTest

2022-02-28 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-26409: - Summary: Remove meaningless Kafka connector test case KafkaConsumerTestBase.runBrokerFailureTest Key: FLINK-26409 URL: https://issues.apache.org/jira/browse/FLINK-26409

[jira] [Created] (FLINK-26379) First message produced via Flink Kafka Connector is slow

2022-02-25 Thread Ragu Krishnamurthy (Jira)
Ragu Krishnamurthy created FLINK-26379: -- Summary: First message produced via Flink Kafka Connector is slow Key: FLINK-26379 URL: https://issues.apache.org/jira/browse/FLINK-26379 Project: Flink

[jira] [Created] (FLINK-26195) Kafka connector tests are mixing JUnit4 and JUnit5

2022-02-16 Thread Alexander Preuss (Jira)
Alexander Preuss created FLINK-26195: Summary: Kafka connector tests are mixing JUnit4 and JUnit5 Key: FLINK-26195 URL: https://issues.apache.org/jira/browse/FLINK-26195 Project: Flink

[jira] [Created] (FLINK-26115) Multiple Kafka connector tests failed due to The topic metadata failed to propagate to Kafka broker

2022-02-14 Thread Yun Gao (Jira)
Yun Gao created FLINK-26115: --- Summary: Multiple Kafka connector tests failed due to The topic metadata failed to propagate to Kafka broker Key: FLINK-26115 URL: https://issues.apache.org/jira/browse/FLINK-26115

[jira] [Created] (FLINK-25701) Add API annotation to some Kafka connector core classes and interface

2022-01-19 Thread Jing Ge (Jira)
Jing Ge created FLINK-25701: --- Summary: Add API annotation to some Kafka connector core classes and interface Key: FLINK-25701 URL: https://issues.apache.org/jira/browse/FLINK-25701 Project: Flink

Re: Flink SQL Kafka connector

2022-01-10 Thread Hang Ruan
Hi, There is already an on-going issue about it. ( https://issues.apache.org/jira/browse/FLINK-24456) Best, hang 聂荧屏 于2022年1月10日周一 10:06写道: > hello > > > Is there any plan to develop batch mode of Flink SQL Kafka connector? > > I would like to use kafka connector for d

Flink SQL Kafka connector

2022-01-09 Thread 聂荧屏
hello Is there any plan to develop batch mode of Flink SQL Kafka connector? I would like to use kafka connector for daily/hourly/minute-by-minute statistics, but currently only supports streaming mode and kafka parameters only support the start parameter setting, not the end parameter setting

[jira] [Created] (FLINK-25344) flink kafka connector Property group.id is required when using committed offset for offsets initializer

2021-12-16 Thread jingzi (Jira)
jingzi created FLINK-25344: -- Summary: flink kafka connector Property group.id is required when using committed offset for offsets initializer Key: FLINK-25344 URL: https://issues.apache.org/jira/browse/FLINK-25344

[jira] [Created] (FLINK-25336) Kafka connector compatible problem in Flink sql

2021-12-15 Thread Yuan Zhu (Jira)
Yuan Zhu created FLINK-25336: Summary: Kafka connector compatible problem in Flink sql Key: FLINK-25336 URL: https://issues.apache.org/jira/browse/FLINK-25336 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-25222) Remove NetworkFailureProxy used for Kafka connector tests

2021-12-08 Thread Fabian Paul (Jira)
Fabian Paul created FLINK-25222: --- Summary: Remove NetworkFailureProxy used for Kafka connector tests Key: FLINK-25222 URL: https://issues.apache.org/jira/browse/FLINK-25222 Project: Flink

[jira] [Created] (FLINK-25106) Support tombstone messages in FLINK's "kafka" connector

2021-11-30 Thread Varun Yeligar (Jira)
Varun Yeligar created FLINK-25106: - Summary: Support tombstone messages in FLINK's "kafka" connector Key: FLINK-25106 URL: https://issues.apache.org/jira/browse/FLINK-25106 Proj

  1   2   3   >