[jira] [Created] (FLINK-35169) Recycle buffers to freeSegments before releasing data buffer for sort accumulator

2024-04-18 Thread Yuxin Tan (Jira)
Yuxin Tan created FLINK-35169: - Summary: Recycle buffers to freeSegments before releasing data buffer for sort accumulator Key: FLINK-35169 URL: https://issues.apache.org/jira/browse/FLINK-35169 Project:

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

2024-04-18 Thread Anupam Aggarwal
Thanks David. That's a great idea. For deserialization the external schema Id will be used to obtain a dynamic message, so in a way it has to be inline with the writer schema. We could limit it to serialization and rename it according to your suggestion. Thanks Anupam On Tue, Apr 16, 2024 at 3:38 

[jira] [Created] (FLINK-35168) Basic State Iterator for async processing

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35168: --- Summary: Basic State Iterator for async processing Key: FLINK-35168 URL: https://issues.apache.org/jira/browse/FLINK-35168 Project: Flink Issue Type: Sub-task

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

2024-04-18 Thread gongzhongqiang
+1 (non-binding) Best, Zhongqiang Gong Xuannan Su 于2024年4月17日周三 13:02写道: > Hi everyone, > > Thanks for all the feedback about the FLIP-442: General Improvement to > Configuration for Flink 2.0 [1] [2]. > > I'd like to start a vote for it. The vote will be open for at least 72 >

Re: [VOTE] FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines

2024-04-18 Thread Jingsong Li
+1 On Thu, Apr 18, 2024 at 11:09 AM Yun Tang wrote: > > +1 (binding) > > Best, > Yun Tang > > From: Jark Wu > Sent: Thursday, April 18, 2024 9:54 > To: dev@flink.apache.org > Subject: Re: [VOTE] FLIP-435: Introduce a New Materialized Table for > Simplifying

[jira] [Created] (FLINK-35167) [CDC] Introduce MaxCompute pipeline DataSink

2024-04-18 Thread zhangdingxin (Jira)
zhangdingxin created FLINK-35167: Summary: [CDC] Introduce MaxCompute pipeline DataSink Key: FLINK-35167 URL: https://issues.apache.org/jira/browse/FLINK-35167 Project: Flink Issue Type: New

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

2024-04-18 Thread Muhammet Orazov
Thanks Xuannan! +1 (non-binding) Best, Muhammet On 2024-04-17 05:00, Xuannan Su wrote: Hi everyone, Thanks for all the feedback about the FLIP-442: General Improvement to Configuration for Flink 2.0 [1] [2]. I'd like to start a vote for it. The vote will be open for at least 72

[jira] [Created] (FLINK-35166) Improve the performance of Hybrid Shuffle when enable memory decoupling

2024-04-18 Thread Jiang Xin (Jira)
Jiang Xin created FLINK-35166: - Summary: Improve the performance of Hybrid Shuffle when enable memory decoupling Key: FLINK-35166 URL: https://issues.apache.org/jira/browse/FLINK-35166 Project: Flink

Re: Question around Flink's AdaptiveBatchScheduler

2024-04-18 Thread Venkatakrishnan Sowrirajan
Filed https://issues.apache.org/jira/browse/FLINK-35165 to address the above described issue. Will share the PR here once it is ready for review. Regards Venkata krishnan On Wed, Apr 17, 2024 at 5:32 AM Junrui Lee wrote: > Thanks Venkata and Xia for providing further clarification. I think

[jira] [Created] (FLINK-35165) AdaptiveBatch Scheduler should not restrict the default source parallelism to the max parallelism set

2024-04-18 Thread Venkata krishnan Sowrirajan (Jira)
Venkata krishnan Sowrirajan created FLINK-35165: --- Summary: AdaptiveBatch Scheduler should not restrict the default source parallelism to the max parallelism set Key: FLINK-35165 URL:

Re: [ DISCUSS ] FLIP-XXX : [Plugin] Enhancing Flink Failure Management in Kubernetes with Dynamic Termination Log Integration

2024-04-18 Thread Swathi C
Hi Gyula and Ahmed, Thanks for reviewing this. @gyula.f...@gmail.com , currently since our aim as part of this FLIP was only to fail the cluster when job manager/flink has issues such that the cluster would no longer be usable, hence, we proposed only related to that. Your right, that it

[jira] [Created] (FLINK-35164) Support `ALTER CATALOG RESET` syntax

2024-04-18 Thread Yubin Li (Jira)
Yubin Li created FLINK-35164: Summary: Support `ALTER CATALOG RESET` syntax Key: FLINK-35164 URL: https://issues.apache.org/jira/browse/FLINK-35164 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-18 Thread Lincoln Lee
Thanks everyone : ) Best, Lincoln Lee Feifan Wang 于2024年4月16日周二 12:44写道: > Congratulations Lincoln ! > > > —— > > Best regards, > > Feifan Wang > > > > > At 2024-04-12 15:59:00, "Jark Wu" wrote: > >Hi everyone, > > > >On behalf of the PMC, I'm very happy to announce that Lincoln

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

2024-04-18 Thread Ahmed Hamdy
+1 (non-binding) Great work! Best Regards Ahmed Hamdy On Wed, 17 Apr 2024 at 14:36, Jeyhun Karimov wrote: > +1 (non binding) > > Regards, > Jeyhun > > On Wed, Apr 17, 2024 at 2:22 PM Zhu Zhu wrote: > > > +1 (binding) > > > > Thanks, > > Zhu > > > > Yuxin Tan 于2024年4月17日周三 18:36写道: > > > >

Re: [ DISCUSS ] FLIP-XXX : [Plugin] Enhancing Flink Failure Management in Kubernetes with Dynamic Termination Log Integration

2024-04-18 Thread Ahmed Hamdy
Hi Swathi! Thanks for the proposal. Could you please elaborate what this FLIP offers more than Flip-304[1]? Flip 304 proposes a Pluggable mechanism for enriching Job failures, If I am not mistaken this proposal looks like a subset of it. 1-

Re: [DISCUSS][QUESTION] Drop jdk 8 support for Flink connector Opensearch

2024-04-18 Thread Danny Cranmer
Hey Sergey, Given the circumstances I think it is ok to drop JDK 8 support with the opensearch v2.0.0 connector. However, in parallel we should still support the 1.x line for Flink 1.x series with JDK 8. This would mean two releases: 1/ flink-connector-opensearch v2.0.0 for Flink 1.18/1.19,

[jira] [Created] (FLINK-35163) Utilize ForSt's native MultiGet API to optimize remote state access

2024-04-18 Thread Jinzhong Li (Jira)
Jinzhong Li created FLINK-35163: --- Summary: Utilize ForSt's native MultiGet API to optimize remote state access Key: FLINK-35163 URL: https://issues.apache.org/jira/browse/FLINK-35163 Project: Flink

[jira] [Created] (FLINK-35162) Support grouping state get and put access

2024-04-18 Thread Jinzhong Li (Jira)
Jinzhong Li created FLINK-35162: --- Summary: Support grouping state get and put access Key: FLINK-35162 URL: https://issues.apache.org/jira/browse/FLINK-35162 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35161) Implement StateExecutor for ForStStateBackend

2024-04-18 Thread Jinzhong Li (Jira)
Jinzhong Li created FLINK-35161: --- Summary: Implement StateExecutor for ForStStateBackend Key: FLINK-35161 URL: https://issues.apache.org/jira/browse/FLINK-35161 Project: Flink Issue Type:

Re: [VOTE] Release flink-connector-mongodb v1.2.0, release candidate #2

2024-04-18 Thread Ahmed Hamdy
+1 (non-binding) - verified hashes and checksums - verified signature - verified source contains no binaries - tag exists in github - reviewed web PR Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 11:21, Danny Cranmer wrote: > Hi everyone, > > Please review and vote on the release

Re: [VOTE] Release flink-connector-jdbc v3.2.0, release candidate #2

2024-04-18 Thread Ahmed Hamdy
+1 (non-binding) - Verified Checksums and hashes - Verified Signatures - No binaries in source - Build source - Github tag exists - Reviewed Web PR Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 11:22, Danny Cranmer wrote: > Sorry for typos: > > > Please review and vote on the release

[jira] [Created] (FLINK-35160) Support for Thread Dump provides a convenient way to display issues of thread deadlocks in tasks

2024-04-18 Thread elon_X (Jira)
elon_X created FLINK-35160: -- Summary: Support for Thread Dump provides a convenient way to display issues of thread deadlocks in tasks Key: FLINK-35160 URL: https://issues.apache.org/jira/browse/FLINK-35160

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-18 Thread Ahmed Hamdy
Hi Xia, I have read through the FLIP and discussion and the new version of the FLIP looks better. +1 for the proposal. Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 12:21, Ron Liu wrote: > Hi, Xia > > Thanks for updating, looks good to me. > > Best, > Ron > > Xia Sun 于2024年4月18日周四 19:11写道:

Re: [DISCUSS] FLIP-438: Make Flink's Hadoop and YARN configuration probing consistent

2024-04-18 Thread Ferenc Csaky
Hi Venkata krishnan, My general point was that personally I do not think that the current implementation is wrong or confusing. And the main thing here is that how we define consistent in this case is subjective. >From the proposed point of view, consistent mean we use the same prefix. But we can

[jira] [Created] (FLINK-35159) CreatingExecutionGraph can leak CheckpointCoordinator and cause JM crash

2024-04-18 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-35159: Summary: CreatingExecutionGraph can leak CheckpointCoordinator and cause JM crash Key: FLINK-35159 URL: https://issues.apache.org/jira/browse/FLINK-35159

[jira] [Created] (FLINK-35158) Error handling in StateFuture's callback

2024-04-18 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-35158: -- Summary: Error handling in StateFuture's callback Key: FLINK-35158 URL: https://issues.apache.org/jira/browse/FLINK-35158 Project: Flink Issue Type: Sub-task

Re: [VOTE] Release flink-connector-gcp-pubsub v3.1.0, release candidate #1

2024-04-18 Thread Ahmed Hamdy
Hi Danny, +1 (non-binding) - verified hashes and checksums - verified signature - verified source contains no binaries - tag exists in github - reviewed web PR Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 11:32, Danny Cranmer wrote: > Hi everyone, > > Please review and vote on release

[jira] [Created] (FLINK-35157) Sources with watermark alignment get stuck once some subtasks finish

2024-04-18 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-35157: -- Summary: Sources with watermark alignment get stuck once some subtasks finish Key: FLINK-35157 URL: https://issues.apache.org/jira/browse/FLINK-35157 Project: Flink

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-18 Thread Ron Liu
Hi, Xia Thanks for updating, looks good to me. Best, Ron Xia Sun 于2024年4月18日周四 19:11写道: > Hi Ron, > Yes, presenting it in a table might be more intuitive. I have already added > the table in the "Public Interfaces | New Config Option" chapter of FLIP. > PTAL~ > > Ron Liu 于2024年4月18日周四

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-18 Thread Xia Sun
Hi Ron, Yes, presenting it in a table might be more intuitive. I have already added the table in the "Public Interfaces | New Config Option" chapter of FLIP. PTAL~ Ron Liu 于2024年4月18日周四 18:10写道: > Hi, Xia > > Thanks for your reply. > > > That means, in terms > of priority,

[VOTE] Release flink-connector-gcp-pubsub v3.1.0, release candidate #1

2024-04-18 Thread Danny Cranmer
Hi everyone, Please review and vote on release candidate #1 for flink-connector-gcp-pubsub v3.1.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This release supports Flink 1.18 and 1.19. The complete staging area is available for

Re: [VOTE] Release flink-connector-jdbc v3.2.0, release candidate #2

2024-04-18 Thread Danny Cranmer
Sorry for typos: > Please review and vote on the release candidate #1 for the version 3.2.0, as follows: Should be "release candidate #2" > * source code tag v3.2.0-rc1 [5], Should be "source code tag v3.2.0-rc2" Thanks, Danny On Thu, Apr 18, 2024 at 11:19 AM Danny Cranmer wrote: > Hi

[VOTE] Release flink-connector-mongodb v1.2.0, release candidate #2

2024-04-18 Thread Danny Cranmer
Hi everyone, Please review and vote on the release candidate #2 for v1.2.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This release supports Flink 1.18 and 1.19. The complete staging area is available for your review, which

[VOTE] Release flink-connector-jdbc v3.2.0, release candidate #2

2024-04-18 Thread Danny Cranmer
Hi everyone, Please review and vote on the release candidate #1 for the version 3.2.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) This release supports Flink 1.18 and 1.19. The complete staging area is available for your review,

[jira] [Created] (FLINK-35156) Wire new operators for async state with DataStream V2

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35156: --- Summary: Wire new operators for async state with DataStream V2 Key: FLINK-35156 URL: https://issues.apache.org/jira/browse/FLINK-35156 Project: Flink Issue

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-18 Thread Ron Liu
Hi, Xia Thanks for your reply. > That means, in terms of priority, `table.exec.hive.infer-source-parallelism` > `table.exec.hive.infer-source-parallelism.mode`. I still have some confusion, if the `table.exec.hive.infer-source-parallelism` >`table.exec.hive.infer-source-parallelism.mode`,

[jira] [Created] (FLINK-35155) Introduce TableRuntimeException

2024-04-18 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-35155: Summary: Introduce TableRuntimeException Key: FLINK-35155 URL: https://issues.apache.org/jira/browse/FLINK-35155 Project: Flink Issue Type: New

Re: [VOTE] Release flink-connector-mongodb v1.2.0, release candidate #1

2024-04-18 Thread Danny Cranmer
-1 (binding) There is a bug [1] in the connector release utils resulting in the build tools [2] not being copied into the source archive. As a result the source archive fails build due to checkstyle errors. This vote is now closed, I will follow up with an RC2. Thanks, Danny [1]

Re: [VOTE] Release flink-connector-jdbc v3.2.0, release candidate #1

2024-04-18 Thread Danny Cranmer
-1 (binding) There is a bug [1] in the connector release utils resulting in the build tools [2] not being copied into the source archive. As a result the source archive fails build due to checkstyle errors. This vote is now closed, I will follow up with an RC2. Thanks, Danny [1]

Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource

2024-04-18 Thread Xia Sun
Hi Ron and Lijie, Thanks for joining the discussion and sharing your suggestions. > the InferMode class should also be introduced in the Public Interfaces > section! Thanks for the reminder, I have now added the InferMode class to the Public Interfaces section as well. >

[jira] [Created] (FLINK-35154) Javadoc aggregate fails

2024-04-18 Thread Dmitriy Linevich (Jira)
Dmitriy Linevich created FLINK-35154: Summary: Javadoc aggregate fails Key: FLINK-35154 URL: https://issues.apache.org/jira/browse/FLINK-35154 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-35153) Internal Async State Implementation and StateDescriptor for Map/List State

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35153: --- Summary: Internal Async State Implementation and StateDescriptor for Map/List State Key: FLINK-35153 URL: https://issues.apache.org/jira/browse/FLINK-35153 Project:

Re: [ DISCUSS ] FLIP-XXX : [Plugin] Enhancing Flink Failure Management in Kubernetes with Dynamic Termination Log Integration

2024-04-18 Thread Gyula Fóra
Hi Swathi! Thank you for creating this proposal. I really like the general idea of increasing the K8s native observability of Flink job errors. I took a quick look at your reference PR, the termination log related logic is contained completely in the ClusterEntrypoint. What type of errors will

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-18 Thread xiangyu feng
Congratulations, Zakelly! Regards, Xiangyu Feng yh z 于2024年4月18日周四 14:27写道: > Congratulations Zakelly! > > Best regards, > Yunhong (swuferhong) > > gongzhongqiang 于2024年4月17日周三 21:26写道: > > > Congratulations, Zakelly! > > > > > > Best, > > Zhongqiang Gong > > > > Yuan Mei 于2024年4月15日周一

[jira] [Created] (FLINK-35152) Flink CDC Doris/Starrocks Sink Auto create table event should support setting auto partition fields for each table

2024-04-18 Thread tumengyao (Jira)
tumengyao created FLINK-35152: - Summary: Flink CDC Doris/Starrocks Sink Auto create table event should support setting auto partition fields for each table Key: FLINK-35152 URL:

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-18 Thread yh z
Congratulations Zakelly! Best regards, Yunhong (swuferhong) gongzhongqiang 于2024年4月17日周三 21:26写道: > Congratulations, Zakelly! > > > Best, > Zhongqiang Gong > > Yuan Mei 于2024年4月15日周一 10:51写道: > > > Hi everyone, > > > > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has >