[jira] [Created] (FLINK-34107) Clearer separation of unit test and integration tests

2024-01-15 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34107: - Summary: Clearer separation of unit test and integration tests Key: FLINK-34107 URL: https://issues.apache.org/jira/browse/FLINK-34107 Project: Flink

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-15 Thread Zakelly Lan
Thanks for the suggestion Rui! The type is added. Best, Zakelly On Tue, Jan 16, 2024 at 2:33 PM Rui Fan <1996fan...@gmail.com> wrote: > Hi Zakelly, > > Would you mind adding the option type in the FLIP doc? > For example, String, Boolean or Enum, etc. Thank you. > > Best, > Rui > > On Tue,

Re: [DISCUSS] FLIP-418: Show data skew score on Flink Dashboard

2024-01-15 Thread Rui Fan
Thanks Emre for driving this proposal! It's very useful for troubleshooting. I have a question: The number_of_records_received_by_each_subtask is the total received records, right? I'm not sure whether we should check data skew based on the latest duration period. In the production, I found

[jira] [Created] (FLINK-34106) user defined source parallelism in ddl can't work if connector ScanRuntimeProvider is SourceFunctionProvider

2024-01-15 Thread Junning Liang (Jira)
Junning Liang created FLINK-34106: - Summary: user defined source parallelism in ddl can't work if connector ScanRuntimeProvider is SourceFunctionProvider Key: FLINK-34106 URL:

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-15 Thread Rui Fan
Hi Zakelly, Would you mind adding the option type in the FLIP doc? For example, String, Boolean or Enum, etc. Thank you. Best, Rui On Tue, Jan 16, 2024 at 2:29 PM Zakelly Lan wrote: > Hi everyone, > > Thanks all for joining the discussion! I'd like to speed this up since it > lasts for nearly

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-15 Thread Zakelly Lan
Hi everyone, Thanks all for joining the discussion! I'd like to speed this up since it lasts for nearly a month. I made changes on this FLIP based on suggestions and compromises acceptable to most people. Please feel free to give your opinion. Thanks! If there are no more suggestions, I will

Re:[DISCUSS] FLIP-418: Show data skew score on Flink Dashboard

2024-01-15 Thread Xuyang
Hi, Emre. In large-scale production jobs, the phenomenon of data skew often occurs. Having an metric on the UI that reflects data skew without the need for manual inspection of each vertex by clicking on them would be quite cool. This could help users quickly identify problematic nodes,

[jira] [Created] (FLINK-34105) Akka timeout happens in TPC-DS benchmarks

2024-01-15 Thread Zhu Zhu (Jira)
Zhu Zhu created FLINK-34105: --- Summary: Akka timeout happens in TPC-DS benchmarks Key: FLINK-34105 URL: https://issues.apache.org/jira/browse/FLINK-34105 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-34104) Improve the ScalingReport format of autoscaling

2024-01-15 Thread Rui Fan (Jira)
Rui Fan created FLINK-34104: --- Summary: Improve the ScalingReport format of autoscaling Key: FLINK-34104 URL: https://issues.apache.org/jira/browse/FLINK-34104 Project: Flink Issue Type:

[jira] [Created] (FLINK-34103) AsyncIO example failed to run as DataGen Connector is not bundled

2024-01-15 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-34103: - Summary: AsyncIO example failed to run as DataGen Connector is not bundled Key: FLINK-34103 URL: https://issues.apache.org/jira/browse/FLINK-34103 Project: Flink

Re: [DISCUSS] FLIP-415: Introduce a new join operator to support minibatch

2024-01-15 Thread shuai xu
Hi Benchao, Do you have any other questions about this issue? Also, I would appreciate your thoughts on the proposal to introduce the new option 'table.exec.mini-batch.compact-changes-enabled'. I’m looking forward your feedback. > 2024年1月12日 15:01,shuai xu 写道: > > Suppose we currently have

[jira] [Created] (FLINK-34102) Invalid configuration when using 'env.log.max' on yarn application mode

2024-01-15 Thread RocMarshal (Jira)
RocMarshal created FLINK-34102: -- Summary: Invalid configuration when using 'env.log.max' on yarn application mode Key: FLINK-34102 URL: https://issues.apache.org/jira/browse/FLINK-34102 Project: Flink

Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-15 Thread Zakelly Lan
Hi everyone, Thanks for all the comments! If there are no objections, I'd like to start a vote after 72 hours. Please let me know if you have any concerns. Thanks! Best, Zakelly On Tue, Jan 16, 2024 at 10:23 AM xiangyu feng wrote: > Hi Zakelly, > > Thanks for driving this. +1 to remove the

[jira] [Created] (FLINK-34101) use flink ui upload jar error, message is 'Exactly 1 file must be sent, received 0'

2024-01-15 Thread blackpighe (Jira)
blackpighe created FLINK-34101: -- Summary: use flink ui upload jar error, message is 'Exactly 1 file must be sent, received 0' Key: FLINK-34101 URL: https://issues.apache.org/jira/browse/FLINK-34101

Re: [VOTE] FLIP-389: Annotate SingleThreadFetcherManager as PublicEvolving

2024-01-15 Thread Leonard Xu
Thanks Hongshun for driving this ! +1(binding) Best, Leonard > 2024年1月3日 下午8:04,Hongshun Wang 写道: > > Dear Flink Developers, > > Thank you for providing feedback on FLIP-389: Annotate > SingleThreadFetcherManager as PublicEvolving[1] on the discussion > thread[2]. The goal of the FLIP is as

[jira] [Created] (FLINK-34100) Support session window table function without pulling up with window agg

2024-01-15 Thread xuyang (Jira)
xuyang created FLINK-34100: -- Summary: Support session window table function without pulling up with window agg Key: FLINK-34100 URL: https://issues.apache.org/jira/browse/FLINK-34100 Project: Flink

[RESULT][VOTE] FLIP-407: Improve Flink Client performance in interactive scenarios

2024-01-15 Thread xiangyu feng
Hi all, Thanks for your review and the votes! I am happy to announce that FLIP-407: Improve Flink Client performance in interactive scenarios[1] has been accepted. There are 3 binding votes and 1 non-binding votes [2]: - Rui Fan (binding) - Yangze Guo (binding) - Weihua Hu (binding) - Chen

Re: [VOTE] FLIP-407: Improve Flink Client performance in interactive scenarios

2024-01-15 Thread xiangyu feng
Thank you all for the votes! I will close the voting thread and summarize the result in a separate email. Regards, Xiangyu Feng Zhanghao Chen 于2024年1月16日周二 10:18写道: > +1 (non-binding) > > Best, > Zhanghao Chen > > From: xiangyu feng > Sent: Thursday, January

Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-15 Thread xiangyu feng
Hi Zakelly, Thanks for driving this. +1 to remove the LEGACY mode. This will make the semantic of snapshot ownership more clear. Best regards, Xiangyu Feng Yuan Mei 于2024年1月15日周一 23:16写道: > +1 > > Thanks for driving this Zakelly! > > Best > Yuan > > On Mon, Jan 15, 2024 at 10:47 PM Piotr

[jira] [Created] (FLINK-34099) CheckpointIntervalDuringBacklogITCase.testNoCheckpointDuringBacklog is unstable on AZP

2024-01-15 Thread Yu Chen (Jira)
Yu Chen created FLINK-34099: --- Summary: CheckpointIntervalDuringBacklogITCase.testNoCheckpointDuringBacklog is unstable on AZP Key: FLINK-34099 URL: https://issues.apache.org/jira/browse/FLINK-34099

Re: [VOTE] FLIP-407: Improve Flink Client performance in interactive scenarios

2024-01-15 Thread Zhanghao Chen
+1 (non-binding) Best, Zhanghao Chen From: xiangyu feng Sent: Thursday, January 11, 2024 19:44 To: dev@flink.apache.org Subject: [VOTE] FLIP-407: Improve Flink Client performance in interactive scenarios Hi all, I would like to start the vote for FLIP-407:

Re: [VOTE] Accept Flink CDC into Apache Flink

2024-01-15 Thread Fabian Hueske
+1 (binding) Cheers, Fabian Timo Walther schrieb am Mo., 15. Jan. 2024, 16:00: > +1 (binding) > > Cheers, > Timo > > > On 09.01.24 10:58, xiangyu feng wrote: > > +1 (non-binding) > > > > Regards, > > Xiangyu Feng > > > > Danny Cranmer 于2024年1月9日周二 17:50写道: > > > >> +1 (binding) > >> > >>

[DISCUSS] FLIP-418: Show data skew score on Flink Dashboard

2024-01-15 Thread Kartoglu, Emre
Hello, I’m opening this thread to discuss a FLIP[1] to make data skew more visible on Flink Dashboard. Data skew is currently not as visible as it should be. Users have to click each operator and check how much data each sub-task is processing and compare the sub-tasks against each other.

[jira] [Created] (FLINK-34098) Not enough Azure Pipeline CI runners available

2024-01-15 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34098: - Summary: Not enough Azure Pipeline CI runners available Key: FLINK-34098 URL: https://issues.apache.org/jira/browse/FLINK-34098 Project: Flink Issue Type:

[jira] [Created] (FLINK-34097) Remove unused JobMasterGateway#requestJobDetails

2024-01-15 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-34097: Summary: Remove unused JobMasterGateway#requestJobDetails Key: FLINK-34097 URL: https://issues.apache.org/jira/browse/FLINK-34097 Project: Flink

Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-15 Thread Yuan Mei
+1 Thanks for driving this Zakelly! Best Yuan On Mon, Jan 15, 2024 at 10:47 PM Piotr Nowojski wrote: > +1 good idea! > > pon., 15 sty 2024 o 05:11 Jinzhong Li > napisał(a): > > > Hi Zakelly, > > > > Thanks for driving the discussion. It makes sense to remove LEGACY mode > in > > Flink 2.0.

Re: [VOTE] Accept Flink CDC into Apache Flink

2024-01-15 Thread Timo Walther
+1 (binding) Cheers, Timo On 09.01.24 10:58, xiangyu feng wrote: +1 (non-binding) Regards, Xiangyu Feng Danny Cranmer 于2024年1月9日周二 17:50写道: +1 (binding) Thanks, Danny On Tue, Jan 9, 2024 at 9:31 AM Feng Jin wrote: +1 (non-binding) Best, Feng Jin On Tue, Jan 9, 2024 at 5:29 PM

[jira] [Created] (FLINK-34096) Upscale of kafka source operator leads to some splits getting lost

2024-01-15 Thread Yang LI (Jira)
Yang LI created FLINK-34096: --- Summary: Upscale of kafka source operator leads to some splits getting lost Key: FLINK-34096 URL: https://issues.apache.org/jira/browse/FLINK-34096 Project: Flink

[jira] [Created] (FLINK-34095) Add a restore test for StreamExecAsyncCalc

2024-01-15 Thread Timo Walther (Jira)
Timo Walther created FLINK-34095: Summary: Add a restore test for StreamExecAsyncCalc Key: FLINK-34095 URL: https://issues.apache.org/jira/browse/FLINK-34095 Project: Flink Issue Type:

[jira] [Created] (FLINK-34094) Document new AsyncScalarFunction

2024-01-15 Thread Timo Walther (Jira)
Timo Walther created FLINK-34094: Summary: Document new AsyncScalarFunction Key: FLINK-34094 URL: https://issues.apache.org/jira/browse/FLINK-34094 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-15 Thread Piotr Nowojski
+1 good idea! pon., 15 sty 2024 o 05:11 Jinzhong Li napisał(a): > Hi Zakelly, > > Thanks for driving the discussion. It makes sense to remove LEGACY mode in > Flink 2.0. > > Best, > Jinzhong Li > > On Mon, Jan 15, 2024 at 10:34 AM Xuannan Su wrote: > > > Hi Zakelly, > > > > Thanks for driving

Re: Permission to add FLIP document

2024-01-15 Thread Jeyhun Karimov
Thank you Leonard. On Mon, Jan 15, 2024 at 2:30 AM Leonard Xu wrote: > Hey, Jeyhun > > I’ve added the permissions for your account. > > Best, > Leonard > > > 2024年1月15日 上午3:42,Jeyhun Karimov 写道: > > > > Dear Flink community, > > > > I would like to create a FLIP document. I assume, for that I

[jira] [Created] (FLINK-34093) RetryingRegistrationTest.testRetryConnectOnFailure failed

2024-01-15 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34093: - Summary: RetryingRegistrationTest.testRetryConnectOnFailure failed Key: FLINK-34093 URL: https://issues.apache.org/jira/browse/FLINK-34093 Project: Flink

Re: [VOTE] Accept Flink CDC into Apache Flink

2024-01-15 Thread Ufuk Celebi
+1 (binding) – Ufuk On Mon, Jan 15, 2024, at 12:38 PM, Congxian Qiu wrote: > +1 (non-binding) > > Best, > Congxian > > > Gyula Fóra 于2024年1月15日周一 15:37写道: > > > +1 (binding) > > > > On Sun, Jan 14, 2024 at 9:01 AM Yun Tang wrote: > > > > > +1 (non-binding) > > > > > > On 2024/01/14

Re: [VOTE] Release flink-connector-rabbitmq, v3.0.2 release candidate #1

2024-01-15 Thread Danny Cranmer
+1 (binding) - Release notes look good - Signatures match for binary/source archive - Checksums match for binary/source archive - Contents of Maven dist look complete - Verified there are no binaries in the source archive - Built src/tests pass running Maven locally - Tag is present in Github -

[jira] [Created] (FLINK-34092) NettyEpollITCase#testNettyEpoll on arrch64 running fail

2024-01-15 Thread baihailiang (Jira)
baihailiang created FLINK-34092: --- Summary: NettyEpollITCase#testNettyEpoll on arrch64 running fail Key: FLINK-34092 URL: https://issues.apache.org/jira/browse/FLINK-34092 Project: Flink Issue

[jira] [Created] (FLINK-34091) Flink pulsar connect adds automatic failover capability

2024-01-15 Thread yangyijun (Jira)
yangyijun created FLINK-34091: - Summary: Flink pulsar connect adds automatic failover capability Key: FLINK-34091 URL: https://issues.apache.org/jira/browse/FLINK-34091 Project: Flink Issue

Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink

2024-01-15 Thread Congxian Qiu
+1 (non-binding) Best, Congxian Gyula Fóra 于2024年1月15日周一 15:37写道: > +1 (binding) > > On Sun, Jan 14, 2024 at 9:01 AM Yun Tang wrote: > > > +1 (non-binding) > > > > On 2024/01/14 02:26:13 Yun Gao wrote: > > > +1 (binding) > > > > > > On Sat, Jan 13, 2024 at 10:08 AM Rodrigo Meneses > >

[jira] [Created] (FLINK-34090) Introduce SerializerConfig for serialization

2024-01-15 Thread Fang Yong (Jira)
Fang Yong created FLINK-34090: - Summary: Introduce SerializerConfig for serialization Key: FLINK-34090 URL: https://issues.apache.org/jira/browse/FLINK-34090 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Hadoop 2 vs Hadoop 3 usage

2024-01-15 Thread Yang Wang
I could share some metrics about Alibaba Cloud EMR clusters. The ratio of Hadoop2 VS Hadoop3 is 1:3. Best, Yang On Thu, Dec 28, 2023 at 8:16 PM Martijn Visser wrote: > Hi all, > > I want to get some insights on how many users are still using Hadoop 2 > vs how many users are using Hadoop 3.

[jira] [Created] (FLINK-34089) Missing check of subscribed and assigned topics at job startup

2024-01-15 Thread tanjialiang (Jira)
tanjialiang created FLINK-34089: --- Summary: Missing check of subscribed and assigned topics at job startup Key: FLINK-34089 URL: https://issues.apache.org/jira/browse/FLINK-34089 Project: Flink

Re: Re: [DISCUSS] FLIP-415: Introduce a new join operator to support minibatch

2024-01-15 Thread Jane Chan
Hi shuai, Thanks for your clarification. The internal behavior of minibatch processing is not well-defined now. I think you're right on this point. If you change the goal of the newly introduced configuration to address this issue, then I'm ok with it. Best, Jane On Mon, Jan 15, 2024 at

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

2024-01-15 Thread Zhu Zhu
Correction: I'm fine to use a void blobService in OLAP scenarios if it works better in most cases. -> I'm fine to use a void blobService in OLAP scenarios *by default* if it works better in most cases. Zhu Zhu 于2024年1月15日周一 17:51写道: > @Yangze > > > (with 128 parallelism WordCount jobs),

Re: [DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

2024-01-15 Thread Zhu Zhu
@Yangze > (with 128 parallelism WordCount jobs), disabling BlobStore resulted in a 100% increase in QPS Did you look into which part takes most of the time? Jar uploading, Jar downloading, JobInformation shipping, TDD shipping, or others? If these objects are large, e.g. a hundreds megabytes

[jira] [Created] (FLINK-34088) [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver.

2024-01-15 Thread blackpighe (Jira)
blackpighe created FLINK-34088: -- Summary: [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver. Key: FLINK-34088 URL: https://issues.apache.org/jira/browse/FLINK-34088

[jira] [Created] (FLINK-34087) [JUnit5 Migration] Module: flink-dist

2024-01-15 Thread Jiabao Sun (Jira)
Jiabao Sun created FLINK-34087: -- Summary: [JUnit5 Migration] Module: flink-dist Key: FLINK-34087 URL: https://issues.apache.org/jira/browse/FLINK-34087 Project: Flink Issue Type: Sub-task

[DISCUSS] Support detecting table schema from external files.

2024-01-15 Thread Yisha Zhou
Hi dev, Currently, we are used to creating a table by listing all physical columns or using like syntax to reuse the table schema in Catalogs. However, in our company there are many cases that the messages in the external systems are with very complex schema. The worst case is that some

[jira] [Created] (FLINK-34086) Native k8s session cannot specify port in nodeport mode

2024-01-15 Thread waywtdcc (Jira)
waywtdcc created FLINK-34086: Summary: Native k8s session cannot specify port in nodeport mode Key: FLINK-34086 URL: https://issues.apache.org/jira/browse/FLINK-34086 Project: Flink Issue Type: