Re: [VOTE] FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records

2022-12-26 Thread Leonard Xu
+1 (binding) Best, Leonard > On Dec 26, 2022, at 4:22 PM, Qingsheng Ren wrote: > > +1 (binding) > > Best, > Qingsheng > Ververica (Alibaba) > > On Wed, Dec 21, 2022 at 3:13 PM Dong Lin wrote: > >> Hi all, >> >> We would like to start the vote for FLIP-208: Add RecordEvaluator to >>

[jira] [Created] (FLINK-30508) CliClientITCase.testSqlStatements failed with output not matched with expected

2022-12-26 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-30508: - Summary: CliClientITCase.testSqlStatements failed with output not matched with expected Key: FLINK-30508 URL: https://issues.apache.org/jira/browse/FLINK-30508

[jira] [Created] (FLINK-30507) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism could not be initialized

2022-12-26 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-30507: - Summary: EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism could not be initialized Key: FLINK-30507 URL:

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

2022-12-26 Thread Yuxin Tan
Hi, Weihua Thanks for your suggestions. > 1. How about reducing ExclusiveBuffersPerChannel to 1 first when the total buffer is not enough? I think it's a good idea. Will try and check the results in PoC. Before all read buffers use floating buffers, I will try to use (ExclusiveBuffersPerChannel

[jira] [Created] (FLINK-30506) Add documentation for writing Table Store with Spark3

2022-12-26 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-30506: --- Summary: Add documentation for writing Table Store with Spark3 Key: FLINK-30506 URL: https://issues.apache.org/jira/browse/FLINK-30506 Project: Flink Issue

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2022-12-26 Thread yuxia
HI, Biao. Thanks for driving this FLIP. After quick look of this FLIP, I have a question about "expose the attempt number which can be used to isolate data produced by speculative executions with the same subtask id". What the sink expect to do to isolate data produced by speculative executions?

Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job

2022-12-26 Thread Zhu Zhu
Hi Biao, Thanks for creating this FLIP! Supporting speculative execution of sinks is important. Also In production we see sinks chaining with other operators, e.g. sources in simple ETL jobs, and currently the task cannot do speculative execution due to sinks are not supported. +1 for the

Re: [DISCUSS] Release Kubernetes Operator 1.3.1

2022-12-26 Thread Őrhidi Mátyás
Hi Gyula, Sounds good! Thanks for volunteering. I'll review [FLINK-30406] tomorrow. Best, Matyas On Sun, Dec 25, 2022 at 1:14 PM Gyula Fóra wrote: > Hi Devs! > > It's been only a few weeks since the 1.3.0 release but we have identified > and fixed a few critical issues that are currently

[DISCUSS] FLIP-280: Introduce a new explain mode to provide SQL advice

2022-12-26 Thread Jane Chan
Hi, devs, I would like to start a discussion on FLIP-280: Introduce a new explain mode to provide SQL advice[1]. Currently, Flink SQL EXPLAIN statement provides three details to display the plan. However, there is a considerable gap between the current explained result and the actual, applicable

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

2022-12-26 Thread Weihua Hu
Hi Yuxin, Thanks for the proposal. "Insufficient number of network buffers" exceptions also bother us. It's too hard for users to figure out how much network buffer they really need. It relates to partitioner type, parallelism, slots per taskmanager. Since streaming jobs are our primary

[DISCUSS] FLIP-282: Introduce Delete & Update API

2022-12-26 Thread yuxia
Hi, devs. I'd like to start a discussion about FLIP-282: Introduce Delete & Update API[1]. Row-Level SQL Delete & Update are becoming more and more important in modern big data workflow. The use cases include deleting a set of rows for regulatory compliance, updating a set of rows for data

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

2022-12-26 Thread Yuxin Tan
Hi, all Thanks for the reply and feedback for everyone! After combining everyone's comments, the main concerns, and corresponding adjustments are as follows. @Guowei Ma, Thanks for your feedback. > should we introduce a _new_ non-orthogonal

[jira] [Created] (FLINK-30505) Close the connection between TM and JM when task executor failed

2022-12-26 Thread Yongming Zhang (Jira)
Yongming Zhang created FLINK-30505: -- Summary: Close the connection between TM and JM when task executor failed Key: FLINK-30505 URL: https://issues.apache.org/jira/browse/FLINK-30505 Project: Flink

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

2022-12-26 Thread Hang Ruan
Hi, thanks for the feedback, Zhu Zhu and Qingsheng. After combining everyone's comments, the main concerns and corresponding adjustments are as follows. Q1: Common metrics are not quite useful. numEventsIn and numEventsOut counters will be removed from the OperatorCoordinatorMetricGroup. These

Re: [DISCUSS] FLIP-266: Simplify network memory configurations for TaskManager

2022-12-26 Thread Zhu Zhu
Hi Yuxin, Thanks for creating this FLIP. It's good if Flink does not require users to set a very large network memory, or tune the advanced(hard-to-understand) per-channel/per-gate buffer configs, to avoid "Insufficient number of network buffers" exceptions which can easily happen for large

[jira] [Created] (FLINK-30504) Fix UnsupportedFileSystemSchemeException when writing Table Store on OSS with other engines

2022-12-26 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-30504: --- Summary: Fix UnsupportedFileSystemSchemeException when writing Table Store on OSS with other engines Key: FLINK-30504 URL: https://issues.apache.org/jira/browse/FLINK-30504

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

2022-12-26 Thread Qingsheng Ren
Thanks for the FLIP, Hang! This FLIP overall looks good to me. Actually I share the same concern with Zhu that numEventsIn and numEventsOut counters are not quite useful to end users. OperatorEvent is a quite low-level abstraction, which requires instantialization in order to be practical to

Re: [VOTE] FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records

2022-12-26 Thread Qingsheng Ren
+1 (binding) Best, Qingsheng Ververica (Alibaba) On Wed, Dec 21, 2022 at 3:13 PM Dong Lin wrote: > Hi all, > > We would like to start the vote for FLIP-208: Add RecordEvaluator to > dynamically stop source based on de-serialized records [1]. This FLIP was > discussed in this thread [2]. > >