[jira] [Created] (FLINK-29913) Shared state would be discarded by mistake when maxConcurrentCheckpoint>1

2022-11-06 Thread Yanfei Lei (Jira)
Yanfei Lei created FLINK-29913: -- Summary: Shared state would be discarded by mistake when maxConcurrentCheckpoint>1 Key: FLINK-29913 URL: https://issues.apache.org/jira/browse/FLINK-29913 Project: Flink

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-06 Thread Yanfei Lei
Hi Max, Thanks for the proposal. This proposal makes Flink better adapted to cloud-native applications! After reading the FLIP, I'm curious about some points: 1) It's said that "The first step is collecting metrics for all JobVertices by combining metrics from all the runtime subtasks and comput

Re: SQL Gateway and SQL Client

2022-11-06 Thread Shengkai Fang
Hi, all. Sorry for the late reply. > Is the gateway mode planned to be supported for SQL Client in 1.17? > Do you have anything you can already share so we can start with your work or just play around with it. Yes. @yzl is working on it and he will list the implementation plan later and share the

Re: [DISCUSS] Repeatable cleanup of checkpoint data

2022-11-06 Thread Yang Wang
Thanks Matthias for continuously improving the clean-up process. Given that we highly depends on K8s APIServer for HA implementation, I am not in favor of storing too many entries in the ConfigMap, as well as adding more update requests to the APIServer. So I lean towards Proposal #2. It just work

[jira] [Created] (FLINK-29912) scan.partition. column can specify any type of field

2022-11-06 Thread waywtdcc (Jira)
waywtdcc created FLINK-29912: Summary: scan.partition. column can specify any type of field Key: FLINK-29912 URL: https://issues.apache.org/jira/browse/FLINK-29912 Project: Flink Issue Type: New

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-06 Thread Gyula Fóra
Hi Dong! Let me try to answer the questions :) 1 : busyTimeMsPerSecond is not specific for CPU, it measures the time spent in the main record processing loop for an operator if I understand correctly. This includes IO operations too. 2: We should add this to the FLIP I agree. It would be a Durat

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-06 Thread Dong Lin
Hi Max, Thank you for the proposal. The proposal tackles a very important issue for Flink users and the design looks promising overall! I have some questions to better understand the proposed public interfaces and the algorithm. 1) The proposal seems to assume that the operator's busyTimeMsPerSe

Re: [DISCUSS] FLIP-271: Autoscaling

2022-11-06 Thread Gyula Fóra
@Pedro: The current design focuses on record processing time metrics. In most cases when we need to scale (such as too much state per operator), record processing time actually slows, so it would detect that. Of course in the future we can add new logic if we see something missing. @ConradJam: We

[jira] [Created] (FLINK-29911) Improve performance of AgglomerativeClustering

2022-11-06 Thread Zhipeng Zhang (Jira)
Zhipeng Zhang created FLINK-29911: - Summary: Improve performance of AgglomerativeClustering Key: FLINK-29911 URL: https://issues.apache.org/jira/browse/FLINK-29911 Project: Flink Issue Type:

[jira] [Created] (FLINK-29910) JdbcNumericBetweenParametersProvider(Variables within the method are extracted)

2022-11-06 Thread zhilinli (Jira)
zhilinli created FLINK-29910: Summary: JdbcNumericBetweenParametersProvider(Variables within the method are extracted) Key: FLINK-29910 URL: https://issues.apache.org/jira/browse/FLINK-29910 Project: Flin