[VOTE] FLIP-309: Support using larger checkpointing interval when source is processing backlog

2023-07-17 Thread Dong Lin
Hi all, We would like to start the vote for FLIP-309: Support using larger checkpointing interval when source is processing backlog [1]. This FLIP was discussed in this thread [2]. The vote will be open until at least July 21st (at least 72 hours), following the consensus voting process.

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-07-17 Thread Dong Lin
Hi all, I have updated FLIP-309 as suggested by Piotr to include a reference to FLIP-328 in the future work section. Piotra, Stephan, and I discussed offline regarding the choice between execution.checkpointing.max-interval and execution.checkpointing.interval-during-backlog. The advantage of

Re: [DISCUSS][2.0] FLIP-337: Remove JarRequestBody#programArgs

2023-07-17 Thread Jing Ge
Hi Chesnay, I am trying to understand what is the right removal process with this concrete example. Given all things about the programArgs are private or package private except the constructor. Will you just mark it as deprecated with constructor overloading in 1.18 and remove it in 2.0? Should

[jira] [Created] (FLINK-32613) Disable check for single rowtime attribute for sinks

2023-07-17 Thread Timo Walther (Jira)
Timo Walther created FLINK-32613: Summary: Disable check for single rowtime attribute for sinks Key: FLINK-32613 URL: https://issues.apache.org/jira/browse/FLINK-32613 Project: Flink Issue

Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement

2023-07-17 Thread Jing Ge
Hi Yunfeng, Thanks for your clarification. It might make sense to add one more performance test with object-reuse disabled alongside to let us know how big the improvement will be. Best regards, Jing On Thu, Jul 13, 2023 at 11:51 AM Matt Wang wrote: > Hi Yunfeng, > > Thanks for the proposal.

[DISCUSS] Flink Kubernetes Operator cleanup procedure

2023-07-17 Thread Wong, Daren
Hi devs, I would like to enquire about the cleanup procedure upon FlinkSessionJob deletion. Currently, FlinkSessionJobController would trigger a cleanup in the SessionJobReconciler which in turn cancels the Flink Job. Link to code:

[jira] [Created] (FLINK-32612) Upgrade Pulsar version to 3.0.0 for the connector

2023-07-17 Thread Zili Chen (Jira)
Zili Chen created FLINK-32612: - Summary: Upgrade Pulsar version to 3.0.0 for the connector Key: FLINK-32612 URL: https://issues.apache.org/jira/browse/FLINK-32612 Project: Flink Issue Type:

[jira] [Created] (FLINK-32611) Redirect to Apache Paimon's link instead of legacy flink table store

2023-07-17 Thread Yun Tang (Jira)
Yun Tang created FLINK-32611: Summary: Redirect to Apache Paimon's link instead of legacy flink table store Key: FLINK-32611 URL: https://issues.apache.org/jira/browse/FLINK-32611 Project: Flink

[jira] [Created] (FLINK-32609) Kafka table source supports projection push down

2023-07-17 Thread dalongliu (Jira)
dalongliu created FLINK-32609: - Summary: Kafka table source supports projection push down Key: FLINK-32609 URL: https://issues.apache.org/jira/browse/FLINK-32609 Project: Flink Issue Type:

[jira] [Created] (FLINK-32610) JSON format supports projection push down

2023-07-17 Thread dalongliu (Jira)
dalongliu created FLINK-32610: - Summary: JSON format supports projection push down Key: FLINK-32610 URL: https://issues.apache.org/jira/browse/FLINK-32610 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32608) Improve source reusing with projection push down

2023-07-17 Thread dalongliu (Jira)
dalongliu created FLINK-32608: - Summary: Improve source reusing with projection push down Key: FLINK-32608 URL: https://issues.apache.org/jira/browse/FLINK-32608 Project: Flink Issue Type:

[jira] [Created] (FLINK-32607) Kafka table source support projection pushdown

2023-07-17 Thread dalongliu (Jira)
dalongliu created FLINK-32607: - Summary: Kafka table source support projection pushdown Key: FLINK-32607 URL: https://issues.apache.org/jira/browse/FLINK-32607 Project: Flink Issue Type:

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

2023-07-17 Thread Maximilian Michels
+1 On Mon, Jul 17, 2023 at 10:45 AM Chesnay Schepler wrote: > > +1 > > On 16/07/2023 08:10, Mohan, Deepthi wrote: > > @Chesnay > > > > Thank you for your feedback. > > > > An important takeaway from the previous discussion [1] and your feedback > > was to keep the design and text/diagram

[jira] [Created] (FLINK-32606) Support namespace and flink version specific default configs

2023-07-17 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-32606: -- Summary: Support namespace and flink version specific default configs Key: FLINK-32606 URL: https://issues.apache.org/jira/browse/FLINK-32606 Project: Flink

Re: [VOTE] Release 2.0 must-have work items

2023-07-17 Thread Yuan Mei
Hey Yun and Xintong, (Had a quick offline discussion with Yun) 1. I agree the current implementation of the queryable state is not a blocker of anything related to disaggregated state management. They are different things. 2. On the other hand, "queryable snapshot" is not a completely equivalent

[jira] [Created] (FLINK-32605) JoinITCase.testFullOuterJoinWithMultipleKeys fails with TimeoutException: Futures timed out after [20 seconds]

2023-07-17 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32605: --- Summary: JoinITCase.testFullOuterJoinWithMultipleKeys fails with TimeoutException: Futures timed out after [20 seconds] Key: FLINK-32605 URL:

[jira] [Created] (FLINK-32604) PyFlink end-to-end fails with kafka-server-stop.sh: No such file or directory

2023-07-17 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32604: --- Summary: PyFlink end-to-end fails with kafka-server-stop.sh: No such file or directory Key: FLINK-32604 URL: https://issues.apache.org/jira/browse/FLINK-32604

Re: [DISCUSS] FLIP-335: Removing Flink's Time classes as part of Flink 2.0

2023-07-17 Thread Chesnay Schepler
I don't understand this bit:/ "One minor Scala change is necessary, though: We need to touch the Scala implementation of the Pattern class (in flink-cep). Pattern requires a new method which needs to be implemented in the Scala Pattern class as well to comply with

[jira] [Created] (FLINK-32603) Avoid consuming twice when two pipelines have the same table source

2023-07-17 Thread Jiang Xin (Jira)
Jiang Xin created FLINK-32603: - Summary: Avoid consuming twice when two pipelines have the same table source Key: FLINK-32603 URL: https://issues.apache.org/jira/browse/FLINK-32603 Project: Flink

Re: [DISCUSS] FLIP 333 - Redesign Apache Flink website

2023-07-17 Thread Chesnay Schepler
+1 On 16/07/2023 08:10, Mohan, Deepthi wrote: @Chesnay Thank you for your feedback. An important takeaway from the previous discussion [1] and your feedback was to keep the design and text/diagram changes separate as each change for text and diagrams likely require deeper discussion.

[jira] [Created] (FLINK-32602) Nightly builds for Elasticsearch are failing with "pull access denied for flink-base"

2023-07-17 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-32602: -- Summary: Nightly builds for Elasticsearch are failing with "pull access denied for flink-base" Key: FLINK-32602 URL: https://issues.apache.org/jira/browse/FLINK-32602

[jira] [Created] (FLINK-32601) Unstable RemoteChannelThroughputBenchmark_remoteRebalance_jmhTest

2023-07-17 Thread Hangxiang Yu (Jira)
Hangxiang Yu created FLINK-32601: Summary: Unstable RemoteChannelThroughputBenchmark_remoteRebalance_jmhTest Key: FLINK-32601 URL: https://issues.apache.org/jira/browse/FLINK-32601 Project: Flink

[jira] [Created] (FLINK-32600) SQL Server JDBC Connector wrong quoteIdentifier + issue when there is no update statement in upsert statement

2023-07-17 Thread Maxime Radigue (Jira)
Maxime Radigue created FLINK-32600: -- Summary: SQL Server JDBC Connector wrong quoteIdentifier + issue when there is no update statement in upsert statement Key: FLINK-32600 URL:

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-07-17 Thread Shammon FY
Hi devs, Thanks for all the valuable feedback. If there are no more concerns about FLIP-314[1], I would like to start voting later, thanks. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener Best, Shammon FY On Wed, Jul 12, 2023 at 11:18 

[jira] [Created] (FLINK-32599) Benchmarks in Slack have failed consistently since 2023-07-14

2023-07-17 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-32599: -- Summary: Benchmarks in Slack have failed consistently since 2023-07-14 Key: FLINK-32599 URL: https://issues.apache.org/jira/browse/FLINK-32599 Project: Flink

Re: [VOTE] Release 2.0 must-have work items

2023-07-17 Thread Xintong Song
Thanks for the clarification. If the list of "Remove deprecated APIs" means, we must remove the code in > Flink-2.0 initial release, I would vote -1 for queryable state before we > get an alternative. FYI, the removal of queryable state is currently marked as the `must-have` priority. Of

[jira] [Created] (FLINK-32598) Spill data from feedback edge to disk to avoid possible OOM

2023-07-17 Thread Zhipeng Zhang (Jira)
Zhipeng Zhang created FLINK-32598: - Summary: Spill data from feedback edge to disk to avoid possible OOM Key: FLINK-32598 URL: https://issues.apache.org/jira/browse/FLINK-32598 Project: Flink

Re: [VOTE] Release 2.0 must-have work items

2023-07-17 Thread Yun Tang
Hi Xintong, If the current implementation of queryable state would not block the implementation of disaggregated state-backends. I prefer to not removing the implementation until we have a better solution (maybe based on the queryable snapshot) cc @Yuan. If the list of "Remove deprecated APIs"

Re: [DISCUSS] FLIP-330: Support specifying record timestamp requirement

2023-07-17 Thread Matt Wang
Hi Yunfeng, Thank you for testing 1 again, and look forward to the performance results of TPC-DS later. We use a latency marker to monitor the end-to-end latency of flink jobs. If the latencyTrackingInterval is set too small(like 5ms), it will have a large impact on performance. But if the