[jira] [Created] (FLINK-32364) Add Rescaling benchmark for ChangelogStateBackend

2023-06-15 Thread Hangxiang Yu (Jira)
Hangxiang Yu created FLINK-32364: Summary: Add Rescaling benchmark for ChangelogStateBackend Key: FLINK-32364 URL: https://issues.apache.org/jira/browse/FLINK-32364 Project: Flink Issue

[jira] [Created] (FLINK-32363) calcite 1.21 supports type coercion but flink don't enable it in validate

2023-06-15 Thread jackylau (Jira)
jackylau created FLINK-32363: Summary: calcite 1.21 supports type coercion but flink don't enable it in validate Key: FLINK-32363 URL: https://issues.apache.org/jira/browse/FLINK-32363 Project: Flink

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-06-15 Thread Dong Lin
Hi again Piotr, Thank you for the reply. Please see my reply inline. On Fri, Jun 16, 2023 at 12:11 AM Piotr Nowojski wrote: > Hi again Dong, > > > I understand that JM will get the backpressure-related metrics every time > > the RestServerEndpoint receives the REST request to get these

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-06-15 Thread Feng Jin
Hi everyone, Thank you all for your valuable input. If there are no further questions or concerns regarding FLIP-308[1], I would like to start voting on Monday, June 19th. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-308%3A+Support+Time+Travel Best, Feng On Mon, Jun 12, 2023 at

[jira] [Created] (FLINK-32362) SourceAlignment announceCombinedWatermark period task maybe lost

2023-06-15 Thread Cai Liuyang (Jira)
Cai Liuyang created FLINK-32362: --- Summary: SourceAlignment announceCombinedWatermark period task maybe lost Key: FLINK-32362 URL: https://issues.apache.org/jira/browse/FLINK-32362 Project: Flink

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-15 Thread Xintong Song
> > Public API is a well defined common concept, and one of its > convention is that it only changes with a major version change. > I agree. And from my understanding, demoting a Public API is also a kind of such change, just like removing one, which can only happen with major version bumps. I'm

[jira] [Created] (FLINK-32361) error after replace dependent jar file

2023-06-15 Thread Spongebob (Jira)
Spongebob created FLINK-32361: - Summary: error after replace dependent jar file Key: FLINK-32361 URL: https://issues.apache.org/jira/browse/FLINK-32361 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] Updates to Flink's external connector CI workflows

2023-06-15 Thread Mason Chen
Can't wait to use this in the Kafka repo, especially the thread dump. Thanks Martjin and Sergey! Best, Mason On Thu, Jun 15, 2023 at 4:05 PM Sergey Nuyanzin wrote: > Thanks a lot Martijn for making it > > On Thu, Jun 15, 2023 at 2:07 PM Leonard Xu wrote: > > > Thanks Martijn for the great

Re: [ANNOUNCE] Updates to Flink's external connector CI workflows

2023-06-15 Thread Sergey Nuyanzin
Thanks a lot Martijn for making it On Thu, Jun 15, 2023 at 2:07 PM Leonard Xu wrote: > Thanks Martijn for the great work ! > > > Best, > Leonard > > -- Best regards, Sergey

Flink-Metrics Prometheus - Native Histograms / Native Counters

2023-06-15 Thread Ryan van Huuksloot
Hello, Internally we use the flink-metrics-prometheus jar and we noticed that the code is a little out of date. Primarily, there are new metric types in Prometheus that would allow for the exporter to write Counters and Histograms as Native metrics in prometheus (vs writing as Gauges). I noticed

[VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

2023-06-15 Thread Mason Chen
Hi all, Thank you to everyone for the feedback on FLIP-246 [1]. Based on the discussion thread [2], we have come to a consensus on the design and are ready to take a vote to contribute this to Flink. This voting thread will be open for at least 72 hours (excluding weekends, until June 20th

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-06-15 Thread Piotr Nowojski
Hi again Dong, > I understand that JM will get the backpressure-related metrics every time > the RestServerEndpoint receives the REST request to get these metrics. But > I am not sure if RestServerEndpoint is already always receiving the REST > metrics at regular interval (suppose there is no

[jira] [Created] (FLINK-32360) Optimize DataStream#coGroup in stream mode when results are emitted at end of input

2023-06-15 Thread Dong Lin (Jira)
Dong Lin created FLINK-32360: Summary: Optimize DataStream#coGroup in stream mode when results are emitted at end of input Key: FLINK-32360 URL: https://issues.apache.org/jira/browse/FLINK-32360 Project:

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-15 Thread Chesnay Schepler
On 13/06/2023 17:26, Becket Qin wrote: It would be valuable if we can avoid releasing minor versions for previous major versions. On paper, /absolutely /agree, but I'm not sure how viable that is in practice. On the current 2.0 agenda is potentially dropping support for Java 8/11, which

Re: AsyncFunction vs Async Sink

2023-06-15 Thread Teoh, Hong
Hi Lu, > 1. Is there any problem if we use Async Function for such a user case? We can > simply drop the output and use Unordered mode. As far as I can tell, it is similar, other than the retry strategy available for AsyncFunctions and batching for Async Sink. Both should work on Flink. >

[jira] [Created] (FLINK-32359) AdaptiveSchedulerBuilder shoudl accept executor service in constructor

2023-06-15 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32359: Summary: AdaptiveSchedulerBuilder shoudl accept executor service in constructor Key: FLINK-32359 URL: https://issues.apache.org/jira/browse/FLINK-32359

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-15 Thread Becket Qin
Hi Xintong, I think the key of stability guarantees is about not breaking the > commitments we made to users. From my understanding, when we mark an API as > Public in for example 1.18, we make the following two commitments. > a) The API will not be removed in all the 1.x serials. > b) The API

[jira] [Created] (FLINK-32358) CI may unintentionally use fallback akka loader

2023-06-15 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32358: Summary: CI may unintentionally use fallback akka loader Key: FLINK-32358 URL: https://issues.apache.org/jira/browse/FLINK-32358 Project: Flink

Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-15 Thread Feng Jin
+1(no-binding) Best, Feng On Thu, Jun 15, 2023 at 8:33 PM liu ron wrote: > +1(no-binding) > > Best, > Ron > > yuxia 于2023年6月15日周四 20:15写道: > > > +1 (binding) > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Leonard Xu" > > 收件人: "dev" > > 发送时间: 星期四, 2023年 6 月 15日 下午

Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-15 Thread liu ron
+1(no-binding) Best, Ron yuxia 于2023年6月15日周四 20:15写道: > +1 (binding) > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Leonard Xu" > 收件人: "dev" > 发送时间: 星期四, 2023年 6 月 15日 下午 8:09:00 > 主题: Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener > > +1 (binding) > > > Best,

Re: [DISCUSS] Status of Statefun Project

2023-06-15 Thread Martijn Visser
Let me know if you have a PR for a Flink update :) On Thu, Jun 8, 2023 at 5:52 PM Galen Warren via user wrote: > Thanks Martijn. > > Personally, I'm already using a local fork of Statefun that is compatible > with Flink 1.16.x, so I wouldn't have any need for a released version > compatible

[DISCUSS] FLIP-303: Support REPLACE TABLE AS SELECT statement

2023-06-15 Thread yuxia
Hi, devs. As the FLIPs FLIP-218[1] & FLIP-305[2] for Flink to supports CREATE TABLE AS SELECT statement has been accepted. I would like to start a discussion about FLIP-303: Support REPLACE TABLE AS SELECT+statement[3] to complete such kinds of statements. With REPLACE TABLE AS SELECT

Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-15 Thread yuxia
+1 (binding) Best regards, Yuxia - 原始邮件 - 发件人: "Leonard Xu" 收件人: "dev" 发送时间: 星期四, 2023年 6 月 15日 下午 8:09:00 主题: Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener +1 (binding) Best, Leonard > On Jun 14, 2023, at 11:21 PM, Jing Ge wrote: > > +1 (binding) > >

Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-15 Thread Leonard Xu
+1 (binding) Best, Leonard > On Jun 14, 2023, at 11:21 PM, Jing Ge wrote: > > +1 (binding) > > Best Regards, > Jing > > On Wed, Jun 14, 2023 at 4:07 PM Benchao Li wrote: > >> +1 (binding) >> >> Shammon FY 于2023年6月14日周三 19:52写道: >> >>> Hi all: >>> >>> Thanks for all the feedback for

Re: [ANNOUNCE] Updates to Flink's external connector CI workflows

2023-06-15 Thread Leonard Xu
Thanks Martijn for the great work ! Best, Leonard

[jira] [Created] (FLINK-32356) Add document for calling procedure

2023-06-15 Thread luoyuxia (Jira)
luoyuxia created FLINK-32356: Summary: Add document for calling procedure Key: FLINK-32356 URL: https://issues.apache.org/jira/browse/FLINK-32356 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32357) Elasticsearch v3.0 won't compile when testing against Flink 1.17.1

2023-06-15 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-32357: -- Summary: Elasticsearch v3.0 won't compile when testing against Flink 1.17.1 Key: FLINK-32357 URL: https://issues.apache.org/jira/browse/FLINK-32357 Project:

[jira] [Created] (FLINK-32355) Support to list procedure

2023-06-15 Thread luoyuxia (Jira)
luoyuxia created FLINK-32355: Summary: Support to list procedure Key: FLINK-32355 URL: https://issues.apache.org/jira/browse/FLINK-32355 Project: Flink Issue Type: Sub-task Components:

[jira] [Created] (FLINK-32353) Make Cassandra connector compatible with Flink 1.18

2023-06-15 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-32353: -- Summary: Make Cassandra connector compatible with Flink 1.18 Key: FLINK-32353 URL: https://issues.apache.org/jira/browse/FLINK-32353 Project: Flink

[jira] [Created] (FLINK-32354) Support to execute the call procedure operation

2023-06-15 Thread luoyuxia (Jira)
luoyuxia created FLINK-32354: Summary: Support to execute the call procedure operation Key: FLINK-32354 URL: https://issues.apache.org/jira/browse/FLINK-32354 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32352) Support convert call procedure statement to correpsonding operation

2023-06-15 Thread luoyuxia (Jira)
luoyuxia created FLINK-32352: Summary: Support convert call procedure statement to correpsonding operation Key: FLINK-32352 URL: https://issues.apache.org/jira/browse/FLINK-32352 Project: Flink

[jira] [Created] (FLINK-32351) Introduce base interfaces for call procedure

2023-06-15 Thread luoyuxia (Jira)
luoyuxia created FLINK-32351: Summary: Introduce base interfaces for call procedure Key: FLINK-32351 URL: https://issues.apache.org/jira/browse/FLINK-32351 Project: Flink Issue Type: Sub-task

[VOTE] Release flink-connector-jdbc v3.1.1, release candidate #2

2023-06-15 Thread Martijn Visser
Hi everyone, Please review and vote on the release candidate #2 for the version 3.1.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

[jira] [Created] (FLINK-32350) FLIP-311: Support Call Stored Procedure

2023-06-15 Thread luoyuxia (Jira)
luoyuxia created FLINK-32350: Summary: FLIP-311: Support Call Stored Procedure Key: FLINK-32350 URL: https://issues.apache.org/jira/browse/FLINK-32350 Project: Flink Issue Type: New Feature

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-15 Thread Lijie Wang
Hi, Benchao and Aitozi, Thanks for your feedback about this FLIP. @Benchao >> I think it would be reasonable to also support "pipeline shuffle" if possible. As I said above, runtime filter can work well with all shuffle mode, including pipeline shuffle. >> if the RuntimeFIlterBuilder could be

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

2023-06-15 Thread Robert Metzger
Thanks for the FLIP. Some comments: 1. Can you specify the full proposed configuration name? " scaling-cooldown-period" is probably not the full config name? 2. Why is the concept of scaling events and a scaling queue needed? If I remember correctly, the adaptive scheduler will just check how

FLIP-303: Support REPLACE TABLE AS SELECT statement

2023-06-15 Thread yuxia
Hi, devs. As the FLIPs FLIP-218[1] & FLIP-305[2] for Flink to supports CREATE TABLE AS SELECT statement has been accepted. I would like to start a discussion about FLIP-303: Support REPLACE TABLE AS SELECT+statement[3] to complete such kinds of statements. With REPLACE TABLE AS SELECT

Re: [VOTE] FLIP-311: Support Call Stored Procedure

2023-06-15 Thread yuxia
Hi, all. FLIP-311 [1] has been accepted. There are 5 binding votes, 3 non-binding votes: - Benchao Li (binding) - Rui Fan (binding) - Jingsong Li (binding) - Lincoln (binding) - Jing Ge (binding) - Mang Zhang (non-binding) - Feng Jin (non-binding) - Jane Chan (non-binding) Thanks everyone.

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-15 Thread Martijn Visser
Hi all, First off, thanks for opening the discussion on this topic. I think it's an important one. >From my perspective, I think that one of the major pain points that users have expressed over time, is the API stability of Flink overall. I think that every Flink user wants to have reliable,

Re: [ANNOUNCE] Updates to Flink's external connector CI workflows

2023-06-15 Thread Ahmed Hamdy
Thanks Martijn Great improvements for the connector repo stability. Best regards Ahmed Hamdy On Thu, 15 Jun 2023, 12:21 Martijn Visser, wrote: > Big thanks to Sergey Nuyanzin for his suggestion on the JVM thread dumps :) > > On Thu, Jun 15, 2023 at 1:19 PM Martijn Visser > wrote: > > > Hi all,

Re: [VOTE] Release flink-connector-jdbc v3.1.1, release candidate #1

2023-06-15 Thread Martijn Visser
Hi all, Apologies for the long delay on this one, but with the help of Sergey Nuyanzin and Joao Boto we managed to resolve this issue. To summarize: 1. The vote for this connector is cancelled, due to https://issues.apache.org/jira/browse/FLINK-31770 2. That Jira ticket has now been resolved 3.

Re: [ANNOUNCE] Updates to Flink's external connector CI workflows

2023-06-15 Thread Martijn Visser
Big thanks to Sergey Nuyanzin for his suggestion on the JVM thread dumps :) On Thu, Jun 15, 2023 at 1:19 PM Martijn Visser wrote: > Hi all, > > I would like to inform you of two changes that have been made to the > shared CI workflow that's used for Flink's externalized connectors. > > 1. Up

[ANNOUNCE] Updates to Flink's external connector CI workflows

2023-06-15 Thread Martijn Visser
Hi all, I would like to inform you of two changes that have been made to the shared CI workflow that's used for Flink's externalized connectors. 1. Up until now, weekly builds were running to validate that connector code (still) works with Flink. However, these builds were only running for code

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

2023-06-15 Thread Shammon FY
Thanks Feng for the FLIP. +1(non-binding) Best, Shammon FY On Thu, Jun 15, 2023 at 6:30 PM Leonard Xu wrote: > Thanks Feng for driving this FLIP forward. > > +1(binding) > > Best, > Leonard > > > On Jun 15, 2023, at 1:42 PM, Dong Lin wrote: > > > > Thanks Feng for the FLIP. > > > >

[jira] [Created] (FLINK-32349) Support atomic for CREATE TABLE AS SELECT(CTAS) statement

2023-06-15 Thread tartarus (Jira)
tartarus created FLINK-32349: Summary: Support atomic for CREATE TABLE AS SELECT(CTAS) statement Key: FLINK-32349 URL: https://issues.apache.org/jira/browse/FLINK-32349 Project: Flink Issue

[RESULT][VOTE] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

2023-06-15 Thread Mang Zhang
Hi, everyone. FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement [1] has been accepted. There are 5 binding votes, 2 non-binding votes [2]. - yuxia(binding) - Jing Ge(binding) - Ron Liu(non-binding) - Rui Fan(binding) - Jingsong Li(binding) - Lincoln Lee(binding) - Mang

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

2023-06-15 Thread Leonard Xu
Thanks Feng for driving this FLIP forward. +1(binding) Best, Leonard > On Jun 15, 2023, at 1:42 PM, Dong Lin wrote: > > Thanks Feng for the FLIP. > > +1(binding) > > Cheers, > Dong > > On Wed, Jun 14, 2023 at 10:35 AM Feng Jin wrote: > >> Hi everyone >> >> Thanks for all the feedback

[jira] [Created] (FLINK-32348) MongoDB tests are flaky and time out

2023-06-15 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-32348: -- Summary: MongoDB tests are flaky and time out Key: FLINK-32348 URL: https://issues.apache.org/jira/browse/FLINK-32348 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-32347) Exceptions from the CompletedCheckpointStore are not registered by the CheckpointFailureManager

2023-06-15 Thread Tigran Manasyan (Jira)
Tigran Manasyan created FLINK-32347: --- Summary: Exceptions from the CompletedCheckpointStore are not registered by the CheckpointFailureManager Key: FLINK-32347 URL:

[jira] [Created] (FLINK-32346) JdbcNumericBetweenParametersProvider Sharding key boundaries large storage long integer overflow, use BigDecimal instead Long

2023-06-15 Thread zhilinli (Jira)
zhilinli created FLINK-32346: Summary: JdbcNumericBetweenParametersProvider Sharding key boundaries large storage long integer overflow, use BigDecimal instead Long Key: FLINK-32346 URL:

[jira] [Created] (FLINK-32345) Improve parallel download of RocksDB incremental state

2023-06-15 Thread Stefan Richter (Jira)
Stefan Richter created FLINK-32345: -- Summary: Improve parallel download of RocksDB incremental state Key: FLINK-32345 URL: https://issues.apache.org/jira/browse/FLINK-32345 Project: Flink

[jira] [Created] (FLINK-32344) MongoDB connector support unbounded streaming read via ChangeStream feature

2023-06-15 Thread Jiabao Sun (Jira)
Jiabao Sun created FLINK-32344: -- Summary: MongoDB connector support unbounded streaming read via ChangeStream feature Key: FLINK-32344 URL: https://issues.apache.org/jira/browse/FLINK-32344 Project:

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-15 Thread Aitozi
Hi Lijie, Nice to see this valuable feature. After reading the FLIP I have some questions below: >Schedule the TableSource(dim) first. How does it know to schedule the TableSource(dim) first ? IMO, In the current implementation two source table operators will be executed simultaneously.

[jira] [Created] (FLINK-32342) SQL Server container behaves unexpected while testing with several surefire forks

2023-06-15 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32342: --- Summary: SQL Server container behaves unexpected while testing with several surefire forks Key: FLINK-32342 URL: https://issues.apache.org/jira/browse/FLINK-32342

[jira] [Created] (FLINK-32343) Fix exception for jdbc tools

2023-06-15 Thread Fang Yong (Jira)
Fang Yong created FLINK-32343: - Summary: Fix exception for jdbc tools Key: FLINK-32343 URL: https://issues.apache.org/jira/browse/FLINK-32343 Project: Flink Issue Type: Bug Components:

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-15 Thread Benchao Li
Hi Lijie, Regarding the shuffle mode, I think it would be reasonable to also support "pipeline shuffle" if possible. "pipeline shuffle" is a essential for OLAP/MPP computing, although this has not been much exposed to users for now, I know a few companies that uses Flink as a MPP computing

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-15 Thread Lijie Wang
Hi Yuxia, I made a mistake in the above response. The runtime filter can work well with all shuffle mode. However, hybrid shuffle and blocking shuffle are currently recommended for batch jobs (piepline shuffle is not recommended). One more thing to mention here is that we will force the edge

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-15 Thread Mason Chen
Hi Gordon, Thanks for your feedback as always. Why not just Map? I think it makes sense to relocate the bootstrapServer field in the Kafka properties (ClusterMetadata abstraction), since that is intuitive based on how it is defined in the Kafka clients library. It also makes the uniqueness of a