Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

2023-06-30 Thread Ryan van Huuksloot
Sounds good - I haven't looked too much into the comparison of the old and new client. Opened a ticket: https://issues.apache.org/jira/browse/FLINK-32508 I may have some time to support this ticket in 2 weeks but anyone can feel free to start working on it. Thanks, Ryan van Huuksloot Sr.

[jira] [Created] (FLINK-32508) Flink-Metrics Prometheus - Native Histograms / Native Counters

2023-06-30 Thread Ryan van Huuksloot (Jira)
Ryan van Huuksloot created FLINK-32508: -- Summary: Flink-Metrics Prometheus - Native Histograms / Native Counters Key: FLINK-32508 URL: https://issues.apache.org/jira/browse/FLINK-32508 Project:

[jira] [Created] (FLINK-32507) Document KafkaSink SinkWriterMetricGroup metrics

2023-06-30 Thread Mason Chen (Jira)
Mason Chen created FLINK-32507: -- Summary: Document KafkaSink SinkWriterMetricGroup metrics Key: FLINK-32507 URL: https://issues.apache.org/jira/browse/FLINK-32507 Project: Flink Issue Type:

[jira] [Created] (FLINK-32506) Add the watermark aggregation benchmark for source coordinator

2023-06-30 Thread Rui Fan (Jira)
Rui Fan created FLINK-32506: --- Summary: Add the watermark aggregation benchmark for source coordinator Key: FLINK-32506 URL: https://issues.apache.org/jira/browse/FLINK-32506 Project: Flink Issue

[jira] [Created] (FLINK-32505) Compilation error in ProducerMergedPartitionFileWriter

2023-06-30 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-32505: - Summary: Compilation error in ProducerMergedPartitionFileWriter Key: FLINK-32505 URL: https://issues.apache.org/jira/browse/FLINK-32505 Project: Flink

Re: [VOTE] FLIP-309: Support using larger checkpointing interval when source is processing backlog

2023-06-30 Thread Chesnay Schepler
-1 (binding) I feel like this FLIP needs a bit more time in the oven. It seems to be very light on actual details; you can summarize the entire changes section as "The enumerator calls this method and then another checkpoint interval is used." I would love to know how this is wired into the

[jira] [Created] (FLINK-32504) DefaultLeaderElectionService#running field can be removed

2023-06-30 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-32504: - Summary: DefaultLeaderElectionService#running field can be removed Key: FLINK-32504 URL: https://issues.apache.org/jira/browse/FLINK-32504 Project: Flink

[jira] [Created] (FLINK-32503) FLIP-285 technical debt

2023-06-30 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-32503: - Summary: FLIP-285 technical debt Key: FLINK-32503 URL: https://issues.apache.org/jira/browse/FLINK-32503 Project: Flink Issue Type: Technical Debt

[jira] [Created] (FLINK-32502) Remove AbstractLeaderElectionService

2023-06-30 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-32502: - Summary: Remove AbstractLeaderElectionService Key: FLINK-32502 URL: https://issues.apache.org/jira/browse/FLINK-32502 Project: Flink Issue Type: Technical

Re: [VOTE] FLIP-309: Support using larger checkpointing interval when source is processing backlog

2023-06-30 Thread Piotr Nowojski
Hey, Sorry to disturb this voting, but after discussing this thoroughly with Chesnay and Stefan Richter I have to vote: -1 (binding) mainly to suspend the current voting thread. Please take a look at my mail at dev mailing list. Best, Piotrek czw., 29 cze 2023 o 14:59 feng xiangyu napisał(a):

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-06-30 Thread Piotr Nowojski
Hey, Sorry for a late reply, I was OoO for a week. I have three things to point out. 1. === The updated proposal is indeed better, but to be honest I still don't like it, for mostly the same reasons that I have mentioned earlier: - only a partial solution, that doesn't address all

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

2023-06-30 Thread Martijn Visser
Hi Ryan, If I look at the changelog for the simpleclient 0.10 [1], they've switched their data model. So if you upgrade to the later version, the data model for existing Flink Prometheus users would be broken IIRC. That's why I think option 1 is more clean: it provides the option to the user to

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

2023-06-30 Thread Ryan van Huuksloot
I'd have to check but the original plan is to upgrade the client but keep the flink-metrics-prometheus implementation the same. This should keep the metrics collection consistent even with the client upgrade - this would need to be verified. But if that is the worry then we could create a new

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

2023-06-30 Thread Martijn Visser
Hi Patrick, Yeah, but you would need the latest version of the client, which would break the implementation for the current, outdated one, wouldn't it? Best regards, Martijn On Fri, Jun 30, 2023 at 3:35 PM Ryan van Huuksloot wrote: > Hi Martijn, > > Option 2 and 3 would use a single client.

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

2023-06-30 Thread Ryan van Huuksloot
Hi Martijn, Option 2 and 3 would use a single client. It would just register the metrics differently. Does that make sense? Does that change your perspective? Thanks, Ryan van Huuksloot Sr. Production Engineer | Streaming Platform [image: Shopify]

[jira] [Created] (FLINK-32501) Wrong execution plan of a proctime window aggregation generated due to incorrect cost evaluation

2023-06-30 Thread lincoln lee (Jira)
lincoln lee created FLINK-32501: --- Summary: Wrong execution plan of a proctime window aggregation generated due to incorrect cost evaluation Key: FLINK-32501 URL: https://issues.apache.org/jira/browse/FLINK-32501

Re: Flink-Metrics Prometheus - Native Histograms / Native Counters

2023-06-30 Thread Martijn Visser
Hi Ryan, I think option 2 and option 3 won't work, because there can be only one version of the client. I don't think we should make a clean break on metrics in a minor version, but only in major. All in all, I think option 1 would be the best. We could deprecate the existing one and remove it

[jira] [Created] (FLINK-32500) Update dependency versions for AWS connectors package

2023-06-30 Thread Aleksandr Pilipenko (Jira)
Aleksandr Pilipenko created FLINK-32500: --- Summary: Update dependency versions for AWS connectors package Key: FLINK-32500 URL: https://issues.apache.org/jira/browse/FLINK-32500 Project: Flink

Re: [DISCUSS] FLIP-278: Hybrid Source Connector

2023-06-30 Thread Ran Tao
Hi, Ilya. thanks for your reply. If your first s3 and second kafka source has same schema. Currently hybrid table source can work. For you question >> But because Flink’s optimizer removes unused fields from internal records in the batch mode, the problem of inconsistent schema arises at

[VOTE] FLIP-321: introduce an API deprecation process

2023-06-30 Thread Becket Qin
Hi folks, I'd like to start the VOTE for FLIP-321[1] which proposes to introduce an API deprecation process to Flink. The discussion thread for the FLIP can be found here[2]. The vote will be open until at least July 4, following the consensus voting process. Thanks, Jiangjie (Becket) Qin [1]

[jira] [Created] (FLINK-32499) Removing dependency of flink-connector-aws on a specific flink-shaded version

2023-06-30 Thread Hong Liang Teoh (Jira)
Hong Liang Teoh created FLINK-32499: --- Summary: Removing dependency of flink-connector-aws on a specific flink-shaded version Key: FLINK-32499 URL: https://issues.apache.org/jira/browse/FLINK-32499

[jira] [Created] (FLINK-32498) array_max return type should always nullable

2023-06-30 Thread Jacky Lau (Jira)
Jacky Lau created FLINK-32498: - Summary: array_max return type should always nullable Key: FLINK-32498 URL: https://issues.apache.org/jira/browse/FLINK-32498 Project: Flink Issue Type:

[jira] [Created] (FLINK-32497) An exception will be thrown when the condition of the IF FUNCTION is FALSE and the false_value parameter is a function.

2023-06-30 Thread Han Zhuo (Jira)
Han Zhuo created FLINK-32497: Summary: An exception will be thrown when the condition of the IF FUNCTION is FALSE and the false_value parameter is a function. Key: FLINK-32497 URL: