[jira] [Created] (FLINK-35247) Upgrade spotless apply to `2.41.1` in flink-connector-parent to work with Java 21

2024-04-26 Thread Mason Chen (Jira)
Mason Chen created FLINK-35247: -- Summary: Upgrade spotless apply to `2.41.1` in flink-connector-parent to work with Java 21 Key: FLINK-35247 URL: https://issues.apache.org/jira/browse/FLINK-35247

[jira] [Created] (FLINK-35122) Implement Watermark Alignment for DynamicKafkaSource

2024-04-16 Thread Mason Chen (Jira)
Mason Chen created FLINK-35122: -- Summary: Implement Watermark Alignment for DynamicKafkaSource Key: FLINK-35122 URL: https://issues.apache.org/jira/browse/FLINK-35122 Project: Flink Issue Type

[RESULT][VOTE] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-02-20 Thread Mason Chen
Hi devs, I'm happy to announce that FLIP-417: Expose JobManagerOperatorMetrics via REST API [1] has been accepted with 6 approving votes (3 binding) [2]: - Hang Ruan (non-binding) - Xuyang (non-binding) - Rui Fan (binding) - Maximilian Michels (binding) - Thomas Weise (binding) - Alexander

[jira] [Created] (FLINK-34445) Integrate new endpoint with Flink UI metrics section

2024-02-14 Thread Mason Chen (Jira)
Mason Chen created FLINK-34445: -- Summary: Integrate new endpoint with Flink UI metrics section Key: FLINK-34445 URL: https://issues.apache.org/jira/browse/FLINK-34445 Project: Flink Issue Type

[jira] [Created] (FLINK-34444) Add new endpoint handler to flink

2024-02-14 Thread Mason Chen (Jira)
Mason Chen created FLINK-3: -- Summary: Add new endpoint handler to flink Key: FLINK-3 URL: https://issues.apache.org/jira/browse/FLINK-3 Project: Flink Issue Type: Sub-task

Re: Re: [VOTE] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-02-14 Thread Mason Chen
needed is the mention of the return data > type. > > I assume since there are potentially multiple coordinators that can > belong > > to the same vertex, the proposed new endpoint will return an array. Is > that > > the plan? > > > > Overall +1 from my side.

Re: Re: [VOTE] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-02-13 Thread Mason Chen
Hi voters and devs, I'm inclined to close the voting thread with the additional minor details to the FLIP. Please chime in if there are any objections! Best, Mason On Wed, Feb 7, 2024 at 11:49 AM Mason Chen wrote: > Hi Voters, > > JFYI, I have modified the proposed REST API path

Re: Re: [VOTE] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-02-07 Thread Mason Chen
ang > > > > > > > > > > > > > > > > > > > > > > > > 在 2024-01-26 10:12:34,"Hang Ruan" 写道: > > > > >Thanks for the FLIP. > > > > > > > > > >+1 (non-binding) > >

Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-02-05 Thread Mason Chen
] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#system-scope Best, Mason On Tue, Jan 23, 2024 at 12:13 PM Mason Chen wrote: > Hi all, > > I hope that everyone has had sufficient time to review the discussion and > the updates on the FLIP doc. If there are n

[jira] [Created] (FLINK-34327) Use maven wrapper in operator build

2024-01-31 Thread Mason Chen (Jira)
Mason Chen created FLINK-34327: -- Summary: Use maven wrapper in operator build Key: FLINK-34327 URL: https://issues.apache.org/jira/browse/FLINK-34327 Project: Flink Issue Type: Improvement

Re: [VOTE] Release flink-connector-kafka v3.1.0, release candidate #1

2024-01-29 Thread Mason Chen
+1 (non-binding) * Verified LICENSE and NOTICE files (this RC has a NOTICE file that points to 2023 that has since been updated on the main branch by Hang) * Verified hashes and signatures * Verified no binaries * Verified poms point to 3.1.0 * Reviewed web PR * Built from source * Verified git

Re: [DISCUSS] Release new version of Flink's Kafka connector

2024-01-25 Thread Mason Chen
Hi Martijn, +1 no objections, thanks for volunteering. I'll definitely help verify the rc when it becomes available. I think FLIP-288 (I assume you meant this) doesn't introduce incompatible changes since the implementation should be state compatible as well as the default changes should be

[VOTE] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-25 Thread Mason Chen
Hi Devs, I would like to start a vote on FLIP-417: Expose JobManagerOperatorMetrics via REST API [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1]

Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-23 Thread Mason Chen
Hi all, I hope that everyone has had sufficient time to review the discussion and the updates on the FLIP doc. If there are no objections, I'll start a voting thread in 2 days. Best, Mason On Thu, Jan 18, 2024 at 2:39 PM Mason Chen wrote: > Hi Lijie, > > That's also a possibility bu

Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-18 Thread Mason Chen
> fields in OperatorQueryScopeInfo and chooses the operatorName instead of > > OperatorID. > > It is fine by my side to change from opertorName to operatorID in this > > FLIP. > > > > Best, > > Hang > > > > Mason Chen 于2024年1月17日周三 09:39写道: > > &g

[jira] [Created] (FLINK-34127) Kafka connector repo runs a duplicate of `IntegrationTests` framework tests

2024-01-16 Thread Mason Chen (Jira)
Mason Chen created FLINK-34127: -- Summary: Kafka connector repo runs a duplicate of `IntegrationTests` framework tests Key: FLINK-34127 URL: https://issues.apache.org/jira/browse/FLINK-34127 Project

Re: [DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-16 Thread Mason Chen
n vertex ID and operator ID, and do we need a url like > > `/jobs//vertices//operators/` > > to list all operator ids under this vertices? > > > > > > > > > > [1] > > > https://github.com/apache/flink/blob/7bebd2d9fac517c28afc24c0c034d77cfe2b43a6/

[DISCUSS] FLIP-417: Expose JobManagerOperatorMetrics via REST API

2024-01-11 Thread Mason Chen
Hi Devs, I'm opening this thread to discuss a short FLIP for exposing JobManagerOperatorMetrics via REST API [1]. The current set of REST APIs make it impossible to query coordinator metrics. This FLIP proposes a new REST API to query the JobManagerOperatorMetrics. [1]

[jira] [Created] (FLINK-34064) Expose JobManagerOperatorMetrics via REST API

2024-01-11 Thread Mason Chen (Jira)
Mason Chen created FLINK-34064: -- Summary: Expose JobManagerOperatorMetrics via REST API Key: FLINK-34064 URL: https://issues.apache.org/jira/browse/FLINK-34064 Project: Flink Issue Type

Re: Re: FLIP-413: Enable unaligned checkpoints by default

2024-01-08 Thread Mason Chen
Hi Piotr, I also agree with Zhanghao's assessment on the limitations of unaligned checkpoints. Some of them are already handled properly by Flink, but in the case of the "Interplay with watermarks" limitation, it is quite confusing for a new user to find that their code doesn't generate

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

2023-12-05 Thread Mason Chen
to > requests to redeploy the job. > > Sorry, I didn't understand what type of benchmarking > we should do, could you elaborate on it? Thanks a lot. > > Best, > Rui > > On Sat, Nov 18, 2023 at 3:32 AM Mason Chen wrote: > >> Hi Rui, >> >> I suppose

Re: [DISCUSS] FLIP-394: Add Metrics for Connector Agnostic Autoscaling

2023-11-27 Thread Mason Chen
its in > splitEnumerator but calculate total unAssignedSplits in SourceReader side > following your design. The action ‘assign’ is happening in SplitEnumerator > sider and should be surely managed by splitEnumerator, > why a SourceReader never assigns any splits need to report its

Re: Request a release of flink-connector-kafka version 3.1.0 (to consume kafka 3.4.0 with Flink 1.18)

2023-11-17 Thread Mason Chen
gt; > > Kind regards > > > > Jean-Marc > > ____ > > From: Martijn Visser > > Sent: Thursday, November 9, 2023 13:51 > > To: dev@flink.apache.org ; Mason Chen < > mas.chen6...@gmail.com> > > Subject: [EXTERNAL] R

Re: [DISCUSS] FLIP-394: Add Metrics for Connector Agnostic Autoscaling

2023-11-17 Thread Mason Chen
Also, it looks like externalizing the Hive connector is unblocked based on the past email thread. https://issues.apache.org/jira/browse/FLINK-30064 seems to have some progress and perhaps we shouldn't touch it for now. On Fri, Nov 17, 2023 at 11:00 AM Mason Chen wrote: > Hi Rui and

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

2023-11-17 Thread Mason Chen
Hi Rui, I suppose we could do some benchmarking on what works well for the resource providers that Flink relies on e.g. Kubernetes. Based on conferences and blogs, it seems most people are relying on Kubernetes to deploy Flink and the restart strategy has a large dependency on how well Kubernetes

Re: [DISCUSS] FLIP-394: Add Metrics for Connector Agnostic Autoscaling

2023-11-17 Thread Mason Chen
f we don't do it, autoscaler > > still cannot fetch this metric, right? > > > > If yes, I suggest this FLIP includes registering metric part, otherwise > > these metrics still cannot work. > > > > Please correct me if I misunderstood anything, thanks~ > &g

[DISCUSS] FLIP-394: Add Metrics for Connector Agnostic Autoscaling

2023-11-16 Thread Mason Chen
Hi all, I would like to start a discussion on FLIP-394: Add Metrics for Connector Agnostic Autoscaling [1]. This FLIP recommends adding two metrics to make autoscaling work for bounded split source implementations like IcebergSource. These metrics are required by the Flink Kubernetes Operator

[jira] [Created] (FLINK-33575) FLIP-394: Add Metrics for Connector Agnostic Autoscaling

2023-11-16 Thread Mason Chen (Jira)
Mason Chen created FLINK-33575: -- Summary: FLIP-394: Add Metrics for Connector Agnostic Autoscaling Key: FLINK-33575 URL: https://issues.apache.org/jira/browse/FLINK-33575 Project: Flink Issue

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Mason Chen
I submitted PR to fix it since I was looking at the Kafka code already: https://github.com/apache/flink-connector-kafka/pull/63 On Mon, Oct 30, 2023 at 12:19 PM Mason Chen wrote: > +1 (non-binding) > > * Verified hashes and signatures > * Verified no binaries > * Verified poms

Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1

2023-10-30 Thread Mason Chen
+1 (non-binding) * Verified hashes and signatures * Verified no binaries * Verified poms point to 3.0.1 * Reviewed web PR * Built from source * Verified git tag @Xianxun, good catch. The datastream docs should be automatically updated via the doc shortcode. However, it seems that the sql

Re: [VOTE] FLIP-375: Built-in cross-platform powerful java profiler

2023-10-18 Thread Mason Chen
+1 (non-binding) Best, Mason On Wed, Oct 18, 2023 at 12:47 AM Yangze Guo wrote: > +1 (binding) > > Best, > Yangze Guo > > On Wed, Oct 18, 2023 at 2:15 AM Alexander Fedulov > wrote: > > > > +1 (non-binding) > > > > Thanks for explicitly addressing my concerns from the previous attempt of > >

[jira] [Created] (FLINK-32893) Make client.id configurable from KafkaSource

2023-08-17 Thread Mason Chen (Jira)
Mason Chen created FLINK-32893: -- Summary: Make client.id configurable from KafkaSource Key: FLINK-32893 URL: https://issues.apache.org/jira/browse/FLINK-32893 Project: Flink Issue Type

[jira] [Created] (FLINK-32892) Upgrade kafka-clients dependency to 3.4.x

2023-08-17 Thread Mason Chen (Jira)
Mason Chen created FLINK-32892: -- Summary: Upgrade kafka-clients dependency to 3.4.x Key: FLINK-32892 URL: https://issues.apache.org/jira/browse/FLINK-32892 Project: Flink Issue Type

Re: FLIP-342: Remove brackets around keys returned by MetricGroup#getAllVariables

2023-07-20 Thread Mason Chen
Hi all, I agree getScopeVariables is in line with the existing terminology but I’ve seen that the existing terminology is a bit confusing with regards to how users end up querying these metrics and building alerts/dashboards. I often get the question how do I add a tag or label to my Flink

Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems

2023-07-07 Thread Mason Chen
Hi all, I also agree with what's been said above. +1, I think the Table API delegation is a good suggestion--it essentially allows a connector to get Python support for free. We've seen that Table/SQL and Python APIs complement each other well and are ideal for data scientists. With respect to

[jira] [Created] (FLINK-32507) Document KafkaSink SinkWriterMetricGroup metrics

2023-06-30 Thread Mason Chen (Jira)
Mason Chen created FLINK-32507: -- Summary: Document KafkaSink SinkWriterMetricGroup metrics Key: FLINK-32507 URL: https://issues.apache.org/jira/browse/FLINK-32507 Project: Flink Issue Type

[ANNOUNCE] Kafka connector code removed from apache/master

2023-06-27 Thread Mason Chen
Hi all, I would like to inform you that we have removed the Kafka connector code from the Flink main repo. This should reduce the developer confusion of which repo to submit PRs. Regarding a few nuances, we have kept the Confluent avro format in the main repo. This is because the format is

[jira] [Created] (FLINK-32452) Refactor SQL Client E2E Test to Remove Kafka SQL Connector Dependency

2023-06-27 Thread Mason Chen (Jira)
Mason Chen created FLINK-32452: -- Summary: Refactor SQL Client E2E Test to Remove Kafka SQL Connector Dependency Key: FLINK-32452 URL: https://issues.apache.org/jira/browse/FLINK-32452 Project: Flink

[jira] [Created] (FLINK-32451) Refactor Confluent Schema Registry E2E Tests to remove Kafka connector dependency

2023-06-27 Thread Mason Chen (Jira)
Mason Chen created FLINK-32451: -- Summary: Refactor Confluent Schema Registry E2E Tests to remove Kafka connector dependency Key: FLINK-32451 URL: https://issues.apache.org/jira/browse/FLINK-32451

[jira] [Created] (FLINK-32449) Refactor state machine examples to remove Kafka dependency

2023-06-27 Thread Mason Chen (Jira)
Mason Chen created FLINK-32449: -- Summary: Refactor state machine examples to remove Kafka dependency Key: FLINK-32449 URL: https://issues.apache.org/jira/browse/FLINK-32449 Project: Flink Issue

[jira] [Created] (FLINK-32417) DynamicKafkaSource User Documentation

2023-06-22 Thread Mason Chen (Jira)
Mason Chen created FLINK-32417: -- Summary: DynamicKafkaSource User Documentation Key: FLINK-32417 URL: https://issues.apache.org/jira/browse/FLINK-32417 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32416) Initial DynamicKafkaSource Implementation

2023-06-22 Thread Mason Chen (Jira)
Mason Chen created FLINK-32416: -- Summary: Initial DynamicKafkaSource Implementation Key: FLINK-32416 URL: https://issues.apache.org/jira/browse/FLINK-32416 Project: Flink Issue Type: Sub-task

[RESULT][VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

2023-06-22 Thread Mason Chen
Hi everyone, I'm happy to announce that FLIP-246: DynamicKafkaSource [1] has been approved! According to the voting thread [2], there are 6 approving votes, 5 of which are binding: - Jing Ge (binding) - Tzu-Li (Gordon) Tai (binding) - Martijn Visser (binding) - Ryan van Huuksloot (non-binding) -

Re: [ANNOUNCE] Updates to Flink's external connector CI workflows

2023-06-15 Thread Mason Chen
Can't wait to use this in the Kafka repo, especially the thread dump. Thanks Martjin and Sergey! Best, Mason On Thu, Jun 15, 2023 at 4:05 PM Sergey Nuyanzin wrote: > Thanks a lot Martijn for making it > > On Thu, Jun 15, 2023 at 2:07 PM Leonard Xu wrote: > > > Thanks Martijn for the great

[VOTE] FLIP-246: Dynamic Kafka Source (originally Multi Cluster Kafka Source)

2023-06-15 Thread Mason Chen
Hi all, Thank you to everyone for the feedback on FLIP-246 [1]. Based on the discussion thread [2], we have come to a consensus on the design and are ready to take a vote to contribute this to Flink. This voting thread will be open for at least 72 hours (excluding weekends, until June 20th

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-15 Thread Mason Chen
te to allow this to move forward. > > Thanks, > Gordon > > On Mon, Jun 12, 2023 at 1:57 PM Mason Chen wrote: > > > > > > > My main worry for doing this as a later iteration is that this would > > > probably be a breaking change for the publ

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-12 Thread Mason Chen
nfusing because it is really similar to the > > KafkaDynamicSource/Sink (table connectors). > > > > The table / sql Kafka connectors (KafkaDynamicTableFactory, > > KafkaDynamicTableSource / KafkaDynamicTableSink) are all internal classes > > not really meant to be exposed

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-07 Thread Mason Chen
Hi Aitozi, I think it makes sense to make it easier for SQL users to make RPCs. Do you think your proposal can extend to the ability to batch data for the RPC? This is also another common strategy to increase throughput. Also, have you considered solving this a bit differently by leveraging

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-07 Thread Mason Chen
Hi Shammon, FLIP generally looks good and I'm excited to see this feature. 1. I'm also curious about default implementations. Would atlas/datahub be supported by default? 2. The FLIP title is confusing to me, especially in distinguishing it from FLIP-314. Would a better FLIP title be "Support

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-06-07 Thread Mason Chen
t; as employing them in SQL Gateway. WRT init containers, I think most > > > users use them just as a workaround. For example, wget a jar from the > > > maven repo. > > > > > > We could implement the functionality in SQL Driver in a more graceful > > > way

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-07 Thread Mason Chen
fluence/display/FLINK/FLIP-246%3A+Multi+Cluster+Kafka+Source?preview=/217389320/255072018/image-2023-6-7_2-29-13.png > > On Wed, Jun 7, 2023 at 3:07 AM Mason Chen wrote: > > > Hi Jing, > > > > Thanks for the prompt feedback! I had some confusion with how to resize > &

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-07 Thread Mason Chen
ibe scenarios when exactly-one is supported and when it is > not? > > Best regards, > Jing > > On Mon, Jun 5, 2023 at 11:41 PM Mason Chen wrote: > > > Hi all, > > > > I'm working on FLIP-246 again, for the Multi Cluster Kafka Source > > contribution. The document ha

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-06-05 Thread Mason Chen
Hi Paul, +1 for this feature and supporting SQL file + JSON plans. We get a lot of requests to just be able to submit a SQL file, but the JSON plan optimizations make sense. +1 for init containers or a more generalized way of obtaining arbitrary files. File fetching isn't specific to just

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-05 Thread Mason Chen
! Best, Mason On Wed, Oct 12, 2022 at 11:12 PM Mason Chen wrote: > Hi Ryan, > > Thanks for the additional context! Yes, the offset initializer would need > to take a cluster as a parameter and the MultiClusterKafkaSourceSplit can > be exposed in an initializer. > > Best, >

[jira] [Created] (FLINK-32197) FLIP 246: Multi Cluster Kafka Source

2023-05-25 Thread Mason Chen (Jira)
Mason Chen created FLINK-32197: -- Summary: FLIP 246: Multi Cluster Kafka Source Key: FLINK-32197 URL: https://issues.apache.org/jira/browse/FLINK-32197 Project: Flink Issue Type: New Feature

Re: Questions on checkpointing mechanism for FLIP-27 Source API

2023-05-24 Thread Mason Chen
Hi Hong, In addition, I guess the inconsistency could also be handled by the reader.start() method such that the operator event is re-sent during restore. Best, Mason On Tue, May 23, 2023 at 7:58 AM Piotr Nowojski wrote: > Hi, > > I vaguely remember someone implementing a mechanism to deal

Re: [ANNOUNCE] New Apache Flink PMC Member - Qingsheng Ren

2023-04-26 Thread Mason Chen
Congrats Qingsheng! Best, Mason On Tue, Apr 25, 2023 at 7:33 PM Shengkai Fang wrote: > Congratulations, Qingsheng! > > > Best regards, > Shengkai > > Qingsheng Ren 于2023年4月24日周一 19:53写道: > > > Thank you all! I'm so honored and happy to work with all contributors in > > the Apache Flink

Re: [VOTE] Release flink-connector-kafka 3.0.0 for Flink 1.17, release candidate #2

2023-04-20 Thread Mason Chen
+1 (non-binding) * Verified hashes and signatures * Verified no binaries * Verified LICENSE and NOTICE files, pointing to 2023 as well * Verified poms point to 3.0.0 * Reviewed web PR * Built from source * Verified git tag Best, Mason On Thu, Apr 20, 2023 at 10:04 AM Ahmed Hamdy wrote: > +1

[jira] [Created] (FLINK-31747) Externalize debezium from flink-json

2023-04-06 Thread Mason Chen (Jira)
Mason Chen created FLINK-31747: -- Summary: Externalize debezium from flink-json Key: FLINK-31747 URL: https://issues.apache.org/jira/browse/FLINK-31747 Project: Flink Issue Type: Improvement

Re: [VOTE] Release flink-connector-kafka, release candidate #1

2023-04-05 Thread Mason Chen
t; > > > > On Thu, Feb 9, 2023 at 3:26 PM Tzu-Li (Gordon) Tai < > tzuli...@apache.org> > > > wrote: > > > > > > > +1 (binding) > > > > > > > > - Verified legals (license headers and root LICENSE / NOTICE file).

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-04-05 Thread Mason Chen
Hi all, I am looking for feedback on removing Kafka code from apache/flink:master: https://issues.apache.org/jira/browse/FLINK-30859. If it wasn't clear, this involves moving more documentation, Kafka examples, Kafka related formats, and test related code to the new repo. Regarding the Kafka

Re: [VOTE] Release flink-connector-parent10.0, release candidate #1

2023-03-20 Thread Mason Chen
+1 (nonbinding) - Verified hashes and signature - Verified git tag - Verified maven staging repo - Read Web PR Best, Mason On Thu, Mar 16, 2023 at 2:07 PM Tzu-Li (Gordon) Tai wrote: > +1 (binding) > > - Verified hashes and signature > - Read web PR > - Staging Maven artifacts

Re: Flink metric collection

2023-03-09 Thread Mason Chen
Hi all, Metric reporter may be useful if you only need per component level metrics like jobmanager and each taskmanager since the metric reporter runs in each component. However, for an aggregated job level view of metrics, there is no better out-of-the-box/user-facing way to get metrics than the

[jira] [Created] (FLINK-31305) KafkaWriter doesn't wait for errors for in-flight records before completing flush

2023-03-02 Thread Mason Chen (Jira)
Mason Chen created FLINK-31305: -- Summary: KafkaWriter doesn't wait for errors for in-flight records before completing flush Key: FLINK-31305 URL: https://issues.apache.org/jira/browse/FLINK-31305

[jira] [Created] (FLINK-30932) Enabling producer metrics for KafkaSink is not documented

2023-02-06 Thread Mason Chen (Jira)
Mason Chen created FLINK-30932: -- Summary: Enabling producer metrics for KafkaSink is not documented Key: FLINK-30932 URL: https://issues.apache.org/jira/browse/FLINK-30932 Project: Flink Issue

Re: [VOTE] Release flink-connector-kafka, release candidate #1

2023-02-06 Thread Mason Chen
regards, > > Martijn > > Op wo 1 feb. 2023 om 09:16 schreef Mason Chen : > > > +1 (non-binding) > > > > * Verified hashes and signatures > > * Verified no binaries > > * Verified LICENSE and NOTICE files > > * Verified poms point to 3.0.0-1.16 > &g

Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator

2023-02-06 Thread Mason Chen
+1 (non-binding) Best, Mason On Thu, Feb 2, 2023 at 6:03 PM Leonard Xu wrote: > +1 > > Best, > Leonard > > > On Feb 3, 2023, at 9:49 AM, Dong Lin wrote: > > > > +1 > > > > On Thu, Feb 2, 2023 at 9:31 PM Hang Ruan wrote: > > > >> Hi all, > >> > >> Thanks for all the help about this FLIP. Now

Re: [VOTE] Release flink-connector-kafka, release candidate #1

2023-02-01 Thread Mason Chen
+1 (non-binding) * Verified hashes and signatures * Verified no binaries * Verified LICENSE and NOTICE files * Verified poms point to 3.0.0-1.16 * Reviewed web PR * Built from source * Verified git tag I think [4] your is a copy-paste error and I did all the verification assuming that

[jira] [Created] (FLINK-30859) Remove flink-connector-kafka rom master branch

2023-01-31 Thread Mason Chen (Jira)
Mason Chen created FLINK-30859: -- Summary: Remove flink-connector-kafka rom master branch Key: FLINK-30859 URL: https://issues.apache.org/jira/browse/FLINK-30859 Project: Flink Issue Type

[jira] [Created] (FLINK-30833) MetricListener supports closing metric groups

2023-01-30 Thread Mason Chen (Jira)
Mason Chen created FLINK-30833: -- Summary: MetricListener supports closing metric groups Key: FLINK-30833 URL: https://issues.apache.org/jira/browse/FLINK-30833 Project: Flink Issue Type

Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator

2023-01-19 Thread Mason Chen
Hi all, I'm also interested in this FLIP! A metric group that extends from the jobmanager group resonates well with me too. I prefer choice 1 since that's consistent with how the other metric groups are designed. In addition, in your example, I think you mean to write

Re: [DISCUSS] Enabling dynamic partition discovery by default in Kafka source

2023-01-19 Thread Mason Chen
Hi all, Sorry to come into the discussion late--I saw the thread earlier. I'm also +1 for the change in general. I think most users have this turned on by default since the overhead is quite low. A default in the two digit seconds range works well for us. However, I do have two main concerns

[jira] [Created] (FLINK-30409) Support reopening closed metric groups

2022-12-13 Thread Mason Chen (Jira)
Mason Chen created FLINK-30409: -- Summary: Support reopening closed metric groups Key: FLINK-30409 URL: https://issues.apache.org/jira/browse/FLINK-30409 Project: Flink Issue Type: Improvement

[DISCUSS] Externalizing Kafka connector

2022-12-06 Thread Mason Chen
Hi all, I've finished the first pass on externalizing the Kafka connector under the release-1.16 branch in this https://github.com/apache/flink-connector-kafka/pull/1. The docs, connectors, and e2e tests have been relocated and CI on my fork has been running green. Thanks for everyone else's work

Re: [DISCUSS] Retroactively externalize some connectors for 1.16

2022-12-06 Thread Mason Chen
Hi all, +1 (non-binding), I agree that syncing the changes going forward would be a huge effort and a cutoff date makes sense. Best, Mason On Tue, Dec 6, 2022 at 12:10 AM Ryan Skraba wrote: > Hello -- this makes sense to me: removing connectors from 1.17 (but not the > 1.16 branch) will still

Re: [VOTE] FLIP-271: Autoscaling

2022-11-29 Thread Mason Chen
+1 (non-binding) On Tue, Nov 29, 2022 at 11:55 AM Ferenc Csaky wrote: > +1 (non-binding) > > > > > --- Original Message --- > On Tuesday, November 29th, 2022 at 15:39, Márton Balassi < > balassi.mar...@gmail.com> wrote: > > > > > > > > +1 (binding) > > > > On Tue, Nov 29, 2022 at 6:13

[jira] [Created] (FLINK-30200) TimestampOffsetsInitializer allows configurable OffsetResetStrategy

2022-11-24 Thread Mason Chen (Jira)
Mason Chen created FLINK-30200: -- Summary: TimestampOffsetsInitializer allows configurable OffsetResetStrategy Key: FLINK-30200 URL: https://issues.apache.org/jira/browse/FLINK-30200 Project: Flink

[jira] [Created] (FLINK-29856) Triggering savepoint does not trigger source operator checkpoint

2022-11-02 Thread Mason Chen (Jira)
Mason Chen created FLINK-29856: -- Summary: Triggering savepoint does not trigger source operator checkpoint Key: FLINK-29856 URL: https://issues.apache.org/jira/browse/FLINK-29856 Project: Flink

Re: [ANNOUNCE] Performance Daily Monitoring Moved from Ververica to Apache Flink Slack Channel

2022-10-31 Thread Mason Chen
+1, thanks for driving this! On a side note, can we also ensure that a performance summary report for Flink major version upgrades is in release notes, once this infrastructure becomes mature? From the user perspective, it would be nice to know what the expected (or unexpected) regressions in a

Re: [VOTE] Release 1.16.0, release candidate #2

2022-10-25 Thread Mason Chen
+1 (non-binding) * Built from source * Verified signature and checksum * Verified behavior/metrics/logs with internal stateful applications using the Kafka source/sink connectors on K8s Best, Mason On Mon, Oct 24, 2022 at 11:16 PM Leonard Xu wrote: > > > BTW, the "Add New" button in "Submit

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2022-10-13 Thread Mason Chen
o pass a map into starting offsets just for demonstrative > purposes, I would be fine with whatever data structure would work best > > Ryan van Huuksloot > Data Developer | Production Engineering | Streaming Capabilities > [image: Shopify] > <https://www.shopify.com/?utm_medium=s

Re: [DISCUSS] Adding client.id.prefix to the KafkaSink

2022-10-12 Thread Mason Chen
Hi Yaroslav, +1 from my end too. I get some questions internally on the warning logs due to conflicting client-id and it would be nice to resolve them. Best, Mason On Wed, Oct 12, 2022 at 2:03 PM Martijn Visser wrote: > Hi Yaroslav, > > +1 from my end. Thanks for bringing this up! > > Best

[jira] [Created] (FLINK-29515) Document KafkaSource behavior with deleted topics

2022-10-05 Thread Mason Chen (Jira)
Mason Chen created FLINK-29515: -- Summary: Document KafkaSource behavior with deleted topics Key: FLINK-29515 URL: https://issues.apache.org/jira/browse/FLINK-29515 Project: Flink Issue Type

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2022-10-03 Thread Mason Chen
e with a single timestamp, the current design may work for your usecase, but I can't tell without more details. Thanks for your interest and sorry for the delay! Best, Mason On Mon, Aug 29, 2022 at 10:02 AM Mason Chen wrote: > Hi Max, > > Thanks for taking a look! > > >> I'

Re: [Discuss] KafkaSink making low level components PublicEvolving

2022-08-30 Thread Mason Chen
conventions and it would make future changes in this connector > much more difficult. It's difficult to support Public APIs, even if they > are `@PublicEvolving`. > > Best, > Piotrek > > pon., 29 sie 2022 o 19:17 Mason Chen napisał(a): > > > Hi all, > > >

Re: [Discuss] KafkaSink making low level components PublicEvolving

2022-08-29 Thread Mason Chen
Hi all, I know everyone is busy with preparing the 1.16 release. Just want to give this thread a gentle ping, please give your +1 if there are no concerns! Best, Mason On Thu, Aug 11, 2022 at 1:27 AM Mason Chen wrote: > I understand that the 1.16 feature freeze is in place so I prop

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2022-08-29 Thread Mason Chen
? Are there any limitations on > the type of configuration changes that we want to allow? For example, is > it allowed to remove a cluster before it has been drained / deactivated? > Is "not active" semantically identical to having the cluster / stream > removed? > > T

Re: [Discuss] KafkaSink making low level components PublicEvolving

2022-08-11 Thread Mason Chen
I understand that the 1.16 feature freeze is in place so I propose for these changes to go in the 1.17 release. Best, Mason On Thu, Aug 11, 2022 at 1:20 AM Mason Chen wrote: > Hi devs and community, > > I would like to discuss making the low level components of the KafkaSink > Pu

[Discuss] KafkaSink making low level components PublicEvolving

2022-08-11 Thread Mason Chen
Hi devs and community, I would like to discuss making the low level components of the KafkaSink PublicEvolving. It looks like all of the classes are package private apart from the public facing API and classes were put in the same package to make this work. My desire to make at least the

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2022-08-11 Thread Mason Chen
appreciate you taking a look at the FLIP! Best, Mason On Thu, Aug 11, 2022 at 1:03 AM Mason Chen wrote: > Hi Qingsheng, > > Thanks for the feedback--these are great points to raise. > > 1. This is something I missed that is now added. More generally, it can > locate multiple to

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2022-08-11 Thread Mason Chen
Could you elaborate more in the FLIP? > > In a nutshell I think the idea of this FLIP is good, which extends the > usage of Kafka source. However as a design doc, some details need to be > enriched for other users and developers to better understand how this > source works. > > Be

Re: [DISCUSS] Bump Kafka to 3.2.1 for 1.16.0

2022-08-09 Thread Mason Chen
+1 We don't have a reproducible test in Flink CI to tell if the issue is definitely solved. However, bumping the version doesn't make matters worse then the current state. Best, Mason On Tue, Aug 9, 2022 at 10:56 AM David Anderson wrote: > I'm in favor of adopting this fix in 1.16.0. > > +1 >

Re: [VOTE] FLIP-217: Support watermark alignment of source splits

2022-08-01 Thread Mason Chen
Hi all, +1 (non-binding). Can't wait to use this feature in Flink! Best, Mason On Sun, Jul 31, 2022 at 10:57 PM Sebastian Mattheis wrote: > Hi everyone, > > I would like to start the vote for FLIP-217 [1]. Thanks for your feedback > and the discussion in [2]. > > FLIP-217 is a follow-up on

[jira] [Created] (FLINK-28722) Hybrid Source should use .equals() for Integer comparison

2022-07-27 Thread Mason Chen (Jira)
Mason Chen created FLINK-28722: -- Summary: Hybrid Source should use .equals() for Integer comparison Key: FLINK-28722 URL: https://issues.apache.org/jira/browse/FLINK-28722 Project: Flink Issue

[DISCUSS] FLIP-246: Multi Cluster Kafka Source

2022-07-20 Thread Mason Chen
Hi all, We would like to start a discussion thread on FLIP-246: Multi Cluster Kafka Source [1] where we propose to provide a source connector for dynamically reading from Kafka multiple clusters, which will not require Flink job restart. This can greatly improve the Kafka migration experience for

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-07-14 Thread Mason Chen
; > > I'm sure there's a PMC (*hint*) out there who can grant you access to > > create a FLIP. Looking forward to it, this sounds like an improvement > that > > users are looking forward to. > > > > Best regards, > > > > Martijn > > > > Op di

Re: [DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-28 Thread Mason Chen
. > > Thanks for the effort on this and looking forward to your FLIP! > > Best, > Qingsheng > > > On Jun 24, 2022, at 09:43, Mason Chen wrote: > > > > Hi community, > > > > We have been working on a Multi Cluster Kafka Source and are looking to > > contrib

[DISCUSS] Contribution of Multi Cluster Kafka Source

2022-06-23 Thread Mason Chen
Hi community, We have been working on a Multi Cluster Kafka Source and are looking to contribute it upstream. I've given a talk about the features and design at a Flink meetup: https://youtu.be/H1SYOuLcUTI. The main features that it provides is: 1. Reading multiple Kafka clusters within a single

Re: Re: [ANNOUNCE] New Apache Flink Committers: Qingsheng Ren, Shengkai Fang

2022-06-21 Thread Mason Chen
Awesome work Qingsheng and Shengkai! Best, Mason On Tue, Jun 21, 2022 at 4:53 AM Zhipeng Zhang wrote: > Congratulations, Qingsheng and ShengKai. > > Yang Wang 于2022年6月21日周二 19:43写道: > > > Congratulations, Qingsheng and ShengKai. > > > > > > Best, > > Yang > > > > Benchao Li 于2022年6月21日周二

[jira] [Created] (FLINK-27479) HybridSource refreshes availability future

2022-05-03 Thread Mason Chen (Jira)
Mason Chen created FLINK-27479: -- Summary: HybridSource refreshes availability future Key: FLINK-27479 URL: https://issues.apache.org/jira/browse/FLINK-27479 Project: Flink Issue Type

  1   2   >