Re: [Discuss] Conventions on driving FLIPs towards consensus

The discussion has been open for quite some time and there seems not to be more inputs. Thanks all for the discussion. I have updated the Flink Improvement Proposals wiki page [1] with the discussed conventions. Best, Xintong [1]

[jira] [Created] (FLINK-30934) Refactor ComputedColumnAndWatermarkTableITCase to get rid of managed table

yuzelin created FLINK-30934: --- Summary: Refactor ComputedColumnAndWatermarkTableITCase to get rid of managed table Key: FLINK-30934 URL: https://issues.apache.org/jira/browse/FLINK-30934 Project: Flink

Re: [VOTE] FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records

Thanks for the update, Dong! +1 for the new table option. Best, Qingsheng On Mon, Feb 6, 2023 at 2:39 PM Dong Lin wrote: > As I am reviewing the FLIP-208 PR here > , I realized that a new public API > change was added to the Proposed Change section in

Re: [VOTE] FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records

> Since this change is relatively minor and looks good to me, I will re-use > this voting thread to confirm this change is OK. > > Qingsheng and Leonard: can you help check whether this public interface > change looks good to you? > Thanks Dong and Hang for driving this. The change looks

Re: Flink Kubernetes Operator 1.4.0 release planning

Thanks Gyula for driving the release again. It is really exciting to see the auto-scaling coming out of the box. Best, Yang Gyula Fóra 于2023年2月6日周一 19:43写道： > Hi Devs! > > Based on the previously agreed upon release schedule ( > >

[jira] [Created] (FLINK-30933) Result of join inside iterationBody loses max watermark

Zhipeng Zhang created FLINK-30933: - Summary: Result of join inside iterationBody loses max watermark Key: FLINK-30933 URL: https://issues.apache.org/jira/browse/FLINK-30933 Project: Flink

Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator

+1 (non-binding) Best regards, Weijie Mason Chen 于2023年2月7日周二 10:11写道： > +1 (non-binding) > > Best, > Mason > > > On Thu, Feb 2, 2023 at 6:03 PM Leonard Xu wrote: > > > +1 > > > > Best, > > Leonard > > > > > On Feb 3, 2023, at 9:49 AM, Dong Lin wrote: > > > > > > +1 > > > > > > On Thu, Feb

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

Hi Feng, I agree with what Jark said. I think what you are looking for is lazy initialization. I don't think we should introduce the new interface CatalogProvider for lazy initialization. What we should do is to store the catalog properties and initialize the catalog when we need it. Could you

Re: [DISCUSS] FLIP-289: Support online inference (Flink ML)

Hi all, If there is no question related to this FLIP, we will start the voting thread on 2/10. Regards, Dong On Wed, Feb 1, 2023 at 8:38 PM Dong Lin wrote: > Hi all, > > Fan, Jiang, Zhipeng, and I have created FLIP-289: Support online inference > (Flink ML). > > The goal of this FLIP is to

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

> Regarding the discussion on global committer [1] for sinks with global transactions, there is no consensus on solving that problem in SinkV2. Will it require any breaking change in SinkV2? Just want to reiterate my earlier question. What is the migration path for the Iceberg sink? [1]

Re: Need help how to use Table API to join two Kafka streams

Hi, could you please share us the root cause? Seems the error message you posted hadn't contained the root cause. Maybe you can post the full error message . Best regards, Yuxia 发件人: "Amir Hossein Sharifzadeh" 收件人: "yuxia" 抄送: "dev" 发送时间: 星期二, 2023年 2 月 07日 上午 10:39:25 主题: Re: Need

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi Piotr Thanks for your feedback. > - stateless operators, could completely ignore the issue and process the records normally, as they are doing right now > - stateful operators, should either: > - if the business doesn't require ordering, they could process the records immediately > -

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

Thanks David for driving this. This is a very valuable work, especially for cloud native environment. >> How about adding some more information such as vertex type (SOURCE/MAP/JOIN and .etc) in the response of `get jobs resource-requirements`? For users, only vertex-id may be difficult to

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

Hi Feng, I think this feature makes a lot of sense. If I understand correctly, what you are looking for is lazy catalog initialization. However, I have some concerns about introducing CatalogProvider, which delegates catalog management to users. It may be hard to avoid conflicts and duplicates

Re: Need help how to use Table API to join two Kafka streams

Thank you for your reply. I tied it with a sample stream but it did not work. I am trying to get the results from my producer here with a very simple query. I want to see results in the console/output. This is my code: // Docker: docker-compose.yml version: '2' services: zookeeper: image:

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

Hi Konstantin, I totally agree with making SinkV2 @Public. I just have concerns about deprecating SinkFunction at this point. Dong Lin has raised the blocker issues of migration multiple times in this thread which I think we should address first. I don't know why we rush to deprecate SinkFunction

[jira] [Created] (FLINK-30932) Enabling producer metrics for KafkaSink is not documented

Mason Chen created FLINK-30932: -- Summary: Enabling producer metrics for KafkaSink is not documented Key: FLINK-30932 URL: https://issues.apache.org/jira/browse/FLINK-30932 Project: Flink Issue

Re: [VOTE] Release flink-connector-kafka, release candidate #1

That makes sense, thanks for the clarification! Best, Mason On Wed, Feb 1, 2023 at 7:16 AM Martijn Visser wrote: > Hi Mason, > > Thanks, [4] is indeed a copy-paste error and you've made the right > assumption that > >

Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator

+1 (non-binding) Best, Mason On Thu, Feb 2, 2023 at 6:03 PM Leonard Xu wrote: > +1 > > Best, > Leonard > > > On Feb 3, 2023, at 9:49 AM, Dong Lin wrote: > > > > +1 > > > > On Thu, Feb 2, 2023 at 9:31 PM Hang Ruan wrote: > > > >> Hi all, > >> > >> Thanks for all the help about this FLIP. Now

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

Hi Konstantin, Just FYI, the FileSystemTableSink are still using SinkFunction. Best regards, Yuxia - 原始邮件 - 发件人: "Dong Lin" 收件人: "dev" 抄送: "Jing Ge" , "Yun Tang" 发送时间: 星期二, 2023年 2 月 07日 上午 9:41:07 主题: Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction Hi Konstantin,

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

Hi Konstantin, Thanks for the reply. Please see my comments inline. On Mon, Feb 6, 2023 at 9:48 PM Konstantin Knauf wrote: > Hi Steven, > > Sink is already deprecated. It was deprecated at the moment where we > introduced SinkV2. > > Hi Jark, Hi Dong, > > My understanding is the SinkV2 is a

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi, Thanks for the answers. >> Are you proposing that all of the inputs to stateful operators would have to be sorted? >> > Records in stream don't need to be sorted, but it should be managed by `Timestamp Barrier`, which means > 1. Records belonging to a specific `Timestamp Barrier` are

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

Hi, We have the same goal to let users migrate to SinkV2. Just marking those interfaces as deprecated or public won't achieve this goal. Frankly speaking, users will start to migrate not because those interfaces are marked as deprecated. They will do it because there is a workable implementation

[NOTICE] Website now has a staging environment

Hello, Just so more people are aware of it, I recently enabled a staging environment for the Flink Website. You can push the rebuilt website to the asf-staging branch in flink-web, and the changes will be visible at https://flink.staged.apache.org shortly. This has been documented at

Re: Reworking the Rescale API

>> I fully agree that in-place scaling is a much harder problem which is out of >> the scope for now. My primary concern here is to be able to rescale with >> upfront reservation of resources before restarting the job, so the job >> doesn't get stuck in case of resource constraints. > Not sure

[jira] [Created] (FLINK-30931) Adaptive scheduler wastes slots for multiple slot sharing groups with different max parallelism

Maximilian Michels created FLINK-30931: -- Summary: Adaptive scheduler wastes slots for multiple slot sharing groups with different max parallelism Key: FLINK-30931 URL:

Re: [Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

Hi Feng, this is indeed a good proposal. 1) It makes sense to improve the catalog listing for platform providers. 2) Other feedback from the past has shown that users would like to avoid the default in-memory catalog and offer their catalog before a TableEnvironment session starts. 3) Also

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

Recently, a critical bug with the Unified Sink committer was reported: [FLINK-30238] Unified Sink committer does not clean up state on final savepoint - ASF JIRA (apache.org) . Fabian Paul reported: Hi folks, > > I did some initial

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

Hi Steven, Sink is already deprecated. It was deprecated at the moment where we introduced SinkV2. Hi Jark, Hi Dong, My understanding is the SinkV2 is a workable interface. The most important connectors have been migrated (Kafka, Filesystem) and more connectors (OpenSearch, ElasticSearch,

Re: [ANNOUNCE] Apache Flink 1.16.1 released

Hi, Thanks to everyone involved. Best Etienne Le 02/02/2023 à 03:55, weijie guo a écrit : Thank Martin for managing the release and all the people involved. Best regards, Weijie Konstantin Knauf 于2023年2月2日周四 06:40写道： Great. Thanks, Martijn for managing the release. Am Mi., 1. Feb.

[jira] [Created] (FLINK-30930) Automatically determine Flink binary download URL from version

Chesnay Schepler created FLINK-30930: Summary: Automatically determine Flink binary download URL from version Key: FLINK-30930 URL: https://issues.apache.org/jira/browse/FLINK-30930 Project:

[jira] [Created] (FLINK-30929) Add helpful message when ElasticsearchSink.Builder.build() throws a IllegalArgumentException.

Kenny Wu created FLINK-30929: Summary: Add helpful message when ElasticsearchSink.Builder.build() throws a IllegalArgumentException. Key: FLINK-30929 URL: https://issues.apache.org/jira/browse/FLINK-30929

[jira] [Created] (FLINK-30928) Use temurin JRE in test hadoop docker image

Gabor Somogyi created FLINK-30928: - Summary: Use temurin JRE in test hadoop docker image Key: FLINK-30928 URL: https://issues.apache.org/jira/browse/FLINK-30928 Project: Flink Issue Type:

[jira] [Created] (FLINK-30927) Several tests fail with two non-abstract methods have the same parameter types, declaring type and return type

Sergey Nuyanzin created FLINK-30927: --- Summary: Several tests fail with two non-abstract methods have the same parameter types, declaring type and return type Key: FLINK-30927 URL:

Re: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction

I agree with Dong Lin. Oracle explains how to use Deprecate API [1]: You are strongly recommended to use the Javadoc @deprecated tag with > appropriate comments explaining how to use the new API. This ensures > developers will *have a workable migration path from the old API to the > new API*.

[jira] [Created] (FLINK-30926) [Umbrella] Test Flink Release 1.17

Qingsheng Ren created FLINK-30926: - Summary: [Umbrella] Test Flink Release 1.17 Key: FLINK-30926 URL: https://issues.apache.org/jira/browse/FLINK-30926 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-30925) Add docs for the SQL Client gateway mode

Shengkai Fang created FLINK-30925: - Summary: Add docs for the SQL Client gateway mode Key: FLINK-30925 URL: https://issues.apache.org/jira/browse/FLINK-30925 Project: Flink Issue Type:

Flink Kubernetes Operator 1.4.0 release planning

Hi Devs! Based on the previously agreed upon release schedule ( https://cwiki.apache.org/confluence/display/FLINK/Release+Schedule+and+Planning) it is almost time for the 1.4.0 release. There are still a number of smaller but important PRs open for some critical fixes. I would like to merge

[jira] [Created] (FLINK-30924) Conversion issues between timestamp and bingint

Feng Jin created FLINK-30924: Summary: Conversion issues between timestamp and bingint Key: FLINK-30924 URL: https://issues.apache.org/jira/browse/FLINK-30924 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-30923) Provide single script for installing Hugo

Matthias Pohl created FLINK-30923: - Summary: Provide single script for installing Hugo Key: FLINK-30923 URL: https://issues.apache.org/jira/browse/FLINK-30923 Project: Flink Issue Type:

[jira] [Created] (FLINK-30922) SQL validate fail to parse writing metadata

tanjialiang created FLINK-30922: --- Summary: SQL validate fail to parse writing metadata Key: FLINK-30922 URL: https://issues.apache.org/jira/browse/FLINK-30922 Project: Flink Issue Type: Bug

RE: Re: Confluent Avro and Debezium formats - default schema name can be incompatible with registered schema name

Hi Dawid, Thanks for the suggestion, it’s worth a try. I’ll have a look at it. I assume this ’schema’ option would not be required and the current logic with the default name ‘record’ would be used, if the schema is not provided explicitly. Best regards, Fruzsina On 2023/01/27 13:14:39 Dawid

[Discuss] :Introduce Catalog dynamic registration in flink catalog manager.

Hi everyone, The original discussion address is https://issues.apache.org/jira/browse/FLINK-30126 Currently, Flink has access to many systems, including kafka, hive, iceberg, hudi, elasticsearch, mysql... The corresponding catalog name might be: kafka_cluster1, kafka_cluster2, hive_cluster1,

Re: [DISCUSS] FLIP-291: Externalized Declarative Resource Management

Hi David Thanks for initiating this discussion. I think declaring job resource requirements by REST API is very valuable. I just left some comments as followed 1) How about adding some more information such as vertex type (SOURCE/MAP/JOIN and .etc) in the response of `get jobs

[jira] [Created] (FLINK-30921) Too many CI failed due to "Could not connect to azure.archive.ubuntu.com"

Rui Fan created FLINK-30921: --- Summary: Too many CI failed due to "Could not connect to azure.archive.ubuntu.com" Key: FLINK-30921 URL: https://issues.apache.org/jira/browse/FLINK-30921 Project: Flink

[jira] [Created] (FLINK-30920) K8 cluster autoscaler | exclude operator id from scaler

Gaurav Miglani created FLINK-30920: -- Summary: K8 cluster autoscaler | exclude operator id from scaler Key: FLINK-30920 URL: https://issues.apache.org/jira/browse/FLINK-30920 Project: Flink

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

Hi Piotr, Thanks for your feedback. In general, I think `Timesamp Barrier` is a special `Watermark` that all sources send watermarks with the same timestamp as `Timestamp Barrier` and aggregation operators will align data by it. For example, all source subtasks are assigned two unified watermarks

[jira] [Created] (FLINK-30919) Fix typo in the document of User-defined Sources & Sinks

luoyuxia created FLINK-30919: Summary: Fix typo in the document of User-defined Sources & Sinks Key: FLINK-30919 URL: https://issues.apache.org/jira/browse/FLINK-30919 Project: Flink Issue Type:

[jira] [Created] (FLINK-30918) Refactor CompositePkAndMultiPartitionedTableITCase to get rid of managed table

yuzelin created FLINK-30918: --- Summary: Refactor CompositePkAndMultiPartitionedTableITCase to get rid of managed table Key: FLINK-30918 URL: https://issues.apache.org/jira/browse/FLINK-30918 Project: Flink

[jira] [Created] (FLINK-30917) The user configured max parallelism does not take effect when using adaptive batch scheduler

Lijie Wang created FLINK-30917: -- Summary: The user configured max parallelism does not take effect when using adaptive batch scheduler Key: FLINK-30917 URL: https://issues.apache.org/jira/browse/FLINK-30917

[jira] [Created] (FLINK-30916) RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp failed with assertion

Matthias Pohl created FLINK-30916: - Summary: RocksDBStateUploaderTest.testUploadedSstCanBeCleanedUp failed with assertion Key: FLINK-30916 URL: https://issues.apache.org/jira/browse/FLINK-30916

[jira] [Created] (FLINK-30915) `flink-fs-hadoop-s3` connector is unable to find IRSA credentials

Samrat Deb created FLINK-30915: -- Summary: `flink-fs-hadoop-s3` connector is unable to find IRSA credentials Key: FLINK-30915 URL: https://issues.apache.org/jira/browse/FLINK-30915 Project: Flink

[jira] [Created] (FLINK-30914) ZooKeeperLeaderElectionTest.testUnExpectedErrorForwarding failed

Matthias Pohl created FLINK-30914: - Summary: ZooKeeperLeaderElectionTest.testUnExpectedErrorForwarding failed Key: FLINK-30914 URL: https://issues.apache.org/jira/browse/FLINK-30914 Project: Flink

[jira] [Created] (FLINK-30913) Various PyFlink tests fail

Matthias Pohl created FLINK-30913: - Summary: Various PyFlink tests fail Key: FLINK-30913 URL: https://issues.apache.org/jira/browse/FLINK-30913 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-30912) CreateTableAsITCase seem to have timed out on Azure

Matthias Pohl created FLINK-30912: - Summary: CreateTableAsITCase seem to have timed out on Azure Key: FLINK-30912 URL: https://issues.apache.org/jira/browse/FLINK-30912 Project: Flink Issue

[jira] [Created] (FLINK-30911) KafkaSinkE2ECase.testStartFromSavepoint fails with TimeoutException because the topic doesn't become available in the meta file

Matthias Pohl created FLINK-30911: - Summary: KafkaSinkE2ECase.testStartFromSavepoint fails with TimeoutException because the topic doesn't become available in the meta file Key: FLINK-30911 URL: