Flink Checkpoint Behavior/Missing Records

2023-10-12 Thread Abhishek SP
I have implemented a Custom File Source(wrapper around native Flink File Source) that signalsNoSplits using a custom strategy which terminates the Flink App automatically. Due to external restrictions, I am bound to using Flink 1.13.3 in Streaming mode with Unbounded Input and cannot use the

Re: Problems sinking a Flink table to Kafka

2023-10-12 Thread yu'an huang
Hi, You are using a special Kafka connector. From the definition in website: "as a sink, the upsert-kafka connector can consume a changelog stream. It will write INSERT/UPDATE_AFTER data as normal Kafka messages value, and write DELETE data as Kafka messages with null values (indicate

Re: 关于Flink SQL无法对带metadata列的表在执行insert into时，指定具体的列名的问题。

2023-10-12 Thread jinzhuguang
感谢大佬！！！ > 2023年10月13日 10:44，tanjialiang 写道： > > Hi, > 这个问题已经在1.17.0修复，详细可以看https://issues.apache.org/jira/browse/FLINK-30922 > > > best wishes, > tanjialiang. > > > 回复的原邮件 > | 发件人 | jinzhuguang | > | 发送日期 | 2023年10月13日 10:39 | > | 收件人 | user-zh | > | 主题 | 关于Flink

回复：关于Flink SQL无法对带metadata列的表在执行insert into时，指定具体的列名的问题。

2023-10-12 Thread tanjialiang
Hi, 这个问题已经在1.17.0修复，详细可以看https://issues.apache.org/jira/browse/FLINK-30922 best wishes, tanjialiang. 回复的原邮件 | 发件人 | jinzhuguang | | 发送日期 | 2023年10月13日 10:39 | | 收件人 | user-zh | | 主题 | 关于Flink SQL无法对带metadata列的表在执行insert into时，指定具体的列名的问题。 | 首先，我的Flink版本为1.16.0 为了方便理解，我以Kafka作为案例来描述:

关于Flink SQL无法对带metadata列的表在执行insert into时，指定具体的列名的问题。

2023-10-12 Thread jinzhuguang
首先，我的Flink版本为1.16.0 为了方便理解，我以Kafka作为案例来描述: 我有以下两个表： CREATE TABLE orders( user_id BIGINT, name STRING, timestamp TIMESTAMP(3) METADATA VIRTUAL )WITH( 'connector'='kafka', 'topic'='orders', 'properties.group.id' = 'test_join_tempral', 'scan.startup.mode'='earliest-offset',

?????? Flink SQL runtime

2023-10-12 Thread Enric Ott
For example,operator state and checkpoint listener of flink sql runtime.I'm trying to modify flink sql compiled behavior programmaticallyand getcorresponding flink sql runtime. ---- ??:

Re: Flink SQL runtime

2023-10-12 Thread liu ron
Hi, What SQL Runtime are you referring to? Why do you need to get it? Best, Ron Enric Ott <243816...@qq.com> 于2023年10月12日周四 14:26写道： > Hi,Team: > Is there any approach to get flink sql runtime via api ? > Any help would be appreciated. >

Kinesis Producer - Support separate Cloudwatch credentials

2023-10-12 Thread Diem, Chase via user
Hey Team, Looking for some thoughts here: * We have a Kinesis Producer that produces to a topic in another AWS account * The producer allows for configurations to set credentials for that account:

"Unable to shutdown event loop"/"Unable to close channel pools"

2023-10-12 Thread Hedden, Ethan
Hello everyone, A little background on the job: relatively simple topology with a single Kinesis ingress stream that utilizes EFO, a few transform operations, and then sinking to DynamoDb. Version: Flink 1.15 I’m running a Flink job on the AWS Managed Service, and am not sure if what I’m

RE: Plans for upgrading Curl with latest 8.4.0

2023-10-12 Thread Singhal, Ankur
Hi Matijn, This is just a reference but we are using it at multiple places. https://github.com/apache/flink/blob/master/tools/ci/maven-utils.sh#L59 Although the hostname we are referring here is hardcoded so it can be mitigated. Thanks and Regards, Ankur Singhal -Original Message-

Re: Plans for upgrading Curl with latest 8.4.0

2023-10-12 Thread Martijn Visser
Hi Ankur, Where do you see Flink using/bundling Curl? Best regards, Martijn On Wed, Oct 11, 2023 at 9:08 AM Singhal, Ankur wrote: > > Hi Team, > > > > Do we have any plans to update flink to support Curl 8.4.0 with earlier > versions having severe vulnerabilities? > > > > Thanks & Regards, >