Re: Quick question about flink document.

Hi, Dongwoo, I think there is no problem in this part. This part describes snapshotting Operator State, which is checkpointing. The checkpoint will store by the JobManager and use the checkpoint storage. Best, Hang Feng Jin 于2023年4月10日周一 00:32写道： > Hi Dongwoo > > > This can be quite

Re: How to set reblance in Flink Sql like Streaming api?

Hi, hiw, IMO, I think the parallelism 1 is enough for you job if we do not consider the sink. I do not know why you need set the lookup join operator's parallelism to 6. The SQL planner will help us to decide the type of the edge and we can not change it. Maybe you could share the Execution graph

Re: Access ExecutionConfig from new Source and Sink API

Hi, christopher, I think there is already about the ExecutionConfig for new Sink API in the FLIP-287[1]. What we actually need is a read-only ExecutionConfig for Source API and Sink API. Maybe we could continue to discuss this topic under FLIP-287. Best, Hang [1]

Re: org.apache.flink.table.api.ValidationException

Hi, This error occurs when the data type can not be parsed. You could read this part to see more details about the User-Defined Data Types[1]. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/types/#user-defined-data-types 柒朵 <1303809...@qq.com>

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

Congratulations! Best, Hang yu zelin 于2023年3月28日周二 10:27写道： > Congratulations! > > Best, > Yu Zelin > > 2023年3月27日 17:23，Yu Li 写道： > > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

Congratulations! Best, Hang yu zelin 于2023年3月28日周二 10:27写道： > Congratulations! > > Best, > Yu Zelin > > 2023年3月27日 17:23，Yu Li 写道： > > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project

Re: Are metadata columns required to get declared in the table's schema?

Hi, Jie, If you don't need these metadata columns, you don't need to declare them for the table. Then metadata columns will not be read from sources and will not be written into the sink. You can query a table that is without the metadata column declaration. It depends on your requests. Best,

Re: Are metadata columns required to get declared in the table's schema?

ps : DDL I said is the CREATE TABLE statements. Best, Hang Hang Ruan 于2023年3月26日周日 21:33写道： > Hi, Jie, > > In Flink, if we want to access a metadata column, we need to declare it in > the DDL. > More details could be found here[1]. > > Best, > Hang > > [1] > h

Re: Are metadata columns required to get declared in the table's schema?

Hi, Jie, In Flink, if we want to access a metadata column, we need to declare it in the DDL. More details could be found here[1]. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/create/#columns Jie Han 于2023年3月26日周日 14:58写道： > Hi community, I want

Re: Table API function and expression vs SQL

Hi, I think the SQL job is better. Flink SQL jobs can be easily shared with others for debugging. And it is more suitable for flow batch integration. For a small part of jobs which can not be expressed through SQL, we will choose a job by DataStream API. Best, Hang ravi_suryavanshi.yahoo.com

Re: [ANNOUNCE] Apache Flink 1.17.0 released

Thanks for the great work ! Congrats all! Best, Hang Panagiotis Garefalakis 于2023年3月25日周六 03:22写道： > Congrats all! Well done! > > Cheers, > Panagiotis > > On Fri, Mar 24, 2023 at 2:46 AM Qingsheng Ren wrote: > > > I'd like to say thank you to all contributors of Flink 1.17. Your support > >

Re: [ANNOUNCE] Apache Flink 1.17.0 released

Thanks for the great work ! Congrats all! Best, Hang Panagiotis Garefalakis 于2023年3月25日周六 03:22写道： > Congrats all! Well done! > > Cheers, > Panagiotis > > On Fri, Mar 24, 2023 at 2:46 AM Qingsheng Ren wrote: > > > I'd like to say thank you to all contributors of Flink 1.17. Your support > >

Re: Is there a way to control the parallelism of auto-generated Flink operators of the FlinkSQL job graph?

Hi, Elkhan, I think this is an intended behavior. If the parallelism of an operator is not specified, it will be the same as the previous one instead of the default parallelism. Actually the table planner will help us to do most jobs. There should not be a way to modify the parallelism for every

Re: Bootstrapping multiple state within same operator

Hi, David, I also read the code about the `SavepointWriter#withOperator`. The transformations are stored in a `Map` whose key is `OperatorID`. I don't come up with a way that we could register multi transformations for one operator with the provided API. Maybe we need a new type of

Re: Unsubscribe

Hi, please send an email to user-unsubscr...@flink.apache.org to unsubscribe . Best, Hang laxmi narayan 于2023年3月21日周二 15:26写道： > Unsubscribe -- > Hi , > > > > Thank you. >

Re: subscribe

Please send an e-mail to user-subscr...@flink.apache.org to subscribe to the flink user mail list. Best, Hang mark 于2023年3月15日周三 22:07写道： > subscribe >

Re: Kafka sql with validation exception

Hi, Lasse, I think you should make sure the situation as Shammon said. Maybe you need to use the maven-shade-plugin like this to package, and make sure files in `META-INF/services` are merged together. org.apache.maven.plugins < > artifactId>maven-shade-plugin 3.2.4 < > executions> package

Re: Are the Table API Connectors production ready?

Hi, yuxia, I would like to help to complete this task. Best, Hang yuxia 于2023年3月13日周一 09:32写道： > Yeah, you're right. We don't provide filtering files with patterns. And > actually we had already a jira[1] for it. > I was intended to do this in the past, but don't have much time. Anyone > who

Re: unsubscribe

Hi, please send to user-unsubscr...@flink.apache.org if you want to unsubscribe user mail list. Best, Hang zhangjunjie 于2023年2月28日周二 00:11写道： > unsubscribe > > >

Re: Could savepoints contain in-flight data?

hanged had in-flight data as part of the > checkpoint, it will lose said data after starting, right? > > I imagine this is not good practice, but it's just a hypothetical scenario > I wanted to understand better. > > Regards, > Alexis. > > > Am Mo., 13. Feb. 2023 um 12:33 Uhr schrie

Re: Could savepoints contain in-flight data?

ps: the savepoint will also not contain in-flight data. Best, Hang Hang Ruan 于2023年2月13日周一 19:31写道： > Hi Alexis, > > No, aligned checkpoint will not contain the in-flight. Aligned checkpoint > makes sure that the data before the barrier has been processed and there is > n

Re: Could savepoints contain in-flight data?

Hi Alexis, No, aligned checkpoint will not contain the in-flight. Aligned checkpoint makes sure that the data before the barrier has been processed and there is no need to store in-flight data for one checkpoint. I think these documents[1][2] will help you to understand it. Best, Hang [1]

Re: I want to subscribe users' questions

Hi, guanyuan, This document( https://flink.apache.org/community.html#how-to-subscribe-to-a-mailing-list) will be helpful. welcome~ Best, Hang guanyuan chen 于2023年2月7日周二 21:37写道： > Hi, > My name is Guanyuan Chen.I am a big data development engineer, tencent > wechat department, china. I

Re: I want to subscribe users' questions

Hi, guanyuan, This document( https://flink.apache.org/community.html#how-to-subscribe-to-a-mailing-list) will be helpful. welcome~ Best, Hang guanyuan chen 于2023年2月7日周二 21:37写道： > Hi, > My name is Guanyuan Chen.I am a big data development engineer, tencent > wechat department, china. I

Re: Standalone cluster memory configuration

Hi, Theodor, The description in https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/config/#memory-configuration map help you to config the memory for flink. > Flink tries to shield users as much as possible from the complexity of > configuring the JVM for data-intensive

Re: Slot sharing behavior in Flink cluster

Hi, Le Xu, If the job is a streaming job, all tasks should be scheduled before any data can flow through the pipeline, and tasks will run in parallel. I think the Execution Mode[1] and FLIP-134[2] will help you to understand more details. Best, Hang [1]

Re: Re: How to set disableChaining like streaming multiple INSERT statements in a StatementSet ?

Hi, Hjw, Maybe this configuration is helpful. https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/config/#pipeline-operator-chaining Best, Hang hjw 于2022年12月8日周四 12:01写道： > hi,yuxia. > > The Dag Image of the job graph is attached in email. > As the code image showed, I

Re: Flink KafkaSource still referencing deleted topic

Hi, Robert, The configuration allowNonRestoredState should be used like this: ./bin/flink run --detached --allowNonRestoredState Best, Hang Robert Cullen 于2022年10月12日周三 23:13写道： > I don't see AllowNonRestoredState in the configuration documentation. How > would it be passed to a job? On the

Re: Regarding Connector Options - value.deserializer

pt > 'value.format' in below code snippet so is that enough and handle > deserializer by itself internally? > > How to create custom format can you please share any link for sample > example for the same ? > > > > Thanks > > Ronak Beejawat > > > > > &

Re: Regarding Connector Options - value.deserializer

Hi, Ronak, I think you should implement a custom format by yourself instead of overriding. The 'value.format' is a required table option. Best, Hang Ronak Beejawat (rbeejawa) 于2022年1月10日周一 17:09写道： > Hi Team, > > Is there any way we use value.deserializer in Connector Options from kafka > via

Re: kafka源码执行测试用例问题

应该先用mvn install 发布到本地仓库，这个依赖才可以被找到 Yuepeng Pan 于2021年12月17日周五 20:28写道： > Hi, Chen. > 如果是idea模式，可以尝试排查下pom中依赖项的scope。 > > > > Best, > Roc. > > > > > > 在 2021-12-17 17:41:32，"陈卓宇" <2572805...@qq.com.INVALID> 写道： > >您好社区： > > > >我在进行flink源码Kafka连接器部分进行测试用例运行 > > > >报错日志： > > > >[ERROR] >

Re: Alternatives of KafkaDeserializationSchema.isEndOfStream()

There is no way to end the kafka stream from the deserializer. When would you want to end the stream? Could you explain why you need to end the kafka stream without using the offset? Ayush Chauhan 于2021年12月8日周三 15:29写道： > >

Re: enable.auto.commit=true and checkpointing turned on

Hi, 1. Yes, the kafka source will use the Kafka committed offset for the group id to start the job. 2. No, the auto.offset.reset is from Kafka consumer config, which defines what to do when there is no initial offset in

Re: KafkaSink.builder setDeliveryGuarantee is not a member

Hi, It seems to be an error in documents. `setDeliverGuarantee` is the method of class `KafkaSinkBuilder`, . It could be used like this : KafkaSink.builder().setDeliverGuarantee(xxx) Lars Skjærven 于2021年12月2日周四 19:34写道： > Hello, > upgrading to 1.14 I bumped into an issue with the kafka sink

Re: How do you configure setCommitOffsetsOnCheckpoints in Flink 1.12 when using KafkaSourceBuilder?

Sorry, I spell it wrong, which I mean the PR. Here it is https://github.com/apache/flink/pull/17276 . Marco Villalobos 于2021年12月1日周三 下午9:18写道： > Thank you. One last question. What is an RP? Where can I read it? > > Marco > > On Nov 30, 2021, at 11:06 PM, Hang Ruan wrote: > &

Re: How do you configure setCommitOffsetsOnCheckpoints in Flink 1.12 when using KafkaSourceBuilder?

th checkpoints? > > > > > On Mon, Nov 29, 2021 at 7:46 PM Hang Ruan wrote: > >> Hi, >> >> Maybe you can write like this : >> builder.setProperty(KafkaSourceOptions.COMMIT_OFFSETS_ON_CHECKPOINT.key(), >> "true"); >> >> Other addit

Re: How do you configure setCommitOffsetsOnCheckpoints in Flink 1.12 when using KafkaSourceBuilder?

Hi, Maybe you can write like this : builder.setProperty(KafkaSourceOptions.COMMIT_OFFSETS_ON_CHECKPOINT.key(), "true"); Other additional properties could be found here : https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/#additional-properties Marco

<    1   2