Re: FlinkKafkaProducer - Avro - Schema Registry

Hello Qingsheng, Removing KafkaAvroSerializer from the producer properties worked, indeed. I validated this by using a FlinkKafkaConsumer, using ConfluentRegistryAvroDeserializationSchema, so it's working properly. The problem I'm still having, is that I will have to use schema registry where

退订

退订

Re: k8s session cluster flink1.13.6创建后提示的地址啥用。

貌似官网对flink k8s情况有2个入口，分别为： https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#session-mode 和 https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/。 分别对应 Resource

Re: k8s session cluster flink1.13.6创建后提示的地址啥用。

集群是3台物理机搭建，非minikube。 不清楚是否和网卡有关，init搭建时就有网络问题，k8s要根据默认路由网卡ip决定监听的地址。 但是我感觉这个场景不应该，因为既然是clusterIp，创建后提示信息就应该提示用clusterIp吧，为啥提示的用了本机的网卡ip呢。 yidan zhao 于2022年4月8日周五 10:38写道： > > 如下是 describe svc my-first-flink-cluster-rest 的结果： > Name: my-first-flink-cluster-rest > Namespace:

Re: k8s session cluster flink1.13.6创建后提示的地址啥用。

如下是 describe svc my-first-flink-cluster-rest 的结果： Name: my-first-flink-cluster-rest Namespace:default Labels: app=my-first-flink-cluster type=flink-native-kubernetes Annotations: Selector:

Re: Official Flink operator additional class paths

Hi Yang, thanks a lot for your help. It ended up being the case that my command in the initContainer was specified incorrectly. On Thu, 7 Apr 2022 at 18:41, Yang Wang wrote: > It seems that you have a typo when specifying the pipeline classpath. >

Re: FlinkKafkaProducer - Avro - Schema Registry

Hi Dan, In FlinkKafkaProducer, records are serialized by the SerializationSchema specified in the constructor, which is the “schema” (ConfluentRegistryAvroSerializationSchema.forSpecific(AvroObject.class)) in your case, instead of the serializer specified in producer properties. The default

退订

退订 发自我的iPhone

yarn api 提交报错

hi, 我在测试使用java api提交flink任务时，遇到了一些问题，需要请教： flink版本1.14.4 Hadoop版本：3.0.0-cdh6.2.1 application模式，使用命令提交正常运行，api提交失败 提交失败，yarn日志： LogType:jobmanager.err LogLastModifiedTime:Fri Apr 08 09:24:01 +0800 2022 LogLength:107

Re: Produnction : Flink 1.14.4 : Kafka reader threads blocked

Hi Vignesh, I think you can check the following things: 1. Check the cpu usage of the workers. Are they close to zero or almost full? 2. Any back pressure happened in downstream tasks? 3. Is the fullGC significant serious? Best, Yuan > On 7 Apr 2022, at 12:33 PM, Vignesh Ramesh wrote: > >

Re: k8s session cluster flink1.13.6创建后提示的地址啥用。

理论上cluster ip是不可能在集群外访问的，你的Kubernetes环境是怎么搭建的呢？Minikube吗？ 方便的话可以分享你运行这个命令的结果吗? 》kubectl describe svc my-first-flink-cluster-rest > On 7 Apr 2022, at 4:44 PM, Zhanghao Chen wrote: > > 你 kubernetes.rest-service.exposed.type 这个参数设置的是什么呢？ > > Best, > Zhanghao Chen >

Re: flink pipeline handles very small amount of messages in a second (only 500)

Hi Sigalit, In your settings, I guess each job will only have one slot (parallelism). So is it too many input for your jobs with parallelism only one? One easy way to confirm is that you increase your slots and job parallelism twice and then see whether the QPS is increased. Hope this would

Re: flink pipeline handles very small amount of messages in a second (only 500)

Hi Sigalit, First of all, did you ensure different source operator consumes different consumer id for the kafka source? Did each flink job share the same data or consumed the data independently? Moreover, was your job behaves back pressured? It might need to break the chained operator to see

flink pipeline handles very small amount of messages in a second (only 500)

hi all I would appreciate some help to understand the pipeline behaviour... We deployed a standalone flink cluster. The pipelines are deployed via the jm rest api. We have 5 task managers with 1 slot each. In total i am deploying 5 pipelines which mainly read from kafka, a simple object

Re: Missing metrics in Flink v 1.15.0 rc-0

Hi, Flink 1.15 has developed a new feature to support different sink pre- and post-topologies[1]. New metrics e.g. NumRecordsSend has been developed to measure records sent to the external system. Metrics like "Bytes Sent" and "Records Sent" measure records sent to the next task. So, in your

RE: python table api

Hello Dian, Indeed. Thank you very much. Now getting +I[2022-01-01T10:00:20, 2022-01-01T10:00:25, 2] +I[2022-01-01T10:00:25, 2022-01-01T10:00:30, 5] +I[2022-01-01T10:00:30, 2022-01-01T10:00:35, 1] from pyflink.table import EnvironmentSettings, TableEnvironment t_env =

Re: Unbalanced distribution of keyed stream to downstream parallel operators

Hello Arvid , thank you for your reply. Actually using a window to aggregate the events for a time period is not applicable to my case since I need the records to be processed immediately. Even if I could I still can not understand how I could forward the aggregated events to lets say 2 parallel

Re: BIGDECIMAL data handling

Is there any reason for not using DECIMAL provided by Flink SQL? On Tue, Apr 5, 2022 at 4:06 PM Anitha Thankappan < anitha.thankap...@quantiphi.com> wrote: > Hi Martijn, > > I am using flink version 1.11.0. > The flink application code snippet is like: > > [image: image.png] > > The Error I am

Re: HOP_PROCTIME is returning null

Maybe the reason is because the HOP_PROCTIME gets the name of the column? Can you share query and plan? On Mon, Apr 4, 2022 at 3:41 PM Surendra Lalwani wrote: > Hi Team, > > HOP_PROCTIME in flink version 1.13.6 is returning null while in previous > versions it used to output a time attribute,

Re: python table api

As Dian sad, your insert into query is just selecting records from source to print, flowing them without any computation whatsoever. Please check out [1] and [2] to learn how to develop queries that perform aggregations over windows. Note that the second method (window tvf) is preferred and

Re: k8s session cluster flink1.13.6创建后提示的地址啥用。

你 kubernetes.rest-service.exposed.type 这个参数设置的是什么呢？ Best, Zhanghao Chen From: yidan zhao Sent: Thursday, April 7, 2022 11:41 To: user-zh Subject: k8s session cluster flink1.13.6创建后提示的地址啥用。 参考

Re: Official Flink operator additional class paths

It seems that you have a typo when specifying the pipeline classpath. "file:///flink-jar/flink-connector-rabbitmq_2.12-1.14.4.jar" -> "file:///flink-jars/flink-connector-rabbitmq_2.12-1.14.4.jar" If this is not the root cause, maybe you could have a try with downloading the connector jars to

flink sql 任务中jm Blob server 总是在凌晨报 java.io.exception :unknow opreation 71

hi,all.想问大家下，是否有遇到过这个问题，flink 1.12 的版本 在线上运行的flink sql 作业，总是在凌晨报错如下： [cid:b11b980a-9bcd-4e7d-993a-e83a9322c66c] blobserver 我理解是传输二进制jar 包，从hdfs 到 本地工作目录。但没发现其他环节出现问题，对任务数据未产生影响。。

Is there any way to get the ExecutionConfigurations in Dynamic factory class

Hi I have developed a BigQuery Flink connector by implementing DynamicTableSourceFactory. I have a requirement to : get the configured parallelism value of StreamExecutionEnvironment in the Factory class. or set the parallelism

Re: flink table store

@tison https://nightlies.apache.org/flink/flink-table-store-docs-release-0.1/docs/try-table-store/quick-start/ Best, Paul Lam > 2022年4月7日 15:05，tison 写道： > > 我有点好奇官网看到的链接在哪，能不能来个链接捏。 >

Official Flink operator additional class paths

Hi all, thanks in advance for any tips. I've been trying to specify some additional classpaths in my kubernetes yaml file when using the official flink operator and nothing seems to work. I know the technique for getting my job jar works fine since it's finding the class ok, but I cannot get the

Re: flink table store

我有点好奇官网看到的链接在哪，能不能来个链接捏。 Best, tison. Leonard Xu 于2022年4月7日周四 14:47写道： > > 项目是开源的[1], 最近快要发布第一个版本了，可以关注下 > > Best, > Leonard > [1] https://github.com/apache/flink-table-store < > https://github.com/apache/flink-table-store> > > > > > 2022年4月7日 上午9:54，Xianxun Ye 写道： > > > > 这里有 flink table

Re: flink table store

项目是开源的[1], 最近快要发布第一个版本了，可以关注下 Best, Leonard [1] https://github.com/apache/flink-table-store > 2022年4月7日 上午9:54，Xianxun Ye 写道： > > 这里有 flink table store 的设计文档，你可以了解下。 >