Re: Apache Atlas - Flink Integration

Hi Shammon, Thank you for your prompt reply.Aslo I'm interested to know if there is an available feature for integrating Apache Flink with Apache Ranger. If so, could you kindly share the relevant documentation with me? Thanks & Regards, Arjun

Re: Kafka Disconnection Error

Hi, Kenan I think you maybe can get help from Kafka community. IMO, it is just an info level log, whether it has a real impact? Best, Ron Kenan Kılıçtepe 于2023年8月2日周三 06:04写道： > I got a lot of these disconnection error logs. Why? My flink and kafka > clusters are running in Google Cloud and

flink1.17.1版本 flink on yarn 提交无法获取配置文件

/opt/flink/flink-1.17.1/bin/flink run-application -t yarn-application -yjm 1024m -ytm 1024m ./xx-1.0.jar ./config.properties以上提交命令制定的配置文件，为什么在容器内找配置文件？file /home/yarn/nm/usercache/root/appcache/application_1690773368385_0092/container_e183_1690773368385_0092_01_01/./config.properties does

Re: Custom connector Update flink 1.13.3 failed notsuchMethod

Hi, longfeng. I think you should rebuild your connector according to the new API. The return type of the method `DynamicTableFactory$Context.getCatalogTable()` is already changed from `CatalogTable` to `ResolvedCatalogTable`[]. Best, Hang [1] https://issues.apache.org/jira/browse/FLINK-21913

Re: Apache Atlas - Flink Integration

Hi arjun, As @Mate mentioned, the discussion of FLIP-314 has been completed and a vote will be initiated soon. We would like to introduce the interfaces for lineage in the next release of Flink after 1.18 Best, Shammon FY On Tue, Aug 1, 2023 at 11:07 PM Mate Czagany wrote: > Hi, > >

Flink operator task opens threads internally

Hello Community, I have a operator pipeline like as below, is it ok if "source" task opens threads by using java thread pool and parallelize the work? This is needed for accepting multiple client socket connections in "single custom source server socket function". Single Custom source server

Re: No. of task managers vs No. of task slots

Hi Kamal, For the three points > 3. What is the difference between high no. of task managers vs high no. of task slots (with low no. of task managers)? I think this is mainly limited by the JVM's efficiency in managing memory. When we use Flink Session cluster as olap engine, we found that when

RE: Flink netty connector for TCP source

Thanks Shammon. Purpose of opening server socket in Split Enumerator was that it has only one instance per source and so the server socket too (port binding can happen only once). And then accepted Socket connections (serversocket.accept()) will act as splits which will be further processed by

Re: Flink netty connector for TCP source

Hi Kamal, It confuses me a little that what's the purpose of opening a server socket in SplitEnumerator? Currently there will be only one SplitEnumerator instance in JobManager for each one source not each source subtask. If there's only one source in your job, no matter how much parallelism this

退订

*退订*

Custom connector Update flink 1.13.3 failed notsuchMethod

Flink 1.13.3 Custom connector Using flink Kafka connector code and little refactoring; And Custom connector can be load in flink 1.12 when using StreamTableEnvironment. Now flink upgrade to 1.13.3, custom connector dependencies also upgraded to 1.13.3 But failed to load:

Kafka Disconnection Error

I got a lot of these disconnection error logs. Why? My flink and kafka clusters are running in Google Cloud and I dont think there is a network issue. Also I got this error even my workload is very low. 2023-08-01 21:54:00,003 INFO org.apache.kafka.clients.NetworkClient [] -

Dependency injection framework for flink

Hi, we are currently migrating some of our jobs into hexagonal architecture and I have seen that we can use spring as dependency injection framework, see: https://getindata.com/blog/writing-flink-jobs-using-spring-dependency-injection-framework/ Has anybody analyzed different JVM DI frameworks

Re: Apache Atlas - Flink Integration

Hi, Unfortunately the Atlas hook you've read about is only available in the Cloudera Flink solution and has not been made open-source. In the future FLIP-314[1] might offer a simple solution to implement the Atlas integration. Best Regards, Mate [1]

Apache Atlas - Flink Integration

I am looking to integrate Apache Atlas with Apache Flink to capture Job lineage. I found some references around it from Cloudera (CDP) and they are using Atlas-flink hook , but I am not able to find any documentation or implementation. I had gone through the JIRA link as mentioned below.But in

Re: Flink Kubernetes Operator autoscaling GPU-based workload

Hi, I've tinkered around a bit more and found that the problem is actually with Native mode vs Standalone mode. In the standalone mode, the pod definition doesn't get a Resource request for nvidia/gpu, whereas in the Native mode it does. I'll open another question since this isn't related to

Is there any way I can do aggregation on DataStream in Apache Flink and store the Aggregated value?

Hi Team, I am new to Flink. I have this use case where I have a dataStream of Doubles and I am trying to get the total sum of whole DataStream. I have used ReduceFunction and AggregateFunction. Case 1: In Reduced function the output is dataStream of rolling Sum. To get the final Sum I have to

Re: Flink Kubernetes Operator autoscaling GPU-based workload

The autoscaler only works for FlinkDeployments in Native mode. You should turn off the reactive scheduler mode as well because that's something completely different. After that you can check the autoscaler logs for more info. Gyula On Tue, Aug 1, 2023 at 10:33 AM Raihan Sunny via user wrote: >

Flink ML

您好 我想知道是否有代码示例，可以在Flink程序中加载预先训练好的编码模型(用python编写)

Flink Kubernetes Operator autoscaling GPU-based workload

Hi, I have a workload that depends on the GPU. I have only 1 GPU card. As per the documentation I have added the necessary configurations and can run the GPU workload in standalone REACTIVE mode with as many taskmanager instances as required. I have set the number of task slots to 1 so that a

Re: No. of task managers vs No. of task slots

Hi, Kamal > How many task managers a job manager can handle? Is there any upper limit also? There is no clear limit to how many TMs a JM can cover, and based on my past experience, it can handle TMs over 1000+, even more. > How to decide no. of task managers, is there any way? I don't think

Re: [EXTERNAL] Re: Query on flink-operator autoscale support

The autoscaler scales jobs based on incoming data and processing throughput. It's completely different from the reactive mod, if the throughput/processing rate doesn't change it will not scale up even if you have more resources available. Also in native mode you cannot add pods to the cluster,

Re: Flink Table API + Jacoco Plugin

up On Tue, 18 Jul 2023 at 09:27, Brendan Cortez wrote: > Hi all! > > I'm trying to use the jacoco-maven-plugin and run unit-tests for Flink > Table API, but they fail with an error (see file error_log_flink_17.txt for > full error stacktrace in attachment): > java.lang.IllegalArgumentException:

No. of task managers vs No. of task slots

Hell Community, Need info. for below - 1. How many task managers a job manager can handle? Is there any upper limit also? 1. How to decide no. of task managers, is there any way? 1. What is the difference between high no. of task managers vs high no. of task slots (with low no.