Re: Flink throws exception when submitting a job through Jenkins and Spinnaker

Hi Shammon, Thanks for your response. If it is a network issue as you have mentioned, how does it read the contents of the jar file, we can see that the code is read and it throws an error only when executing the SQL. Also can you let us know exactly what address could be wrong here, so that we c

Re: [Question] Good way to monitor data skewness

Hi, Dennis, Although all operators are chained together, each operator metrics is there, you can view the metrcis related to the corresponding operator's input and output records through the UI, as following: [image: image.png] Best, Ron Dennis Jung 于2023年8月16日周三 14:13写道： > Hello people, > I'

[Question] Good way to monitor data skewness

Hello people, I'm trying to monitor data skewness with 'web-ui', between TaskManagers. Currently all operators has been chained, so I cannot find how data has been skewed to TaskManagers (or subtasks). But if I disable chaining, AFAIK, it can degrade performance. https://nightlies.apache.org/flin

Re: [Issue] Repeatedly receiving same message from Kafka

Hello, Thank you. I think it is a problem caused by Kafka configuration, not Flink. I'll take a look and let you know if there's an issue in Flink. BR, Jung 2023년 8월 15일 (화) 오후 9:40, Hector Rios 님이 작성: > Hi there > > It would be helpful if you could include the code for your pipeline. One > su

Re: [E] RE: Recommendations on using multithreading in flink map functions in java

Thanks Ron and Thias, I understand flink parallelism works at task level. Distribute n subtasks of the operator across your cluster and parallel process the elements by distributing them across multiple instances of your operator. This gives me very high throughput. What I want to know is how to

[ANNOUNCE] Apache flink-connector-mongodb 1.0.2 released

The Apache Flink community is very happy to announce the release of Apache flink-connector-mongodb 1.0.2. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download

[ANNOUNCE] Apache Flink Kubernetes Operator 1.6.0 released

The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.6.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Release highlights: - Improved rollback me

Re: 404 Jar File Not Found w/ Web Submit Disabled

Hi, Patricia Maybe you should check the code carefully, a reply code of 404 typically indicates that an error occured in client side.Alternatively,could you share a snippet code where the error occured? This would greatly assist us in addressing your issue effectively. Best Jiadong Lu On 20

Re: [Issue] Repeatedly receiving same message from Kafka

Hi there It would be helpful if you could include the code for your pipeline. One suggestion, can you disable the "EXACTLY_ONCE" semantics on the producer. Using EXACTLY_ONCE will leverage Kafka transactions and thus add overhead. I would disable it to see if you still get the same situation. Als

[Issue] Repeatedly receiving same message from Kafka

Sorry, I've forgot putting title, so sending again. 2023년 8월 15일 (화) 오후 6:27, Dennis Jung 님이 작성: > (this is issue from Flink 1.14) > > Hello, > > I've set up following logic to consume messages from kafka, and produce > them to another kafka broker. For producer, I've configured > `Semantics.EXAC

[no subject]

(this is issue from Flink 1.14) Hello, I've set up following logic to consume messages from kafka, and produce them to another kafka broker. For producer, I've configured `Semantics.EXACTLY_ONCE` to send messages exactly once. (also setup 'StreamExecutionEnvironment::enableCheckpointing' as 'Chec

Re: Question about serialization of java.util classes

Check this answer: https://stackoverflow.com/a/64721838/5793905 You could then use, for example, something like: new SetTypeInfo(Types.STRING) instead of Types.LIST(Types.STRING) Am Di., 15. Aug. 2023 um 10:40 Uhr schrieb : > Hello Alexis, > > Thank you for sharing the helper classes this but un

Re: Question about serialization of java.util classes

Hello Alexis, Thank you for sharing the helper classes this but unfortunately I have no idea how to use these classes or how they might be able to help me. This is all very new to me and I honestly can't wrap my head around Flink's type information system. Best regards, Saleh. > On 14 Aug 202

RE: Recommendations on using multithreading in flink map functions in java

Hi Vignesh, In addition to what Ron has said, there are a number of options to consider, depending on the nature of your calculations: Given that your main focus seems to be latency: * As Ron has said, Flink manages parallelism in a coarse grained way that is optimized for spending as lit