Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

Hi Flink Users, We are trying to upgrade Flink from 1.12.7 to 1.16.0. But we run into the following issue: We are running Flink job through application mode. After the upgrade, when we submit the job and now it gets this exception: *org.apache.flink.client.deployment.ClusterDeploymentException:

Updating Parallelism based on Traffic and Max Parallelism

Hello Team, I am fine tuning my application which can adjust parallelism automatically based on traffic and EMR auto-scaling will be adding more cores as needed. I am trying to set max Parallelism but I always see its working with default parallelism irrespective of traffic. Can you please let

Re: OOM taskmanager

Hi Marco, When you say OOM, I assume you mean TM pod being OOMKilled, is that correct? If so, this usually means that the TM is using more than the actual memory allocated to the pod. First I would check your memory configuration to figure out where this extra memory use is coming from. This

Re: Docker image Flink 1.15.4

1.15.4 is not released yet. On 26/01/2023 16:06, Peng Zhang wrote: Hi, We would like to use Flink 1.15.4 docker image. The latest seems 1.15.3. Could you make a docker release Flink 1.1.5.4? Thanks! There is a blocking bug https://issues.apache.org/jira/browse/FLINK-28695

Re: Using pyflink from flink distribution

Ah, oops and my original email had a typo: > Some python dependencies are not included in the flink distribution tarballs: cloudpickle, py4j and pyflink are in opt/python. Should read: > Some python dependencies ARE included in the flink distribution tarballs: cloudpickle, py4j and pyflink are in

Re: Using pyflink from flink distribution

Let me ask a related question: We are building our own base Flink docker image. We will be deploying both JVM and python apps via flink-kubernetes-operator. Is there any reason not to install Flink in this image via `pip install apache-flink` and use it for JVM apps? -Andrew Otto Wikimedia

Docker image Flink 1.15.4

Hi, We would like to use Flink 1.15.4 docker image. The latest seems 1.15.3. Could you make a docker release Flink 1.1.5.4? Thanks! There is a blocking bug https://issues.apache.org/jira/browse/FLINK-28695 in 1.5.3 and fixed in 1.15.4 BR, Peng

Re: Using S3 as stream source in Flink

I saw in aws-samples https://github.com/aws-samples/flink-stream-processing-refarch/blob/master/kinesis-taxi-stream-producer/src/main/java/com/amazonaws/flink/refarch/utils/TaxiEventReader.java they are not using FileSource. Now I got it. Thanks, Martijn. On Wed, Jan 25, 2023 at 9:07 PM Martijn