OOM taskmanager

Hello, We are deploying a flink application cluster in kubernetes, 2 pods one for the JM and the other for the TM. The problem is when we launch load tests we see that task manager memory usage increases, after the tests are finished and flink stop processing data the memory usage never comes d

Re: Job gets stuck when using kafka transactions and eventually crashes

If data flows normally, watermarks will progress to downstream operators. If you have no records flowing in a partition of a stream and you don't have an idleness configured, that partition will hold back the progress of watermarks. Watermarks are essential for the checkpointing mechanism, which ul

Re: Job gets stuck when using kafka transactions and eventually crashes

Can you elaborate a bit more? While idleness is not what we’re seeing now, it could perhaps be an issue later on. What about a certain partition going idle will result in state buildup? Thanks, Vishal On 25 Jan 2023 at 9:14 PM +0530, Martijn Visser , wrote: > Hi Vishal, > > Could idleness be an

Re: Job gets stuck when using kafka transactions and eventually crashes

Hi Vishal, Could idleness be an issue? I could see that if idleness occurs and the Kafka Source not going in an idle state, that more internal state (to commit Kafka transactions) can build up over time that ultimately causes an out of memory problem. See https://nightlies.apache.org/flink/flink-d

Re: Using S3 as stream source in Flink

Hi Sriram G, Both the DataStream and Table API support filesystem as a source in unbounded (streaming mode) with exactly once guarantees. This is documented at https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/filesystem/ and https://nightlies.apache.org/flink/flink-d

Re: PyFlink job in kubernetes operator

No, just check documentation on https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/ and found no mention of python Thank you Apache Flink Kubernetes Operator | Apache Flink Kubernetes Operator nightlie

Re: PyFlink job in kubernetes operator

Did you check the Python example? https://github.com/apache/flink-kubernetes-operator/tree/main/examples/flink-python-example Gyula On Wed, Jan 25, 2023 at 2:54 PM Evgeniy Lyutikov wrote: > Hello > > Is there a way to run PyFlink jobs in k8s with flink kubernetes operator? > And if not, is it p

PyFlink job in kubernetes operator

Hello Is there a way to run PyFlink jobs in k8s with flink kubernetes operator? And if not, is it planned to add such functionality? "This message contains confidential information/commercial secret. If you are not the intended addressee of this message you may