hi all I would appreciate some help to understand the pipeline behaviour...
We deployed a standalone flink cluster. The pipelines are deployed via the jm rest api. We have 5 task managers with 1 slot each. In total i am deploying 5 pipelines which mainly read from kafka, a simple object conversion and either write back to kafka or GCP pub/sub or save in the DB. These jobs run "forever" and basically each task manager runs a specific job (this is how flink handled it). We have a test that sends to kafka 10k messages per second. but according to the metrics exposed by flink i see that the relevant job handles only 500 messages per second. I would expect all the 10K to be handled. I guess the setup is not correct. The messages are in avro format Currently we are not using checkpoints at all. Any suggestions are welcome. Thanks alot Sigalit
