Hi all, We are currently using beam to create a few pipelines, and then deploy them on our on-prem Flink cluster.
we have a few questions regarding the automation of the pipelines deployment: We have beam running as a k8s pod which starts a java process for each pipeline and has the flinkrunner parameters. the java processes deploy the pipelines as flink jobs (via rest to a flink cluster deployed on k8s as well) The main issues we have with the current deployment is – every time the beam pod gets restarted, it re-creates the flink jobs, and this causes duplications and errors. We were wondering what the recommended way to automate deploying the beam pipelines to flink, and is there any documentation on this? We also tried to see how to generate jars from the beam pipelines and then running them on flink, but we had issues with the Fat Jars, as they included some flink libraries, and we were not able to run them as is on flink, is there a way to get a ‘leaner’ fat jar? Thanks
