Thanks Alexander for looking into email. - Should this reusable ProcessFunction filter out data or simply tag > them? For example, filtering data for green deployments would be > suitable > in the case of Async IO outputs.
Consider a Flink job that mutates data in an external service using Async IO. If we are running this job with a blue-green deployment, it might be crucial to ensure that the Green job does not produce such mutations. This could be due to performance or quality considerations. Then there are could be different mechanism/APIs to provide such behaviour, e.g. wrapping the data into the tuple with color and it is up to user to have specific logic in order to process blue/green messages. And there are another question, which have been lost due to the formatting issues: - Have you considered adding a validation mechanism for Blue-Green deployments? For instance, you could allow users to perform automatic validation before promoting the Green Flink job to Blue. Simple solution: After deploying of the Green job, k8s operator could wait for an external ConfigMap change or a property change in the FlinkBlueGreenDeployment CRD to initiate the promotion. Such ConfigMap change/CRD change, could be initiated by another external system which ensures that Green job is performing correctly. Regards Alex On Tue, Dec 17, 2024 at 2:16 PM Alexander Fedulov < alexander.fedu...@gmail.com> wrote: > Hi Alex, > > - Should this reusable ProcessFunction filter out data or simply tag >> them? For example, filtering data for green deployments would be >> suitable >> in the case of Async IO outputs. > > > Could you clarify the idea a bit more, it is not quite clear what you mean. > > - Should we also modify the Source API to allow it to provide COLOR to >> the Source functions? This could be used, for instance, to generate a >> different groupId for Kafka. > > > That is a good point, and can definitely be useful, especially given the > concerns that both jobs can consume at significantly different rates, > particularly if the blue job is backpressured by the sink. > > Best, > Alex > > On Mon, 9 Dec 2024 at 14:36, Yang LI <yang.hunter...@gmail.com> wrote: > >> Hello Sergio and Jose, >> >> Apologies for jumping into the conversation. I’m very interested in what >> you’ve learned about blue/green Flink deployments. I’m currently working >> on >> a project aimed at achieving no-downtime Flink deployments, so your >> insights would be highly valuable. >> >> Thank you, >> Yang >> >
