I am not sure if we should solve this problem in Flink. This is more like a dynamic config problem that probably should be solved by some configuration framework. Here is one post from google search: https://medium.com/twodigits/dynamic-app-configuration-inject-configuration-at-run-time-using-spring-boot-and-docker-ffb42631852a
On Fri, Jun 4, 2021 at 7:09 AM 刘建刚 <liujiangangp...@gmail.com> wrote: > Hi everyone, > > Flink jobs are always long-running. When the job is running, users > may want to control the job but not stop it. The control reasons can be > different as following: > > 1. > > Change data processing’ logic, such as filter condition. > 2. > > Send trigger events to make the progress forward. > 3. > > Define some tools to degrade the job, such as limit input qps, > sampling data. > 4. > > Change log level to debug current problem. > > The common way to do this is to stop the job, do modifications and > start the job. It may take a long time to recover. In some situations, > stopping jobs is intolerable, for example, the job is related to money or > important activities.So we need some technologies to control the running > job without stopping the job. > > > We propose to add control mode for flink. A control mode based on the > restful interface is first introduced. It works by these steps: > > > 1. The user can predefine some logic which supports config control, > such as filter condition. > 2. Run the job. > 3. If the user wants to change the job's running logic, just send a > restful request with the responding config. > > Other control modes will also be considered in the future. More > introduction can refer to the doc > https://docs.google.com/document/d/1WSU3Tw-pSOcblm3vhKFYApzVkb-UQ3kxso8c8jEzIuA/edit?usp=sharing > . If the community likes the proposal, more discussion is needed and a more > detailed design will be given later. Any suggestions and ideas are welcome. > >
