Hi everyone,
Flink jobs are always long-running. When the job is running, users
may want to control the job but not stop it. The control reasons can be
different as following:
1.
Change data processing’ logic, such as filter condition.
2.
Send trigger events to make the progress forward.
3.
Define some tools to degrade the job, such as limit input qps, sampling
data.
4.
Change log level to debug current problem.
The common way to do this is to stop the job, do modifications and
start the job. It may take a long time to recover. In some situations,
stopping jobs is intolerable, for example, the job is related to money or
important activities.So we need some technologies to control the running
job without stopping the job.
We propose to add control mode for flink. A control mode based on the
restful interface is first introduced. It works by these steps:
1. The user can predefine some logic which supports config control, such
as filter condition.
2. Run the job.
3. If the user wants to change the job's running logic, just send a
restful request with the responding config.
Other control modes will also be considered in the future. More
introduction can refer to the doc
https://docs.google.com/document/d/1WSU3Tw-pSOcblm3vhKFYApzVkb-UQ3kxso8c8jEzIuA/edit?usp=sharing
. If the community likes the proposal, more discussion is needed and a more
detailed design will be given later. Any suggestions and ideas are welcome.
