aloyszhang opened a new pull request, #9865: URL: https://github.com/apache/inlong/pull/9865
- Fixes #9862 ### Background For real-time data sync, the whole procedure is as follows:  Currently, InLong does not support submitting Flink batch jobs for offline data sync. The main differences between offline sync and real-time sync are as follows: - Offline synchronization requires corresponding scheduling information to be configured - Offline task submission is triggered by the scheduling system instead of manager The interaction process with the scheduling system is as follows: - Register the scheduling information at the point of approval - The scheduling system generates an instance to submit Flink batch tasks by InLong callback. Design principles for the association between synchronization tasks and the scheduling system are as follows: - Management of task configuration and scheduling configuration are independent of each other. - The granularity of scheduling information configuration is at the stream level: there are multiple streams under a group, fine-grained scheduling control should be at the stream level, and the scheduling information registered with the scheduling system should include groupID and streamID. - Whether to register scheduling information with the scheduling system depends on the approval status of the task: - If the task is in an approved state: save the scheduling information and register it with the scheduling system. - If it is in a non-approved state: only save the scheduling information and wait to register the scheduling information when the task is approved. Furthermore, it is important to note the distinction between configuring streams and scheduling. In the configuration process, Flink tasks are not submitted, while in the scheduling process, Flink tasks are submitted. This is identified by the status of the stream. PS: Scheduling management is not covered by this pull request. ### Motivation Support submit flink job for offline sync ### Modifications Ensure the Flink batch task will be submitted only if the stream is in the CONFIG_SUCCESSFUL state. ### Verifying this change *(Please pick either of the following options)* - [ ] This change is a trivial rework/code cleanup without any test coverage. ### Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
