dijiekstra commented on issue #18627: URL: https://github.com/apache/dolphinscheduler/issues/18627#issuecomment-5615548984
> This function belongs to the category of data synchronization, and DS is a scheduling tool. On the contrary, the scheduling system should do these things even more. In the past, we usually detected whether tasks were scheduled based on time perspectives. However, in the context of modern data warehouses, data freshness has further evolved from days and hours to other time units, such as 1 minute, 5 minutes, and so on. If we rely solely on time-based scheduling, I might miss some submitted data within the current window, and accumulate multiple data in the next window. In fact, we should consider event-based scheduling, such as snapshot submissions from Paimon. If you position it as data synchronization, how do we organize and orchestrate the data development of a subject domain? We still need to rely on the workflow model concept of DS to connect the entire data warehouse subject domain development. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
