Hi all, I would like to provide a summary of the discussion regarding the integration of Flink Materialized Table refresh tasks (DSIP-90) into DolphinScheduler.
We have received positive feedback with three PMC members expressing support (+1) for this proposal. The community has provided valuable input on the implementation details, particularly regarding the UI elements and integration approach. With this community consensus, I will proceed with the code development for this feature. I'll create a PR in the coming weeks and welcome further feedback during the code review process. Thank you to everyone who participated in this discussion and provided valuable input. The proposal details: https://github.com/apache/dolphinscheduler/issues/17086 Best regards, Feng Jin On Tue, Apr 15, 2025 at 8:06 PM Feng Jin <jinfeng1...@gmail.com> wrote: > Hi, all > > I'd like to follow up on our proposal regarding the integration of Flink > Materialized Table refresh tasks (DSIP-90). We've had some initial > discussions on both the mailing list and the GitHub issue, but we're hoping > to gather more community input before moving forward. > > For those who may have missed the earlier discussions, we propose adding > support for refreshing Flink Materialized Tables (introduced in Flink 1.20) > within DolphinScheduler through a new task type. This would enable better > integration between DolphinScheduler and Flink Materialized Tables. > > Some technical points that have been discussed on GitHub: > > 1. For job status monitoring, we plan to use a single task instance that > monitors the job until completion by leveraging the operation handle > (containing jobId and clusterInfo) obtained after submitting a refresh > request. > > 2. Our current implementation focuses specifically on scheduled batch jobs > rather than streaming jobs. > > The proposal details are available at: > https://github.com/apache/dolphinscheduler/issues/17086 > > We welcome any additional feedback, questions, or suggestions from the > community. Your input is valuable in ensuring this integration meets the > needs of DolphinScheduler users. > > Best regards, > Feng Jin > > > On Wed, Apr 9, 2025 at 8:42 PM Feng Jin <jinfeng1...@gmail.com> wrote: > >> Hi, all >> >> I'd like to kindly follow up on our proposal regarding the integration of >> Flink Materialized Table refresh tasks into DolphinScheduler (DSIP-90). >> >> We've received some positive initial feedback (thank you Chufeng for the >> +1), but we're hoping to gather more community input before proceeding >> further. The proposal aims to enhance DolphinScheduler's integration with >> Flink 1.20's Materialized Tables feature, which would benefit users working >> with both systems. >> >> For those who haven't had a chance to review, the proposal is available >> at: >> https://github.com/apache/dolphinscheduler/issues/17086 >> >> We welcome any thoughts, concerns, or suggestions on this integration. >> Your input will be valuable in shaping this feature to best serve the >> community's needs. >> >> Best regards, >> Feng Jin >> >> >> On Tue, Apr 8, 2025 at 12:10 AM Chufeng Gao <chufeng...@gmail.com> wrote: >> >>> Hi Feng, >>> >>> Thanks for bringing this up! Glad to see better integration with >>> Apache DolphinScheduler and Apache Flink. >>> >>> Since both projects are open-source, it is possible for us to have >>> automated integration tests to cover this new plugin so that it will >>> not increase the burden of maintenance for the community. >>> >>> I'm +1 to this. >>> >>> Best Regards, >>> >>> Chufeng (Eric) Gao >>> >>> >>> >>> On Wed, Mar 26, 2025 at 5:16 PM Feng Jin <jinfeng1...@gmail.com> wrote: >>> > >>> > Dear DolphinScheduler Development Community, >>> > >>> > I hope this message finds you well. >>> > >>> > Ron(cc'd) and I would like to initiate a discussion regarding the >>> > integration of Flink Materialized Table refresh tasks into >>> DolphinScheduler. >>> > >>> > >>> > Flink introduced Materialized Tables[1] in version 1.20, and we would >>> like >>> > to propose adding support for refreshing these tables within >>> > DolphinScheduler through a new task type. >>> > >>> > >>> > *Background:* >>> > >>> > Materialized Tables rely on scheduled refresh jobs to keep data >>> updated. To >>> > enable this functionality in DolphinScheduler, we propose creating a >>> new >>> > task type for executing Flink refresh jobs. This would provide a >>> seamless >>> > way to integrate Flink Materialized Table refresh operations into our >>> > existing data pipeline workflows. >>> > >>> > >>> > >>> > *What’s Needed:* >>> > >>> > Introduce a new task type in DolphinScheduler to periodically trigger >>> the >>> > refresh of Flink Materialized Tables. >>> > >>> > >>> > >>> > I have opened an issue to track further discussions and gather >>> community >>> > feedback: [https://github.com/apache/dolphinscheduler/issues/17086] >>> > >>> > Looking forward to your thoughts and suggestions. >>> > >>> > >>> > >>> > >>> > [1]. >>> > >>> https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/materialized-table/overview/ >>> > >>> > >>> > >>> > Best regards, >>> > >>> > Feng Jin >>> >>
