Hi Feng, Thanks for bringing this up! Glad to see better integration with Apache DolphinScheduler and Apache Flink.
Since both projects are open-source, it is possible for us to have automated integration tests to cover this new plugin so that it will not increase the burden of maintenance for the community. I'm +1 to this. Best Regards, Chufeng (Eric) Gao On Wed, Mar 26, 2025 at 5:16 PM Feng Jin <jinfeng1...@gmail.com> wrote: > > Dear DolphinScheduler Development Community, > > I hope this message finds you well. > > Ron(cc'd) and I would like to initiate a discussion regarding the > integration of Flink Materialized Table refresh tasks into DolphinScheduler. > > > Flink introduced Materialized Tables[1] in version 1.20, and we would like > to propose adding support for refreshing these tables within > DolphinScheduler through a new task type. > > > *Background:* > > Materialized Tables rely on scheduled refresh jobs to keep data updated. To > enable this functionality in DolphinScheduler, we propose creating a new > task type for executing Flink refresh jobs. This would provide a seamless > way to integrate Flink Materialized Table refresh operations into our > existing data pipeline workflows. > > > > *What’s Needed:* > > Introduce a new task type in DolphinScheduler to periodically trigger the > refresh of Flink Materialized Tables. > > > > I have opened an issue to track further discussions and gather community > feedback: [https://github.com/apache/dolphinscheduler/issues/17086] > > Looking forward to your thoughts and suggestions. > > > > > [1]. > https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/materialized-table/overview/ > > > > Best regards, > > Feng Jin
