Dear DolphinScheduler Development Community,

I hope this message finds you well.

Ron(cc'd) and I would like to initiate a discussion regarding the
integration of Flink Materialized Table refresh tasks into DolphinScheduler.


Flink introduced Materialized Tables[1] in version 1.20, and we would like
to propose adding support for refreshing these tables within
DolphinScheduler through a new task type.


*Background:*

Materialized Tables rely on scheduled refresh jobs to keep data updated. To
enable this functionality in DolphinScheduler, we propose creating a new
task type for executing Flink refresh jobs. This would provide a seamless
way to integrate Flink Materialized Table refresh operations into our
existing data pipeline workflows.



*What’s Needed:*

Introduce a new task type in DolphinScheduler to periodically trigger the
refresh of Flink Materialized Tables.



I have opened an issue to track further discussions and gather community
feedback: [https://github.com/apache/dolphinscheduler/issues/17086]

Looking forward to your thoughts and suggestions.




[1].
https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/materialized-table/overview/



Best regards,

Feng Jin

Reply via email to