Hi, Dev Thank you all for joining this thread and giving your comments and suggestions, they have helped improve this proposal and I look forward to further feedback. If there are no further comments, I'd like to close the discussion and start the voting one day later.
Best, Ron Ron Liu <ron9....@gmail.com> 于2024年5月7日周二 20:51写道： > Hi, dev > > Following the recent PoC[1], and drawing on the excellent code design > within Flink, I have made the following optimizations to the Public > Interfaces section of FLIP: > > 1. I have renamed WorkflowOperation to RefreshWorkflow. This change better > conveys its purpose. RefreshWorkflow is used to provide the necessary > information required for creating, modifying, and deleting workflows. Using > WorkflowOperation could mislead people into thinking it is a command > operation, whereas in fact, it does not represent an operation but merely > provides the essential context information for performing operations on > workflows. The specific operations are completed within WorkflowScheduler. > Additionally, I felt that using WorkflowOperation could potentially > conflict with the Operation[2] interface in the table. > 2. I have refined the signatures of the modifyRefreshWorkflow and > deleteRefreshWorkflow interface methods in WorkflowScheduler. The parameter > T refreshHandler is now provided by ModifyRefreshWorkflow and > deleteRefreshWorkflow, which makes the overall interface design more > symmetrical and clean. > > [1] https://github.com/lsyldliu/flink/tree/FLIP-448-PoC > [2] > https://github.com/apache/flink/blob/29736b8c01924b7da03d4bcbfd9c812a8e5a08b4/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/Operation.java > > Best, > Ron > > Ron Liu <ron9....@gmail.com> 于2024年5月7日周二 14:30写道： > >> > 4. It appears that in the section on `public interfaces`, within >> `WorkflowOperation`, `CreatePeriodicWorkflowOperation` should be changed to >> >> `CreateWorkflowOperation`, right? >> >> After discussing with Xuyang offline, we need to support periodic >> workflow and one-time workflow, they need different information, for >> example, periodic workflow needs cron expression, one-time workflow needs >> refresh partition, downstream cascade materialized table, etc. Therefore, >> CreateWorkflowOperation correspondingly will have two different >> implementation classes, which will be cleaner for both the implementer and >> the caller. >> >> Best, >> Ron >> >> Ron Liu <ron9....@gmail.com> 于2024年5月6日周一 20:48写道： >> >>> Hi, Xuyang >>> >>> Thanks for joining this discussion >>> >>> > 1. In the sequence diagram, it appears that there is a missing step >>> for obtaining the refresh handler from the catalog during the suspend >>> operation. >>> >>> Good catch >>> >>> > 2. The term "cascade refresh" does not seem to be mentioned in >>> FLIP-435. The workflow it creates is marked as a "one-time workflow". This >>> is different >>> >>> from a "periodic workflow," and it appears to be a one-off execution. Is >>> this actually referring to the Refresh command in FLIP-435? >>> >>> The cascade refresh is a future work, we don't propose the corresponding >>> syntax in FLIP-435. However, intuitively, it would be an extension of the >>> Refresh command in FLIP-435. >>> >>> > 3. The workflow-scheduler.type has no default value; should it be set >>> to CRON by default? >>> >>> Firstly, CRON is not a workflow scheduler. Secondly, I believe that >>> configuring the Scheduler should be an action that users are aware of, and >>> default values should not be set. >>> >>> > 4. It appears that in the section on `public interfaces`, within >>> `WorkflowOperation`, `CreatePeriodicWorkflowOperation` should be changed to >>> >>> `CreateWorkflowOperation`, right? >>> >>> Sorry, I don't get your point. Can you give more description? >>> >>> Best, >>> Ron >>> >>> Xuyang <xyzhong...@163.com> 于2024年5月6日周一 20:26写道： >>> >>>> Hi, Ron. >>>> >>>> Thanks for driving this. After reading the entire flip, I have the >>>> following questions: >>>> >>>> >>>> >>>> >>>> 1. In the sequence diagram, it appears that there is a missing step for >>>> obtaining the refresh handler from the catalog during the suspend >>>> operation. >>>> >>>> >>>> >>>> >>>> 2. The term "cascade refresh" does not seem to be mentioned in >>>> FLIP-435. The workflow it creates is marked as a "one-time workflow". This >>>> is different >>>> >>>> from a "periodic workflow," and it appears to be a one-off execution. >>>> Is this actually referring to the Refresh command in FLIP-435? >>>> >>>> >>>> >>>> >>>> 3. The workflow-scheduler.type has no default value; should it be set >>>> to CRON by default? >>>> >>>> >>>> >>>> >>>> 4. It appears that in the section on `public interfaces`, within >>>> `WorkflowOperation`, `CreatePeriodicWorkflowOperation` should be changed to >>>> >>>> `CreateWorkflowOperation`, right? >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Best！ >>>> Xuyang >>>> >>>> >>>> >>>> >>>> >>>> At 2024-04-22 14:41:39, "Ron Liu" <ron9....@gmail.com> wrote: >>>> >Hi, Dev >>>> > >>>> >I would like to start a discussion about FLIP-448: Introduce Pluggable >>>> >Workflow Scheduler Interface for Materialized Table. >>>> > >>>> >In FLIP-435[1], we proposed Materialized Table, which has two types of >>>> data >>>> >refresh modes: Full Refresh & Continuous Refresh Mode. In Full Refresh >>>> >mode, the Materialized Table relies on a workflow scheduler to perform >>>> >periodic refresh operation to achieve the desired data freshness. >>>> > >>>> >There are numerous open-source workflow schedulers available, with >>>> popular >>>> >ones including Airflow and DolphinScheduler. To enable Materialized >>>> Table >>>> >to work with different workflow schedulers, we propose a pluggable >>>> workflow >>>> >scheduler interface for Materialized Table in this FLIP. >>>> > >>>> >For more details, see FLIP-448 [2]. Looking forward to your feedback. >>>> > >>>> >[1] https://lists.apache.org/thread/c1gnn3bvbfs8v1trlf975t327s4rsffs >>>> >[2] >>>> > >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-448%3A+Introduce+Pluggable+Workflow+Scheduler+Interface+for+Materialized+Table >>>> > >>>> >Best, >>>> >Ron >>>> >>>
