Dear All, Sorry for the inaccuracy of the previous description about Issue #11652 <https://github.com/apache/dolphinscheduler/issues/11652> .
*What is the purpose of this feature？* DS can support the task instance running on the remote servers (task server), not just worker nodes. - Users can manage these task servers on the page, such as create, edit, delete and test connect. - Each task node in DAG can specify the task server that needs to be executed. - The task server property belongs only to task instances, not workflow instances. *MOP* 1. Add *TaskServer *Entity in *dolphinscheduler-dao* model, and create a table named *t_ds_task_server *in DB. Add TaskServer API. [image: image.png] 2. Add a task server manage page in Secerity menu, such as create, delete, edit and test connect. 3. Add a task server select input (field: taskServerCode) in Shell and Python task form. 4. Add *taskServerCode *column in* t_ds_task_instance、t_ds_task_definition、t_ds_task_definition_log* tables. And change these APIs. 5. Add *taskServerCode *field in *TaskInstance, TaskDefintion, TaskDefinitionLog, TaskNode*, and add *taskServerInfo *field in *TaskInstance *(not a table field). 6. Add *taskServerInfo *(just contain ip, user, password, name) entity in *dolphinscheduler-task-plugin* model. And add taskServerInfo field in *TaskExecutionContext*. 7. Shell and Python task will check TaskExecutionContext.getTaskServerInfo(), if not null, it will scp the command files and resource files to the task server, and send the start command to remote to exec it. If other task plugin need this feture, can also check this task server field. 8. The task will ssh and scp to the task server as the DS running user, not tenant. Look forward to your comments. If no any comment, I will try to complete the test and submit a PR. Thanks B. R. Yann On Sat, Aug 27, 2022 at 9:31 PM yann ann <[email protected]> wrote: > Hi, > > The current DS only supports tasks running on the Worker servers, but in > most real business scenarios, tasks need to be run on the specified server, > and the server does not need to deploy the Worker service. Please check the > Issue > #11652 <https://github.com/apache/dolphinscheduler/issues/11652> . > Thanks to @SbloodyS for suggesting me to provide the details of the > architecture design. > > Please check the architecture design. I'm not sure if DS needs to use a > fixed design template, so I'll try to be as detailed as possible. > This design mainly consists of four parts： > > *Part 1: Create a host table.* > Create a table named t_ds_host in db. Please check the following E-R > diagram. T_DS_HOST table The purpose of this table design is to save the > host information configured in the UI. > > [image: image.png] > > *Part 2: Add host manage page.* > Add a *Host Manage* module in the Security Page, such as the > *Environment Manage* page. Users can add the host records and test > connect it. > [image: image.png] > *Part 3: Add host manage API* > Add host manage API, same as environment manage model. > About the connect test function, I think that DS needs to test SSH and > SFTP at the same time. Because we need to scp our task script to this host > and run shell commands. I plan to use the JSch or SSNJ 3rd. > > *Part 4: Add host selection* > Add host selection on Shell and Python task pages. > Then shellTask or pythonTask can check this host param, if host param is > not null, the task will scp the command files to the remote server and > dispatch the exec command. > > Why just add to Shell and Python tasks? > Because I think each task needs to decide for itself if it needs to be > executed on the remote server, I think shell and python should add it. > > look forward to your comments. > > B. R. > Yann > >
