[GitHub] [dolphinscheduler] Radeity opened a new pull request, #13343: [Improvement][Task] Support end-to-end transfer file between tasks

[GitBox]

Radeity opened a new pull request, #13343:
URL: https://github.com/apache/dolphinscheduler/pull/13343

   <!--Thanks very much for contributing to Apache DolphinScheduler. Please 
review 
https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html
 before opening a pull request.-->
   
   ## Purpose of the pull request
   - This PR closes https://github.com/apache/dolphinscheduler/issues/12849.
   <!--(For example: This pull request adds checkstyle plugin).-->
   
   ## Brief change log
   - Add configuration `tmp.transfer.file.size`, default size 100MB.
   - Add `TmpDirClearProcessor` to clean temporary directory after process is 
finished.
   
   - If usage of temporary storage doesn't reach the limit, transfer files will 
be stored in local temporary path: 
`DATA_BASEDIR/tmp/{tenandCode}/{projectCode}/{processDefineCode}/{processDefineVersion}/{processInstanceId}`
   - If downloaded file is scp command template file(with `.template` suffix), 
complete and execute scp command to fetch files from upstream worker.
   <!--*(for example:)*
   - *Add maven-checkstyle-plugin to root pom.xml*
   -->
   
   ## Verify this pull request
   
   - Add UT for `FileUtils` and exist UT covers changes in 
`TaskFileTransferUtils`.
   
   - Manually test scp way can work in standalone mode. **However, it has to be 
verified that deploy DS in cluster mode and k8s mode, can workers communicate 
with each other without password? Otherwise, scp command will not execute 
successfully.**
   
   Log in upstream task:
   
   <img width="827" alt="image" 
src="https://user-images.githubusercontent.com/45198818/210716315-3f0f7c91-8a63-4780-9321-46d59ed0b72d.png";>
   
   Log in downstream task:
   
   
   <img width="821" alt="image" 
src="https://user-images.githubusercontent.com/45198818/210716512-713aeeb5-e960-420c-a46e-d58ef70efabc.png";>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org