EricGao888 commented on code in PR #10337:
URL: https://github.com/apache/dolphinscheduler/pull/10337#discussion_r893242034
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-jupyter/src/main/java/org/apache/dolphinscheduler/plugin/task/jupyter/JupyterConstants.java:
##########
@@ -33,6 +33,18 @@ private JupyterConstants() {
*/
public static final String CONDA_ACTIVATE = "conda activate";
+ /**
+ * create and activate conda env from tar
+ */
+ public static final String CREATE_ENV_FROM_TAR = "mkdir jupyter_env && " +
+ "tar -xzf %s -C jupyter_env && " +
+ "source jupyter_env/bin/activate";
Review Comment:
Currently, `env tar` will be downloaded from `resource center` before task
execution and removed when task execution completed.
https://github.com/apache/dolphinscheduler/blob/f90f0f8490d8f4cd87d2368cd0264a4837918c7d/dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java#L141-L145
https://github.com/apache/dolphinscheduler/blob/f90f0f8490d8f4cd87d2368cd0264a4837918c7d/dolphinscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/runner/TaskExecuteThread.java#L199-L203
However, I think it is a good idea to give a choice to define the path and
install the env on workers without getting removed after task execution. In
this way, the same env will not need to be downloaded every time.
WDYT @SbloodyS @zhongjiajie
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
