zjffdu edited a comment on pull request #4097: URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-842054131
@Reamer Actually `YarnRemoteInterpreterProcess` doesn't do the downloading, it just upload the conda to hdfs as yarn app resource, and yarn will download it from hdfs before starting yarn container. There're 2 benefits of using yarn app resource: * Don't need to update conda archives to hdfs, just use the local file system to store the conda env. This would make the development much smooth, user use the local conda env to verify it in local environment and then move it to yarn environment in production environment. * We can leverage yarn's resource cache mechanism. That means the same conda env downloaded by yarn_app_1 can be reused by yarn_app_2. If we download it in `JupyterKernelInterpreter.java`, it may cause network congestion if many python interpreters runs at the same time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
