zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-864567409
@Reamer Do you have any more comment ? Otherwise I will merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-854380073
CI is fixed now, ready for review @Reamer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-854380073
CI is fixed now, ready for review @Reamer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-852767277
CI is failed, I am looking at that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-843283503
> I was aware of this, but it seems that downloading dependencies several
times is the way of `spark.archives`. It is clear that this is not optimal.
If I read spark
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-842054131
@Reamer Actually `YarnRemoteInterpreterProcess` doesn't do the downloading,
it just upload the conda to hdfs as yarn app resource, and yarn will download
it from hdfs before
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-842054131
@Reamer Actually `YarnRemoteInterpreterProcess` doesn't do the downloading,
it just upload the conda to hdfs as yarn app resource, and yarn will download
it from hdfs before
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-836747887
Thanks @Reamer for the investigation, so how about using `python.archives`
for specifying conda env archives in both yarn and k8s, in this PR, I will make
it work in yarn and
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-830638631
For spark interpreter, we can leverage `spark.archives` to download and
setup conda environment in both driver(spark interpreter) and executor. But for
python interpreter, I
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-826297779
@Reamer `spark.archives` works, but it is only available after spark 3.1,
and I think it is better to put conda env in cloud storage and then specify it
via `spark.archives`
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-826297779
@Reamer `spark.archives` works, but it is only available after spark 3.1,
and I think it is better to put conda env in cloud storage and then specify it
via `spark.archives`
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-825500280
Thanks @Reamer `spark.archives` seems to be able to work both in yarn and
k8s, let me try this.
--
This is an automated message from the Apache Git Service.
To respond to
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-825324712
> > I mean allowing user to choose different image for their notes.
Customizing image won't be a frequent operation IIUC
>
> Maintaining different Zeppelin interpreter
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-824603467
> Updates of the image with a changed Python version and libraries are not
really possible because you don't know which other notebooks might be broken
afterwards.
I
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-824601359
@Reamer Actually the approach here also apply for pyspark interpreter, I
have verified that
--
This is an automated message from the Apache Git Service.
To respond to the
zjffdu commented on pull request #4097:
URL: https://github.com/apache/zeppelin/pull/4097#issuecomment-824535028
@Reamer Regarding your experience, would building conda env into docker
image works for users ?
User can use a base docker image with lots of common used python packages
16 matches
Mail list logo
