[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-563925261 @yantzu Hi, the target of this PR is remove the livy.rsc.sparkr.packge in livy.conf. Because I think it's a conf a client rather than server. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-563913256 @yantzu The reason of: set livy.rsc.sparkr.package in sever side as default value doesn't work, is livy will not read livy-client.conf. So you can config it livy.rsc.sparkr.packge=xx in livy.conf This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-563912375 @yantzu If you want to config it in livy rather than client, you can put livy.rsc.sparkr.packge=xx in livy.conf. And livy will not read livy-client.conf at all, so your config in livy-client.conf cannot valid. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-563910338 @yantzu I think livy-client.conf should place in the src/main/resources of your client rather than in the conf of livy. You can see the example in LivySparkR.zip I uploaded before. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-563907946 @yantzu Hi, The first log "ERROR InteractiveSession$: None" shows the config in your livy-client.conf was not passed to livy at all. Where your livy-client.conf placed ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-563902348 @yantzu Hi, could you ` error(builderProperties.get(RSCConf.Entry.SPARKR_PACKAGE.key()))` before `val sparkRArchive = if (!LivyConf.TEST_MODE) findSparkRArchive() else None` ? Then let's see what's the output. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-563650967 @yantzu I have test both TEST_MODE=true and TEST_MODE=false, they both will not upload sparkr. I think you can debug the code by `print(builderProperties.get(RSCConf.Entry.SPARKR_PACKAGE.key()))` before https://github.com/apache/incubator-livy/blob/master/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L317, and see whether it print the path which config in livy-client.conf. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-563026014 @yantzu Hi, I have test it again, and sparkr will not upload when set livy.rsc.sparkr.package in livy-client.conf. The first image shows sparkr was uploaded when not set livy.rsc.sparkr.package, and the second image shows sparkr was not uploaded when set it. And the attach file is the source code, you can change livy.rsc.sparkr.package to your hdfs path, and test it again.   [LivySparkR.zip](https://github.com/apache/incubator-livy/files/3937502/LivySparkR.zip) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-531539288 @shanyu I think we cannot achieve "spark submit won't upload the zip file for every session and every batch", because spark upload pyspark.zip in the following code and we cannot stop it in livy: https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L862. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-531536697 @RogPodge pyspark.zip is uploaded by spark, not livy. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-livy] runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package
runzhiwang commented on issue #196: [LIVY-606][LIVY-152] fix the invalid conf of pyspark.archives and sparkr.package URL: https://github.com/apache/incubator-livy/pull/196#issuecomment-531528011 @RogPodge 1. This config should be defined by user, so in livy-client.conf. 2. Actually, this config in livy.conf has not been read in livy, so remove or not does not matter. 3. Sorry, what do you mean "I can't seem to replicate the stated behavior on my machine after trying to incorporate this PR"? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
