idzikovsky commented on a change in pull request #249:
URL: https://github.com/apache/incubator-livy/pull/249#discussion_r493348458
##########
File path: rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java
##########
@@ -169,6 +169,13 @@ private void initializeServer() throws Exception {
// on the cluster, it would be tricky to solve that problem in a generic
way.
livyConf.set(RPC_SERVER_ADDRESS, null);
+ // If we are running on Kubernetes, set RPC_SERVER_ADDRESS from
"spark.driver.host" option,
+ // which is set in class
org.apache.spark.deploy.k8s.features.DriverServiceFeatureStep:
+ // line 61: val driverHostname =
s"$resolvedServiceName.${kubernetesConf.namespace()}.svc"
+ if (livyConf.isRunningOnKubernetes()) {
+ livyConf.set(RPC_SERVER_ADDRESS, conf.get("spark.driver.host"));
+ }
+
Review comment:
By some reason this version does not work for me like it was with the
same piece from https://github.com/apache/incubator-livy/pull/167.
I got the same exception as here:
https://github.com/apache/incubator-livy/pull/167#issuecomment-485878996
I'm not sure why, but it seems like it's because the `livy.spark.master`
property is not set inside of Spark Driver of Livy Session on my env.
While the same piece from https://github.com/apache/incubator-livy/pull/167
works fine:
```
if (conf.get("spark.master").startsWith("k8s")) {
livyConf.set(RPC_SERVER_ADDRESS, conf.get("spark.driver.host"));
}
```
But I'm not using code from your branch, but rather backporting your patch
to our Livy build which is based on older Livy version, so maybe it's the cause.
On the other hand, during quick lookup I've not found any code that bypass
`livy.spark.master` property into driver, so I see no reason why it should work
here.
If I get a chance to test this issue on build from this PR, I'll share what
get.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
