Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5060#issuecomment-96062921
I thought more about this particular issue. These [two
lines](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L361)
in the SparkContext actually sets the two parameters explicitly with host =
local hostname and port = 0, ONLY IF they are not set from the user-provided
conf. For each of them here is my observation.
1. spark.driver.host = This should never be recovered through checkpoints.
Because if the driver is restarted on a different machine, and the old value of
spark.driver.host (that is, the old hostname) is recovered from checkpoint ,
then it will be set incorrectly as the recreated SparkContext will not set
spark.driver.host to the new hostname. So we should never recover the host
name, and always let the new SparkContext set a new hostname.
2. spark.driver.port = The sparkcontext sets the value to 0 if not
specified
([see](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L362)).
So if we always save the value of sparkContext.conf (that is, after 0 is set
in SparkContext initialization) in checkpoint, then it suffices the purpose -
if set explicitly by user, then that port is used. If not set explicitly by
user, then 0 stays, in which case it will be random even after restart.
I think this simplifies the whole idea - never save host, always save port.
Isnt it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
