Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/5060#issuecomment-121402196
@jerryshao let finalize this PR. I think what we should do is.
Currently, we remove spark.driver.* from conf used to create the recovered
streaming context, ignoring the fact that the user may be explicitly setting
those conf in spark-defaults.conf. That is wrong, the general policy should be
never recover from spark.driver.* from checkpointed conf. Then if the those
properties are set in the defaults, they would be present in the final conf for
restarting context, other they wont be.
This solves the original problem in the JIRA. If someone wants to set the
port explicitly, then they can set if in the spark-defaults.conf. With the
above change, it will not be explicitly deleted when recovering and will be
automatically used in the recovered context.
Sounds good? If so, please update the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
