GitHub user hustfxj opened a pull request:
https://github.com/apache/spark/pull/16525
the new leader will lost the statistics of the driver's resource on the
worker When the leader master has changed.
I deploy the standalone cluster with two masters. and utilize zooKeeper to
provide leader election. Firstly, I submit the application with cluster mode.
Then I kill the leader master, and the standby master will be the leader. But
the new leader will lost the statistics of the driver's resource. Then I stop
the application, we will see the negative used resource at the worker from
masterPage. Like that:
```
Workers
Worker Id Address State Cores Memory
worker-20161220162751-10.125.6.222-59295 10.125.6.222:59295 ALIVE
4 (-1 Used) 6.8 GB (-1073741824.0 B Used)
worker-20161220164233-10.218.135.80-10944 10.218.135.80:10944 ALIVE
4 (0 Used) 6.8 GB (0.0 B Used)
```
Because the new leader forget calculate the driverâ resource when the
master receive the "WorkerLatestState" message. At the same time we can set
RUNNING state for the app after the master receive the messageï¼ otherwise the
app' state will still be WAITTING.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hustfxj/spark 2017
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16525.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16525
----
commit 764731714bfc2f07dcc5090b977af0e22301ea47
Author: xiaojian.fxj <[email protected]>
Date: 2017-01-10T01:42:35Z
the new leader will lost the statistics of the driver's resource on the
worker When the leader master has changed.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
