Jacek Lewandowski created SPARK-7172:
----------------------------------------
Summary: Worker includes ExecutorRunner and DriverRunner in Akka
messages
Key: SPARK-7172
URL: https://issues.apache.org/jira/browse/SPARK-7172
Project: Spark
Issue Type: Bug
Components: Spark Core, Web UI
Affects Versions: 1.3.1, 1.2.2, 1.4.0
Reporter: Jacek Lewandowski
Worker recognises {{RequestWorkerState}} messages which are sent by the UI to
get the detailed information about the current state of the worker. The worker
responds with {{WorkerStateResponse}} which includes the requested information.
h5.Problem
{{WorkerStateResponse}} includes two lists of {{ExecutorRunner}} objects and
two lists of {{DriverRunner}} objects. These objects are mutable and actually
performs some work. They shouldn't be used to move the information through the
Akka actor system. The first of all, this is discouraged practise. The second
is that when we want to get the information about the worker from some other
actor system, Akka needs to serialise it it shows that these objects are not
serialisable. This would be particularly useful for examining the state of
Spark cluster in some test suites.
h5.Proposed solution
Implemented dedicated, immutable case classes to reflect the snapshot of the
worker state and use them instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
