GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/10857
[SPARK-12896] [WIP] Send only accumulator updates to driver, not TaskMetrics
This patch builds on top of #10835, so the actual diff should be much
smaller. This would complete SPARK-10620, which replaces much of the logic in
TaskMetrics with accumulators. This was once part of #10717. It was split into
its own patch so it's more reviewable.
**The high level idea is that instead of having the executors send both
accumulator updates and TaskMetrics, we should have them send only accumulator
updates.** This is because TaskMetrics can be represented as a collection of
accumulators (as we have done so in #10835) and we can just reconstruct them on
the driver based on the accumulator updates.
While an effort has been made to preserve as much of the public API as
possible, there were a few known breaking @DeveloperApi changes that would be
very awkward to maintain. I will gather the full list shortly and post it here.
Lastly, this is still WIP because more tests are pending.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark dont-send-task-metrics
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10857.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10857
----
commit 42ca72de7031281119877ada77d8e734a7f45028
Author: Andrew Or <[email protected]>
Date: 2016-01-18T22:55:44Z
Change types of some signatures
commit 2c62000769e21b0f206b637ffefeb2671780fa9b
Author: Andrew Or <[email protected]>
Date: 2016-01-18T23:02:14Z
Boiler plate for all the new internal accums
commit 11677227827735ee4999d4c8d536f17391c4ffac
Author: Andrew Or <[email protected]>
Date: 2016-01-19T00:34:16Z
Squashed commit of the following:
commit 269031f162cbce031efd4cdce55908f46569a8c8
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 16:33:12 2016 -0800
Remove unused method
commit c04b5df944e32d6854ab5ed4a282b77df889d481
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 16:13:08 2016 -0800
Review comments
commit d2e4e23be82a0afb2f39d629ee7413591bc08c8d
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 14:42:19 2016 -0800
One more
commit 202d48e5ceab044e941f8f5a2d866982a0072637
Merge: e99b9af 4f11e3f
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 14:27:47 2016 -0800
Merge branch 'master' of github.com:apache/spark into
get-or-create-metrics
commit e99b9af23b8135b312bc4a968ba9d2ba7d71e127
Merge: 34c7ce5 b8cb548
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 13:56:41 2016 -0800
Merge branch 'master' of github.com:apache/spark into
get-or-create-metrics
Conflicts:
core/src/main/scala/org/apache/spark/CacheManager.scala
core/src/main/scala/org/apache/spark/memory/StorageMemoryPool.scala
core/src/test/scala/org/apache/spark/CacheManagerSuite.scala
commit 34c7ce5bf724c781a37c352277f7c5cd86d33c9a
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 12:46:42 2016 -0800
Hide updatedBlocks
commit ad094f071472b9cf7b9f9bdb7cd00d88c402995d
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 12:30:59 2016 -0800
Clean up JsonProtocol
This commit collapsed 10 methods into 2. The 8 that were inlined
were only used in 1 place each, and the body of each was quite
small. The additional level of abstraction did not add much value
and made the code verbose.
commit 078598409225224f0532a45f34dae533695b25df
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 12:20:28 2016 -0800
Replace set with register
JsonProtocol remains the only place where we still call set
on each of the *Metrics classes.
commit b9d7fbf37cc410d44e462d9d08650a20decc8fc9
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 12:10:17 2016 -0800
Clean up places where we set OutputMetrics
Note: there's one remaining place, which is JsonProtocol.
commit 62c96e1cdc472356dfbfb24cf9650a8f36017224
Author: Andrew Or <[email protected]>
Date: Mon Jan 18 11:50:04 2016 -0800
Add register* methods (get or create)
commit 144df46ba989e507470c9a5026874b15b34792ad
Author: Andrew Or <[email protected]>
Date: 2016-01-19T01:06:13Z
Implement TaskMetrics using Accumulators
commit 5ec17c1020f8987d801791d7e091d53b675a461b
Author: Andrew Or <[email protected]>
Date: 2016-01-19T01:26:08Z
Fix accums not set on the driver
Previously we would always zero out an accumulator when we
deserialize it. Certainly we don't want to do that on the driver.
The changes in this commit are temporary and will be reverted
in SPARK-12896.
commit 362cde55486c51cc1644299626f71c60f4aab820
Author: Andrew Or <[email protected]>
Date: 2016-01-19T01:35:10Z
Fix test compile
Tests don't pass yet, obviously...
commit e43e8be0fbbb71e27485a7b4167389d331a9e95a
Author: Andrew Or <[email protected]>
Date: 2016-01-19T02:11:14Z
Make accum updates read from all registered accums
+ fix tests, which are still failing
commit 2330a377a3268d17b7dad60965baf6f4e3d97ec6
Author: Andrew Or <[email protected]>
Date: 2016-01-19T18:55:24Z
Fix metrics being double counted on driver
... by setting it to zero on the executors, always.
commit cca87cc63a9bef485aa1736df4ff73ced336ddf6
Author: Andrew Or <[email protected]>
Date: 2016-01-19T19:09:53Z
Merge branch 'master' of github.com:apache/spark into
task-metrics-use-accums
Conflicts:
core/src/main/scala/org/apache/spark/CacheManager.scala
core/src/main/scala/org/apache/spark/executor/Executor.scala
core/src/main/scala/org/apache/spark/executor/ShuffleWriteMetrics.scala
core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala
core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
core/src/test/scala/org/apache/spark/ui/jobs/JobProgressListenerSuite.scala
core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala
commit 4ead1ba25bf73bf5dda6a96e2aca32ce49787f9b
Author: Andrew Or <[email protected]>
Date: 2016-01-19T19:45:59Z
Miscellaneous updates; make diff smaller
commit 0f40753ae932f31f3576c1046368cbfeea942975
Author: Andrew Or <[email protected]>
Date: 2016-01-19T19:55:17Z
Fix JsonProtocolSuite
Also add some useful error printing if things don't match.
commit 76b605cc7d4fb5f7e0bcd3337cdd688e82b6d766
Author: Andrew Or <[email protected]>
Date: 2016-01-19T19:58:42Z
Fix SQLQuerySuite
commit 7b5d840cda21c74b1f50f36d62aa19f096958b46
Author: Andrew Or <[email protected]>
Date: 2016-01-19T20:03:07Z
Fix style
commit 2069a784dbb419d20f4a58ca93d897e0e657ef60
Author: Andrew Or <[email protected]>
Date: 2016-01-19T21:57:17Z
Fix MiMa
commit d9813b1aa585ac88394e01fc43753165e965aa85
Author: Andrew Or <[email protected]>
Date: 2016-01-19T23:27:48Z
Add test on accum values being zero'ed out + cleanups
commit 40fd853168af8db18b4b33b7d62e349d930a34a6
Author: Andrew Or <[email protected]>
Date: 2016-01-19T23:41:04Z
Add tests for TaskMetrics, which uncovered a bug
We never updated the original ShuffleReadMetrics accumulators;
we were updating a copy.
commit cdb32797d301a8017ef851f50fe3153cf545b22f
Author: Andrew Or <[email protected]>
Date: 2016-01-19T23:54:45Z
Minor test changes
commit 8e46ee3c6745edc797182e540825fd1fcfd7e9c1
Author: Andrew Or <[email protected]>
Date: 2016-01-19T23:55:01Z
Merge branch 'master' of github.com:apache/spark into
task-metrics-use-accums
commit 628076d01900ec252a2d8200132cdd669d5ce6be
Author: Andrew Or <[email protected]>
Date: 2016-01-19T23:55:48Z
Fix style
commit 2a3cd27089f4c12e5856d8465722942ee29cd1e6
Author: Andrew Or <[email protected]>
Date: 2016-01-19T23:57:28Z
Fix MiMa
commit 67a1bee3bfb4f4b954105aaa9e62843ca885ad6d
Author: Andrew Or <[email protected]>
Date: 2016-01-20T01:57:33Z
Fix InputOutputMetricsSuite
commit ec6ea448babad287597c34d0f05b8e9720556e31
Author: Andrew Or <[email protected]>
Date: 2016-01-20T02:02:10Z
Fix TaskContextSuite
commit 6355dbd36e266ee296e91b34ba8c0d105b3e26e9
Author: Andrew Or <[email protected]>
Date: 2016-01-20T02:06:49Z
Fix ReplayListenerSuite
commit ed815843a607d8c626ada8c6f0f29d0f8916591e
Author: Andrew Or <[email protected]>
Date: 2016-01-20T02:31:14Z
Fix SparkListenerSuite
commit 641f7365df3b0913e7e709671cc6f640c1effb56
Author: Andrew Or <[email protected]>
Date: 2016-01-20T02:31:31Z
Merge branch 'master' of github.com:apache/spark into
task-metrics-use-accums
commit 4ca7328fd5b0d3b9f6185885f36d95358e9101a3
Author: Andrew Or <[email protected]>
Date: 2016-01-20T02:37:21Z
Minor comment correction
commit 17db1c95629aaa19040c7d2b1bf1c6412036b907
Author: Andrew Or <[email protected]>
Date: 2016-01-20T02:52:03Z
Add test to verify internal accums are cleaned up
commit 308db4cb4712893da43fe8589ee3bbfe31b7958b
Author: Andrew Or <[email protected]>
Date: 2016-01-20T03:17:34Z
Add deprecated matching methods for Input/OutputMetrics
commit a591119ef963fedbbaf9c8ee99686bc43cd7774e
Author: Andrew Or <[email protected]>
Date: 2016-01-20T20:11:16Z
Merge branch 'task-metrics-use-accums' of github.com:andrewor14/spark into
dont-send-task-metrics
commit 393ec19c297a1b8a7ebb47d5f85f481195507b58
Author: Andrew Or <[email protected]>
Date: 2016-01-20T22:37:48Z
Add "countFailedValues" flag in Accumulable[Info]
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
