Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21688#discussion_r224864545
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -341,7 +341,9 @@ private class LiveExecutorStageSummary(
metrics.shuffleWriteMetrics.recordsWritten,
metrics.memoryBytesSpilled,
metrics.diskBytesSpilled,
- isBlacklisted)
+ isBlacklisted,
--- End diff --
We can look into it more, I can't image its that hard to join, its just
that much more data that is going over the wire, you are sending the entire
ExecutorSummary for all executors when you really just need 2 fields out of it
of some executors. Previously this was happening on the driver side so it
didn't have to transfer the data. We can look at the data size and if it
doesn't seem to bad we can do that for now and can always revisit if needed.
The stage page executor table which uses this I wouldn't expect to be as used
as the task one so probably not bad for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
