Imran Rashid created SPARK-26697:
------------------------------------
Summary: ShuffleBlockFetcherIterator can log block sizes in
addition to num blocks
Key: SPARK-26697
URL: https://issues.apache.org/jira/browse/SPARK-26697
Project: Spark
Issue Type: Improvement
Components: Shuffle, Spark Core
Affects Versions: 2.4.0
Reporter: Imran Rashid
Every so often I find myself looking at executor logs, wondering why something
is going wrong (failed exec, or seems to be stuck etc) and I wish I had a bit
more info about shuffle sizes. {{ShuffleBlockFetcherIterator}} logs the number
of local & remote blocks, but not their sizes. It would be really easy to add
in size info too.
eg. instead of
{noformat}
19/01/22 23:10:49.978 Executor task launch worker for task 8 INFO
ShuffleBlockFetcherIterator: Getting 2 non-empty blocks including 1 local
blocks and 1 remote blocks
{noformat}
it should be
{noformat}
19/01/22 23:10:49.978 Executor task launch worker for task 8 INFO
ShuffleBlockFetcherIterator: Getting 2 (194.0 B) non-empty blocks including 1
(97.0 B) local blocks and 1 (97.0 B) remote blocks
{noformat}
I know this is a really minor change, but I've wanted it multiple times, seems
worth it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
