[
https://issues.apache.org/jira/browse/SPARK-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-26697:
------------------------------------
Assignee: (was: Apache Spark)
> ShuffleBlockFetcherIterator can log block sizes in addition to num blocks
> -------------------------------------------------------------------------
>
> Key: SPARK-26697
> URL: https://issues.apache.org/jira/browse/SPARK-26697
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Spark Core
> Affects Versions: 2.4.0
> Reporter: Imran Rashid
> Priority: Major
>
> Every so often I find myself looking at executor logs, wondering why
> something is going wrong (failed exec, or seems to be stuck etc) and I wish I
> had a bit more info about shuffle sizes. {{ShuffleBlockFetcherIterator}}
> logs the number of local & remote blocks, but not their sizes. It would be
> really easy to add in size info too.
> eg. instead of
> {noformat}
> 19/01/22 23:10:49.978 Executor task launch worker for task 8 INFO
> ShuffleBlockFetcherIterator: Getting 2 non-empty blocks including 1 local
> blocks and 1 remote blocks
> {noformat}
> it should be
> {noformat}
> 19/01/22 23:10:49.978 Executor task launch worker for task 8 INFO
> ShuffleBlockFetcherIterator: Getting 2 (194.0 B) non-empty blocks including 1
> (97.0 B) local blocks and 1 (97.0 B) remote blocks
> {noformat}
> I know this is a really minor change, but I've wanted it multiple times,
> seems worth it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
