yucai created SPARK-22540: ----------------------------- Summary: HighlyCompressedMapStatus's avgSize is incorrect Key: SPARK-22540 URL: https://issues.apache.org/jira/browse/SPARK-22540 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.3.0 Reporter: yucai
The calculation of HighlyCompressedMapStatus's avgSize is incorrect. Currently, it looks like "sum of small blocks / count of all non empty blocks", all non empty blocks contains huge blocks number also, actually we need the count of small blocks only. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org