mridulm commented on a change in pull request #35559:
URL: https://github.com/apache/spark/pull/35559#discussion_r813606907
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ShuffleIndexInformation.java
##########
@@ -46,8 +45,11 @@ public ShuffleIndexInformation(File indexFile) throws
IOException {
* Size of the index file
* @return size
*/
- public int getSize() {
- return size;
+ public int getRetainedMemorySize() {
+ // SPARK-33206: here the offsets' capacity is multiplied by 8 as offsets
stores long values.
+ // And the extra 176 bytes is the estimate of the
`ShuffleIndexInformation` memory footprint
+ // which is relevant in case of small index files (i.e. storing only 2
offsets = 16 bytes).
+ return (offsets.capacity() << 3) + 176;
}
Review comment:
nit: pull the `176` as a package private constant and have the Suite
depend on it as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
