zhengruifeng commented on PR #57389: URL: https://github.com/apache/spark/pull/57389#issuecomment-5041458984
Memory estimate for the leaf-prediction path (measured with JOL on Java 11 / Scala 2.13, with compressed references): | Leaves | Old Scala map cache | Stored `leafIndex` fields | Net saved | | ---: | ---: | ---: | ---: | | 1,000 | 52,480 B | 8,000 B | 44,480 B | | 10,000 | 464,280 B | 80,000 B | 384,280 B | | 100,000 | 5,302,184 B | 800,000 B | 4,502,184 B | This is approximately **40-45 bytes per leaf**, or **40-45 MiB for one million leaves**, on this JVM. The actual number will vary with JVM and Scala collection implementation. The normal `Int` field adds about 8 B per leaf even when leaf prediction is unused; the net saving above applies when `predictLeaf` would otherwise construct the map. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
