attilapiros edited a comment on pull request #32790: URL: https://github.com/apache/spark/pull/32790#issuecomment-865506524
> (Ignoring ephemenral nodes, dockerized deployments, etc - and using more real world scenarios) The feature ([Serve local disk persisted blocks by the external service](https://github.com/apache/spark/pull/24499)) only works when external shuffle service is available. > For example, assuming 1k blocks per node on average - we end up with the map using approximately: ~ 5000 * 2048 * (~8 bytes) == ~80MB : this is the driver cost with zero blocks being used. I do not think the number of blocks matters at all. So `blockStatusByShuffleService` is the following data structure: https://github.com/apache/spark/blob/6de20f10ca60c778db25ba3d5975f85e5c23aedb/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L64-L66 It maps a special `BlockManagerId` instances which are created for the external shuffle service (so for the nodes) to a map which maps `BlockId` instances to a BlockStatus which is a simple case class: https://github.com/apache/spark/blob/37ef7bb98cdb1a8eefa06677f119a4d97e242097/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L788-L790 So the number is ~ 5000 * 2 * (~8 bytes) = 80KB = 0.08MB PS: I have adapted your calculation although I believe an empty hashmap is not 8 bytes but still some low constant number and I do not get why the 2 multiplier used as we should talk about only the external shuffle service map (the other map is for each executors but its lifecycle is handled correctly and when they removed the `BlockManagerInfo` instances are cleaned up too). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
