attilapiros commented on pull request #32790: URL: https://github.com/apache/spark/pull/32790#issuecomment-865506524
> (Ignoring ephemenral nodes, dockerized deployments, etc - and using more real world scenarios) The feature ([Serve local disk persisted blocks by the external service](https://github.com/apache/spark/pull/24499)) only works when external shuffle service is available. > For example, assuming 1k blocks per node on average - we end up with the map using approximately: ~ 5000 * 2048 * (~8 bytes) == ~80MB : this is the driver cost with zero blocks being used. I do not think the number of blocks matters at all. So `blockStatusByShuffleService` is the following data structure: https://github.com/apache/spark/blob/6de20f10ca60c778db25ba3d5975f85e5c23aedb/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L64-L66 It maps a special `BlockManagerId` instances which are created for the external shuffle service (so for the nodes) to a map which maps `BlockId` instances to a BlockStatus which is a simple case class: https://github.com/apache/spark/blob/37ef7bb98cdb1a8eefa06677f119a4d97e242097/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L788-L790 So the number is ~ 5000 * 2 * (~8 bytes) = 80KB = 0.08MB -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
