smallzhongfeng commented on code in PR #39879:
URL: https://github.com/apache/spark/pull/39879#discussion_r1096494814
##########
core/src/main/scala/org/apache/spark/resource/ResourceAllocator.scala:
##########
@@ -38,11 +37,13 @@ private[spark] trait ResourceAllocator {
* For task resources ([[org.apache.spark.scheduler.ExecutorResourceInfo]]),
this value
* can be a multiple, such that each address can be allocated up to
[[slotsPerAddress]]
* times.
- *
- * TODO Use [[org.apache.spark.util.collection.OpenHashMap]] instead to gain
better performance.
*/
private lazy val addressAvailabilityMap = {
- mutable.HashMap(resourceAddresses.map(_ -> slotsPerAddress): _*)
+ val map: OpenHashMap[String, Int] = new OpenHashMap[String, Int]()
Review Comment:
Thank you for your comments. I make a mirco-benchmark. I used `OpenHashMap`
and `HashMap` to calculate the time consumed by the `acquire` and `release`
methods of `ResourceAllocator` at the same time. It was found that
`OpenHashMap` spent about 5 percent more time than `HashMap`. My test method is
to cycle 10000 times, 1000 times, and 100 times respectively.
I'm not sure whether this is reasonable, but the current situation shows
that the performance of `OpenHashMap` is not as good as that of `HashMap` in
the current case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
