tgravescs commented on a change in pull request #25047:
[WIP][SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone
URL: https://github.com/apache/spark/pull/25047#discussion_r305870426
##########
File path: core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
##########
@@ -70,6 +93,276 @@ private[spark] object ResourceUtils extends Logging {
// internally we currently only support addresses, so its just an integer
count
val AMOUNT = "amount"
+ /**
+ * Assign resources to workers/drivers from the same host to avoid address
conflict.
+ *
+ * This function works in three steps. First, acquiring the lock on
RESOURCES_LOCK_FILE
+ * to achieve synchronization among workers and drivers. Second, getting all
allocated
+ * resources from ALLOCATED_RESOURCES_FILE and assigning isolated resources
to the worker
+ * or driver after differentiating available resources in discovered
resources from
+ * allocated resources. If available resources don't meet worker's or
driver's requirement,
+ * try to update allocated resources by excluding the resource allocation if
its related
+ * process has already terminated and do the assignment again. If still
don't meet requirement,
+ * exception would be threw. Third, updating ALLOCATED_RESOURCES_FILE with
new allocated
Review comment:
nit: s/threw/thrown
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
