ivoson commented on code in PR #37268:
URL: https://github.com/apache/spark/pull/37268#discussion_r980043684
##########
core/src/main/scala/org/apache/spark/resource/ResourceProfileManager.scala:
##########
@@ -59,35 +59,65 @@ private[spark] class ResourceProfileManager(sparkConf:
SparkConf,
private val testExceptionThrown =
sparkConf.get(RESOURCE_PROFILE_MANAGER_TESTING)
/**
- * If we use anything except the default profile, it's only supported on
YARN and Kubernetes
- * with dynamic allocation enabled. Throw an exception if not supported.
+ * If we use anything except the default profile, it's supported on YARN,
Kubernetes and
+ * Standalone with dynamic allocation enabled, and task resource profile
with dynamic allocation
+ * disabled on Standalone. Throw an exception if not supported.
*/
private[spark] def isSupported(rp: ResourceProfile): Boolean = {
- val isNotDefaultProfile = rp.id !=
ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
- val notYarnOrK8sOrStandaloneAndNotDefaultProfile =
- isNotDefaultProfile && !(isYarn || isK8s || isStandalone)
- val YarnOrK8sOrStandaloneNotDynAllocAndNotDefaultProfile =
- isNotDefaultProfile && (isYarn || isK8s || isStandalone) &&
!dynamicEnabled
- // We want the exception to be thrown only when we are specifically
testing for the
- // exception or in a real application. Otherwise in all other testing
scenarios we want
- // to skip throwing the exception so that we can test in other modes to
make testing easier.
- if ((notRunningUnitTests || testExceptionThrown) &&
+ if (rp.isInstanceOf[TaskResourceProfile] && !dynamicEnabled) {
Review Comment:
Hey @Ngone51, thanks for the feedback, and for your concerns:
1. `TaskResourceProfile` can be used when dynamic allocation is enabled.
2. When dynamic allocation is enabled, `TaskResourceProfile` will be treated
as a normal `ResourceProfile` with no specific executor resource requirements,
and dynamic allocation manager will also request executors for
`TaskResourceProfile`, so we'll have executors matching the same resource
profile id. The behavior will be the same with what we have in master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
