xintongsong commented on a change in pull request #11320:
[FLINK-16437][runtime] Make SlotManager allocate resource from ResourceManager
at the worker granularity.
URL: https://github.com/apache/flink/pull/11320#discussion_r403823418
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ActiveResourceManagerFactory.java
##########
@@ -81,4 +83,14 @@ private static Configuration
createActiveResourceManagerConfiguration(Configurat
ClusterInformation clusterInformation,
@Nullable String webInterfaceUrl,
ResourceManagerMetricGroup resourceManagerMetricGroup) throws
Exception;
+
+ protected WorkerResourceSpec
createDefaultWorkerResourceSpec(Configuration configuration) {
+ final TaskExecutorProcessSpec taskExecutorProcessSpec =
TaskExecutorProcessUtils
+ .newProcessSpecBuilder(configuration)
+ .withCpuCores(getDefaultCpus(configuration))
+ .build();
+ return
WorkerResourceSpec.fromTaskExecutorProcessSpec(taskExecutorProcessSpec);
+ }
+
+ protected abstract CPUResource getDefaultCpus(Configuration
configuration);
Review comment:
If the deriving of cpu cores and default worker spec is too detailed for
`ActiveResourceManager`, I would suggest to add an abstract method
`getDefaultWorkerSpecFactory` to return different `WorkerResourceSpecFactory`
for different `ActiveResourceManager` implementations.
My concern for passing `WorkerResourceSpecFactory` or `WorkerResourceSpec`
to `ActiveResourceManager` is that, neither the constructor nor the
`createResourceManager` method seems to be a good place to pass them in IMO.
- For the constructor, `ActiveResourceManager` is abstract and its
implementations are either singleton (K8s/Yarn) or created by cluster entry
point (Mesos). I think it is too detail for the cluster entry point to pass in
different `WorkerResourceSpecFactory`/`WorkerResourceSpec` to the resource
manager factory, not to mention breaking singleton properties on K8s/Yarn.
- For `createResourceManager`, it is called by
`DefaultDispatcherResourceManagerComponentFactory`, which is not aware of the
different deployments either.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
