craigcondit commented on a change in pull request #113: URL: https://github.com/apache/incubator-yunikorn-site/pull/113#discussion_r792753686
########## File path: docs/design/scheduler_plugin.md ########## @@ -0,0 +1,112 @@ +--- +id: scheduler_plugin +title: K8s Scheduler Plugin +--- + +<!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +## Background + +YuniKorn (on Kubernetes) is traditionally implemented as a ground-up implementation of a Kubernetes scheduler. +This has allowed us to innovate rapidly, but is not without its problems; we currently have numerous places +where we call into non-public K8S source code APIs with varying levels of (code) stability, requiring +sometimes very disruptive code changes when we switch to new Kubernetes releases. + +Ideally, we should be able to take advantage of enhancements to new Kubernetes releases automatically. +Using the plugin model enables us to enhance the Kubernetes scheduling logic with YuniKorn features. +This also helps keep YuniKorn compatible with new Kubernetes releases with minimal effort. + +Additionally, it is desirable in many cases to allow non-batch workloads to bypass the YuniKorn scheduling +functionality and use default scheduling logic. However, we have no way to do that today as the default +scheduling functionality is not present in the YuniKorn scheduler binary. + +Since Kubernetes 1.19, the Kubernetes project has created a stable API for the +[Scheduling Framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/), +which allows plugins to be created which implement various extension points. Plugins implement one or more +of these extension points, and are then compiled into a scheduler binary which contains the default +scheduler and plugin code, configured to call into the plugins during normal scheduling flow. + +## Design Review comment: > 1. We need a high-level design to explain how this works together. A workflow will be helpful. For example, when a pod gets submitted, what happens in the scheduler-plugin mode until it gets allocated? As the document indicates, YuniKorn runs as normal in the background with very little change (it just defers the actual binding until the scheduler framework calls `PreFilter()` / `Filter()`. I can take a stab at a diagram (though my illustration skills are not that great), but I'm not really sure what it would convey. > 2. Can you help to list the features that cannot be supported in the plugin mode? N/A, as there are no unsupported features with the plugin mode. > 3. I think we need to mentioned the plugin mode will have worse performance than the standalone mode This has yet to be determined, and I'm not sure it's actually the case. I'm not going to speculate about it here. ########## File path: docs/design/scheduler_plugin.md ########## @@ -0,0 +1,112 @@ +--- +id: scheduler_plugin +title: K8s Scheduler Plugin +--- + +<!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +* limitations under the License. +--> + +## Background + +YuniKorn (on Kubernetes) is traditionally implemented as a ground-up implementation of a Kubernetes scheduler. +This has allowed us to innovate rapidly, but is not without its problems; we currently have numerous places +where we call into non-public K8S source code APIs with varying levels of (code) stability, requiring +sometimes very disruptive code changes when we switch to new Kubernetes releases. + +Ideally, we should be able to take advantage of enhancements to new Kubernetes releases automatically. +Using the plugin model enables us to enhance the Kubernetes scheduling logic with YuniKorn features. +This also helps keep YuniKorn compatible with new Kubernetes releases with minimal effort. + +Additionally, it is desirable in many cases to allow non-batch workloads to bypass the YuniKorn scheduling +functionality and use default scheduling logic. However, we have no way to do that today as the default +scheduling functionality is not present in the YuniKorn scheduler binary. + +Since Kubernetes 1.19, the Kubernetes project has created a stable API for the +[Scheduling Framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/), +which allows plugins to be created which implement various extension points. Plugins implement one or more +of these extension points, and are then compiled into a scheduler binary which contains the default +scheduler and plugin code, configured to call into the plugins during normal scheduling flow. + +## Design Review comment: > 1. We need a high-level design to explain how this works together. A workflow will be helpful. For example, when a pod gets submitted, what happens in the scheduler-plugin mode until it gets allocated? As the document indicates, YuniKorn runs as normal in the background with very little change (it just defers the actual binding until the scheduler framework calls `PreFilter()` / `Filter()`. I can take a stab at a diagram (though my illustration skills are not that great), but I'm not really sure what it would convey. > 2. Can you help to list the features that cannot be supported in the plugin mode? N/A, as there are no unsupported features with the plugin mode. > 3. I think we need to mentioned the plugin mode will have worse performance than the standalone mode This has yet to be determined, and I'm not sure it's actually the case. I'm not going to speculate about it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
