This is an automated email from the ASF dual-hosted git repository.
wwei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new 871787e [YUNIKORN-240] Move missing documents from core to web repo.
871787e is described below
commit 871787e71736b6710dd5f5191421b545e195b171
Author: Weiwei Yang <[email protected]>
AuthorDate: Thu Jul 16 23:50:11 2020 -0700
[YUNIKORN-240] Move missing documents from core to web repo.
---
README.md | 2 +
docs/configuration/acls.md | 4 +-
docs/configuration/placement_rules.md | 4 +-
docs/configuration/queue_config.md | 6 +-
docs/configuration/sorting_policies.md | 154 +++++++++++++++
docs/design/{design.md => architecture.md} | 19 +-
docs/design/cross_queue_preemption.md | 4 +-
docs/design/namespace_resource_quota.md | 2 -
docs/design/pluggable_app_management.md | 2 -
docs/design/resilience.md | 2 -
docs/design/scheduler_configuration.md | 1 -
docs/design/scheduler_core_design.md | 219 +++++++++++++++++++++
docs/design/scheduler_object_states.md | 122 ++++++++++++
docs/design/state_aware_scheduling.md | 112 +++++++++++
.../deployment.md} | 41 +++-
.../developer_guide.md | 91 +++++----
docs/{setup => developer_guide}/env_setup.md | 2 +-
docs/get_started/get_started.md | 70 +++++++
docs/get_started/run_non_modified_k8s_apps.md | 91 +++++++++
docs/get_started/run_sample_workloads.md | 56 ++++++
docs/get_started/run_spark.md | 49 +++++
docs/get_started/run_tensorflow.md | 38 ++++
docs/get_started/user_guide.md | 197 ------------------
.../evaluate_perf_function_with_kubemark.md | 4 +-
docs/performance/metrics.md | 4 +-
docs/performance/profiling.md | 2 -
docs/setup/build_local.md | 74 -------
docs/setup/deployment.md | 81 --------
docusaurus.config.js | 2 +-
local-build.sh | 75 +++++++
sidebars.js | 27 +--
static/img/application-state.png | Bin 0 -> 119630 bytes
static/img/fifo-state-example.png | Bin 0 -> 239302 bytes
static/img/object-state.png | Bin 0 -> 39732 bytes
34 files changed, 1115 insertions(+), 442 deletions(-)
diff --git a/README.md b/README.md
index 2c29946..60a6688 100644
--- a/README.md
+++ b/README.md
@@ -16,6 +16,8 @@
* limitations under the License.
-->
# Apache YuniKorn website
+---
+
This readme will walk you through building the Apache YuniKorn website
diff --git a/docs/configuration/acls.md b/docs/configuration/acls.md
index 5a9d1f6..422d38a 100644
--- a/docs/configuration/acls.md
+++ b/docs/configuration/acls.md
@@ -1,6 +1,6 @@
---
id: acls
-title: Acls
+title: ACLs
---
<!--
@@ -22,8 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# YuniKorn Scheduler ACLs
-
## Usage
Access Control Lists are generic for YuniKorn.
They can be used in multiple places in YuniKorn.
diff --git a/docs/configuration/placement_rules.md
b/docs/configuration/placement_rules.md
index aee8a54..bf70bfa 100644
--- a/docs/configuration/placement_rules.md
+++ b/docs/configuration/placement_rules.md
@@ -1,6 +1,6 @@
---
id: placement_rules
-title: Placement Rules
+title: App Placement Rules
---
<!--
@@ -22,8 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# YuniKorn Scheduler Placement Rules
-
The basics for the placement rules are described in the [scheduler design
document](e_scheduler_configuration.md#placement-rules-definition).
Multiple rules can be chained to form a placement policy.
[Access control lists](configuration/acls.md) and rule filters are defined per
rule and enforced per rule.
diff --git a/docs/configuration/queue_config.md
b/docs/configuration/queue_config.md
index 37dc766..e1b8cbc 100644
--- a/docs/configuration/queue_config.md
+++ b/docs/configuration/queue_config.md
@@ -1,6 +1,6 @@
---
id: queue_config
-title: Queue Config
+title: Partition and Queue Configuration
---
<!--
@@ -22,9 +22,7 @@ specific language governing permissions and limitations
under the License.
-->
-# YuniKorn Partition and Queue Configuration
-
-The basis for the queue configuration is given in the [scheduler design
document](setup/configure_scheduler.md#queue-configuration).
+The basis for the queue configuration is given in the [scheduler design
document](design/architecture.md#configurations--semantics).
This document provides the generic queue configuration.
It references both the [Access control lists](configuration/acls.md) and
[placement rule](configuration/placement_rules.md) documentation.
diff --git a/docs/configuration/sorting_policies.md
b/docs/configuration/sorting_policies.md
new file mode 100644
index 0000000..f969cc0
--- /dev/null
+++ b/docs/configuration/sorting_policies.md
@@ -0,0 +1,154 @@
+---
+id: sorting_policies
+title: Sorting Policies
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+The scheduler uses policies allow changing the scheduling behaviour without
code changes.
+Policies can be set for:
+* [Applications](#application-sorting)
+* [Nodes](#node-sorting)
+* [Requests](#request-sorting)
+
+## Application sorting
+The application sorting policy is set for each queue via the config.
+A sorting policy setting is only effective on a `leaf` queue.
+Each `leaf` queue can use a different policy.
+
+A sorting policy only specifies the order in which the applications are sorted
within a queue.
+That order is crucial in specifying which application is considered first when
assigning resources.
+Sorting policies do _not_ affect the number of applications that are scheduled
or active in the queue at the same time.
+All applications that have pending resource requests can and will be scheduled
in a queue unless specifically filtered out.
+Even when applications are sorted using a first in first out policy multiple
applications will run in a queue in parallel.
+
+A `parent` queue will always use the fair policy to sort the child queues.
+
+The following configuration entry sets the application sorting policy to
`fifo` for the queue `root.sandbox`:
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: root
+ queues:
+ - name: sandbox
+ properties:
+ application.sort.policy: fifo
+```
+
+The only applications that are considered during scheduling must have
outstanding requests.
+A filter is applied _while_ sorting the applications to remove all that do not
have outstanding requests.
+
+### FifoSortPolicy
+Short description: first in first out, based on application create time
+Config value: fifo (default)
+Behaviour:
+Before sorting the applications are filtered and must have pending resource
requests.
+
+After filtering the applications left are sorted based on the application
create time stamp only, no other filtering is applied.
+Since applications can only be added while the system is locked there can
never be two applications with the exact same time stamp.
+
+The result is that the oldest application that requests resources gets
resources.
+Younger applications will be given resources when all the current requests of
older applications have been fulfilled.
+
+### FairSortPolicy
+Short description: fair based on usage
+Config value: fair
+Behaviour:
+Before sorting the applications are filtered and must have pending resource
requests.
+
+After filtering the applications left are sorted based on the application
usage.
+The usage of the application is defined as all confirmed and unconfirmed
allocations for the applications.
+All resources defined on the application will be taken into account when
calculating the usage.
+
+The result is that the resources available are spread equally over all
applications that request resources.
+
+### StateAwarePolicy
+Short description: limit of one (1) application in Starting or Accepted state
+Config value: stateaware
+Behaviour:
+This sorting policy requires an understanding of the application states.
+Applications states are described in the [application
states](design/scheduler_object_states.md#application-state) documentation.
+
+Before sorting applications the following filters are applied to all
applications in the queue:
+The first filter is based on the application state.
+The following applications pass through the filter and generate the first
intermediate list:
+* all applications in the state _running_
+* _one_ (1) application in the _starting_ state
+* if there are _no_ applications in the _starting_ state _one_ (1) application
in the _accepted_ state is added
+
+The second filter takes the result of the first filter as an input.
+The preliminary list is filtered again: all applications _without_ a pending
request are removed.
+
+After filtering based on status and pending requests the applications that
remain are sorted.
+The final list is thus filtered twice with the remaining applications sorted
on create time.
+
+To recap the _staring_ and _accepted_ state interactions:
+The application in the _accepted_ state is only added if there is no
application in the _starting_ state.
+The application in the _starting_ state does not have to have pending requests.
+Any application in the _starting_ state will prevent _accepted_ applications
from being added to the filtered list.
+
+For further details see the [Example
run](design/state_aware_scheduling.md#example-run) in the design document.
+
+The result is that already running applications that request resources will
get resources first.
+A drip feed of one new applications is added to the list of running
applications to be allocated after all running applications.
+
+## Node sorting
+The node sorting policy is set for a partition via the config.
+Each partition can use a different policy.
+
+The following configuration entry sets the node sorting policy to `fair` for
the partition `default`:
+```yaml
+partitions:
+ - name: default
+ nodesortpolicy:
+ type: fair
+```
+
+### FairnessPolicy
+Short description: available resource, descending order
+Config value: fair (default)
+Behaviour:
+Sort the list of nodes by the amount of available resources so that the node
with the _highest_ amount of available resource is the first in the list.
+All resources defined on a node will be taken into account when calculating
the usage.
+Resources of the same type are compared for the nodes.
+
+This results in a node with the lowest utilisation to be considered first for
assigning new allocation.
+Resulting in a spread of allocations over all available nodes.
+Leading to an overall lower utilisation of the individual available nodes,
unless the whole environment is highly utilised.
+Keeping the load on all nodes at a similar level does help
+In an environment that auto scales by adding new nodes this could trigger
unexpected auto scale requests.
+
+### BinPackingPolicy
+Short description: available resource, ascending order
+Config value: binpacking
+Behaviour:
+Sort the list of nodes by the amount of available resources so that the node
with the _lowest_ amount of available resource is the first in the list.
+All resources defined on a node will be taken into account when calculating
the usage.
+Resources of the same type are compared for the nodes.
+
+This results in a node with the highest utilisation to be considered first for
assigning new allocation.
+Resulting in a high(er) utilisation of a small(er) number of nodes, better
suited for cloud deployments.
+
+## Request sorting
+There is currently one policy for sorting requests within an application.
+This policy is not configurable.
+Sorting requests is only possible based on the priority of the request.
+If there are multiple requests within an application that have the same
priority the order of the requests is undetermined.
+This means that the order of requests with the same priority can, and most
likely will, change between runs.
\ No newline at end of file
diff --git a/docs/design/design.md b/docs/design/architecture.md
similarity index 90%
rename from docs/design/design.md
rename to docs/design/architecture.md
index 1c7d683..26fcfe6 100644
--- a/docs/design/design.md
+++ b/docs/design/architecture.md
@@ -1,6 +1,6 @@
---
-id: design
-title: Design
+id: architecture
+title: Architecture
---
<!--
@@ -22,13 +22,22 @@ specific language governing permissions and limitations
under the License.
-->
-# Design of YuniKorn scheduler
+Apache YuniKorn (Incubating) is a light-weight, universal resource scheduler
for container orchestrator systems.
+It was created to achieve fine-grained resource sharing for various workloads
efficiently on a large scale, multi-tenant,
+and cloud-native environment. YuniKorn brings a unified, cross-platform,
scheduling experience for mixed workloads that
+consist of stateless batch workloads and stateful services.
-## Overall
+YuniKorn now supports K8s and can be deployed as a custom K8s scheduler.
YuniKorn's architecture design also allows
+adding different shim layer and adopt to different ResourceManager
implementation including Apache Hadoop YARN,
+or any other systems.
## Architecture
-### Components:
+Following chart illustrates the high-level architecture of YuniKorn.
+
+
+
+## Components:
**Scheduler API Server**:
diff --git a/docs/design/cross_queue_preemption.md
b/docs/design/cross_queue_preemption.md
index a2a1177..51c8033 100644
--- a/docs/design/cross_queue_preemption.md
+++ b/docs/design/cross_queue_preemption.md
@@ -22,9 +22,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Preemption Logic Of YuniKorn
-
-## To be solved problems:
+## Problems:
According to lessons we learned from YARN Scheduler preemption.
diff --git a/docs/design/namespace_resource_quota.md
b/docs/design/namespace_resource_quota.md
index d1fec5a..90830b6 100644
--- a/docs/design/namespace_resource_quota.md
+++ b/docs/design/namespace_resource_quota.md
@@ -22,8 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# Use K8s Namespace in YuniKorn
-
In K8s, user can setup namespace with [resource
quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) to limit
aggregated resource consumption in this namespace. The validation of namespace
resource quotas is handled in api-server directly, therefore YuniKorn simply
honors the quotas like the default scheduler.
## Best practice
diff --git a/docs/design/pluggable_app_management.md
b/docs/design/pluggable_app_management.md
index 45c3872..681e7bb 100644
--- a/docs/design/pluggable_app_management.md
+++ b/docs/design/pluggable_app_management.md
@@ -22,8 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# Pluggable App Management
-
## The Problem
Currently, we schedule and group an application is based on a label on the pod.
diff --git a/docs/design/resilience.md b/docs/design/resilience.md
index 7c3834a..46aac13 100644
--- a/docs/design/resilience.md
+++ b/docs/design/resilience.md
@@ -22,8 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# Resilience Design
-
This is not a HA (High-availability) design, HA implies that a service can
survive from a fatal software/hardware failure. That requires one or more
standby instances providing same services to take over active instance on
failures.
diff --git a/docs/design/scheduler_configuration.md
b/docs/design/scheduler_configuration.md
index 5b7511a..3193771 100644
--- a/docs/design/scheduler_configuration.md
+++ b/docs/design/scheduler_configuration.md
@@ -22,7 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# Yunikorn Scheduler Configuration
The Yunikorn core scheduler configuration has two separate areas that need to
be configured. The scheduler service itself, things like web service ports etc,
and the queue configuration. The split between the two types of configuration
is proposed with two points in mind:
* Separation of duty
* Dynamic vs Static
diff --git a/docs/design/scheduler_core_design.md
b/docs/design/scheduler_core_design.md
new file mode 100644
index 0000000..a941228
--- /dev/null
+++ b/docs/design/scheduler_core_design.md
@@ -0,0 +1,219 @@
+---
+id: scheduler_core_design
+title: Scheduler Core Design
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+## Overall
+
+## Architecture
+
+### Components:
+
+```
+
+ +---------------+ +--------------+
+ |K8s Shim | |YARN Shim |
+ +---------------+ +--------------+
+
+ +--------------+ +------------+
+ Scheduler- | GRPC Protocol| |Go API |
+ Interface: +--------------+ +------------+
+ +----------------------------- YuniKorn-Core
-------------------------------+
+ +--------------------+
+ | RMProxy |
+ +---------+----------+
+ |
+ |Write Ops
+----------------+
+ +-------------+ V ++Scheduler
|
+ |ConfigWacher + +---------------+ Allocate || And
|
+ +-------------+ |Scheduler Cache| <-----------------|
|
+ +----------> +---------------+ Preempt ++Preemptor
|
+ Update Cfg ^
+----------------+
+ |
+ |
+ +-------------------+-------------------------+
+ |--------+ +------+ +----------+ +----------+ |
+ ||Node | |Queue | |Allocation| |Requests | | +
+ |--------+ +------+ +----------+ +----------+ |
+ +---------------------------------------------+
+```
+
+#### RMProxy
+
+Responsible for communication between RM and Scheduler, which implements
scheduler-interface GRPC protocol, or just APIs. (For intra-process
communication w/o Serde).
+
+#### Scheduler Cache
+
+Caches all data related to scheduler state, such as used resources of each
queues, nodes, allocations. Relationship between allocations and nodes, etc.
+
+Should not include in-flight data for resource allocation. For example
to-be-preempted allocation candidates. Fair share resource of queues, etc.
+
+#### Configuration
+
+Handles configuration for YuniKorn scheduler, reload configuration, etc.
ConfigWatcher is responsible for watching changes of config and reload the
config, and ConfigValidator is responsible for validate if a config is valid or
not.
+
+#### Scheduler and Preemptor
+
+Handles Scheduler's internal state. (Which is not belong to scheduelr cache),
such as internal reservations, etc. Scheduler and preemptor will work together,
make scheduling or preemption decisions.
+
+##### Scheduler's responsibility
+
+- According to resource usages between queues, sort queues, applications, and
figure out order of application allocation. (This will be used by preemption as
well).
+- It is possible that we cannot satisfy some of the allocation request, we
need to skip them and find next request.
+- It is possible that some allocation request cannot be satisfied because of
resource fragmentation. We need to reserve room for such requests.
+- Different nodes may belong to different disjoint partitions, we can make
independent scheduler runs
+- Be able to config and change ordering policies for apps, queues.
+- Application can choose their own way to manage sort of nodes.
+
+##### Preemptor's responsibility
+
+- It is important to know "who wants the resource", so we can do preemption
based on allocation orders.
+- When do preemption, it is also efficient to trigger allocation op. Think
about how to do it.
+- Preemption needs to take care about queue resource balancing.
+
+#### EventHandler
+
+All events exchange between RMProxy, Cache, Scheduler are handled by
asynchonrized event handler.
+
+#### Communication between Shim and Core
+
+YuniKorn-Shim (like https://github.com/apache/incubator-yunikorn-k8shim)
communicates with core by using scheduler-interface
(https://github.com/apache/incubator-yunikorn-scheduler-interface). Scheduler
interface has Go API or GRPC. Currently, yunikorn-k8shim is using Go API to
communicate with yunikorn-core to avoid extra overhead introduced by GRPC.
+
+**Shim (like K8shim) first need to register with core:**
+
+```go
+func (m *RMProxy) RegisterResourceManager(request
*si.RegisterResourceManagerRequest, callback api.ResourceManagerCallback)
(*si.RegisterResourceManagerResponse, error)
+```
+
+Which indicate ResourceManager's name, a callback function for updateResponse.
The design of core is be able to do scheduling for multiple clusters (such as
multiple K8s cluster) just with one core instance.
+
+**Shim interacts with core by invoking RMProxy's Update API frequently, which
updates new allocation request, allocation to kill, node updates, etc.**
+
+```go
+func (m *RMProxy) Update(request *si.UpdateRequest) error
+```
+
+Response of update (such as new allocated container) will be received by
registered callback.
+
+## Configurations & Semantics
+
+Please refer to [scheduler-configuration](setup/configure_scheduler.md) to
better understand configuration.
+
+## How scheduler do allocation
+
+Scheduler runs a separate goroutine to look at asks and available resources,
and do resource allocation. Here's allocation logic in pseudo code:
+
+Entry point of scheduler allocation is `scheduler.go: func (s *Scheduler)
schedule()`
+
+```
+# First of all, YuniKorn has partition concept, a logical resource pool can
consists
+# of one of multiple physical dis-joint partitions. It is similar to YARN's
node
+# partition concept.
+
+for partition : partitions:
+ # YuniKorn can reserve allocations for picky asks (such as large request,
etc.)
+ # Before doing regular allocation, YuniKorn look at reservedAllocations
first.
+ for reservedAllocation : partition.reservedAllocations:
+ reservedAllocation.tryAllocate(..)
+
+ # After tried all reserved allocation, YuniKorn will go to regular allocation
+ partition.tryAllocate(..)
+
+ # If there's any allocation created, scheduler will create an
AllocationProposal
+ # and send to Cache to "commit" the AllocationProposal
+```
+
+**Allocation by hierchical of queues**
+
+Inside `partition.tryAllocate`
+
+It recursively traverse from root queue and down to lower level, for each
level, logic is inside `pkg/scheduler/scheduling_queue.go func (sq
*SchedulingQueue) tryAllocate`
+
+Remember YuniKorn natively supports hierarchical of queues. For ParentQueue
(which has sub queues under the parent queue), it uses queue's own sorting
policy to sort subqueues and try to allocate from most preferred queue to
least-preferred queue.
+
+For LeafQueue (which has applications inside the queue), it uses queue's own
sorting policy to sort applications belong to the queue and allocate based on
the sorted order.
+
+(All sorting policies can be configured differently at each level.)
+
+**Allocation by application**
+
+When it goes to Application, see (`scheduler_application.go: func (sa
*SchedulingApplication) tryAllocate`), It first sort the pending resource
requests belong to the application (based on requests' priority). And based on
the selected request, and configured node-sorting policy, it sorts nodes belong
to the partition and try to allocate resources on the sorted nodes.
+
+When application trying to allocate resources on nodes, it will invokes
PredicatePlugin to make sure Shim can confirm the node is good. (For example
K8shim runs predicates check for allocation pre-check).
+
+**Allocation completed by scheduler**
+
+Once allocation is done, scheduler will create an AllocationProposal and send
to Cache to do further check, we will cover details in the upcoming section.
+
+## Flow of events inside YuniKorn-core
+
+Like mentioned before, all communications between components like
RMProxy/Cache/Schedulers are done by using async event handler.
+
+RMProxy/Cache/Scheduler include local event queues and event handlers. RMProxy
and Scheduler have only one queue (For example: `pkg/scheduler/scheduler.go:
handleSchedulerEvent`), and Cache has two queues (One for events from RMProxy,
and one for events from Scheduler, which is designed for better performance).
+
+We will talk about how events flowed between components:
+
+**Events for ResourceManager registration and updates:**
+
+```
+Update from ResourceManager -> RMProxy -> RMUpdateRequestEvent Send to Cache
+New ResourceManager registration -> RMProxy -> RegisterRMEvent Send to Cache
+```
+
+**Cache Handles RM Updates**
+
+There're many fields inside RM Update event (`RMUpdateRequestEvent`), among
them, we have following categories:
+
+```
+1) Update for Application-related updates
+2) Update for New allocation ask and release.
+3) Node (Such as kubelet) update (New node, remove node, node resource change,
etc.)
+```
+
+More details can be found at:
+
+```
+func (m *ClusterInfo) processRMUpdateEvent(event
*cacheevent.RMUpdateRequestEvent)
+
+inside cluster_info.go
+```
+
+**Cache send RM updates to Scheduler**
+
+For most cases, Cache propagate updates from RM to scheduler directly
(including Application, Node, Asks, etc.). And it is possible that some updates
from RM is not valid (such as adding an application to a non-existed queue),
for such cases, cache can send an event back to RMProxy and notify the
ResourceManager. (See `RMApplicationUpdateEvent.RejectedApplications`)
+
+**Cache handles scheduler config**
+
+Cache also handles scheduler's config changes, see
+
+```go
+func (m *ClusterInfo) processRMConfigUpdateEvent(event
*commonevents.ConfigUpdateRMEvent)
+```
+
+Similar to other RM updates, it propages news to scheduelr.
+
+**Scheduler do allocation**
+
+Once an AllocationProposal created by scheduler, scheduler sends
`AllocationProposalBundleEvent` to Cache to commit.
+
+Cache look at AllocationProposal under lock, and commit these proposals. The
reason to do proposal/commit is Scheduler can run in multi-threads which could
cause conflict for resource allocation. This approach is inspired by
Borg/Omega/YARN Global Scheduling.
+
+Cache checks more states such as queue resources, node resources (we cannot
allocate more resource than nodes' available), etc. Once check is done, Cache
updates internal data strcture and send confirmation to Scheduler to update the
same, and scheduler sends allocated Allocation to RMProxy so Shim can do
further options. For example, K8shim will `bind` an allocation (POD) to kubelet.
\ No newline at end of file
diff --git a/docs/design/scheduler_object_states.md
b/docs/design/scheduler_object_states.md
new file mode 100644
index 0000000..d761658
--- /dev/null
+++ b/docs/design/scheduler_object_states.md
@@ -0,0 +1,122 @@
+---
+id: scheduler_object_states
+title: Scheduler Object States
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+The YuniKorn project uses state machines to track the states of different
objects.
+This ranges from applications in the core to nodes in the k8shim.
+The state machines are independent and not shared between the resource
managers and core.
+A resource manager shim, and the core can thus have an independent idea of the
state of a similar object.
+
+## Core Scheduler
+State change are triggered by events that get processed.
+One event can cause a change for multiple states or no change at all.
+
+### Application State
+Applications have a complex state model.
+An application when created starts ain the new state.
+
+An application can have the following states:
+* New: A new application that is being submitted or created, from here the
application transitions into the accepted state when it is ready for scheduling.
+The first ask to be added will trigger the transition.
+* Accepted: The application is ready and part of the scheduling cycle.
+On allocation of the first ask the application moves into a starting state.
+This state is part of the normal scheduling cycle.
+* Starting: The application has exactly one allocation confirmed this
corresponds to one running container/pod.
+The application transitions to running if and when more allocations are added
to the application.
+This state times out automatically to prevent applications that consist of
just one allocation from getting stuck in this state.
+The current time out is set to 5 minutes, and cannot be changed.
+If after the timeout expires the application will auto transition to running.
+The state change on time out is independent of the number of allocations
added.
+This state is part of the normal scheduling cycle.
+* Running: The state in which the application will spend most of its time.
+Containers/pods can be added to and removed from the application.
+This state is part of the normal scheduling cycle.
+* Waiting: An application that has no pending requests or running
containers/pod will be waiting.
+This state shows that the application has not been marked completed yet but
currently is not actively being scheduled.
+* Completed: The resource manager has signalled that the application is done.
+This is a final state. The application cannot change state after entering.
+* Killed: Removed by the resource manager at the request of an administrator
or the user running the application.
+This is a final state. The application cannot change state after entering.
+* Rejected: The application was rejected when it was added to the scheduler.
+This only happens when a resource manager tries to add a new application, when
it gets created in a New state, and the scheduler rejects the creation.
+Applications can be rejected due ACLs denying access to a queue the
application has specified, or a placement via placement rules has failed.
+This is a final state. The application cannot change state after entering.
+
+The events that can trigger a state change:
+* Reject: rejecting the application by the scheduler (source: core scheduler)
+* Run: progress an application to the next active state (source: core
scheduler)
+* Wait: mark an application as idle (source: core scheduler)
+* Complete: mark an application as complete (source: resource manager)
+* Kill: kill an application (source: resource manager)
+
+Here is a diagram that shows the states with the event that causes the state
to change:
+
+
+### Object State
+<!-- fix the draining to stopped transition -->
+The object state is used by the following objects:
+* queues
+* partitions
+
+The object states are as follows:
+* Active: The object is active and used during the scheduling cycle.
+This is the starting and normal state of an object.
+An active object transitions to draining when it is removed.
+* Stopped: The object is stopped and no longer actively scheduled.
+The object if empty is ready to be removed from the scheduler.
+The object can transition back into active state if it gets re-started.
+* Draining: Before an object can be removed it needs to be cleaned up.
+The cleanup starts with placing the object in the draining state.
+In this state it does not accept additions or changes but is still actively
being scheduled.
+This allows for a graceful shutdown, cleanup and removal of the object.
+This is the final state.
+
+The events that can trigger a state change:
+* Start: make the object active (source: core scheduler)
+* Stop: make the object inactive (source: core scheduler)
+* Remove: mark an object for removal (source: core scheduler)
+
+Here is a diagram that shows the states with the event that causes the state
to change:
+
+
+### Node
+<!-- should start using object state -->
+Node objects in the core are not using a state machine but do have a state.
+A node can have one of two states: `schedulable` or `not schedulable`.
+There is no complex state model or complex transition logic.
+The scheduler can either use the node or not.
+
+The node status changes based on the status provided by the resource manager
(shim) that owns the node.
+
+## K8Shim Resource Manager
+
+### Application
+To be added
+
+### Task
+To be added
+
+### Node
+To be added
+
+### Scheduler
+To be added
\ No newline at end of file
diff --git a/docs/design/state_aware_scheduling.md
b/docs/design/state_aware_scheduling.md
new file mode 100644
index 0000000..7cc35a1
--- /dev/null
+++ b/docs/design/state_aware_scheduling.md
@@ -0,0 +1,112 @@
+---
+id: state_aware_scheduling
+title: Batch Workloads Ordering with StateAware Policy
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+## The problem
+A common pattern while processing data is that the application can be divided
into multiple stages.
+Another way to look at this is the fact that processing needs to be kicked off
and that the first step is to start a driver or manager for the application.
+Later stages might depend on the previous stages.
+When running applications in a size limited environment this could lead to a
resource exhaustion when submitting multiple applications at the same time.
+These first stages might consume all available resources leaving no room for
the next stage(s) to start.
+Often this issue is caused by having a high number of applications start
simultaneous and trying to get resources in parallel.
+### Example issue
+When submitting numerous Spark applications in a short amount of time the
drivers will all be started shortly after each other.
+The drivers consume all available resources in a queue or in the whole cluster.
+After starting the drivers they will request resources for the executors.
+Since the queue or cluster has no resources left the executors will not be
started.
+The driver cannot progress.
+The only way that progress would be made is if and when one of the drivers
finishes or fails and frees up resources for executors to be started.
+
+## Design
+### Design goals
+1. Prevent resource exhaustion by first stage allocations
+1. Improve chance for jobs to get minimal required resources over others
+
+### None goals
+1. This is NOT an implementation of Gang scheduling.
+1. No change to the currently supported FAIR or FIFO scheduling algorithms
+1. Fix resource quota usage outside of the core scheduler for submitted but
waiting applications
+
+### Possible solutions
+Other batch schedulers like the YARN schedulers use a limit on the number of
simultaneous running applications.
+They use either resource constraints on the driver or management stage or set
a hard limit of the number of applications that can run in a queue.
+The draw back of that solution is that it does not work well in a cluster that
can scale up or down automatically in a cloud environment.
+To work around that percentage based limits could be set on the consumed
resources for the driver or management stage.
+This does not alleviate the fact that driver or management stages can be of
any size, large and or small, which complicates the percentage scenario further
as it does not give a predictable behaviour.
+
+A different solution would be to assume a specific behaviour of the
applications.
+Using that assumption a limit on the applications could be set based on the
state it is in.
+The spark driver and executor behaviour is the most usual use case.
+This would provide a way to limit scheduling to existing applications and only
drip feed new applications into the list of applications to schedule when there
are resources available.
+
+### Algorithm
+The algorithm described here is based on the drip feed of new applications
into the applications to schedule based on the states of all applications.
+Scheduling is based on the applications in a queue.
+The algorithm will be applied at a queue level.
+This is not a cluster wide setup.
+
+What we want to achieve is the following behaviour: only schedule one (1)
application that is in its early stage(s) (called a starting state) at the same
time.
+Only consider another new application if and when the previous application has
transitioned out of the starting state.
+Applications will always be allocated resources on a first in first out basis
based on submission time.
+That means that an application that is newly added and in its starting phase
will only get resources if applications in the later stages do not need any
resources.
+
+This algorithm will be implemented as an application sorting policy on a queue.
+This allows specific queues to limit parallel application startup while other
queues with different work loads can schedule without or with different
limitations.
+
+### Fallback mechanism
+A fallback mechanism has to be built into the algorithm.
+Not all applications will request more than one allocation.
+The other case that has to be accounted for could be a misbehaving or a slow
application.
+Having an application stuck in the starting state could cause a scheduler
livelock and starvation of other applications.
+
+The fall back mechanism proposed is as simple as a time limit on the starting
state.
+This means that any application auto progresses out of the starting state.
+The time limit will be set to five (5) minutes hard coded as a starting point
and will not be made configurable.
+
+The other fallback mechanism considered was making the number of allocations
for the starting state configurable.
+This option provided a number of issues which made it difficult to implement.
+One of the main stumbling blocks is the fact that it requires the application
submitter to specify the value.
+It also does not guarantee that the application will leave the starting state
either and does not fix the livelock issue.
+If an application was submitted with five required allocation but due to
issues during the run never asked for more than four then the livelock would
still occur.
+
+Setting a default of zero (0) would also not work as it would bypass the
starting state.
+It would make the sorting policy an opt-in instead of an opt-out.
+Setting a default of one (1) does not give us much enhancement to what we
currently propose.
+It makes the sorting policy an opt-out but does not give the cluster
administrator any control over the scheduling behaviour.
+Weighing those against each other the proposal is to not make this
configurable.
+
+### Example run
+Using Spark applications as an example: a new application can only be
scheduled if the previous application has at least one (1) executor allocated.
+
+
+
+Assume we have the following Spark apps: App1 & App2 as in the diagram above.
The applications were submitted in that order: App1 first, then App2. They were
both submitted to the same queue.
+
+1. Both applications are in the queue waiting for the first allocation:
accepted by the scheduler. App1 has requested driver D1 and App2 has requested
driver D2.
+1. The scheduler sorts the application and allows 1 accepted application to be
scheduled (no starting applications yet): App1 as the oldest applications with
an outstanding request is scheduled.
+App1 is allocated its driver (D1) and progresses to starting.
+App2 request for a driver is ignored as the scheduler is starting App1 (only 1
application in starting or accepted state is scheduled).
+1. App1 requests executors E11 and E12. The scheduler assigns E11 and E12. At
this point the application state changes to running when it has at least 1
executor allocated.
+1. App2 has been waiting to get the driver allocated. Since there are no
applications in a starting state the scheduler looks at App2 which is in an
accepted state. App2 moves from the accepted state to starting when the driver
is allocated.
+1. App2 requests its executor E21. The application state changes to running
when E21 is allocated.
+
+This process would repeat itself for any new application submitted.
\ No newline at end of file
diff --git a/docs/setup/configure_scheduler.md
b/docs/developer_guide/deployment.md
similarity index 66%
rename from docs/setup/configure_scheduler.md
rename to docs/developer_guide/deployment.md
index 463145e..4b99400 100644
--- a/docs/setup/configure_scheduler.md
+++ b/docs/developer_guide/deployment.md
@@ -1,6 +1,6 @@
---
-id: configure_scheduler
-title: Configure Scheduler
+id: deployment
+title: Build and Run the Scheduler
---
<!--
@@ -22,7 +22,9 @@ specific language governing permissions and limitations
under the License.
-->
-# Deployment of YuniKorn using a ConfigMap
+The easiest way to deploy YuniKorn is to leverage our [helm
charts](https://hub.helm.sh/charts/yunikorn/yunikorn),
+you can find the guide [here](get_started/get_started.md). This document
describes the manual process to deploy YuniKorn
+scheduler and it is majorly for developers.
## Build docker image
@@ -35,6 +37,15 @@ This command will build an image. The image will be tagged
with a default versio
**Note** the default build uses a hardcoded user and tag. You *must* update
the `IMAGE_TAG` variable in the `Makefile` to push to an appropriate
repository.
+
+## Setup RBAC
+
+The first step is to create the RBAC role for the scheduler, see
[yunikorn-rbac.yaml](https://github.com/apache/incubator-yunikorn-k8shim/blob/master/deployments/scheduler/yunikorn-rbac.yaml)
+```
+kubectl create -f scheduler/yunikorn-rbac.yaml
+```
+The role is a requirement on the current versions of kubernetes.
+
## Create the ConfigMap
This must be done before deploying the scheduler. It requires a correctly
setup kubernetes environment.
@@ -55,7 +66,7 @@ kubectl describe configmaps yunikorn-configs
**Note** if name of the ConfigMap is changed the volume in the scheduler yaml
file must be updated to reference the new name otherwise the changes to the
configuration will not be picked up.
-## Attach ConfigMap Volume to the Scheduler Pod
+## Attach ConfigMap to the Scheduler Pod
The ConfigMap is attached to the scheduler as a special volume. First step is
to specify where to mount it in the pod:
```yaml
@@ -74,13 +85,33 @@ Second step is to link the mount point back to the
configuration map created in
Both steps are part of the scheduler yaml file, an example can be seen at
[scheduler.yaml](https://github.com/apache/incubator-yunikorn-k8shim/blob/master/deployments/scheduler/scheduler.yaml)
for reference.
-
## Deploy the Scheduler
+
The scheduler can be deployed with following command.
```
kubectl create -f deployments/scheduler/scheduler.yaml
```
+The deployment will run 2 containers from your pre-built docker images in 1
pod,
+
+* yunikorn-scheduler-core (yunikorn scheduler core and shim for K8s)
+* yunikorn-scheduler-web (web UI)
+
+The pod is deployed as a customized scheduler, it will take the responsibility
to schedule pods which explicitly specifies `schedulerName: yunikorn` in pod's
spec.
+
+## Access to the web UI
+
+When the scheduler is deployed, the web UI is also deployed in a container.
+Port forwarding for the web interface on the standard ports can be turned on
via:
+
+```
+POD=`kubectl get pod -l app=yunikorn -o jsonpath="{.items[0].metadata.name}"`
&& \
+kubectl port-forward ${POD} 9889 9080
+```
+
+`9889` is the default port for Web UI, `9080` is the default port of
scheduler's Restful service where web UI retrieves info from.
+Once this is done, web UI will be available at: http://localhost:9889.
+
## Configuration Hot Refresh
YuniKorn supports to load configuration changes automatically from attached
configmap. Simply update the content in the configmap,
diff --git a/docs/get_started/developer_guide.md
b/docs/developer_guide/developer_guide.md
similarity index 65%
rename from docs/get_started/developer_guide.md
rename to docs/developer_guide/developer_guide.md
index 7f1ccee..2578751 100644
--- a/docs/get_started/developer_guide.md
+++ b/docs/developer_guide/developer_guide.md
@@ -28,7 +28,7 @@ This document describes resources how to setup dev
environment and how to do the
## Development Environment setup
-Read the [environment setup guide](setup/env_setup.md) first to setup Docker
and Kubernetes development environment.
+Read the [environment setup guide](developer_guide/env_setup.md) first to
setup Docker and Kubernetes development environment.
## Build YuniKorn for Kubernetes
@@ -57,7 +57,7 @@ make image
The image with the build in configuration can be deployed directly on
kubernetes.
Some sample deployments that can be used are found under
[deployments](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/scheduler)
directory.
For the deployment that uses a config map you need to set up the ConfigMap in
kubernetes.
-How to deploy the scheduler with a ConfigMap is explained in the [scheduler
configuration deployment](setup/configure_scheduler.md) document.
+How to deploy the scheduler with a ConfigMap is explained in the [scheduler
configuration deployment](developer_guide/deployment.md) document.
The image build command will first build the integrated executable and then
create the docker image.
Currently, there are some published docker images under [this docker hub
repo](https://hub.docker.com/r/apache/yunikorn), you are free to fetch and use.
@@ -94,60 +94,73 @@ The dependencies in the projects are managed using [go
modules](https://blog.gol
Go Modules require at least Go version 1.11 to be installed on the development
system.
If you want to modify one of the projects locally and build with your local
dependencies you will need to change the module file.
-Changing dependencies uses mod `replace` directives as explained in the [local
build document](setup/build_local.md).
+Changing dependencies uses mod `replace` directives as explained in the
[Update dependencies](#Updating dependencies).
-## Build the web UI
+The YuniKorn project has four repositories three of those repositories have a
dependency at the go level.
+These dependencies are part of the go modules and point to the github
repositories.
+During development it can be required to break the dependency on the committed
version from github.
+This requires making changes in the module file to allow loading a local copy
or a forked copy from a different repository.
-Example deployments reference the [YuniKorn web
UI](https://github.com/apache/incubator-yunikorn-web).
-The YuniKorn web UI has its own specific requirements for the build. The
project has specific requirements for the build follow the steps in the README
to prepare a development environment and build how to build the projects.
-The scheduler is fully functional without the web UI.
+#### Affected repositories
+The following dependencies exist between the repositories:
-## Locally run the integrated scheduler
+| repository| depends on |
+| --- | --- |
+| yunikorn-core | yunikorn-scheduler-interface |
+| yunikorn-k8shim | yunikorn-scheduler-interface, yunikorn-core |
+| yunikorn-scheduler-interface | none |
+| yunikorn-web | yunikorn-core |
-When you have a local development environment setup you can run the scheduler
in your local kubernetes environment.
-This has been tested in a Docker desktop with docker for desktop and Minikube.
See the [environment setup guide](setup/env_setup.md) for further details.
+The `yunikorn-web` repository has no direct go dependency on the other
repositories. However any change to the `yunikorn-core` webservices can affect
the web interface.
-```
-make run
-```
-It will connect with the kubernetes cluster using the users configured
configuration located in `$HOME/.kube/config`.
+#### Making local changes
-You can also use the same approach to run the scheduler locally but connecting
to a remote kubernetes cluster,
-as long as the `$HOME/.kube/config` file is pointing to that remote cluster.
+To make sure that the local changes will not break other parts of the build
you should run:
+- A full build `make` (build target depends on the repository)
+- A full unit test run `make test`
-## Core component build
+Any test failures should be fixed before proceeding.
-The scheduler core, this repository build, by itself does not provide a
functional scheduler.
-It just builds the core scheduler functionality without any resource managers
or shims.
-A functional scheduler must have at least one resource manager that registers.
+#### Updating dependencies
-### Build steps
-The core component contains two command line tools: the `simplescheduler` and
the `schedulerclient`.
-The two command line tools have been provided as examples only and are not
supposed to implement all functionality.
+The simplest way is to use the `replace` directive in the module file. The
`replace` directive allows you to override the import path with a new (local)
path.
+There is no need to change any of the imports in the source code. The change
must be made in the `go.mod` file of the repository that has the dependency.
-Building the example command line tools:
+Using `replace` to use of a forked dependency, such as:
+```
+replace github.com/apache/incubator-yunikorn-core =>
example.com/some/forked-yunikorn
```
-make commands
-```
-Run all unit tests for the core component:
+There is no requirement to fork and create a new repository. If you do not
have a repository you can use a local checked out copy too.
+Using `replace` to use of a local directory as a dependency:
```
-make test
+replace github.com/apache/incubator-yunikorn-core =>
/User/example/local/checked-out-yunikorn
```
-Any changes made to the core code should not cause any existing tests to fail.
-
-Running the lint tool over the current code:
+and for the same dependency using a relative path:
```
-make lint
-```
-See the [coding guidelines documentation](community/coding_guidelines.md) for
more details.
+replace github.com/apache/incubator-yunikorn-core => ../checked-out-yunikorn
+```
+Note: if the `replace` directive is using a local filesystem path, then the
target must have the `go.mod` file at that location.
+
+Further details on the modules wiki: [When should I use the 'replace'
directive?](https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive).
+
+## Build the web UI
+
+Example deployments reference the [YuniKorn web
UI](https://github.com/apache/incubator-yunikorn-web).
+The YuniKorn web UI has its own specific requirements for the build. The
project has specific requirements for the build follow the steps in the README
to prepare a development environment and build how to build the projects.
+The scheduler is fully functional without the web UI.
+
+## Locally run the integrated scheduler
+
+When you have a local development environment setup you can run the scheduler
in your local kubernetes environment.
+This has been tested in a Docker desktop with docker for desktop and Minikube.
See the [environment setup guide](developer_guide/env_setup.md) for further
details.
-As a utility target you can check that all files that must have a license have
the correct license by running:
```
-make common-check-license
+make run
```
+It will connect with the kubernetes cluster using the users configured
configuration located in `$HOME/.kube/config`.
+
+You can also use the same approach to run the scheduler locally but connecting
to a remote kubernetes cluster,
+as long as the `$HOME/.kube/config` file is pointing to that remote cluster.
-## Design documents
-All design documents are located in a central location per component. The core
component design documents also contains the design documents for cross
component designs.
-[List of design documents](design/README.md)
diff --git a/docs/setup/env_setup.md b/docs/developer_guide/env_setup.md
similarity index 99%
rename from docs/setup/env_setup.md
rename to docs/developer_guide/env_setup.md
index e59bb27..1b723c2 100644
--- a/docs/setup/env_setup.md
+++ b/docs/developer_guide/env_setup.md
@@ -1,6 +1,6 @@
---
id: env_setup
-title: Environment Setup
+title: Dev Environment Setup
---
<!--
diff --git a/docs/get_started/get_started.md b/docs/get_started/get_started.md
new file mode 100644
index 0000000..ba8babf
--- /dev/null
+++ b/docs/get_started/get_started.md
@@ -0,0 +1,70 @@
+---
+id: get_started
+title: Get Started
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Before reading this guide, we assume you either have a Kubernetes cluster, or
a local Kubernetes dev environment, e.g MiniKube.
+It is also assumed that `kubectl` is on your path and properly configured.
+Follow this [guide](developer_guide/env_setup.md) on how to setup a local
Kubernetes cluster using docker-desktop.
+
+All files mentioned in this user guide are part of the
[yunikorn-k8shim](https://github.com/apache/incubator-yunikorn-k8shim)
repository.
+They are located in the
[deployments](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/scheduler)
sub directory. The command given assume that you are located in that directory.
+
+## Install
+
+The easiest way to get started is to use our Helm Charts to deploy YuniKorn on
an existing Kubernetes cluster.
+It is recommended to use Helm 3 or later versions.
+
+```shell script
+helm repo add yunikorn https://apache.github.io/incubator-yunikorn-release
+helm repo update
+kubectl create namespace yunikorn
+helm install yunikorn/yunikorn --namespace yunikorn
+```
+
+By default, the helm chart will install the scheduler, web-server and the
admission-controller in the cluster.
+When `admission-controller` is installed, it simply routes all traffic to
YuniKorn. That means the resource scheduling
+is delegated to YuniKorn. You can disable it by setting
`embedAdmissionController` flag to false during the helm install.
+
+If you don't want to use helm charts, you can find our step-by-step
+tutorial [here](developer_guide/deployment.md).
+
+## Uninstall
+
+Run the following command to uninstall YuniKorn:
+```shell script
+helm uninstall yunikorn --namespace yunikorn
+```
+
+## Access the Web UI
+
+When the scheduler is deployed, the web UI is also deployed in a container.
+Port forwarding for the web interface on the standard ports can be turned on
via:
+
+```
+kubectl port-forward svc/yunikorn-service 9889:9889 -n yunikorn
+kubectl port-forward svc/yunikorn-service 9080:9080 -n yunikorn
+```
+
+`9889` is the default port for Web UI, `9080` is the default port of
scheduler's Restful service where web UI retrieves info from.
+Once this is done, web UI will be available at: http://localhost:9889.
+
diff --git a/docs/get_started/run_non_modified_k8s_apps.md
b/docs/get_started/run_non_modified_k8s_apps.md
new file mode 100644
index 0000000..cf1090d
--- /dev/null
+++ b/docs/get_started/run_non_modified_k8s_apps.md
@@ -0,0 +1,91 @@
+---
+id: run_non_modified_k8s_apps
+title: Run Unmodified K8s Applications
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+YuniKorn is fully K8s compatible, K8s applications can run with YuniKorn
without any modification.
+All volumes, predicates are supported.
+
+## Affinity scheduling
+The scheduler supports affinity and ati affinity scheduling on kubernetes
using predicates:
+```
+kubectl create -f examples/predicates/pod-anti-affinity-example.yaml
+```
+This deployment ensures 2 pods cannot be co-located together on same node.
+If this yaml is deployed on 1 node cluster, expect 1 pod to be started and the
other pod should stay in a pending state.
+More examples on affinity and anti affinity scheduling in the predicates
section of the [README
predicates](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#predicates)
+
+## Volume examples
+There are three examples with volumes available. The NFS example does not work
on docker desktop and requires
[minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/).
+The EBS volume requires a kubernetes cluster running on AWS (EKS).
+Further instructions for the volume examples in the section of the [README
Volumes](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#volumes).
+
+CAUTION: All examples will generate an unending stream of data in a file
called `dates.txt` on the mounted volume. This could cause a disk to fill up
and execution time should be limited.
+
+### Local volume
+* create the local volume and volume claim
+```
+kubectl create -f examples/volume/local-pv.yaml
+```
+* create the pod that uses the volume
+```
+kubectl create -f examples/volume/pod-local.yaml
+```
+
+### NFS volume
+* create the NFS server
+```
+kubectl create -f nfs-server.yaml
+```
+* get the IP address for the NFS server and update the pod yaml by replacing
the existing example IP with the one returned:
+```
+kubectl get services | grep nfs-server | awk '{print $3}'
+```
+* create the pod that uses the volume
+```
+kubectl create -f pod-nfs.yaml
+```
+
+### EBS volume
+The Volume for the first two examples must be created before you can run the
examples. The `VolumeId` must be updated in the yaml files to get this to work.
+To create a volume you can use the command line or web UI:
+```
+aws ec2 create-volume --volume-type gp2 --size 10 --availability-zone us-west-1
+```
+The `VolumeId` is part of the returned information of the create command.
+
+* create the pod that uses a direct volume reference:
+```
+kubectl create -f pod-ebs-direct.yaml
+```
+* create the persistent volume (pv) and a pod that uses a persistent volume
claim (pvc) to claim the existing volume:
+```
+kubectl create -f ebs-pv.yaml
+kubectl create -f pod-ebs-exist.yaml
+```
+* create a storage class to allow dynamic provisioning and create the pod that
uses this mechanism:
+```
+kubectl create -f storage-class.yaml
+kubectl create -f pod-ebs-dynamic.yaml
+```
+Dynamic provisioning has a number of pre-requisites for it to work, see
[Dynamic Volume
Provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/)
in the kubernetes docs.
+The dynamically created volume will be automatically destroyed as soon as the
pod is stopped.
diff --git a/docs/get_started/run_sample_workloads.md
b/docs/get_started/run_sample_workloads.md
new file mode 100644
index 0000000..46b7fe9
--- /dev/null
+++ b/docs/get_started/run_sample_workloads.md
@@ -0,0 +1,56 @@
+---
+id: run_sample_workloads
+title: Run Sample Workloads
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Run application with YuniKorn scheduler
+
+Unlike the default Kubernetes scheduler, YuniKorn has `application` notion in
order to support batch workloads better.
+There are a few ways to run batch workloads with YuniKorn scheduler
+
+- Add labels `applicationId` and `queue` in pod's spec.
+- Pods that have the same applicationId will be considered as tasks from 1
application.
+
+Here is an example of the entry to add:
+```yaml
+ labels:
+ applicationId: "MyOwnApplicationId"
+ queue: "root.sandbox"
+```
+All examples provided in the next section have the labels already set. The
`queue` name must be a known queue name from the configuration.
+Unknown queue names will cause the pod to be rejected by the YuniKorn
scheduler.
+
+## Run sample applications
+
+All sample deployments can be found under `examples` directory.
+The list of all examples is in the
[README](https://github.com/apache/incubator-yunikorn-k8shim/blob/master/deployments/examples).
+Not all examples are given here. Further details can be found in that README.
+
+A single pod based on a standard nginx image:
+```
+kubectl create -f examples/nginx/nginx.yaml
+```
+A simple sleep job example:
+```
+kubectl create -f examples/sleep/sleeppods.yaml
+```
+The files for these examples can be found in the [README
nginx](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#nginx)
and the [README
sleep](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#sleep)
sections.
diff --git a/docs/get_started/run_spark.md b/docs/get_started/run_spark.md
new file mode 100644
index 0000000..2eec9c3
--- /dev/null
+++ b/docs/get_started/run_spark.md
@@ -0,0 +1,49 @@
+---
+id: run_spark
+title: Run Spark Jobs
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Kubernetes support for Apache Spark is not part of all releases. You must have
a current release of Apache Spark with Kubernetes support built in.
+
+The `examples/spark` directory contains pod template files for the Apache
Spark driver and executor, they can be used if you want to run Spark on K8s
using this scheduler.
+
+* Get latest spark from github (only latest code supports to specify pod
template), URL: https://github.com/apache/spark
+* Build spark with Kubernetes support:
+```
+mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.4 -Phive -Pkubernetes
-Phive-thriftserver -DskipTests package
+```
+* Run spark submit
+```
+spark-submit --master k8s://http://localhost:8001 --deploy-mode cluster --name
spark-pi \
+ --class org.apache.spark.examples.SparkPi \
+ --conf spark.executor.instances=1 \
+ --conf spark.kubernetes.container.image=yunikorn/spark:latest \
+ --conf spark.kubernetes.driver.podTemplateFile=examples/spark/driver.yaml \
+ --conf
spark.kubernetes.executor.podTemplateFile=examples/spark/executor.yaml \
+ local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar
+```
+
+Spark uses its own version of the application ID tag called *spark-app-id*.
This tags is required for the pods to be recognised as specific spark pods.
+* examples/spark/driver.yaml
+* examples/spark/executor.yaml
+When you run Spark on Kubernetes with pod templates, *spark-app-id* is
considered the applicationId.
+A script to run the spark application and the yaml files are in the [README
spark](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#spark)
section.
diff --git a/docs/get_started/run_tensorflow.md
b/docs/get_started/run_tensorflow.md
new file mode 100644
index 0000000..190fe4f
--- /dev/null
+++ b/docs/get_started/run_tensorflow.md
@@ -0,0 +1,38 @@
+---
+id: run_tf
+title: Run Tensorflow Jobs
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Here is an example for Tensorflow job. You must install tf-operator first.
+You can install tf-operator by applying all yaml from two website down below:
+* CRD:
https://github.com/kubeflow/manifests/tree/master/tf-training/tf-job-crds/base
+* Deployment:
https://github.com/kubeflow/manifests/tree/master/tf-training/tf-job-operator/base
+Also you can install kubeflow which can auto install tf-operator for you, URL:
https://www.kubeflow.org/docs/started/getting-started/
+
+A simple Tensorflow job example:
+
+You need to [build the
image](https://github.com/kubeflow/tf-operator/tree/master/examples/v1/dist-mnist)
which used in example yaml.
+```
+kubectl create -f examples/tfjob/tf-job-mnist.yaml
+```
+
+The file for this example can be found in the [README Tensorflow
job](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#Tensorflow-job)
section.
diff --git a/docs/get_started/user_guide.md b/docs/get_started/user_guide.md
deleted file mode 100644
index d4f6c42..0000000
--- a/docs/get_started/user_guide.md
+++ /dev/null
@@ -1,197 +0,0 @@
----
-id: user_guide
-title: User Guide
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-Before reading this guide, we assume you either have a Kubernetes cluster, or
a local Kubernetes dev environment, e.g MiniKube.
-It is also assumed that `kubectl` is on your path and properly configured.
-Follow this [guide](setup/env_setup.md) on how to setup a local Kubernetes
cluster using docker-desktop.
-
-All files mentioned in this user guide are part of the
[yunikorn-k8shim](https://github.com/apache/incubator-yunikorn-k8shim)
repository.
-They are located in the
[deployments](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/scheduler)
sub directory. The command given assume that you are located in that directory.
-
-## Quick Start
-
-The easiest way to get started is to use our Helm Charts to deploy YuniKorn on
an existing Kubernetes cluster. Recommand to use Helm 3 or later versions.
-
-```shell script
-helm repo add yunikorn https://apache.github.io/incubator-yunikorn-release
-helm repo update
-helm install yunikorn/yunikorn
-```
-
-it will firstly create a `configmap` where stores YuniKorn configuration, and
then deploy YuniKorn scheduler
-and web UI containers in a pod as a `deployment` in the `default` namespace.
If you want to deploy YuniKorn to another namespace, you can do following:
-
-```shell script
-kubectl create namespace yunikorn-ns
-helm install yunikorn yunikorn/yunikorn --namespace yunikorn-ns
-```
-
-Uninstall yunikorn:
-```shell script
-helm uninstall yunikorn --namespace yunikorn-ns
-```
-
-If you don't want to use helm charts, you can find our step-by-step
-tutorial [here](setup/deployment.md).
-
-## Run workloads with YuniKorn Scheduler
-
-Unlike default Kubernetes scheduler, YuniKorn has `application` notion in
order to support batch workloads better.
-There are a few ways to run batch workloads with YuniKorn scheduler
-
-- Add labels `applicationId` and `queue` in pod's spec.
-- Pods that have the same applicationId will be considered as tasks from 1
application.
-
-Here is an example of the entry to add:
-```yaml
- labels:
- applicationId: "MyOwnApplicationId"
- queue: "root.sandbox"
-```
-All examples provided in the next section have the labels already set. The
`queue` name must be a known queue name from the configuration.
-Unknown queue names will cause the pod to be rejected by the YuniKorn
scheduler.
-
-### Running simple sample applications
-
-All sample deployments can be found under `examples` directory.
-The list of all examples is in the
[README](https://github.com/apache/incubator-yunikorn-k8shim/blob/master/deployments/examples).
-Not all examples are given here. Further details can be found in that README.
-
-A single pod based on a standard nginx image:
-```
-kubectl create -f examples/nginx/nginx.yaml
-```
-A simple sleep job example:
-```
-kubectl create -f examples/sleep/sleeppods.yaml
-```
-The files for these examples can be found in the [README
nginx](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#nginx)
and the [README
sleep](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#sleep)
sections.
-
-### Running a spark application
-Kubernetes support for Apache Spark is not part of all releases. You must have
a current release of Apache Spark with Kubernetes support built in.
-
-The `examples/spark` directory contains pod template files for the Apache
Spark driver and executor, they can be used if you want to run Spark on K8s
using this scheduler.
-
-* Get latest spark from github (only latest code supports to specify pod
template), URL: https://github.com/apache/spark
-* Build spark with Kubernetes support:
-```
-mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.4 -Phive -Pkubernetes
-Phive-thriftserver -DskipTests package
-```
-* Run spark submit
-```
-spark-submit --master k8s://http://localhost:8001 --deploy-mode cluster --name
spark-pi \
- --class org.apache.spark.examples.SparkPi \
- --conf spark.executor.instances=1 \
- --conf spark.kubernetes.container.image=yunikorn/spark:latest \
- --conf spark.kubernetes.driver.podTemplateFile=examples/spark/driver.yaml \
- --conf
spark.kubernetes.executor.podTemplateFile=examples/spark/executor.yaml \
- local:///opt/spark/examples/jars/spark-examples_2.12-3.0.0-SNAPSHOT.jar
-```
-
-Spark uses its own version of the application ID tag called *spark-app-id*.
This tags is required for the pods to be recognised as specific spark pods.
-* examples/spark/driver.yaml
-* examples/spark/executor.yaml
-When you run Spark on Kubernetes with pod templates, *spark-app-id* is
considered the applicationId.
-A script to run the spark application and the yaml files are in the [README
spark](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#spark)
section.
-
-### Running a simple Tensorflow job
-There is an example for Tensorflow job. You must install tf-operator first.
-You can install tf-operator by applying all yaml from two website down below:
-* CRD:
https://github.com/kubeflow/manifests/tree/master/tf-training/tf-job-crds/base
-* Deployment:
https://github.com/kubeflow/manifests/tree/master/tf-training/tf-job-operator/base
-Also you can install kubeflow which can auto install tf-operator for you, URL:
https://www.kubeflow.org/docs/started/getting-started/
-
-A simple Tensorflow job example:
-
-You need to [build the
image](https://github.com/kubeflow/tf-operator/tree/master/examples/v1/dist-mnist)
which used in example yaml.
-```
-kubectl create -f examples/tfjob/tf-job-mnist.yaml
-```
-
-The file for this example can be found in the [README Tensorflow
job](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#Tensorflow-job)
section.
-
-### Affinity scheduling
-The scheduler supports affinity and ati affinity scheduling on kubernetes
using predicates:
-```
-kubectl create -f examples/predicates/pod-anti-affinity-example.yaml
-```
-This deployment ensures 2 pods cannot be co-located together on same node.
-If this yaml is deployed on 1 node cluster, expect 1 pod to be started and the
other pod should stay in a pending state.
-More examples on affinity and anti affinity scheduling in the predicates
section of the [README
predicates](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#predicates)
-
-### Volume examples
-There are three examples with volumes available. The NFS example does not work
on docker desktop and requires
[minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/).
-The EBS volume requires a kubernetes cluster running on AWS (EKS).
-Further instructions for the volume examples in the section of the [README
Volumes](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/deployments/examples#volumes).
-
-CAUTION: All examples will generate an unending stream of data in a file
called `dates.txt` on the mounted volume. This could cause a disk to fill up
and execution time should be limited.
-
-#### Local volume
-* create the local volume and volume claim
-```
-kubectl create -f examples/volume/local-pv.yaml
-```
-* create the pod that uses the volume
-```
-kubectl create -f examples/volume/pod-local.yaml
-```
-
-#### NFS volume
-* create the NFS server
-```
-kubectl create -f nfs-server.yaml
-```
-* get the IP address for the NFS server and update the pod yaml by replacing
the existing example IP with the one returned:
-```
-kubectl get services | grep nfs-server | awk '{print $3}'
-```
-* create the pod that uses the volume
-```
-kubectl create -f pod-nfs.yaml
-```
-
-#### EBS volume
-The Volume for the first two examples must be created before you can run the
examples. The `VolumeId` must be updated in the yaml files to get this to work.
-To create a volume you can use the command line or web UI:
-```
-aws ec2 create-volume --volume-type gp2 --size 10 --availability-zone us-west-1
-```
-The `VolumeId` is part of the returned information of the create command.
-
-* create the pod that uses a direct volume reference:
-```
-kubectl create -f pod-ebs-direct.yaml
-```
-* create the persistent volume (pv) and a pod that uses a persistent volume
claim (pvc) to claim the existing volume:
-```
-kubectl create -f ebs-pv.yaml
-kubectl create -f pod-ebs-exist.yaml
-```
-* create a storage class to allow dynamic provisioning and create the pod that
uses this mechanism:
-```
-kubectl create -f storage-class.yaml
-kubectl create -f pod-ebs-dynamic.yaml
-```
-Dynamic provisioning has a number of pre-requisites for it to work, see
[Dynamic Volume
Provisioning](https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/)
in the kubernetes docs.
-The dynamically created volume will be automatically destroyed as soon as the
pod is stopped.
diff --git a/docs/performance/evaluate_perf_function_with_kubemark.md
b/docs/performance/evaluate_perf_function_with_kubemark.md
index 07abced..e4a6a6c 100644
--- a/docs/performance/evaluate_perf_function_with_kubemark.md
+++ b/docs/performance/evaluate_perf_function_with_kubemark.md
@@ -1,6 +1,6 @@
---
id: evaluate_perf_function_with_kubemark
-title: Perf With Kubemark
+title: Evaluate YuniKorn function & performance with Kubemark
---
<!--
@@ -22,8 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# Evaluate YuniKorn function & performance with Kubemark
-
All the following tests are done with
[Kubemark](https://github.com/kubernetes/kubernetes/blob/release-1.3/docs/devel/kubemark-guide.md#starting-a-kubemark-cluster),
a tool helps us to simulate large K8s cluster and run experimental workloads.
There were 18 bare-metal servers being used to simulate 2000/4000 nodes for
these tests.
diff --git a/docs/performance/metrics.md b/docs/performance/metrics.md
index 6eea6be..2b37c1c 100644
--- a/docs/performance/metrics.md
+++ b/docs/performance/metrics.md
@@ -1,6 +1,6 @@
---
id: metrics
-title: Metrics
+title: Scheduler Metrics
---
<!--
@@ -22,8 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# YuniKorn Metrics
-
YuniKorn leverages [Prometheus](https://prometheus.io/) to record metrics. The
metrics system keeps tracking of
scheduler's critical execution paths, to reveal potential performance
bottlenecks. Currently, there are two categories
for these metrics:
diff --git a/docs/performance/profiling.md b/docs/performance/profiling.md
index e224d57..dbbff84 100644
--- a/docs/performance/profiling.md
+++ b/docs/performance/profiling.md
@@ -22,8 +22,6 @@ specific language governing permissions and limitations
under the License.
-->
-# Profiling Scheduler
-
Use [pprof](https://github.com/google/pprof) to do CPU, Memory profiling can
help you understand the runtime status of YuniKorn scheduler. Profiling
instruments have been
added to YuniKorn rest service, we can easily retrieve and analyze them from
HTTP
endpoints.
diff --git a/docs/setup/build_local.md b/docs/setup/build_local.md
deleted file mode 100644
index 4e5e556..0000000
--- a/docs/setup/build_local.md
+++ /dev/null
@@ -1,74 +0,0 @@
----
-id: build_local
-title: Build Local
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-# Building with local dependencies
-
-The YuniKorn project has four repositories three of those repositories have a
dependency at the go level.
-These dependencies are part of the go modules and point to the github
repositories.
-During development it can be required to break the dependency on the committed
version from github.
-This requires making changes in the module file to allow loading a local copy
or a forked copy from a different repository.
-
-## Affected repositories
-The following dependencies exist between the repositories:
-
-| repository| depends on |
-| --- | --- |
-| yunikorn-core | yunikorn-scheduler-interface |
-| yunikorn-k8shim | yunikorn-scheduler-interface, yunikorn-core |
-| yunikorn-scheduler-interface | none |
-| yunikorn-web | yunikorn-core |
-
-The `yunikorn-web` repository has no direct go dependency on the other
repositories. However any change to the `yunikorn-core` webservices can affect
the web interface.
-
-## Making local changes
-
-To make sure that the local changes will not break other parts of the build
you should run:
-- A full build `make` (build target depends on the repository)
-- A full unit test run `make test`
-
-Any test failures should be fixed before proceeding.
-
-## Updating dependencies
-
-The simplest way is to use the `replace` directive in the module file. The
`replace` directive allows you to override the import path with a new (local)
path.
-There is no need to change any of the imports in the source code. The change
must be made in the `go.mod` file of the repository that has the dependency.
-
-Using `replace` to use of a forked dependency, such as:
-```
-replace github.com/apache/incubator-yunikorn-core =>
example.com/some/forked-yunikorn
-```
-
-There is no requirement to fork and create a new repository. If you do not
have a repository you can use a local checked out copy too.
-Using `replace` to use of a local directory as a dependency:
-```
-replace github.com/apache/incubator-yunikorn-core =>
/User/example/local/checked-out-yunikorn
-```
-and for the same dependency using a relative path:
-```
-replace github.com/apache/incubator-yunikorn-core => ../checked-out-yunikorn
-```
-Note: if the `replace` directive is using a local filesystem path, then the
target must have the `go.mod` file at that location.
-
-
-Further details on the modules wiki: [When should I use the 'replace'
directive?](https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive).
\ No newline at end of file
diff --git a/docs/setup/deployment.md b/docs/setup/deployment.md
deleted file mode 100644
index a737f60..0000000
--- a/docs/setup/deployment.md
+++ /dev/null
@@ -1,81 +0,0 @@
----
-id: deployment
-title: Deployment Guide
----
-
-<!--
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
--->
-
-The easiest way to deploy YuniKorn is to leverage our [helm
charts](https://github.com/apache/incubator-yunikorn-k8shim/tree/master/helm-charts),
-you can find the guide [here](get_started/user_guide.md). However, if you want
to explore the deployment procedure
-step by step, here are the instructions.
-
-## Setup RBAC
-
-The first step is to create the RBAC role for the scheduler, see
[yunikorn-rbac.yaml](https://github.com/apache/incubator-yunikorn-k8shim/blob/master/deployments/scheduler/yunikorn-rbac.yaml)
-```
-kubectl create -f scheduler/yunikorn-rbac.yaml
-```
-The role is a requirement on the current versions of kubernetes.
-
-## Create the ConfigMap
-
-YuniKorn loads its configuration from a K8s configmap, so it is required to
create the configmap before launching the scheduler.
-
-- download a sample configuration file:
-```
-curl -o queues.yaml
https://raw.githubusercontent.com/apache/incubator-yunikorn-k8shim/master/conf/queues.yaml
-```
-- create ConfigMap in kubernetes:
-```
-kubectl create configmap yunikorn-configs --from-file=queues.yaml
-```
-- check if the ConfigMap was created correctly:
-```
-kubectl describe configmaps yunikorn-configs
-```
-
-For more information about how to manage scheduler's configuration via
configmap, see more from [here](setup/configure_scheduler.md).
-
-## Deploy the scheduler on k8s
-
-Before you can deploy the scheduler the image for the scheduler and the web
interface must be build with the appropriate tags.
-The procedure on how to build the images is explained in the [build
document](a_developer_guide.md). See
[scheduler.yaml](https://github.com/apache/incubator-yunikorn-k8shim/blob/master/deployments/scheduler/scheduler.yaml)
-```
-kubectl create -f scheduler/scheduler.yaml
-```
-The deployment will run 2 containers from your pre-built docker images in 1
pod,
-
-* yunikorn-scheduler-core (yunikorn scheduler core and shim for K8s)
-* yunikorn-scheduler-web (web UI)
-
-The pod is deployed as a customized scheduler, it will take the responsibility
to schedule pods which explicitly specifies `schedulerName: yunikorn` in pod's
spec.
-
-## Access to the web UI
-
-When the scheduler is deployed, the web UI is also deployed in a container.
-Port forwarding for the web interface on the standard ports can be turned on
via:
-
-```
-POD=`kubectl get pod -l app=yunikorn -o jsonpath="{.items[0].metadata.name}"`
&& \
-kubectl port-forward ${POD} 9889 9080
-```
-
-`9889` is the default port for Web UI, `9080` is the default port of
scheduler's Restful service where web UI retrieves info from.
-Once this is done, web UI will be available at: http://localhost:9889.
diff --git a/docusaurus.config.js b/docusaurus.config.js
index de269e7..b59b4c8 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -38,7 +38,7 @@ module.exports = {
'@docusaurus/preset-classic',
{
docs: {
- homePageId: 'get_started/user_guide',
+ homePageId: 'get_started/get_started',
path: 'docs',
sidebarPath: require.resolve('./sidebars.js'),
},
diff --git a/local-build.sh b/local-build.sh
new file mode 100644
index 0000000..f2c49fd
--- /dev/null
+++ b/local-build.sh
@@ -0,0 +1,75 @@
+# this script must be running from this dir
+
+
+function build() {
+ # build local docker image
+ cat <<EOF >.dockerfile.tmp
+FROM node:12.18.0
+ADD . /incubator-yunikorn-site
+WORKDIR /incubator-yunikorn-site
+EOF
+
+ docker build -t yunikorn/yunikorn-website:latest -f .dockerfile.tmp .
+ [ "$?" -ne 0 ] && echo "docker image build failed" && rm -rf .dockerfile.tmp
&& exit 1
+ rm -rf .dockerfile.tmp
+}
+
+function run() {
+ # run docker container
+ # mount the repo root to the container working dir,
+ # so that changes made in the repo can trigger the web-server auto-update
+ docker run -it --name yunikorn-site-local -d \
+ -p 3000:3000 \
+ -v $PWD:/incubator-yunikorn-site \
+ -v $PWD/yunikorn-site:/incubator-yunikorn-site/build \
+ yunikorn/yunikorn-website:latest
+ [ "$?" -ne 0 ] && echo "run local docker image failed" && exit 1
+
+ # install dependency in docker container, run build locally
+ docker exec -it yunikorn-site-local /bin/bash -c "yarn install"
+ [ "$?" -ne 0 ] && echo "yarn install failed" && exit 1
+
+ docker exec -it yunikorn-site-local /bin/bash -c "yarn build"
+ [ "$?" -ne 0 ] && echo "yarn build failed" && exit 1
+
+ docker exec -it yunikorn-site-local /bin/bash -c "yarn start --host 0.0.0.0"
+ [ "$?" -ne 0 ] && echo "start web-server failed" && exit 1
+}
+
+function print_usage() {
+ cat <<EOF
+Usage: $(basename "$0") run | logs | help
+ run build the website, and launch the server locally in dev mode.
+ logs dump server logs.
+
+Description:
+ This command builds and luanches the website server locally, the server is
built with the latest content within
+ this repo. When changes are made in this directory, it can be automatically
refreshed to the local server. Be aware
+ that some of the changes require the restart of the server, just run the
command again.
+
+Have fun!!!
+EOF
+}
+
+if [ $# -ne 1 ]; then
+ print_usage
+ exit 1
+fi
+
+opt=$1
+if [ "$opt" == "run" ]; then
+ docker stop yunikorn-site-local &>/dev/null
+ docker rm -f yunikorn-site-local &>/dev/null
+ docker rmi yunikorn/yunikorn-website:latest &>/dev/null
+
+ build
+ run
+elif [ "$opt" == "logs" ]; then
+ docker logs yunikorn-site-local
+elif [ "$opt" == "help" ]; then
+ print_usage
+ exit 0
+else
+ print_usage
+ exit 0
+fi
diff --git a/sidebars.js b/sidebars.js
index 5bcb52c..1597984 100644
--- a/sidebars.js
+++ b/sidebars.js
@@ -15,22 +15,24 @@
*/
module.exports = {
-
docs: {
'Get Started': [
- 'get_started/user_guide',
- 'get_started/developer_guide'
+ 'get_started/get_started',
+ 'get_started/run_sample_workloads',
+ 'get_started/run_non_modified_k8s_apps',
+ 'get_started/run_spark',
+ 'get_started/run_tf'
],
- 'Setup': [
- 'setup/build_local',
- 'setup/configure_scheduler',
- 'setup/deployment',
- 'setup/env_setup'
+ 'Developer Guide': [
+ 'developer_guide/developer_guide',
+ 'developer_guide/env_setup',
+ 'developer_guide/deployment'
],
'Configuration': [
'configuration/queue_config',
'configuration/placement_rules',
- 'configuration/acls'
+ 'configuration/acls',
+ 'configuration/sorting_policies'
],
'Performance': [
'performance/evaluate_perf_function_with_kubemark',
@@ -38,12 +40,15 @@ module.exports = {
'performance/profiling'
],
'Design': [
- 'design/design',
+ 'design/architecture',
+ 'design/scheduler_core_design',
'design/cross_queue_preemption',
'design/namespace_resource_quota',
'design/pluggable_app_management',
'design/resilience',
- 'design/scheduler_configuration'
+ 'design/scheduler_configuration',
+ 'design/state_aware_scheduling',
+ 'design/scheduler_object_states'
],
},
};
diff --git a/static/img/application-state.png b/static/img/application-state.png
new file mode 100644
index 0000000..fc53f3f
Binary files /dev/null and b/static/img/application-state.png differ
diff --git a/static/img/fifo-state-example.png
b/static/img/fifo-state-example.png
new file mode 100644
index 0000000..ca04c17
Binary files /dev/null and b/static/img/fifo-state-example.png differ
diff --git a/static/img/object-state.png b/static/img/object-state.png
new file mode 100644
index 0000000..9baca07
Binary files /dev/null and b/static/img/object-state.png differ
