This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new b4c67cd SUBMARINE-365. Exposing the Submarine in the Kind
b4c67cd is described below
commit b4c67cdd664cbe9b9bab90836f343bfcd6a7b789
Author: Xun Liu <[email protected]>
AuthorDate: Wed Jan 29 16:03:25 2020 +0800
SUBMARINE-365. Exposing the Submarine in the Kind
### What is this PR for?
submarine runs on the private network of k8s,
We expose the port of the submarine workbench through k8s ingress nginx,
This allows users to access outside the k8s cluster or directly.
#### Execution kind-cluster-build.sh
1. Create a new kind cluster, If you already have kind cluster, you need
deleted before.
```
kind delete cluster
./submarine-cloud/hack/kind-cluster-build.sh
```
2. Do you want to run submarine in kind cluster now? [y/n] `y`
+ Automation pull `submarine-operation` & `submarine-server` docker image
from https://hub.docker.com/r/apache/submarine/tags
+ Automation push `submarine-operation` & `submarine-server` docker image
in Kind cluster
+ launch `submarine-operation` & `submarine-server` by yaml
+ You can open your browser and access the submarine workbench at
`http://127.0.0.1/`
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/SUBMARINE-365
### How should this be tested?
* https://travis-ci.org/liuxunorg/submarine/builds/643354689
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Xun Liu <[email protected]>
Closes #168 from liuxunorg/SUBMARINE-365 and squashes the following commits:
4e941a6 [Xun Liu] SUBMARINE-365. Exposing the Submarine Workbench in the
Kind
---
submarine-cloud/hack/ingress/mandatory.yaml | 307 +++++++++++++++++++++
submarine-cloud/hack/ingress/service-nodeport.yaml | 38 +++
submarine-cloud/hack/kind-cluster-build.sh | 55 +++-
.../manifests/submarine-cluster/deployment.yaml | 17 --
.../manifests/submarine-cluster/service.yaml | 45 +++
5 files changed, 438 insertions(+), 24 deletions(-)
diff --git a/submarine-cloud/hack/ingress/mandatory.yaml
b/submarine-cloud/hack/ingress/mandatory.yaml
new file mode 100644
index 0000000..699d13a
--- /dev/null
+++ b/submarine-cloud/hack/ingress/mandatory.yaml
@@ -0,0 +1,307 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+ name: nginx-configuration
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+ name: tcp-services
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+ name: udp-services
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+ name: nginx-ingress-serviceaccount
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+ name: nginx-ingress-clusterrole
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+rules:
+ - apiGroups:
+ - ""
+ resources:
+ - configmaps
+ - endpoints
+ - nodes
+ - pods
+ - secrets
+ verbs:
+ - list
+ - watch
+ - apiGroups:
+ - ""
+ resources:
+ - nodes
+ verbs:
+ - get
+ - apiGroups:
+ - ""
+ resources:
+ - services
+ verbs:
+ - get
+ - list
+ - watch
+ - apiGroups:
+ - ""
+ resources:
+ - events
+ verbs:
+ - create
+ - patch
+ - apiGroups:
+ - "extensions"
+ - "networking.k8s.io"
+ resources:
+ - ingresses
+ verbs:
+ - get
+ - list
+ - watch
+ - apiGroups:
+ - "extensions"
+ - "networking.k8s.io"
+ resources:
+ - ingresses/status
+ verbs:
+ - update
+
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: Role
+metadata:
+ name: nginx-ingress-role
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+rules:
+ - apiGroups:
+ - ""
+ resources:
+ - configmaps
+ - pods
+ - secrets
+ - namespaces
+ verbs:
+ - get
+ - apiGroups:
+ - ""
+ resources:
+ - configmaps
+ resourceNames:
+ # Defaults to "<election-id>-<ingress-class>"
+ # Here: "<ingress-controller-leader>-<nginx>"
+ # This has to be adapted if you change either parameter
+ # when launching the nginx-ingress-controller.
+ - "ingress-controller-leader-nginx"
+ verbs:
+ - get
+ - update
+ - apiGroups:
+ - ""
+ resources:
+ - configmaps
+ verbs:
+ - create
+ - apiGroups:
+ - ""
+ resources:
+ - endpoints
+ verbs:
+ - get
+
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: RoleBinding
+metadata:
+ name: nginx-ingress-role-nisa-binding
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: Role
+ name: nginx-ingress-role
+subjects:
+ - kind: ServiceAccount
+ name: nginx-ingress-serviceaccount
+ namespace: ingress-nginx
+
+---
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+ name: nginx-ingress-clusterrole-nisa-binding
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: ClusterRole
+ name: nginx-ingress-clusterrole
+subjects:
+ - kind: ServiceAccount
+ name: nginx-ingress-serviceaccount
+ namespace: ingress-nginx
+
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: nginx-ingress-controller
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+spec:
+ replicas: 1
+ selector:
+ matchLabels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+ annotations:
+ prometheus.io/port: "10254"
+ prometheus.io/scrape: "true"
+ spec:
+ # wait up to five minutes for the drain of connections
+ terminationGracePeriodSeconds: 300
+ serviceAccountName: nginx-ingress-serviceaccount
+ nodeSelector:
+ beta.kubernetes.io/os: linux
+ containers:
+ - name: nginx-ingress-controller
+ image:
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master
+ args:
+ - /nginx-ingress-controller
+ - --configmap=$(POD_NAMESPACE)/nginx-configuration
+ - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
+ - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
+ - --publish-service=$(POD_NAMESPACE)/ingress-nginx
+ - --annotations-prefix=nginx.ingress.kubernetes.io
+ securityContext:
+ allowPrivilegeEscalation: true
+ capabilities:
+ drop:
+ - ALL
+ add:
+ - NET_BIND_SERVICE
+ # www-data -> 101
+ runAsUser: 101
+ env:
+ - name: POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+ - name: POD_NAMESPACE
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.namespace
+ ports:
+ - name: http
+ containerPort: 80
+ protocol: TCP
+ - name: https
+ containerPort: 443
+ protocol: TCP
+ livenessProbe:
+ failureThreshold: 3
+ httpGet:
+ path: /healthz
+ port: 10254
+ scheme: HTTP
+ initialDelaySeconds: 10
+ periodSeconds: 10
+ successThreshold: 1
+ timeoutSeconds: 10
+ readinessProbe:
+ failureThreshold: 3
+ httpGet:
+ path: /healthz
+ port: 10254
+ scheme: HTTP
+ periodSeconds: 10
+ successThreshold: 1
+ timeoutSeconds: 10
+ lifecycle:
+ preStop:
+ exec:
+ command:
+ - /wait-shutdown
+
+---
+apiVersion: v1
+kind: LimitRange
+metadata:
+ name: ingress-nginx
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+spec:
+ limits:
+ - default:
+ min:
+ memory: 90Mi
+ cpu: 100m
+ type: Container
diff --git a/submarine-cloud/hack/ingress/service-nodeport.yaml
b/submarine-cloud/hack/ingress/service-nodeport.yaml
new file mode 100644
index 0000000..84331e9
--- /dev/null
+++ b/submarine-cloud/hack/ingress/service-nodeport.yaml
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: v1
+kind: Service
+metadata:
+ name: ingress-nginx
+ namespace: ingress-nginx
+ labels:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
+spec:
+ type: NodePort
+ ports:
+ - name: http
+ port: 80
+ targetPort: 80
+ protocol: TCP
+ - name: https
+ port: 443
+ targetPort: 443
+ protocol: TCP
+ selector:
+ app.kubernetes.io/name: ingress-nginx
+ app.kubernetes.io/part-of: ingress-nginx
diff --git a/submarine-cloud/hack/kind-cluster-build.sh
b/submarine-cloud/hack/kind-cluster-build.sh
index 5e01fed..0e8b0db 100755
--- a/submarine-cloud/hack/kind-cluster-build.sh
+++ b/submarine-cloud/hack/kind-cluster-build.sh
@@ -76,7 +76,7 @@ esac
done
clusterName=${clusterName:-kind}
-nodeNum=${nodeNum:-1}
+nodeNum=${nodeNum:-3}
k8sVersion=${k8sVersion:-v1.12.8}
volumeNum=${volumeNum:-1}
@@ -120,14 +120,27 @@ configFile=${workDir}/kind-config.yaml
cat <<EOF > ${configFile}
kind: Cluster
-apiVersion: kind.sigs.k8s.io/v1alpha3
+apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
+ kubeadmConfigPatches:
+ - |
+ kind: InitConfiguration
+ nodeRegistration:
+ kubeletExtraArgs:
+ node-labels: "ingress-ready=true"
+ authorization-mode: "AlwaysAllow"
extraPortMappings:
- containerPort: 5000
hostPort: 5000
listenAddress: 127.0.0.1
protocol: TCP
+ - containerPort: 80
+ hostPort: 80
+ protocol: TCP
+ - containerPort: 443
+ hostPort: 443
+ protocol: TCP
EOF
for ((i=0;i<${nodeNum};i++))
@@ -149,7 +162,7 @@ done
echo "start to create k8s cluster"
kind create cluster --config ${configFile} --image kindest/node:${k8sVersion}
--name=${clusterName}
-export KUBECONFIG="$(kind get kubeconfig-path --name=${clusterName})"
+export KUBECONFIG="${HOME}/.kube/kind-config-${clusterName}"
echo "deploy docker registry in kind"
registryNode=${clusterName}-control-plane
@@ -226,15 +239,26 @@ spec:
EOF
kubectl apply -f ${registryFile}
-echo "init submarine env"
-kubectl create ns submarine-e2e
+# https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx
+echo "setting up ingress on a kind cluster."
+
+# load ingress denpendence docker-image into kind
+docker pull registry:2
+kind load docker-image registry:2
+
+docker pull
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master
+kind load docker-image
quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master
+
+kubectl apply -f $ROOT/hack/ingress/mandatory.yaml
+kubectl apply -f $ROOT/hack/ingress/service-nodeport.yaml
+kubectl patch deployments -n ingress-nginx nginx-ingress-controller -p
'{"spec":{"template":{"spec":{"containers":[{"name":"nginx-ingress-controller","ports":[{"containerPort":80,"hostPort":80},{"containerPort":443,"hostPort":443}]}],"nodeSelector":{"ingress-ready":"true"},"tolerations":[{"key":"node-role.kubernetes.io/master","operator":"Equal","effect":"NoSchedule"}]}}}}'
echo "############# success create cluster:[${clusterName}] #############"
echo "To start using your cluster, run:"
-echo " export KUBECONFIG=$(kind get kubeconfig-path --name=${clusterName})"
+echo " export KUBECONFIG=\"${HOME}/.kube/kind-config-${clusterName}\""
echo ""
-echo <<EOF
+cat <<EOF
NOTE: In kind, nodes run docker network and cannot access host network.
If you configured local HTTP proxy in your docker, images may cannot be pulled
because http proxy is inaccessible.
@@ -243,3 +267,20 @@ If you cannot remove http proxy settings, you can either
whitelist image
domains in NO_PROXY environment or use 'docker pull <image> && kind load
docker-image <image>' command to load images into nodes.
EOF
+
+# Run submarine in kind cluster
+echo -n "Do you want to run submarine in kind cluster now? [y/n]"
+read myselect
+if [[ "$myselect" == "y" || "$myselect" == "Y" ]]; then
+ docker pull apache/submarine:operator-0.3.0-SNAPSHOT
+ kind load docker-image apache/submarine:operator-0.3.0-SNAPSHOT
+ kubectl apply -f $ROOT/manifests/submarine-operator/
+
+ docker pull apache/submarine:server-0.3.0-SNAPSHOT
+ kind load docker-image apache/submarine:server-0.3.0-SNAPSHOT
+ kubectl apply -f $ROOT/manifests/submarine-cluster/
+
+ cat <<EOF
+NOTE: You can open your browser and access the submarine workbench at
http://127.0.0.1/
+EOF
+fi
diff --git a/submarine-cloud/manifests/submarine-cluster/deployment.yaml
b/submarine-cloud/manifests/submarine-cluster/deployment.yaml
index a8dca24..99634eb 100644
--- a/submarine-cloud/manifests/submarine-cluster/deployment.yaml
+++ b/submarine-cloud/manifests/submarine-cluster/deployment.yaml
@@ -68,20 +68,3 @@ spec:
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
-
----
-apiVersion: v1
-kind: Service
-metadata:
- name: submarine-svc
-spec:
- ports:
- - name: submarine-test-svc
- port: 8080
- targetPort: 8080
- nodePort: 30001
- selector:
- app: submarine-test
- type: NodePort
-
-# kubectl port-forward svc/submarine-svc 8080:8080 --address 0.0.0.0
diff --git a/submarine-cloud/manifests/submarine-cluster/service.yaml
b/submarine-cloud/manifests/submarine-cluster/service.yaml
new file mode 100644
index 0000000..7697101
--- /dev/null
+++ b/submarine-cloud/manifests/submarine-cluster/service.yaml
@@ -0,0 +1,45 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+kind: Service
+apiVersion: v1
+metadata:
+ name: submarine-svc
+spec:
+ selector:
+ app: cluster-test
+ ports:
+ - port: 8080
+
+---
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+ name: submarine-ingress
+ annotations:
+ ingress.kubernetes.io/rewrite-target: /
+spec:
+ rules:
+ - http:
+ paths:
+ - path: /
+ backend:
+ serviceName: submarine-svc
+ servicePort: 8080
+
+---
+# You can also access the submarine workbench via port-forward
+# kubectl port-forward svc/submarine-svc 18080:8080 --address 0.0.0.0
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
