Hello Michael,
onap-dcaegen2-dcae pod/pods are coming up after restarting the containers (as you see the below query raised by Viji). We want to ask one basic Question here: We are doing VNF certification testing for a VNF ( *vFirewall* ) being created using ONAP and have following below use cases required to demonstrate using Robot Platform. - vFirewall VNF policy update - Service Function Chaining (SFC) using the vFirewall VNF - Auto-Scaling of vFirewall VNF *Query* – Please do let me know ONAP components which are required/Essential to demonstrate the above use cases of a vFirewall VNF. Thank you. Regards, Pranjal *From:* Viji J. *Sent:* Thursday, March 28, 2019 7:34 PM *To:* Michael O'Brien <[email protected]>; Viji Jeyaraman < [email protected]>; [email protected] *Cc:* Pranjal Sharma <[email protected]>; [email protected] *Subject:* RE: Onap installation steps query Hi Michael, Thanks for the input. We were able bring up 1(ranchermaster)+14 VMs successfully. OOM repo branch *Casablanca*. We were deploying onap each component one by one. While deploying aaf pods, we see onap-aaf-aaf-sshsm-testca is in error state. onap-aaf-aaf-sshsm-testca initially was in podintializing state. Then, it went to error state and few more container of it showed. root@rancheroom:~# *kubectl get pods --namespace onap | grep aaf* onap-aaf-aaf-cm-5999cf85df-49fns 1/1 Running 0 2h onap-aaf-aaf-cs-5c7586f97d-fk2l8 1/1 Running 0 34m onap-aaf-aaf-fs-67dc4f5fd4-vzkkk 1/1 Running 0 2h onap-aaf-aaf-gui-6558cc4d6b-srdkp 1/1 Running 0 34m onap-aaf-aaf-hello-5fdc4c766f-xdrgq 1/1 Running 0 2h onap-aaf-aaf-locate-654c764588-4qqds 1/1 Running 0 2h onap-aaf-aaf-oauth-76c559cc48-85rjq 1/1 Running 6 34m onap-aaf-aaf-service-7f54674f88-g6mc4 1/1 Running 6 34m onap-aaf-aaf-sms-8647967f-zbns8 1/1 Running 0 2h onap-aaf-aaf-sms-preload-rk9sb 0/1 Completed 0 2h onap-aaf-aaf-sms-quorumclient-0 1/1 Running 0 2h onap-aaf-aaf-sms-quorumclient-1 1/1 Running 0 2h onap-aaf-aaf-sms-quorumclient-2 1/1 Running 0 2h onap-aaf-aaf-sms-vault-0 2/2 Running 1 2h onap-aaf-aaf-sshsm-distcenter-2qb8p 0/1 Completed 0 2h onap-aaf-aaf-sshsm-testca-4tj6l 0/1 Error 0 1h onap-aaf-aaf-sshsm-testca-7nf74 0/1 Error 0 1h onap-aaf-aaf-sshsm-testca-8g2nl 0/1 Error 0 1h onap-aaf-aaf-sshsm-testca-fm7jj 0/1 Error 0 2h onap-aaf-aaf-sshsm-testca-kh25n 0/1 Error 0 1h onap-aaf-aaf-sshsm-testca-s6d6w 0/1 Error 0 1h The following is the onap-aaf-aaf-sshsm-testca container logs. prepareHWPlugin() called Couldn't open the directory HW plugin NOT available to use ! prepareHWPlugin() called Couldn't open the directory HW plugin NOT available to use ! The token has been initialized and is reassigned to slot 852610267 The slot ID used is 852610267 TPM hardware unavailable. Using SoftHSM implementation cat: passphrase: No such file or directory gpg: directory `/root/.gnupg' created gpg: new configuration file `/root/.gnupg/gpg.conf' created gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this run gpg: keyring `/root/.gnupg/secring.gpg' created gpg: keyring `/root/.gnupg/pubring.gpg' created gpg: can't open `privkey.pem.gpg' We see the issue related to this in following links. https://jira.onap.org/browse/AAF-376?src=confmacro https://gerrit.onap.org/r/#/c/56423/5/bin/distcenter/create_ca.sh We are following below order. As of now, we proceeded to deploy dmaap,msb. In dcae, 2 container are not up. EPLOY_ORDER_POD_NAME_ARRAY=('robot consul aaf dmaap dcaegen2 msb aai esr multicloud oof so sdc sdnc vid policy portal log vfc uui vnfsdk appc clamp cli pomba vvp contrib sniro-emulator') root@rancheroom:~# *kubectl get pods --namespace onap | grep dcae* onap-dcaegen2-dcae-bootstrap-8564488869-jszcj 0/1 Init:0/1 5 1h onap-dcaegen2-dcae-cloudify-manager-548c847788-qdvf7 0/1 ImagePullBackOff 0 1h onap-dcaegen2-dcae-db-0 1/1 Running 0 1h Can you please let us know the workaround. Thanks, Viji J On Thu, 21 Mar 2019 at 21:49, Michael O'Brien <[email protected]> wrote: > Adding the list > > Guys, you only need to install tiller (either automatically through the > rancher install or as part of the OOM-1598 RKE install) on the master – not > the other 1-14 slave nodes – those run only the rancher agent docker > container and a nfs mount. > > Kubectl can be installed anywhere as it is a wrapper on the K8S Rest API – > I usually install it directly on the same master – so the master VM acts as > a sort of localized bastion node – the 2 scripts do this. > > > > If you refer to the automated installation scripts for the verified > installation steps you will have your answers – I don’t recommend > installing k8s manually. > > Just run the following script on your master and then add your 14 hosts – > you can remove the host created on the master after that > > > > sudo ./oom_rancher_setup.sh -b master -s <your domain/ip> -e onap > > > > From your steps – you did not need to rerun the kubectl/helm steps on the > slaves (they only need nfs, docker and the rancher agent docker container) > – your master was ok > > > > The single node install is good for any environment - there is a 110 limit > step needed for rancher - for RKE it is part of the script > > > https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-Scriptedundercloud(Helm/Kubernetes/Docker)andONAPinstall-SingleVM > > > https://git.onap.org/logging-analytics/tree/deploy/rancher/oom_rancher_setup.sh > > > > > The clustered example is either for Openstack or AWS > > > https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-Scriptedundercloud(Helm/Kubernetes/Docker)andONAPinstall-clustered > > > > But the steps are the same for the slaves - see the script > > > https://git.onap.org/logging-analytics/tree/deploy/aws/oom_cluster_host_install.sh > > > https://git.onap.org/logging-analytics/tree/deploy/heat/logging_openstack_13_16g.yaml > > > > install docker, run the nfs client script, run the slave docker client - > that is it. > > > > if [[ "$IS_NODE" != false ]]; then > > sudo curl > https://releases.rancher.com/install-docker/$DOCKER_VER.sh | sh > > sudo usermod -aG docker $USERNAME > > fi > > sudo apt-get install nfs-common -y > > sudo mkdir /$DOCKERDATA_NFS > > sudo mount -t nfs4 -o > nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport > $AWS_EFS.efs.$AWS_REGION.amazonaws.com:/ /$DOCKERDATA_NFS > > if [[ "$IS_NODE" != false ]]; then > > echo "Running agent docker..." > > if [[ "$COMPUTEADDRESS" != false ]]; then > > echo "sudo docker run --rm --privileged -v > /var/run/docker.sock:/var/run/docker.sock -v > /var/lib/rancher:/var/lib/rancher rancher/agent:v$AGENT_VER http:// > $MASTER:$PORT/v1/scripts/$TOKEN" > > sudo docker run --rm --privileged -v > /var/run/docker.sock:/var/run/docker.sock -v > /var/lib/rancher:/var/lib/rancher rancher/agent:v$AGENT_VER http:// > $MASTER:$PORT/v1/scripts/$TOKEN > > else > > echo "sudo docker run -e CATTLE_AGENT_IP=\"$ADDRESS\" --rm > --privileged -v /var/run/docker.sock:/var/run/docker.sock -v > /var/lib/rancher:/var/lib/rancher rancher/agent:v$AGENT_VER http:// > $MASTER:$PORT/v1/scripts/$TOKEN" > > sudo docker run -e CATTLE_AGENT_IP="$ADDRESS" --rm --privileged > -v /var/run/docker.sock:/var/run/docker.sock -v > /var/lib/rancher:/var/lib/rancher rancher/agent:v$AGENT_VER http:// > $MASTER:$PORT/v1/scripts/$TOKEN > > fi > > fi > > > > > > After everything is up - run the following script on the master to > optionally sequence in the onap pods > > https://git.onap.org/logging-analytics/tree/deploy/cd.sh > > > > Alternatively you can use the 0.1.16 RKE install - I will be adding HA and > move up to 2.0 - but for now you can provision a 128g+ VM and run that one > as well - tiller/helm is automated there - it is not merged and is in review > > > https://gerrit.onap.org/r/#/c/79067/5/kubernetes/contrib/tools/rke/rke_setup.sh > > subject to change in > > https://jira.onap.org/browse/OOM-1670 > > > > > > for your kubectl/helm steps - from the oom_rancher_setup.sh script - on > the master > > sudo curl -LO > https://storage.googleapis.com/kubernetes-release/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl > > sudo chmod +x ./kubectl > > sudo mv ./kubectl /usr/local/bin/kubectl > > sudo mkdir ~/.kube > > wget > http://storage.googleapis.com/kubernetes-helm/helm-v${HELM_VERSION}-linux-amd64.tar.gz > > sudo tar -zxvf helm-v${HELM_VERSION}-linux-amd64.tar.gz > > sudo mv linux-amd64/helm /usr/local/bin/helm > > > > # create kubernetes environment on rancher using cli > > RANCHER_CLI_VER=0.6.7 > > KUBE_ENV_NAME=$ENVIRON > > wget > https://releases.rancher.com/cli/v${RANCHER_CLI_VER}/rancher-linux-amd64-v${RANCHER_CLI_VER}.tar.gz > > sudo tar -zxvf rancher-linux-amd64-v${RANCHER_CLI_VER}.tar.gz > > sudo cp rancher-v${RANCHER_CLI_VER}/rancher . > > sudo chmod +x ./rancher > > > > echo "install jq for json parsing" > > sudo wget > https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 -O jq > > sudo chmod 777 jq > > # not +x or jq will not be runnable from your non-root user > > sudo mv jq /usr/local/bin > > echo "wait for rancher server container to finish - 3 min" > > echo "if you are planning on running a co-located host to bring up more > than 110 pods on a single vm - you have 3 min to add --max-pods=900 in > additional kublet flags - in the k8s template" > > sleep 60 > > echo "2 more min" > > sleep 60 > > echo "1 min left" > > sleep 60 > > echo "get public and private tokens back to the rancher server so we can > register the client later" > > API_RESPONSE=`curl -s 'http://127.0.0.1:8880/v2-beta/apikey' -d > '{"type":"apikey","accountId":"1a1","name":"autoinstall","description":"autoinstall","created":null,"kind":null,"removeTime":null,"removed":null,"uuid":null}'` > > # Extract and store token > > echo "API_RESPONSE: $API_RESPONSE" > > KEY_PUBLIC=`echo $API_RESPONSE | jq -r .publicValue` > > KEY_SECRET=`echo $API_RESPONSE | jq -r .secretValue` > > echo "publicValue: $KEY_PUBLIC secretValue: $KEY_SECRET" > > > > export RANCHER_URL=http://${SERVER}:$PORT > > export RANCHER_ACCESS_KEY=$KEY_PUBLIC > > export RANCHER_SECRET_KEY=$KEY_SECRET > > ./rancher env ls > > echo "wait 60 sec for rancher environments to settle before we create the > onap kubernetes one" > > sleep 60 > > > > echo "Creating kubernetes environment named ${KUBE_ENV_NAME}" > > ./rancher env create -t kubernetes $KUBE_ENV_NAME > kube_env_id.json > > PROJECT_ID=$(<kube_env_id.json) > > echo "env id: $PROJECT_ID" > > export RANCHER_HOST_URL=http://${SERVER}:$PORT/v1/projects/$PROJECT_ID > > echo "you should see an additional kubernetes environment usually with id > 1a7" > > ./rancher env ls > > # optionally disable cattle env > > > > # add host registration url > > # https://github.com/rancher/rancher/issues/2599 > > # wait for REGISTERING to ACTIVE > > echo "sleep 90 to wait for REG to ACTIVE" > > ./rancher env ls > > sleep 30 > > echo "check on environments again before registering the URL response" > > ./rancher env ls > > sleep 30 > > ./rancher env ls > > echo "60 more sec" > > sleep 60 > > > > REG_URL_RESPONSE=`curl -X POST -u $KEY_PUBLIC:$KEY_SECRET -H 'Accept: > application/json' -H 'ContentType: application/json' -d > '{"name":"$SERVER"}' "http:// > $SERVER:8880/v1/projects/$PROJECT_ID/registrationtokens"` > > echo "REG_URL_RESPONSE: $REG_URL_RESPONSE" > > echo "wait for server to finish url configuration - 5 min" > > sleep 240 > > echo "60 more sec" > > sleep 60 > > # see registrationUrl in > > REGISTRATION_TOKENS=`curl http://127.0.0.1: > $PORT/v2-beta/registrationtokens` > > echo "REGISTRATION_TOKENS: $REGISTRATION_TOKENS" > > REGISTRATION_URL=`echo $REGISTRATION_TOKENS | jq -r > .data[0].registrationUrl` > > REGISTRATION_DOCKER=`echo $REGISTRATION_TOKENS | jq -r .data[0].image` > > REGISTRATION_TOKEN=`echo $REGISTRATION_TOKENS | jq -r .data[0].token` > > echo "Registering host for image: $REGISTRATION_DOCKER url: > $REGISTRATION_URL registrationToken: $REGISTRATION_TOKEN" > > HOST_REG_COMMAND=`echo $REGISTRATION_TOKENS | jq -r .data[0].command` > > echo "Running agent docker..." > > if [[ "$COMPUTEADDRESS" != false ]]; then > > echo "sudo docker run --rm --privileged -v > /var/run/docker.sock:/var/run/docker.sock -v > /var/lib/rancher:/var/lib/rancher $REGISTRATION_DOCKER > $RANCHER_URL/v1/scripts/$REGISTRATION_TOKEN" > > sudo docker run --rm --privileged -v > /var/run/docker.sock:/var/run/docker.sock -v > /var/lib/rancher:/var/lib/rancher $REGISTRATION_DOCKER > $RANCHER_URL/v1/scripts/$REGISTRATION_TOKEN > > else > > echo "sudo docker run -e CATTLE_AGENT_IP=\"$ADDRESS\" --rm > --privileged -v /var/run/docker.sock:/var/run/docker.sock -v > /var/lib/rancher:/var/lib/rancher rancher/agent:v$AGENT_VERSION http:// > $SERVER:$PORT/v1/scripts/$TOKEN" > > sudo docker run -e CATTLE_AGENT_IP="$ADDRESS" --rm --privileged -v > /var/run/docker.sock:/var/run/docker.sock -v > /var/lib/rancher:/var/lib/rancher rancher/agent:v$AGENT_VERSION http:// > $SERVER:$PORT/v1/scripts/$REGISTRATION_TOKEN > > fi > > echo "waiting 8 min for host registration to finish" > > sleep 420 > > echo "1 more min" > > sleep 60 > > > > # base64 encode the kubectl token from the auth pair > > # generate this after the host is registered > > KUBECTL_TOKEN=$(echo -n 'Basic '$(echo -n > "$RANCHER_ACCESS_KEY:$RANCHER_SECRET_KEY" | base64 -w 0) | base64 -w 0) > > echo "KUBECTL_TOKEN base64 encoded: ${KUBECTL_TOKEN}" > > # add kubectl config - NOTE: the following spacing has to be "exact" or > kubectl will not connect - with a localhost:8080 error > > cat > ~/.kube/config <<EOF > > apiVersion: v1 > > kind: Config > > clusters: > > - cluster: > > api-version: v1 > > insecure-skip-tls-verify: true > > server: "https://$SERVER:$PORT/r/projects/$PROJECT_ID/kubernetes:6443"; > > name: "${ENVIRON}" > > contexts: > > - context: > > cluster: "${ENVIRON}" > > user: "${ENVIRON}" > > name: "${ENVIRON}" > > current-context: "${ENVIRON}" > > users: > > - name: "${ENVIRON}" > > user: > > token: "$KUBECTL_TOKEN" > > > > EOF > > > > > > echo "Verify all pods up on the kubernetes system - will return > localhost:8080 until a host is added" > > echo "kubectl get pods --all-namespaces" > > kubectl get pods --all-namespaces > > echo "upgrade server side of helm in kubernetes" > > if [ "$USERNAME" == "root" ]; then > > helm version > > else > > sudo helm version > > fi > > echo "sleep 90" > > sleep 90 > > if [ "$USERNAME" == "root" ]; then > > helm init --upgrade > > else > > sudo helm init --upgrade > > fi > > echo "sleep 90" > > sleep 90 > > echo "verify both versions are the same below" > > if [ "$USERNAME" == "root" ]; then > > helm version > > else > > sudo helm version > > fi > > echo "start helm server" > > if [ "$USERNAME" == "root" ]; then > > helm serve & > > else > > sudo helm serve & > > fi > > echo "sleep 30" > > sleep 30 > > echo "add local helm repo" > > if [ "$USERNAME" == "root" ]; then > > helm repo add local http://127.0.0.1:8879 > > helm repo list > > else > > sudo helm repo add local http://127.0.0.1:8879 > > sudo helm repo list > > fi > > echo "To enable grafana dashboard - do this after running cd.sh which > brings up onap - or you may get a 302xx port conflict" > > echo "kubectl expose -n kube-system deployment monitoring-grafana > --type=LoadBalancer --name monitoring-grafana-client" > > echo "to get the nodeport for a specific VM running grafana" > > echo "kubectl get services --all-namespaces | grep graf" > > kubectl get pods --all-namespaces > > echo "finished!" > > > > *From: *Viji Jeyaraman <[email protected]> > *Date: *Thursday, March 21, 2019 at 09:08 > *To: *Michael O'Brien <[email protected]> > *Cc: *Pranjal Sharma <[email protected]>, "[email protected]" < > [email protected]>, "[email protected]" <[email protected]> > *Subject: *Onap installation steps query > > > > To Michael, > > > > We are currently doing ONAP installation Casablanca version following > https://docs.onap.org/en/casablanca/submodules/oom.git/docs/oom_setup_kubernetes_rancher.html#onap-on-kubernetes-with-rancher > > documentation. > > > > Our deployment VMS consists of > > 1. 1 Rancher master node > 2. 14 kubernetes nodes which is internally called 1 masterNFS +13 > SlaveNFS VMS > > > > We have created the VMS, and through Rancher UI we have added 14 > kubernetes nodes as hosts and created the kubernetes cluster. > > > > In the documentation we got stuck in “Configure kubectl and helm” step. > > > > We need a clarification from you, the following configuration steps > should we run in all the 15VMs including rancher or we should run only in > k8s hosts(14VMS) or only in masterNFS VM > > 1. vi .kube/config -- add config generated from rancher UI > 2. kubectl config get-contexts > 3. kubectl get pods --all-namespaces -o=wide > 4. helm list > 5. helm init –upgrade > > > > Also, when we ran the above configuration steps at first in masterNFS VM, > things went fine. “tiller-deploy-76747d6678-455n2” pod came up and in > running state and also “rancher/pause-amd64” container was seen running in > “kube system” namespace in masterNFS VM. > > root@masterNFS:~# kubectl get pods --all-namespaces -o=wide > > NAMESPACE NAME READY STATUS > RESTARTS AGE IP NODE NOMINATED NODE > > kube-system heapster-5c6fddd5b-7k988 1/1 Running > 0 18h 10.42.230.106 masternfs <none> > > kube-system kube-dns-8587b597fc-49mwv 3/3 Running > 0 18h 10.42.240.2 masternfs <none> > > kube-system kubernetes-dashboard-79599f58bc-zhz8k 1/1 Running > 0 18h 10.42.115.179 masternfs <none> > > kube-system monitoring-grafana-74c4f86f9c-tx6d8 1/1 Running > 0 18h 10.42.53.121 masternfs <none> > > kube-system monitoring-influxdb-f78c85b98-8jvd4 1/1 Running > 0 18h 10.42.188.87 masternfs <none> > > kube-system tiller-deploy-b5f895978-95ctj 1/1 Running > 0 18h 10.42.6.233 masternfs <none> > > > > Then, same steps executed in slaveNFS VM, “tiller-deploy-76747d6678-455n2” > was not up and “rancher/pause-amd64” container now seen in slavenfs which > is also not in running state. > > > > root@slaveNFS:~# kubectl get pods --all-namespaces -o=wide > > NAMESPACE NAME READY > STATUS RESTARTS AGE IP NODE > NOMINATED NODE > > kube-system heapster-5c6fddd5b-7k988 1/1 > Running 0 18h 10.42.230.106 masternfs <none> > > kube-system kube-dns-8587b597fc-49mwv 3/3 > Running 0 18h 10.42.240.2 masternfs <none> > > kube-system kubernetes-dashboard-79599f58bc-zhz8k 1/1 > Running 0 18h 10.42.115.179 masternfs <none> > > kube-system monitoring-grafana-74c4f86f9c-tx6d8 1/1 > Running 0 18h 10.42.53.121 masternfs <none> > > kube-system monitoring-influxdb-f78c85b98-8jvd4 1/1 > Running 0 18h 10.42.188.87 masternfs <none> > > kube-system tiller-deploy-76747d6678-4zljw 0/1 > ImagePullBackOff 0 1m <none> slavenfs <none> > > root@slaveNFS:~# > > root@slaveNFS:~# helm list > > Error: could not find a ready tiller pod > > root@slaveNFS:~# > > > > root@k8s-onap2-2:~# kubectl get pods --all-namespaces -o=wide > > NAMESPACE NAME READY > STATUS RESTARTS AGE IP NODE > NOMINATED NODE > > kube-system heapster-5c6fddd5b-7k988 1/1 > Running 0 20h 10.42.230.106 masternfs > <none> > > kube-system kube-dns-8587b597fc-49mwv 3/3 > Running 15 20h 10.42.186.119 masternfs > <none> > > kube-system kubernetes-dashboard-79599f58bc-zhz8k 1/1 > Running 0 20h 10.42.115.179 masternfs > <none> > > kube-system monitoring-grafana-74c4f86f9c-tx6d8 1/1 > Running 0 20h 10.42.53.121 masternfs > <none> > > kube-system monitoring-influxdb-f78c85b98-8jvd4 1/1 > Running 0 20h 10.42.188.87 masternfs > <none> > > kube-system tiller-deploy-76747d6678-455n2 0/1 > ContainerCreating 0 8m <none> k8s-onap-12 > <none> > > > > So ,we need a clarification, On which nodes the tiller helm server should > be running ? Should the tiller container run only in masterNFS VM or all > 14VMS k8 hosts ? > > > > Thanks, > > Viji J > > > > *This email and the information contained herein is proprietary and > confidential and subject to the Amdocs Email Terms of Service, which you > may review at* *https://www.amdocs.com/about/email-terms-of-service* > <https://www.amdocs.com/about/email-terms-of-service> > -- Pranjal Dadhich -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#16388): https://lists.onap.org/g/onap-discuss/message/16388 Mute This Topic: https://lists.onap.org/mt/30696319/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
