Team, Kubernetes 1.8.6 looks OK - all containers up (except the expected aaf one) under master - Notice the new pod names for SDNC under SDNC-163 a current cluster of 1 for the new HA setup.
Job 933 had SDNC issues http://jenkins.onap.info/job/oom-cd/933/console job 934 was ok - so this is the normal intermittent failures of SDNC at about 4% http://kibana.onap.info:5601/app/kibana#/dashboard/AWAtvpS63NTXK5mX2kuS?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-7d,mode:quick,to:now))&_a=(description:'',filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:healthcheck,index:AWAtYVuQYNh0ncCb5IdF,key:healthcheck,negate:!f,type:phrase,value:FAIL),script:(script:(inline:'boolean%20compare(Supplier%20s,%20def%20v)%20%7Breturn%20s.get()%20%3D%3D%20v;%7Dcompare(()%20-%3E%20%7B%20def%20msg%20%3D%20doc%5B!'message.keyword!'%5D.value;%0Aif%20(doc%5B!'message.keyword!'%5D.value%20!!%3D%20null)%20%7B%0Aif%20(doc%5B!'message.keyword!'%5D.value%20%3D~%20%2FHealth%20Check%2F)%20%7B%0A%20%20if%20(doc%5B!'message.keyword!'%5D.value%20%3D~%20%2FPASS%2F)%20%7B%20%0A%20%20%20%20return%20%22PASS%22;%0A%20%20%7D%0A%20%20else%20%7B%0A%20%20%20%20return%20%22FAIL%22;%0A%20%20%7D%0A%7D%0A%7D%0Areturn%20null;%20%7D,%20params.value);',lang:painless,params:(value:FAIL)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:component,index:AWAtYVuQYNh0ncCb5IdF,key:component,negate:!f,type:phrase,value:'Basic%20SDNGC'),script:(script:(inline:'boolean%20compare(Supplier%20s,%20def%20v)%20%7Breturn%20s.get()%20%3D%3D%20v;%7Dcompare(()%20-%3E%20%7B%20if%20(doc%5B!'message.keyword!'%5D.value%20!!%3D%20null)%20%7B%0Aif%20(doc%5B!'message.keyword!'%5D.value%20%3D~%20%2FHealth%20Check%2F)%20%7B%0A%20%20def%20m%20%3D%20%2F%5E(.%2B%3F)(API)%3F%5CsHealth%20Check.*$%2F.matcher(doc%5B!'message.keyword!'%5D.value);%0A%20%20if%20(%20m.matches()%20)%20%7B%0A%20%20%20return%20m.group(1)%0A%20%20%7D%0A%20%20else%20%7B%0A%20%20%20%20return%20%22unknown%22;%0A%20%20%7D%0A%7D%0A%7D%0Areturn%20null;%20%7D,%20params.value);',lang:painless,params:(value:'Basic%20SDNGC'))))),options:(darkTheme:!f),panels:!((col:1,id:AWAts77k3NTXK5mX2kuM,panelIndex:1,row:1,size_x:8,size_y:3,type:visualization),(col:9,id:AWAtuTVI3NTXK5mX2kuP,panelIndex:2,row:1,size_x:4,size_y:3,type:visualization),(col:1,id:AWAtuBTY3NTXK5mX2kuO,panelIndex:3,row:7,size_x:6,size_y:3,type:visualization),(col:1,id:AWAttmqB3NTXK5mX2kuN,panelIndex:4,row:4,size_x:6,size_y:3,type:visualization),(col:7,id:AWAtvHtY3NTXK5mX2kuR,panelIndex:6,row:4,size_x:6,size_y:6,type:visualization)),query:(match_all:()),timeRestore:!f,title:'CD%20Health%20Check',uiState:(),viewMode:view)<http://kibana.onap.info:5601/app/kibana#/dashboard/AWAtvpS63NTXK5mX2kuS?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-7d,mode:quick,to:now))&_a=(description:'',filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:healthcheck,index:AWAtYVuQYNh0ncCb5IdF,key:healthcheck,negate:!f,type:phrase,value:FAIL),script:(script:(inline:'boolean%20compare(Supplier%20s,%20def%20v)%20%7Breturn%20s.get()%20%3D%3D%20v;%7Dcompare(()%20-%3E%20%7B%20def%20msg%20%3D%20doc%5B!'message.keyword!'%5D.value;%0Aif%20(doc%5B!'message.keyword!'%5D.value%20!!%3D%20null)%20%7B%0Aif%20(doc%5B!'message.keyword!'%5D.value%20%3D~%20%2FHealth%20Check%2F)%20%7B%0A%20%20if%20(doc%5B!'message.keyword!'%5D.value%20%3D~%20%2FPASS%2F)%20%7B%20%0A%20%20%20%20return%20%22PASS%22;%0A%20%20%7D%0A%20%20else%20%7B%0A%20%20%20%20return%20%22FAIL%22;%0A%20%20%7D%0A%7D%0A%7D%0Areturn%20null;%20%7D,%20params.value);',lang:painless,params:(value:FAIL)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:component,index:AWAtYVuQYNh0ncCb5IdF,key:component,negate:!f,type:phrase,value:'Basic%20SDNGC'),script:(script:(inline:'boolean%20compare(Supplier%20s,%20def%20v)%20%7Breturn%20s.get()%20%3D%3D%20v;%7Dcompare(()%20-%3E%20%7B%20if%20(doc%5B!'message.keyword!'%5D.value%20!!%3D%20null)%20%7B%0Aif%20(doc%5B!'message.keyword!'%5D.value%20%3D~%20%2FHealth%20Check%2F)%20%7B%0A%20%20def%20m%20%3D%20%2F%5E(.%2B%3F)(API)%3F%5CsHealth%20Check.*$%2F.matcher(doc%5B!'message.keyword!'%5D.value);%0A%20%20if%20(%20m.matches()%20)%20%7B%0A%20%20%20return%20m.group(1)%0A%20%20%7D%0A%20%20else%20%7B%0A%20%20%20%20return%20%22unknown%22;%0A%20%20%7D%0A%7D%0A%7D%0Areturn%20null;%20%7D,%20params.value);',lang:painless,params:(value:'Basic%20SDNGC'))))),options:(darkTheme:!f),panels:!((col:1,id:AWAts77k3NTXK5mX2kuM,panelIndex:1,row:1,size_x:8,size_y:3,type:visualization),(col:9,id:AWAtuTVI3NTXK5mX2kuP,panelIndex:2,row:1,size_x:4,size_y:3,type:visualization),(col:1,id:AWAtuBTY3NTXK5mX2kuO,panelIndex:3,row:7,size_x:6,size_y:3,type:visualization),(col:1,id:AWAttmqB3NTXK5mX2kuN,panelIndex:4,row:4,size_x:6,size_y:3,type:vi> ubuntu@ip-172-31-82-112:~/oom/kubernetes/oneclick$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AG onap-aaf aaf-1993711932-qc7qt 0/1 Running 0 54m onap-aaf aaf-cs-1310404376-nsjzl 1/1 Running 0 54m onap-aai aai-resources-1039856271-3w9ph 2/2 Running 0 54m onap-aai aai-service-749944520-hq3ch 1/1 Running 0 54m onap-aai aai-traversal-143081090-zp9mh 2/2 Running 0 54m onap-aai data-router-3434587794-j15sb 1/1 Running 0 54m onap-aai elasticsearch-622738319-qxtng 1/1 Running 0 54m onap-aai hbase-1949550546-5qbvr 1/1 Running 0 54m onap-aai model-loader-service-4144225433-sqg0g 2/2 Running 0 54m onap-aai search-data-service-3842430948-7b394 2/2 Running 0 54m onap-aai sparky-be-4222608366-hrqqc 2/2 Running 0 54m onap-appc appc-1828810488-81fh9 2/2 Running 0 54m onap-appc appc-dbhost-2793739621-jz4lt 1/1 Running 0 54m onap-appc appc-dgbuilder-2298093128-ms5z0 1/1 Running 0 54m onap-clamp clamp-2211988013-kgnf5 1/1 Running 0 54m onap-clamp clamp-mariadb-1812977665-71ftc 1/1 Running 0 54m onap-cli cli-595710742-dv5f5 1/1 Running 0 54m onap-consul consul-agent-3312409084-k3w2j 1/1 Running 8 55m onap-consul consul-server-1173049560-q8hl7 1/1 Running 8 55m onap-consul consul-server-1173049560-s1h9l 1/1 Running 8 55m onap-consul consul-server-1173049560-w9wwl 1/1 Running 7 55m onap-kube2msb kube2msb-registrator-4293827076-6pl0v 1/1 Running 0 54m onap-log elasticsearch-1942187295-zdz5f 1/1 Running 0 54m onap-log kibana-3372627750-zt591 1/1 Running 0 54m onap-log logstash-1708188010-tgm61 1/1 Running 0 54m onap-message-router dmaap-3126594942-96lsq 1/1 Running 0 55m onap-message-router global-kafka-3848542622-gq9lx 1/1 Running 0 55m onap-message-router zookeeper-624700062-5kdg2 1/1 Running 0 55m onap-msb msb-consul-3334785600-st0kp 1/1 Running 0 55m onap-msb msb-discovery-196547432-jgvls 1/1 Running 0 55m onap-msb msb-eag-1649257109-wrsx7 1/1 Running 0 55m onap-msb msb-iag-1033096170-gs700 1/1 Running 0 55m onap-mso mariadb-829081257-4rzj1 1/1 Running 0 55m onap-mso mso-736294529-l5m3r 2/2 Running 0 55m onap-multicloud framework-2273343137-pwkx6 1/1 Running 0 54m onap-multicloud multicloud-ocata-1517639325-pvctj 1/1 Running 0 54m onap-multicloud multicloud-vio-4239509896-llftl 1/1 Running 0 54m onap-multicloud multicloud-windriver-3629763724-dtm0b 1/1 Running 0 54m onap-policy brmsgw-1909438199-3z4jp 1/1 Running 0 54m onap-policy drools-2600956298-zz82h 2/2 Running 0 54m onap-policy mariadb-2660273324-mqw55 1/1 Running 0 54m onap-policy nexus-3663640793-cl5kl 1/1 Running 0 54m onap-policy pap-466625067-c4m67 2/2 Running 0 54m onap-policy pdp-2354817903-38234 2/2 Running 0 54m onap-portal portalapps-1783099045-3c28b 2/2 Running 0 54m onap-portal portaldb-1451233177-3r64j 1/1 Running 0 54m onap-portal portalwidgets-2060058548-zgf9s 1/1 Running 0 54m onap-portal vnc-portal-3136230744-6rxdf 1/1 Running 0 54m onap-robot robot-2551980890-6d903 1/1 Running 0 54m onap-sdc sdc-be-2336519847-qs51r 2/2 Running 0 54m onap-sdc sdc-cs-1151560586-jf74t 1/1 Running 0 54m onap-sdc sdc-es-3319302712-3243g 1/1 Running 0 54m onap-sdc sdc-fe-2862673798-d69lc 2/2 Running 0 54m onap-sdc sdc-kb-1258596734-mz6p3 1/1 Running 0 54m onap-sdnc nfs-provisioner-4051743034-kc99z 1/1 Running 0 16m onap-sdnc sdnc-0 2/2 Running 0 16m onap-sdnc sdnc-dbhost-0 2/2 Running 0 16m onap-sdnc sdnc-dgbuilder-4267203648-tfr1s 1/1 Running 0 16m onap-sdnc sdnc-portal-2558294154-p7pdb 1/1 Running 0 16m onap-uui uui-4267149477-dj4kx 1/1 Running 0 54m onap-uui uui-server-3441797946-mp39k 1/1 Running 0 54m onap-vfc vfc-catalog-840807183-w7sv6 1/1 Running 0 54m onap-vfc vfc-emsdriver-2936953408-mk596 1/1 Running 0 54m onap-vfc vfc-gvnfmdriver-2866216209-3rkh6 1/1 Running 0 54m onap-vfc vfc-hwvnfmdriver-2588350680-rx4d8 1/1 Running 0 54m onap-vfc vfc-jujudriver-406795794-822hp 1/1 Running 0 54m onap-vfc vfc-nokiavnfmdriver-1760240499-nmp6r 1/1 Running 0 54m onap-vfc vfc-nslcm-3756650867-2q6zz 1/1 Running 0 54m onap-vfc vfc-resmgr-1409642779-r9wqv 1/1 Running 0 54m onap-vfc vfc-vnflcm-3340104471-nv9sw 1/1 Running 0 54m onap-vfc vfc-vnfmgr-2823857741-fff9t 1/1 Running 0 54m onap-vfc vfc-vnfres-1792029715-18kjn 1/1 Running 0 54m onap-vfc vfc-workflow-3450325534-97nh7 1/1 Running 0 54m onap-vfc vfc-workflowengineactiviti-4110617986-3lmdd 1/1 Running 0 54m onap-vfc vfc-ztesdncdriver-1452986549-kmhn4 1/1 Running 0 54m onap-vfc vfc-ztevmanagerdriver-2080553526-nl0nv 1/1 Running 0 54m onap-vid vid-mariadb-3318685446-d3qgt 1/1 Running 0 54m onap-vid vid-server-3026751708-xh563 2/2 Running 0 54m onap-vnfsdk postgres-436836560-04cmv 1/1 Running 0 54m onap-vnfsdk refrepo-1924147637-zlgdh 1/1 Running 0 54m thank you /michael From: [email protected] [mailto:[email protected]] On Behalf Of Michael O'Brien Sent: Thursday, December 21, 2017 18:25 To: [email protected] Subject: [onap-discuss] Heads up: OOM supports Kubernetes 1.8.4 - not 1.9.0 yet which came out of RC 6 days ago - OOM-522 and Kubernetes/57528 ONAP members, Hi, stay on kubernetes 1.8.4 - dont use stable 1.9.0 yet - it just came out of RC - as there are problems generating secrets for authenticated docker registries in this release we need to get fixed first before we upgrade to 1.9 (the upgrade from 1.7 to 1.8 looks good so far) updating jiras/mails https://wiki.onap.org/display/DW/ONAP+on+Kubernetes#ONAPonKubernetes-InstallKubectl https://wiki.onap.org/display/DW/3.+Set+Up+the+Undercloud#id-3.SetUptheUndercloud-Installkubectl The CNCF Kubernetes project and ourselves work together - https://github.com/kubernetes/kubernetes/issues/57528 and https://jira.onap.org/browse/OOM-522 Officially supported ONAP Kubernetes install version curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.4/bin/linux/amd64/kubectl Downgrade procedure for those that ran the stable.txt download version of the Kubernetes curl - for 1.9.0 to 1.8.4 - in the last 6 days - Rerun the kubernetes part of the bootstrap install with v1.8.4, all the way to the ~/.kube/config token curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.8.4/bin/linux/amd64/kubectl chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl vi ~/.kube/config Going forward The script and wiki instructions are modified to pull only v1.8.4 not vStable Myself and a couple others had this issue only on new machines/VMs that did not have populated docker repos yet including Huabing, some SDNC team, a DevOps - when Gordon and I worked together we narrowed it down to secret generation in kubectl - Mike mentioned a week ago that we should be stabilizing on Kubernetes 1.8 as 1.9 has a lot of changes - so this is the first issue with the new K8S release out of RC. Stay on 1.8.4 as the CD server runs this fine until we validate 1.8.6 We will address the formal upgrade from 1.8 to 1.9 for Kubernetes in story https://jira.onap.org/browse/OOM-535 When installing OOM via the steps in the wiki or via the provided rancher bootstrap script there - verify you are running Kubernetes 1.8.4 (1.8.6 came out 2 hours ago but we still need to test it on SDNC) (any 1.8.4- version). 6 days ago Kubernetes 1.9.0 was marked stable - so using the script up to a couple hours ago you would be running a version that just came out of RC and has issues with the kubectl secret command required to authenticate with secured nexus docker repositories. If you use 1.9.0 you will see ImageBackoff exceptions for nexus3 repo docker pulls. With 1.8.4 you will be OK on either the master or amsterdam branches. The following https://git.onap.org/oom/tree/kubernetes/oneclick/createAll.bash fails to create the secret properly cmd=`echo kubectl --namespace $1-$2 create secret docker-registry $3 --docker-server=$4 --docker-username=$5 --docker-password=$6 --docker-email=$7` note: 1.9.0 was released 6 days ago https://github.com/kubernetes/kubernetes/releases/tag/v1.9.0 Good version check ubuntu@ip-172-31-83-79:~$ kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:28:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.7-rancher1", GitCommit:"a1ea37c6f6d21f315a07631b17b9537881e1986a", GitTreeState:"clean", BuildDate:"2017-10-02T21:33:08Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Verifying 1.8.6 on lab.onap.info running master:20171221:1700 - stick to 1.8.4 for now Michael O'Brien Amdocs Technology 16135955268 55268 [amdocs-a] This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer>
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
