Re: [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not accessible
3cadbcNode Not > Found:object located at service-design-and-creation/ > models/model/827361dd-931e-4468-a8ac-d77785039938/model- > vers/model-ver/6e064bcd-b574-49cf-9846-48f9c52e7ce8#model-version not > foundERR.5.4.6114", > "precentProgress": "100" > > 03/05/18 10:52:56 HTTP Status: Accepted (202) > { > "requestReferences": { > "instanceId": "c52f17cf-fbd6-42c0-8019-aa8eeb3cadbc", > "requestId": "1e8a37c9-4028-45c7-acd3-282b5600b1ba" > } > } > > *VID Screen shot* > > > > > > On Mon, Mar 5, 2018 at 4:20 PM, Borislav Glozman < > borislav.gloz...@amdocs.com> wrote: > >> Please try to redeploy sdc: >> deleteAll.sh -n yournamespace -a sdc >> createAll.sh -n yournamespace -a sdc >> Wait that all pods are running and retry the workaround I wrote in the >> previous email. >> >> >> Thanks, >> Borislav Glozman >> O. +972-9-7669188 >> M. +972-52-2835726 >> >> >> -Original Message- >> *From:* Vivekanandan Muthukrishnan [vmuthukrish...@aarnanetworks.com] >> *Received:* Monday, 05 Mar 2018, 9:59 >> *To:* Borislav Glozman [borislav.gloz...@amdocs.com] >> *CC:* onap-discuss@lists.onap.org [onap-discuss@lists.onap.org] >> *Subject:* Re: [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not >> accessible >> >> Hi Borislav, >> >> Now the SDC UI is not loading and the below are the UI console image and >> logs. >> >> I did the steps that you have suggested. Am i missing something here? >> >> I would really appreciate your help to over come this issue. >> >> Regards >> Vivek >> >> >> >> *SDC-BE Log snippet (/var/log/onap/sdc/sdc-be/error.log)* >> >> 2018-03-05T07:53:30.316Z [BE-Health-Check-Task] INFO >> c.d.d.c.p.DCAwareRoundRobinPolicy Using data-center name 'SDC-CS-AUTO' >> for DCAwareRoundRobinPolicy (if this is incorrect, please provide the >> correct datacenter name with DCAwareRoundRobinPolicy constructor) >> 2018-03-05T07:53:30.316Z [BE-Health-Check-Task] INFO >> com.datastax.driver.core.Cluster New Cassandra host sdc-cs.onap-sdc/ >> 10.42.139.62:9042 added >> 2018-03-05T07:53:30.452Z [BE-Health-Check-Task] INFO >> o.o.s.b.c.impl.CassandraHealthCheck The number of cassandra nodes is:1 >> 2018-03-05T07:53:30.452Z [BE-Health-Check-Task] INFO >> o.o.s.b.c.impl.CassandraHealthCheck The cassandra down nodes number is 0 >> 2018-03-05T07:53:31.418Z [qtp561247961-19] INFO >> o.o.sdc.be.filters.BeServletFilter serviceInstanceID=null userId=null >> localAddr=10.42.94.114 uuid=5bdb4381-f45f-477a-a665-003f8a314702 >> remoteAddr=10.42.123.31 No requestID provided -> Generated UUID >> 5bdb4381-f45f-477a-a665-003f8a314702 >> 2018-03-05T07:53:33.126Z [BE-Health-Check-Task] INFO >> o.o.s.b.c.impl.CassandraHealthCheck creating cluster for Cassandra for >> monitoring. >> 2018-03-05T07:53:33.126Z [BE-Health-Check-Task] INFO >> o.o.s.b.d.c.schema.SdcSchemaUtils connecting to node:[sdc-cs.onap-sdc]. >> 2018-03-05T07:53:33.168Z [ES-Health-Check-Thread] ERROR >> o.o.sdc.be.dao.impl.ESCatalogDAO Error while trying to connect to >> elasticsearch. host: [sdc-es.onap-sdc:9300] | port: 9200 | error: None of >> the configured nodes are available: [{#transport#-1}{10.42.103.200 >> }{sdc-es.onap-sdc/10.42.103.200:9300}] >> org.elasticsearch.client.transport.NoNodeAvailableException: None of the >> configured nodes are available: [{#transport#-1}{10.42.103.200 >> }{sdc-es.onap-sdc/10.42.103.200:9300}] >> at org.elasticsearch.client.transport.TransportClientNodesServi >> ce.ensureNodesAreAvailable(TransportClientNodesService.java:290) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at org.elasticsearch.client.transport.TransportClientNodesServi >> ce.execute(TransportClientNodesService.java:207) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at org.elasticsearch.client.transport.support.TransportProxyCli >> ent.execute(TransportProxyClient.java:55) ~[elasticsearch-2.1.0.jar:2.1. >> 0] >> at >> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:283) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at >> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at >> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:336) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at org.elasticsearch.client.support.AbstractClient$ClusterAdmin >> .execute(AbstractClient.java:836
Re: [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not accessible
Hi, This is AAI model-loader that failed to initialized correctly. Please bounce model-loader pod. After that, you should be fine. See https://jira.onap.org/browse/OOM-766 Alexis > On Mar 5, 2018, at 6:03 AM, Vivekanandan Muthukrishnan > <vmuthukrish...@aarnanetworks.com> wrote: > > Hi Borislav, > > Thanks for the support. > > Yes, we did the same and we are able to create a new service, but it fails at > the deployment time. > > Here is the UI logs, i guess this is something to do with the service type? I > can only see the only vLB, vFWCL, vCPE, vIMS and the there is not vFW in the > drop down box. > > > Deploy instance failed > > "requestId": "1e8a37c9-4028-45c7-acd3-282b5600b1ba", > "requestType": "createInstance", > "timestamp": undefined, > "requestState": "FAILED", > "requestStatus": "SVC3001Resource not found for %1 using id %2 (msg=%3) > (ec=%4)PUTbusiness/customers/customer/Demonstration/service-subscriptions/service-subscription/vFWCL/service-instances/service-instance/c52f17cf-fbd6-42c0-8019-aa8eeb3cadbcNode > Not Found:object located at > service-design-and-creation/models/model/827361dd-931e-4468-a8ac-d77785039938/model-vers/model-ver/6e064bcd-b574-49cf-9846-48f9c52e7ce8#model-version > not foundERR.5.4.6114", > "precentProgress": "100" > > 03/05/18 10:52:56 HTTP Status: Accepted (202) > { > "requestReferences": { > "instanceId": "c52f17cf-fbd6-42c0-8019-aa8eeb3cadbc", > "requestId": "1e8a37c9-4028-45c7-acd3-282b5600b1ba" > } > } > > VID Screen shot > > > > > > On Mon, Mar 5, 2018 at 4:20 PM, Borislav Glozman <borislav.gloz...@amdocs.com > <mailto:borislav.gloz...@amdocs.com>> wrote: > Please try to redeploy sdc: > deleteAll.sh -n yournamespace -a sdc > createAll.sh -n yournamespace -a sdc > Wait that all pods are running and retry the workaround I wrote in the > previous email. > > > Thanks, > Borislav Glozman > O. +972-9-7669188 > M. +972-52-2835726 > > > -Original Message- > From: Vivekanandan Muthukrishnan [vmuthukrish...@aarnanetworks.com > <mailto:vmuthukrish...@aarnanetworks.com>] > Received: Monday, 05 Mar 2018, 9:59 > To: Borislav Glozman [borislav.gloz...@amdocs.com > <mailto:borislav.gloz...@amdocs.com>] > CC: onap-discuss@lists.onap.org <mailto:onap-discuss@lists.onap.org> > [onap-discuss@lists.onap.org <mailto:onap-discuss@lists.onap.org>] > Subject: Re: [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not accessible > > Hi Borislav, > > Now the SDC UI is not loading and the below are the UI console image and logs. > > I did the steps that you have suggested. Am i missing something here? > > I would really appreciate your help to over come this issue. > > Regards > Vivek > > > > SDC-BE Log snippet (/var/log/onap/sdc/sdc-be/error.log) > > 2018-03-05T07:53:30.316Z [BE-Health-Check-Task] > INFO c.d.d.c.p.DCAwareRoundRobinPolicy > Using data-center name 'SDC-CS-AUTO' for DCAwareRoundRobinPolicy (if this is > incorrect, please provide the correct datacenter name with > DCAwareRoundRobinPolicy constructor) > 2018-03-05T07:53:30.316Z [BE-Health-Check-Task] > INFO com.datastax.driver.core.Cluster > New Cassandra host sdc-cs.onap-sdc/10.42.139.62:9042 > <http://10.42.139.62:9042/> added > 2018-03-05T07:53:30.452Z [BE-Health-Check-Task] > INFO o.o.s.b.c.impl.CassandraHealthCheck > The number of cassandra nodes is:1 > 2018-03-05T07:53:30.452Z [BE-Health-Check-Task] > INFO o.o.s.b.c.impl.CassandraHealthCheck > The cassandra down nodes number is 0 > 2018-03-05T07:53:31.418Z [qtp561247961-19] > INFO o.o.sdc.be.filters.BeServletFilter > serviceInstanceID=null userId=null > localAddr=10.42.94.114 uuid=5bdb4381-f45f-477a-a665-003f8a314702 > remoteAddr=10.42.123.31 No requestID provided -> Generated UUID > 5bdb4381-f45f-477a-a665-003f8a314702 > 2018-03-05T07:53:33.126Z [BE-Health-Check-Task] > INFO o.o.s.b.c.impl.CassandraHealthCheck > creating cluster for Cassandra for monitoring. > 2018-03-05T07:53:33.126Z [BE-Health-Check-Task] > INFO o.o.s.b.d.c.schema.SdcSchemaUtils > connecting to node:[sdc-cs.onap-sdc]. > 2018-03-05T07:53:33.168Z [ES-Health-Check-Thread] > ERROR o.o.sdc.be.dao.impl.ESCatalogDAO > Error while trying to connect to elasticsearch. host: [sdc-es.onap-sdc:9300] > | port: 9200 | error: None of the configured nodes are available: > [{#transport#-1}{10.42.103.200}{sdc-es.onap-sdc/10.42.103.200:9300 > <http:
Re: [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not accessible
Borislav Thanks for the quick response, let me try this one and let you know the results. Regards Vivek On Sun, Mar 4, 2018 at 9:42 PM, Borislav Glozman < borislav.gloz...@amdocs.com> wrote: > Hi Vivek, > > > > This is a known problem with SDC initialization in amsterdam. > > In order to overcome it, you should recreate SDC pods in the following > order: cs -> be -> fe. > > Example, if my namespace is onap, I would: > > # kubectl delete pod -n onap-sdc sdc-cs-fd787c9b6-nt6f2 > > Wait for another instance of sdc-cs pod to be created and running (1/1) > > # kubectl delete pod -n onap-sdc sdc-be-6644d5cd67-vg4xw > > Wait for another instance of sdc-be pod to be created and running (2/2) > > #kubectl delete pod -n onap-sdc sdc-fe-7c4d59c5dc-5d5qw > > Wait for another instance of sdc-fe pod to be created and running (2/2) > > > > Hope this helps. > > > > Thanks, > > Borislav Glozman > > O:+972.9.776.1988 > > M:+972.52.2835726 > > [image: amdocs-a] > > *Amdocs a **Platinum member of ONAP* > <https://www.amdocs.com/open-network/nfv-powered-by-onap> > > > > *From:* onap-discuss-boun...@lists.onap.org [mailto:onap-discuss-bounces@ > lists.onap.org] *On Behalf Of *Vivekanandan Muthukrishnan > *Sent:* Friday, March 2, 2018 6:41 AM > *To:* onap-discuss@lists.onap.org > *Subject:* [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not > accessible > > > > > > Hi All, > > > > We are setting up Amsterdam kuberneter oom. The deployment went fine, even > the ONAP portal login works fine. > > > > However when we try to access the SDC portal it is keep throwing 503 error > message. And we looked at the SDC-BE and SDC-ES logs and it seems like the > elastic serach container failed to initialise "Error : > Chef::Exceptions::FileNotFound: > Cookbook 'sdc-elasticsearch' (0.0.0)" > > > > Did anyone face this issue? Is there any workaround to over come this > issue? > > > > Thanks & Regards > > Vivek > > > > > > *SDC-ES log sinppets* > > > > [2018-02-28T12:24:38+00:00] INFO: Running queued delayed notifications > before re-raising exception > > > > Running handlers: > > [2018-02-28T12:24:38+00:00] ERROR: Running exception handlers > > Running handlers complete > > [2018-02-28T12:24:38+00:00] ERROR: Exception handlers complete > > Chef Client failed. 0 resources updated in 01 seconds > > [2018-02-28T12:24:38+00:00] FATAL: Stacktrace dumped to > /root/chef-solo/cache/chef-stacktrace.out > > [2018-02-28T12:24:38+00:00] FATAL: Please provide the contents of the > stacktrace.out file if you file a bug report > > [2018-02-28T12:24:38+00:00] ERROR: cookbook_file[/usr/share/ > elasticsearch/config/kibana_dashboard_virtualization.json] > (sdc-elasticsearch::ES_6_create_kibana_dashboard_virtualization line 1) > had an error: Chef::Exceptions::FileNotFound: Cookbook > 'sdc-elasticsearch' (0.0.0) does not contain a file at any of these > locations: > > files/debian-8.6/kibana_dashboard_virtualization.json > > files/debian/kibana_dashboard_virtualization.json > > files/default/kibana_dashboard_virtualization.json > > files/kibana_dashboard_virtualization.json > > > > This cookbook _does_ contain: ['files/default/dashboard_BI- > Dashboard.json','files/default/dashboard_Monitoring- > Dashboared.json','files/default/logging.yml','files/ > default/visualization_JVM-used-CPU.json','files/default/ > visualization_JVM-used-Memory.json','files/default/visualization_Show-all- > certified-services-ampersand-resources-(per-day).json',' > files/default/visualization_Show-all-created-Resources- > slash-Services-slash-Products.json','files/default/ > visualization_host-used-CPU.json','files/default/visualization_host-used- > Threads-Num.json','files/default/visualization_number- > of-user-accesses.json','files/default/visualization_Show- > all-distributed-services.json','files/default/visualization_ > JVM-used-Threads-Num.json'] > > [2018-02-28T12:24:38+00:00] FATAL: Chef::Exceptions::ChildConvergeError: > Chef run process exited unsuccessfully (exit code 1) > > [2018-02-28 12:39:06,607][INFO ][cluster.metadata ] > [sdc-es-3319302712-dn5h1] [.kibana] creating index, cause [api], templates > [], shards [1]/[1], mappings [config] > > > > *SDC-BE log snippets* > > > > 2018-02-28T17:40:21.169Z[ES-Health-Check-Thread]ERROR > o.o.sdc.be.dao.impl.ESCatalogDAOError while trying to c > > onnect to elasticsearch. host: [sdc-es.onap-sdc:9300] | port: 9200 | > error: None of the configured nodes are available: [{#transport#-1}{10.42. > > 82.207}{sdc-es.onap-sdc/10.42.82.207:9300}] > > > > *ONAP Portl UI* > > > > > > > > > > > This message and the information contained herein is proprietary and > confidential and subject to the Amdocs policy statement, > you may review at https://www.amdocs.com/about/email-disclaimer > ___ onap-discuss mailing list onap-discuss@lists.onap.org https://lists.onap.org/mailman/listinfo/onap-discuss
Re: [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not accessible
Hi Vivek, This is a known problem with SDC initialization in amsterdam. In order to overcome it, you should recreate SDC pods in the following order: cs -> be -> fe. Example, if my namespace is onap, I would: # kubectl delete pod -n onap-sdc sdc-cs-fd787c9b6-nt6f2 Wait for another instance of sdc-cs pod to be created and running (1/1) # kubectl delete pod -n onap-sdc sdc-be-6644d5cd67-vg4xw Wait for another instance of sdc-be pod to be created and running (2/2) #kubectl delete pod -n onap-sdc sdc-fe-7c4d59c5dc-5d5qw Wait for another instance of sdc-fe pod to be created and running (2/2) Hope this helps. Thanks, Borislav Glozman O:+972.9.776.1988 M:+972.52.2835726 [amdocs-a] Amdocs a Platinum member of ONAP<https://www.amdocs.com/open-network/nfv-powered-by-onap> From: onap-discuss-boun...@lists.onap.org [mailto:onap-discuss-boun...@lists.onap.org] On Behalf Of Vivekanandan Muthukrishnan Sent: Friday, March 2, 2018 6:41 AM To: onap-discuss@lists.onap.org Subject: [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not accessible Hi All, We are setting up Amsterdam kuberneter oom. The deployment went fine, even the ONAP portal login works fine. However when we try to access the SDC portal it is keep throwing 503 error message. And we looked at the SDC-BE and SDC-ES logs and it seems like the elastic serach container failed to initialise "Error : Chef::Exceptions::FileNotFound: Cookbook 'sdc-elasticsearch' (0.0.0)" Did anyone face this issue? Is there any workaround to over come this issue? Thanks & Regards Vivek SDC-ES log sinppets [2018-02-28T12:24:38+00:00] INFO: Running queued delayed notifications before re-raising exception Running handlers: [2018-02-28T12:24:38+00:00] ERROR: Running exception handlers Running handlers complete [2018-02-28T12:24:38+00:00] ERROR: Exception handlers complete Chef Client failed. 0 resources updated in 01 seconds [2018-02-28T12:24:38+00:00] FATAL: Stacktrace dumped to /root/chef-solo/cache/chef-stacktrace.out [2018-02-28T12:24:38+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report [2018-02-28T12:24:38+00:00] ERROR: cookbook_file[/usr/share/elasticsearch/config/kibana_dashboard_virtualization.json] (sdc-elasticsearch::ES_6_create_kibana_dashboard_virtualization line 1) had an error: Chef::Exceptions::FileNotFound: Cookbook 'sdc-elasticsearch' (0.0.0) does not contain a file at any of these locations: files/debian-8.6/kibana_dashboard_virtualization.json files/debian/kibana_dashboard_virtualization.json files/default/kibana_dashboard_virtualization.json files/kibana_dashboard_virtualization.json This cookbook _does_ contain: ['files/default/dashboard_BI-Dashboard.json','files/default/dashboard_Monitoring-Dashboared.json','files/default/logging.yml','files/default/visualization_JVM-used-CPU.json','files/default/visualization_JVM-used-Memory.json','files/default/visualization_Show-all-certified-services-ampersand-resources-(per-day).json','files/default/visualization_Show-all-created-Resources-slash-Services-slash-Products.json','files/default/visualization_host-used-CPU.json','files/default/visualization_host-used-Threads-Num.json','files/default/visualization_number-of-user-accesses.json','files/default/visualization_Show-all-distributed-services.json','files/default/visualization_JVM-used-Threads-Num.json'] [2018-02-28T12:24:38+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1) [2018-02-28 12:39:06,607][INFO ][cluster.metadata ] [sdc-es-3319302712-dn5h1] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [config] SDC-BE log snippets 2018-02-28T17:40:21.169Z[ES-Health-Check-Thread]ERROR o.o.sdc.be.dao.impl.ESCatalogDAOError while trying to c onnect to elasticsearch. host: [sdc-es.onap-sdc:9300] | port: 9200 | error: None of the configured nodes are available: [{#transport#-1}{10.42. 82.207}{sdc-es.onap-sdc/10.42.82.207:9300<http://10.42.82.207:9300/>}] ONAP Portl UI This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer> ___ onap-discuss mailing list onap-discuss@lists.onap.org https://lists.onap.org/mailman/listinfo/onap-discuss
[onap-discuss] Amsterdam Kubernetes OOM SDC UI is not accessible
Hi All, We are setting up Amsterdam kuberneter oom. The deployment went fine, even the ONAP portal login works fine. However when we try to access the SDC portal it is keep throwing 503 error message. And we looked at the SDC-BE and SDC-ES logs and it seems like the elastic serach container failed to initialise "Error : Chef::Exceptions::FileNotFound: Cookbook 'sdc-elasticsearch' (0.0.0)" Did anyone face this issue? Is there any workaround to over come this issue? Thanks & Regards Vivek *SDC-ES log sinppets* [2018-02-28T12:24:38+00:00] INFO: Running queued delayed notifications before re-raising exception Running handlers: [2018-02-28T12:24:38+00:00] ERROR: Running exception handlers Running handlers complete [2018-02-28T12:24:38+00:00] ERROR: Exception handlers complete Chef Client failed. 0 resources updated in 01 seconds [2018-02-28T12:24:38+00:00] FATAL: Stacktrace dumped to /root/chef-solo/cache/chef-stacktrace.out [2018-02-28T12:24:38+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report [2018-02-28T12:24:38+00:00] ERROR: cookbook_file[/usr/share/elast icsearch/config/kibana_dashboard_virtualization.json] (sdc-elasticsearch::ES_6_create_kibana_dashboard_virtualization line 1) had an error: Chef::Exceptions::FileNotFound: Cookbook 'sdc-elasticsearch' (0.0.0) does not contain a file at any of these locations: files/debian-8.6/kibana_dashboard_virtualization.json files/debian/kibana_dashboard_virtualization.json files/default/kibana_dashboard_virtualization.json files/kibana_dashboard_virtualization.json This cookbook _does_ contain: ['files/default/dashboard_BI-D ashboard.json','files/default/dashboard_Monitoring-Dashboare d.json','files/default/logging.yml','files/default/ visualization_JVM-used-CPU.json','files/default/visualiza tion_JVM-used-Memory.json','files/default/visualization_ Show-all-certified-services-ampersand-resources-(per-day). json','files/default/visualization_Show-all-created-Resources-slash- Services-slash-Products.json','files/default/visualization_ host-used-CPU.json','files/default/visualization_host- used-Threads-Num.json','files/default/visualization_number-o f-user-accesses.json','files/default/visualization_Show-all- distributed-services.json','files/default/visualization_JV M-used-Threads-Num.json'] [2018-02-28T12:24:38+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1) [2018-02-28 12:39:06,607][INFO ][cluster.metadata ] [sdc-es-3319302712-dn5h1] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [config] *SDC-BE log snippets* 2018-02-28T17:40:21.169Z[ES-Health-Check-Thread]ERROR o.o.sdc.be.dao.impl.ESCatalogDAOError while trying to c onnect to elasticsearch. host: [sdc-es.onap-sdc:9300] | port: 9200 | error: None of the configured nodes are available: [{#transport#-1}{10.42. 82.207}{sdc-es.onap-sdc/10.42.82.207:9300}] *ONAP Portl UI* ___ onap-discuss mailing list onap-discuss@lists.onap.org https://lists.onap.org/mailman/listinfo/onap-discuss