Hi Alex, We see the below error in aai-model container. Do we need to delete model-loader-service container? So that Kubernetes can bring it back again.
Thanks for your support. Regards Vivek *model-loader-service log (/var/log/onap/aai/aai-ml/error.log)* 2018-03-05T07:22:27.036Z [Timer-0] ERROR org.openecomp.sdc.http.AsdcConnectorClient status from ASDC is org.openecomp.sdc.http.HttpAsdcResponse@3bdfffb4 2018-03-05T07:22:27.036Z [Timer-0] ERROR org.openecomp.sdc.http.AsdcConnectorClient DistributionClientResultImpl [responseStatus=ASDC_CONNECTION_FAILED, responseMessage=ASDC server problem] 2018-03-05T07:22:27.051Z [Timer-0] ERROR org.onap.aai.modelloader.service.SdcConnectionJob MDLSVC2001E|MDLSVC2001E Unable to register with ASDC: Failed to initialize distribution client: ASDC server problem| 2018-03-05T07:22:27.051Z [Timer-0] INFO org.openecomp.sdc.impl.DistributionClientImpl DistributionClient - init 2018-03-05T07:24:34.409Z [Timer-0] ERROR org.openecomp.sdc.http.HttpAsdcClient failed to connect to url: /sdc/v1/distributionUebCluster org.apache.http.conn.HttpHostConnectException: Connect to sdc-be.onap-sdc:8443 [sdc-be.onap-sdc/10.43.27.92] failed: Connection timed out (Connection timed out) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.openecomp.sdc.http.HttpAsdcClient.getRequest(HttpAsdcClient.java:278) at org.openecomp.sdc.http.AsdcConnectorClient.performAsdcServerRequest(AsdcConnectorClient.java:143) at org.openecomp.sdc.http.AsdcConnectorClient.getServerList(AsdcConnectorClient.java:94) at org.openecomp.sdc.impl.DistributionClientImpl.initUebServerList(DistributionClientImpl.java:325) at org.openecomp.sdc.impl.DistributionClientImpl.init(DistributionClientImpl.java:241) at org.onap.aai.modelloader.service.SdcConnectionJob.run(SdcConnectionJob.java:59) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: java.net.ConnectException: Connection timed out (Connection timed out) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134) ... 17 common frames omitted 2018-03-05T07:24:34.410Z [Timer-0] ERROR org.openecomp.sdc.http.AsdcConnectorClient status from ASDC is org.openecomp.sdc.http.HttpAsdcResponse@7e0e523e On Mon, Mar 5, 2018 at 6:26 PM, Alexis de Talhouët <[email protected]> wrote: > Hi, > > This is AAI model-loader that failed to initialized correctly. Please > bounce model-loader pod. After that, you should be fine. > See https://jira.onap.org/browse/OOM-766 > > Alexis > > On Mar 5, 2018, at 6:03 AM, Vivekanandan Muthukrishnan < > [email protected]> wrote: > > Hi Borislav, > > Thanks for the support. > > Yes, we did the same and we are able to create a new service, but it fails > at the deployment time. > > Here is the UI logs, i guess this is something to do with the service > type? I can only see the only vLB, vFWCL, vCPE, vIMS and the there is not > vFW in the drop down box. > > > *Deploy instance failed* > > "requestId": "1e8a37c9-4028-45c7-acd3-282b5600b1ba", > "requestType": "createInstance", > "timestamp": undefined, > "requestState": "FAILED", > "requestStatus": "SVC3001Resource not found for %1 using id %2 (msg=%3) > (ec=%4)PUTbusiness/customers/customer/Demonstration/ > service-subscriptions/service-subscription/vFWCL/service- > instances/service-instance/c52f17cf-fbd6-42c0-8019-aa8eeb3cadbcNode Not > Found:object located at service-design-and-creation/ > models/model/827361dd-931e-4468-a8ac-d77785039938/model- > vers/model-ver/6e064bcd-b574-49cf-9846-48f9c52e7ce8#model-version not > foundERR.5.4.6114", > "precentProgress": "100" > > 03/05/18 10:52:56 HTTP Status: Accepted (202) > { > "requestReferences": { > "instanceId": "c52f17cf-fbd6-42c0-8019-aa8eeb3cadbc", > "requestId": "1e8a37c9-4028-45c7-acd3-282b5600b1ba" > } > } > > *VID Screen shot* > > <image.png> > > > > On Mon, Mar 5, 2018 at 4:20 PM, Borislav Glozman < > [email protected]> wrote: > >> Please try to redeploy sdc: >> deleteAll.sh -n yournamespace -a sdc >> createAll.sh -n yournamespace -a sdc >> Wait that all pods are running and retry the workaround I wrote in the >> previous email. >> >> >> Thanks, >> Borislav Glozman >> O. +972-9-7669188 >> M. +972-52-2835726 >> >> >> -----Original Message----- >> *From:* Vivekanandan Muthukrishnan [[email protected]] >> *Received:* Monday, 05 Mar 2018, 9:59 >> *To:* Borislav Glozman [[email protected]] >> *CC:* [email protected] [[email protected]] >> *Subject:* Re: [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not >> accessible >> >> Hi Borislav, >> >> Now the SDC UI is not loading and the below are the UI console image and >> logs. >> >> I did the steps that you have suggested. Am i missing something here? >> >> I would really appreciate your help to over come this issue. >> >> Regards >> Vivek >> >> >> >> *SDC-BE Log snippet (/var/log/onap/sdc/sdc-be/error.log)* >> >> 2018-03-05T07:53:30.316Z [BE-Health-Check-Task] INFO >> c.d.d.c.p.DCAwareRoundRobinPolicy Using data-center name 'SDC-CS-AUTO' >> for DCAwareRoundRobinPolicy (if this is incorrect, please provide the >> correct datacenter name with DCAwareRoundRobinPolicy constructor) >> 2018-03-05T07:53:30.316Z [BE-Health-Check-Task] INFO >> com.datastax.driver.core.Cluster New Cassandra host sdc-cs.onap-sdc/ >> 10.42.139.62:9042 added >> 2018-03-05T07:53:30.452Z [BE-Health-Check-Task] INFO >> o.o.s.b.c.impl.CassandraHealthCheck The number of cassandra nodes is:1 >> 2018-03-05T07:53:30.452Z [BE-Health-Check-Task] INFO >> o.o.s.b.c.impl.CassandraHealthCheck The cassandra down nodes number is 0 >> 2018-03-05T07:53:31.418Z [qtp561247961-19] INFO >> o.o.sdc.be.filters.BeServletFilter serviceInstanceID=null userId=null >> localAddr=10.42.94.114 uuid=5bdb4381-f45f-477a-a665-003f8a314702 >> remoteAddr=10.42.123.31 No requestID provided -> Generated UUID >> 5bdb4381-f45f-477a-a665-003f8a314702 >> 2018-03-05T07:53:33.126Z [BE-Health-Check-Task] INFO >> o.o.s.b.c.impl.CassandraHealthCheck creating cluster for Cassandra for >> monitoring. >> 2018-03-05T07:53:33.126Z [BE-Health-Check-Task] INFO >> o.o.s.b.d.c.schema.SdcSchemaUtils connecting to node:[sdc-cs.onap-sdc]. >> 2018-03-05T07:53:33.168Z [ES-Health-Check-Thread] ERROR >> o.o.sdc.be.dao.impl.ESCatalogDAO Error while trying to connect to >> elasticsearch. host: [sdc-es.onap-sdc:9300] | port: 9200 | error: None of >> the configured nodes are available: [{#transport#-1}{10.42.103.200 >> }{sdc-es.onap-sdc/10.42.103.200:9300}] >> org.elasticsearch.client.transport.NoNodeAvailableException: None of the >> configured nodes are available: [{#transport#-1}{10.42.103.200 >> }{sdc-es.onap-sdc/10.42.103.200:9300}] >> at org.elasticsearch.client.transport.TransportClientNodesServi >> ce.ensureNodesAreAvailable(TransportClientNodesService.java:290) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at org.elasticsearch.client.transport.TransportClientNodesServi >> ce.execute(TransportClientNodesService.java:207) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at org.elasticsearch.client.transport.support.TransportProxyCli >> ent.execute(TransportProxyClient.java:55) ~[elasticsearch-2.1.0.jar:2.1. >> 0] >> at >> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:283) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at >> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at >> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:336) >> ~[elasticsearch-2.1.0.jar:2.1.0] >> at org.elasticsearch.client.support.AbstractClient$ClusterAdmin >> .execute(AbstractClient.java:836) ~[elasticsearch-2.1.0.jar:2.1.0] >> at org.elasticsearch.client.support.AbstractClient$ClusterAdmin >> .health(AbstractClient.java:856) ~[elasticsearch-2.1.0.jar:2.1.0] >> at >> org.openecomp.sdc.be.dao.impl.ESCatalogDAO.checkHealth(ESCatalogDAO.java:104) >> ~[catalog-dao-1.1.0.jar:na] >> at >> org.openecomp.sdc.be.dao.impl.ESCatalogDAO.access$100(ESCatalogDAO.java:52) >> ~[catalog-dao-1.1.0.jar:na] >> at org.openecomp.sdc.be.dao.impl.ESCatalogDAO$HealthCheckSchedu >> ledTask.run(ESCatalogDAO.java:75) ~[catalog-dao-1.1.0.jar:na] >> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> [na:1.8.0_141] >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> [na:1.8.0_141] >> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_141] >> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu >> tureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_141] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> [na:1.8.0_141] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> [na:1.8.0_141] >> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141] >> 2018-03-05T07:53:33.313Z [BE-Health-Check-Task] INFO >> c.d.d.c.p.DCAwareRoundRobinPolicy Using data-center name 'SDC-CS-AUTO' >> for DCAwareRoundRobinPolicy (if this is incorrect, please provide the >> correct datacenter name with DCAwareRoundRobinPolicy constructor) >> 2018-03-05T07:53:33.313Z [BE-Health-Check-Task] INFO >> com.datastax.driver.core.Cluster New Cassandra host sdc-cs.onap-sdc/ >> 10.42.139.62:9042 added >> 2018-03-05T07:53:33.429Z [BE-Health-Check-Task] INFO >> o.o.s.b.c.impl.CassandraHealthCheck The number of cassandra nodes is:1 >> 2018-03-05T07:53:33.429Z [BE-Health-Check-Task] INFO >> o.o.s.b.c.impl.CassandraHealthCheck The cassandra down nodes number >> >> >> SDC UI Load error >> >> <image.png> >> >> >> >> On Mon, Mar 5, 2018 at 1:08 PM, Vivekanandan Muthukrishnan < >> [email protected]> wrote: >> >>> Borislav >>> >>> Thanks for the quick response, let me try this one and let you know the >>> results. >>> >>> Regards >>> Vivek >>> >>> >>> On Sun, Mar 4, 2018 at 9:42 PM, Borislav Glozman < >>> [email protected]> wrote: >>> >>>> Hi Vivek, >>>> >>>> >>>> >>>> This is a known problem with SDC initialization in amsterdam. >>>> >>>> In order to overcome it, you should recreate SDC pods in the following >>>> order: cs -> be -> fe. >>>> >>>> Example, if my namespace is onap, I would: >>>> >>>> # kubectl delete pod -n onap-sdc sdc-cs-fd787c9b6-nt6f2 >>>> >>>> Wait for another instance of sdc-cs pod to be created and running (1/1) >>>> >>>> # kubectl delete pod -n onap-sdc sdc-be-6644d5cd67-vg4xw >>>> >>>> Wait for another instance of sdc-be pod to be created and running (2/2) >>>> >>>> #kubectl delete pod -n onap-sdc sdc-fe-7c4d59c5dc-5d5qw >>>> >>>> Wait for another instance of sdc-fe pod to be created and running (2/2) >>>> >>>> >>>> >>>> Hope this helps. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Borislav Glozman >>>> >>>> O:+972.9.776.1988 >>>> >>>> M:+972.52.2835726 >>>> >>>> <image001.jpg> >>>> >>>> *Amdocs a **Platinum member of ONAP* >>>> <https://www.amdocs.com/open-network/nfv-powered-by-onap> >>>> >>>> >>>> >>>> *From:* [email protected] [mailto: >>>> [email protected]] *On Behalf Of *Vivekanandan >>>> Muthukrishnan >>>> *Sent:* Friday, March 2, 2018 6:41 AM >>>> *To:* [email protected] >>>> *Subject:* [onap-discuss] Amsterdam Kubernetes OOM SDC UI is not >>>> accessible >>>> >>>> >>>> >>>> >>>> >>>> Hi All, >>>> >>>> >>>> >>>> We are setting up Amsterdam kuberneter oom. The deployment went fine, >>>> even the ONAP portal login works fine. >>>> >>>> >>>> >>>> However when we try to access the SDC portal it is keep throwing 503 >>>> error message. And we looked at the SDC-BE and SDC-ES logs and it seems >>>> like the elastic serach container failed to initialise "Error : >>>> Chef::Exceptions::FileNotFound: Cookbook 'sdc-elasticsearch' (0.0.0)" >>>> >>>> >>>> >>>> Did anyone face this issue? Is there any workaround to over come this >>>> issue? >>>> >>>> >>>> >>>> Thanks & Regards >>>> >>>> Vivek >>>> >>>> >>>> >>>> >>>> >>>> *SDC-ES log sinppets* >>>> >>>> >>>> >>>> [2018-02-28T12:24:38+00:00] INFO: Running queued delayed notifications >>>> before re-raising exception >>>> >>>> >>>> >>>> Running handlers: >>>> >>>> [2018-02-28T12:24:38+00:00] ERROR: Running exception handlers >>>> >>>> Running handlers complete >>>> >>>> [2018-02-28T12:24:38+00:00] ERROR: Exception handlers complete >>>> >>>> Chef Client failed. 0 resources updated in 01 seconds >>>> >>>> [2018-02-28T12:24:38+00:00] FATAL: Stacktrace dumped to >>>> /root/chef-solo/cache/chef-stacktrace.out >>>> >>>> [2018-02-28T12:24:38+00:00] FATAL: Please provide the contents of the >>>> stacktrace.out file if you file a bug report >>>> >>>> [2018-02-28T12:24:38+00:00] ERROR: cookbook_file[/usr/share/elast >>>> icsearch/config/kibana_dashboard_virtualization.json] >>>> (sdc-elasticsearch::ES_6_create_kibana_dashboard_virtualization line >>>> 1) had an error: Chef::Exceptions::FileNotFound: Cookbook >>>> 'sdc-elasticsearch' (0.0.0) does not contain a file at any of these >>>> locations: >>>> >>>> files/debian-8.6/kibana_dashboard_virtualization.json >>>> >>>> files/debian/kibana_dashboard_virtualization.json >>>> >>>> files/default/kibana_dashboard_virtualization.json >>>> >>>> files/kibana_dashboard_virtualization.json >>>> >>>> >>>> >>>> This cookbook _does_ contain: ['files/default/dashboard_BI-D >>>> ashboard.json','files/default/dashboard_Monitoring-Dashboare >>>> d.json','files/default/logging.yml','files/default/visualiza >>>> tion_JVM-used-CPU.json','files/default/visualization_JVM- >>>> used-Memory.json','files/default/visualization_Show-all- >>>> certified-services-ampersand-resources-(per-day).json','fil >>>> es/default/visualization_Show-all-created-Resources-slash- >>>> Services-slash-Products.json','files/default/visualization_ >>>> host-used-CPU.json','files/default/visualization_host- >>>> used-Threads-Num.json','files/default/visualization_number- >>>> of-user-accesses.json','files/default/visualization_Show- >>>> all-distributed-services.json','files/default/visualization_ >>>> JVM-used-Threads-Num.json'] >>>> >>>> [2018-02-28T12:24:38+00:00] FATAL: Chef::Exceptions::ChildConvergeError: >>>> Chef run process exited unsuccessfully (exit code 1) >>>> >>>> [2018-02-28 12:39:06,607][INFO ][cluster.metadata ] >>>> [sdc-es-3319302712-dn5h1] [.kibana] creating index, cause [api], templates >>>> [], shards [1]/[1], mappings [config] >>>> >>>> >>>> >>>> *SDC-BE log snippets* >>>> >>>> >>>> >>>> 2018-02-28T17:40:21.169Z [ES-Health-Check-Thread] ERROR >>>> o.o.sdc.be.dao.impl.ESCatalogDAO Error while trying to >>>> c >>>> >>>> onnect to elasticsearch. host: [sdc-es.onap-sdc:9300] | port: 9200 | >>>> error: None of the configured nodes are available: [{#transport#-1}{10.42. >>>> >>>> 82.207}{sdc-es.onap-sdc/10.42.82.207:9300}] >>>> >>>> >>>> >>>> *ONAP Portl UI* >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> This message and the information contained herein is proprietary and >>>> confidential and subject to the Amdocs policy statement, >>>> you may review at https://www.amdocs.com/about/email-disclaimer >>>> >>> >>> >> This message and the information contained herein is proprietary and >> confidential and subject to the Amdocs policy statement, >> you may review at https://www.amdocs.com/about/email-disclaimer >> > > _______________________________________________ > onap-discuss mailing list > [email protected] > https://lists.onap.org/mailman/listinfo/onap-discuss > > >
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
