Hi Roger, Pulled latest OOM code this morning. I still got same error in kubeadm environment.
The aai-resource try to connect to Cassandra via hostname. But got time out. Here is log from aai-resource container. Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=onap-aai-cassandra-2(onap-aai-cassandra-2):9160, latency=30001(30001), attempts=3]Timed out waiting for connection So I checked the hostname, onap-aai-cassandra-1, realized that that hostname is unresolvable in kubeadm environment. aaiadmin@onap-aai-babel-8c6784976-96ph2:/$ ping onap-aai-cassandra-1 ping: unknown host onap-aai-cassandra-1 I also deployed same code in a Rancher cluster, that works. I checked the hostname, in rancher that is reachable. aaiadmin@onap-aai-babel-8c6784976-256g6:/$ ping onap-aai-cassandra-0 PING onap-aai-cassandra-0.rancher.internal (10.42.245.107) 56(84) bytes of data. 64 bytes from 10.42.245.107: icmp_seq=1 ttl=62 time=0.558 ms 64 bytes from 10.42.245.107: icmp_seq=2 ttl=62 time=0.526 ms Do you have any ideal no that? I compared kubedns log between 2 environments, seems similar. Here is some reference may relate to this issue https://github.com/kubernetes/kubernetes/issues/45779<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_issues_45779&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=dohYfIx845zs5L0UQZsaWWeaS3-RUF4VjkiC4bvJ3k0&e=> https://github.com/kubernetes/kubernetes/issues/39197<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_issues_39197&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=zEi_K3fHMgeQFqLuAGsJxQmS7sjHjiC-EIhvnobuvOc&e=> Borislav, Daniel How you set the environment. And could you verify the connection to the pod hostname. Thanks, Nicolas From: Roger Maitland [mailto:[email protected]] Sent: Thursday, May 10, 2018 4:36 PM To: HU, JUN NICOLAS <[email protected]>; Borislav Glozman <[email protected]>; SILVERTHORN, DANIEL <[email protected]>; [email protected]; BLIMKIE, STEVEN <[email protected]>; FORSYTH, JAMES <[email protected]>; Mandeep Khinda <[email protected]> Subject: Re: [onap-discuss] [oom] [aai] Resources fails to start Be aware that a following fix went in yesterday (https://gerrit.onap.org/r/#/c/45941/<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.onap.org_r_-23_c_45941_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=nvHpi3p-BEb8BMtZT91WWMF5WXt-TUHFS5SSdzlWqmI&e=>). With a fresh pull do you still see the problem? Thanks, Roger From: <[email protected]<mailto:[email protected]>> on behalf of "HU, JUN NICOLAS" <[email protected]<mailto:[email protected]>> Date: Thursday, May 10, 2018 at 3:37 PM To: Borislav Glozman <[email protected]<mailto:[email protected]>>, Daniel Silverthorn <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, Steven Blimkie <[email protected]<mailto:[email protected]>>, "FORSYTH, JAMES" <[email protected]<mailto:[email protected]>>, Mandeep Khinda <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [oom] [aai] Resources fails to start Hi Borislav, I have the same issue in my Kubeadm Cluster. I created a JIRA ticket for that. https://jira.onap.org/browse/OOM-1067<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_OOM-2D1067&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=gfgtDWEz2NC_jCLZ72Y8cXp86pzLa3-dWU2Cue_5Om0&e=> I believe that the reason why we got this issue is the POD cannot resolve each other by expected hostnames. The aai-resource try to use cassandra default hostnames, which is the Cassandra pod name, to connect Cassandra. However, aai-resource pod cannot resolve that. I logged in to the container (aai-resource is not running, I use aai-babel to test), try to ping Cassandra hostnames. aaiadmin@onap-aai-babel-8c6784976-5544q:/$ ping onap-aai-cassandra-0 ping: unknown host onap-aai-cassandra-0 But I can ping it by pod ip. onap-aai-cassandra-0 1/1 Running 0 11m 192.168.183.72 kubeadm-test aaiadmin@onap-aai-babel-8c6784976-5544q:/$ ping 192.168.183.72 PING 192.168.183.72 (192.168.183.72) 56(84) bytes of data. 64 bytes from 192.168.183.72: icmp_seq=1 ttl=63 time=0.097 ms 64 bytes from 192.168.183.72: icmp_seq=2 ttl=63 time=0.113 ms 64 bytes from 192.168.183.72: icmp_seq=3 ttl=63 time=0.058 ms Here is some reference may relate to this issue https://github.com/kubernetes/kubernetes/issues/45779<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_issues_45779&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=dohYfIx845zs5L0UQZsaWWeaS3-RUF4VjkiC4bvJ3k0&e=> https://github.com/kubernetes/kubernetes/issues/39197<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes_kubernetes_issues_39197&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=zEi_K3fHMgeQFqLuAGsJxQmS7sjHjiC-EIhvnobuvOc&e=> Thanks, Nicolas From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Borislav Glozman Sent: Thursday, May 10, 2018 12:15 PM To: SILVERTHORN, DANIEL <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; BLIMKIE, STEVEN <[email protected]<mailto:[email protected]>>; FORSYTH, JAMES <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [oom] [aai] Resources fails to start Hi, Same problem here. Did anyone manage to resolve this? Thanks, Borislav Glozman O:+972.9.776.1988 M:+972.52.2835726 [amdocs-a] Amdocs a Platinum member of ONAP<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_open-2Dnetwork_nfv-2Dpowered-2Dby-2Donap&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=esi3MN_9IbBmi4PVHc282U-nZXyug6Mc4mh4VE-i5UY&e=> From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Daniel Silverthorn Sent: Monday, April 23, 2018 5:40 PM To: [email protected]<mailto:[email protected]> Subject: [onap-discuss] [oom] [aai] Resources fails to start Using the latest oom as of Friday, I am able to start aai on a single node k8s cluster, but when trying it on a multimode cluster, resources fails to start up with the following error: Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:594) Caused by: java.lang.ExceptionInInitializerError at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:98) at org.onap.aai.dbgen.GenTester.main(GenTester.java:118) ... 8 more Caused by: java.lang.RuntimeException: Failed to instantiate graphs at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:83) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:55) at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:88) ... 10 more Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69) at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477) at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1376) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113) at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:110) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:80) ... 12 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58) ... 20 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:619) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.<init>(AstyanaxStoreManager.java:314) ... 25 more Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=aai-aai-cassandra-1(aai-aai-cassandra-1):9160, latency=30001(30001), attempts=3]Timed out waiting for connection at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231) at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198) at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:84) at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:117) at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352) at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146) at com.netflix.astyanax.thrift.ThriftClusterImpl.internalCreateKeyspace(ThriftClusterImpl.java:321) at com.netflix.astyanax.thrift.ThriftClusterImpl.addKeyspace(ThriftClusterImpl.java:294) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:614) I have tried modifying the Cassandra deployment to only start one instance of Cassandra and got the same result. Has anyone seen something like this? Thanks, Daniel Silverthorn Software Developer Network R&D 613-595-5259 x55259 (office) [amdocs-2017-brand-mark-rgb] Read the latest on Amdocs.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.amdocs.com_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=HRvdXbAePkL-jAlt2FnxcyR1XRLDyVfqA_Ed0PyaLaE&e=> and the Amdocs blog network<https://urldefense.proofpoint.com/v2/url?u=http-3A__blogs.amdocs.com_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=ogcb2gu-cvqvX0KXoqa7VfH9kl-6jwdyIO1_G3MhUsg&e=> – and follow us on Facebook<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_Amdocs&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=0iOaqB0sLnyUIEmgMsK9czrlCuZXugjFy3rfMXSmxRA&e=>, Twitter<https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_Amdocs&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=yBXd2TPeTMO_VBFH-AFvKHi9co_w5Fo0vJBzIAIpMk8&e=>, LinkedIn<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_amdocs&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=6lc3xV1Ota0CsP7YV4lsiSV8UgEvWWJJ3i-_6O7j0Ec&e=> and YouTube<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_amdocs&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=93ZqSsAs1YtUR_f3OfqLli-Pjfv-HBCsMuTOF7a_qZ4&e=>. This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=hy4pjtsN3rtfEwg_oh0rbt8xidlWCp6PN5D68a2ChBM&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=_GZeCv6HIMYVjl_nEIJoeljq51QP1yPfxMr9qada6N8&s=hy4pjtsN3rtfEwg_oh0rbt8xidlWCp6PN5D68a2ChBM&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=qxToYrwGC0Ebi3fnN8QCAw&m=HfOWhvzCtvUWjtduPm5ExHNTAphre2Lm4Rrfcw1fghI&s=i8aTqZIt2eLgxNTbusL7MLblS14MCXjwSmz8KIR7vOY&e=>
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
