Using today’s everything (helm charts, docker images) I am not seeing the same problem in my multi-node k8s cluster:
root@aai-resources:/opt/app/aai-resources# ping dev-aai-cassandra-0 PING dev-aai-cassandra-0.rancher.internal (10.42.62.210) 56(84) bytes of data. 64 bytes from 10.42.62.210: icmp_seq=1 ttl=62 time=0.428 ms 64 bytes from 10.42.62.210: icmp_seq=2 ttl=62 time=0.226 ms ^C --- dev-aai-cassandra-0.rancher.internal ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 0.226/0.327/0.428/0.101 ms root@aai-resources:/opt/app/aai-resources# ping dev-aai-cassandra-1 PING dev-aai-cassandra-1.rancher.internal (10.42.26.92) 56(84) bytes of data. 64 bytes from 10.42.26.92: icmp_seq=1 ttl=64 time=0.065 ms ^C --- dev-aai-cassandra-1.rancher.internal ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.065/0.065/0.065/0.000 ms root@aai-resources:/opt/app/aai-resources# ping dev-aai-cassandra-2 PING dev-aai-cassandra-2.rancher.internal (10.42.57.146) 56(84) bytes of data. 64 bytes from 10.42.57.146: icmp_seq=1 ttl=62 time=0.559 ms ^C --- dev-aai-cassandra-2.rancher.internal ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.559/0.559/0.559/0.000 ms Each SS instance is on a different K8s node too. kubectl get pods -o=wide|grep aai-cass dev-aai-cassandra-0 1/1 Running 0 1h 10.42.62.210 k8s-5 dev-aai-cassandra-1 1/1 Running 0 1h 10.42.26.92 k8s-10 dev-aai-cassandra-2 1/1 Running 0 1h 10.42.57.146 k8s-4 From: <[email protected]> on behalf of Borislav Glozman <[email protected]> Date: Thursday, May 10, 2018 at 12:15 PM To: Daniel Silverthorn <[email protected]>, "[email protected]" <[email protected]>, Steven Blimkie <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [onap-discuss] [oom] [aai] Resources fails to start Hi, Same problem here. Did anyone manage to resolve this? Thanks, Borislav Glozman O:+972.9.776.1988 M:+972.52.2835726 [amdocs-a] Amdocs a Platinum member of ONAP<https://www.amdocs.com/open-network/nfv-powered-by-onap> From: [email protected] [mailto:[email protected]] On Behalf Of Daniel Silverthorn Sent: Monday, April 23, 2018 5:40 PM To: [email protected] Subject: [onap-discuss] [oom] [aai] Resources fails to start Using the latest oom as of Friday, I am able to start aai on a single node k8s cluster, but when trying it on a multimode cluster, resources fails to start up with the following error: Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:594) Caused by: java.lang.ExceptionInInitializerError at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:98) at org.onap.aai.dbgen.GenTester.main(GenTester.java:118) ... 8 more Caused by: java.lang.RuntimeException: Failed to instantiate graphs at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:83) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:55) at org.onap.aai.dbmap.AAIGraph$Helper.<clinit>(AAIGraph.java:88) ... 10 more Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:69) at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477) at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409) at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1376) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133) at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113) at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:110) at org.onap.aai.dbmap.AAIGraph.<init>(AAIGraph.java:80) ... 12 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58) ... 20 more Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:619) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.<init>(AstyanaxStoreManager.java:314) ... 25 more Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=aai-aai-cassandra-1(aai-aai-cassandra-1):9160, latency=30001(30001), attempts=3]Timed out waiting for connection at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:231) at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:198) at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:84) at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:117) at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:352) at com.netflix.astyanax.thrift.ThriftClusterImpl.executeSchemaChangeOperation(ThriftClusterImpl.java:146) at com.netflix.astyanax.thrift.ThriftClusterImpl.internalCreateKeyspace(ThriftClusterImpl.java:321) at com.netflix.astyanax.thrift.ThriftClusterImpl.addKeyspace(ThriftClusterImpl.java:294) at org.janusgraph.diskstorage.cassandra.astyanax.AstyanaxStoreManager.ensureKeyspaceExists(AstyanaxStoreManager.java:614) I have tried modifying the Cassandra deployment to only start one instance of Cassandra and got the same result. Has anyone seen something like this? Thanks, Daniel Silverthorn Software Developer Network R&D 613-595-5259 x55259 (office) [amdocs-2017-brand-mark-rgb] Read the latest on Amdocs.com<http://www.amdocs.com/> and the Amdocs blog network<http://blogs.amdocs.com/> – and follow us on Facebook<http://www.facebook.com/Amdocs>, Twitter<http://twitter.com/Amdocs>, LinkedIn<http://www.linkedin.com/company/amdocs> and YouTube<http://www.youtube.com/amdocs>. This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer>
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
