Hi Team, Hazelcast is causing problem. Since we have 3 servers clustered. Earlier we have tomcat memory of 512mb min and max. I posted in CAS community and they suggested to increase memory by max 2gb. It is running fine for 12 days and stopped showing heap memory issue. The following are capturing during heap error 1. Catalina.out.log size is more than 500 mb 2. facing above hazelcast errors in logs
KINDLY DO NEEDFUL On Tuesday, November 14, 2023 at 4:11:54 PM UTC+5:30 Sathish Sekar wrote: > Hi Team, > > Prod server stopped due to following issues. Kindly help. I'm facing heap > memory issue for long time :( > > > > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread > "hz.10.34.196.43.MetricsRegistry.thread-2" > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "NioReceiver[Catalina-Channel]" > cpa-cas.tomcat is down code=28 . > cpa-cas.tomcat is down code=28 . > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread > "hz.10.34.196.43.MetricsRegistry.thread-1" > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "Catalina-utility-1" > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "Catalina-utility-2" > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread > "Tribes-MembershipReceiver[Catalina-Channel]" > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "idle-timeout-task" > java.lang.OutOfMemoryError: Java heap space > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread > "Tribes-MembershipReceiver[Catalina-Channel]" > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread > "hz.10.34.196.41.MetricsRegistry.thread-2" > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread "PooledConnectionFactory@1754546700" > Exception: java.lang.OutOfMemoryError thrown from the > UncaughtExceptionHandler in thread > "Tribes-MembershipSender[Catalina-Channel]" > > > Caused by: org.springframework.beans.BeanInstantiationException: Failed to > instantiate [com.hazelcast.core.HazelcastInstance]: Factory method > 'casTicketRegistryHazelcastInstance' threw exception; nested exception is > java.lang.IllegalStateException: Node failed to start! > at > org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) > at > org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) > ... 131 more > Caused by: java.lang.IllegalStateException: Node failed to start! > at > com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126) > at > com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217) > at > com.hazelcast.instance.impl.HazelcastInstanceFactory.getOrCreateHazelcastInstance(HazelcastInstanceFactory.java:114) > at > org.apereo.cas.config.HazelcastTicketRegistryConfiguration.casTicketRegistryHazelcastInstance(HazelcastTicketRegistryConfiguration.java:73) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at > java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) > > On Thursday, November 2, 2023 at 1:27:56 AM UTC+5:30 Jonathon Taylor wrote: > >> Not sure why it's complaining about an IP address that's not in your >> configuration. I wonder if it's picking up a second IP that's configured? >> You could try setting this: >> >> cas.ticket.registry.hazelcast.cluster.core.instance-name=localhost >> cas.ticket.registry.hazelcast.cluster.network.public-address= >> 101.34.194.57:5701 >> >> On Mon, Oct 30, 2023 at 12:03 PM Sathish Sekar <[email protected]> >> wrote: >> >>> Hi Team, >>> >>> I'm upgrading CAS 6.3.7.4 to 6.6.9. The below properties I have added >>> for Hazelcast and Ticket Registry in upgraded version. I'm seeing this >>> error which is printed in logs often. Kindly check below properties and >>> guide me if any modification/addition/deletion of properties. >>> >>> ERROR [org.apereo.cas.ticket.registry.HazelcastTicketRegistry] - Unable >>> to locate ticket map for ticket definition >>> [DefaultTicketDefinition(implementationClass=class >>> org.apereo.cas.ticket.TicketGrantingTicketImpl, apiClass=interface >>> org.apereo.cas.ticket.TicketGrantingTicket, prefix=TGT, >>> properties=DefaultTicketDefinitionProperties(cascadeRemovals=false, >>> storageName=ticketGrantingTicketsCache, storageTimeout=9223372036854775807, >>> storagePassword=null, excludeFromCascade=false), order=2147483647 >>> <(214)%20748-3647>)] >>> 2023-10-26 21:14:08,694 WARN >>> [org.apereo.cas.ticket.registry.HazelcastTicketRegistry] - No ticket >>> definition could be found in the catalog to match [TGT-16318-********] >>> 2023-10-26 21:14:08,694 ERROR >>> [org.apereo.cas.ticket.registry.HazelcastTicketRegistry] - >>> HazelcastInstance[[101.34.196.41]:5701] is not active! >>> ProxyRegistry.java:getService:91 >>> ProxyRegistry.java:<init>:73 >>> ProxyServiceImpl.java:lambda$new$0:84 >>> >>> #hazelcast properties >>> cas.ticket.registry.hazelcast.page-size=500 >>> cas.ticket.registry.hazelcast.cluster.network.members >>> =101.34.194.57,101.34.194.58,101.34.194.59 >>> cas.ticket.registry.hazelcast.cluster.core.instance-name=101.34.194.57 >>> cas.ticket.registry.hazelcast.cluster.network.port=5701 >>> cas.ticket.registry.hazelcast.core.enable-compression=false >>> cas.ticket.registry.hazelcast.core.enable-management-center-scripting >>> =true >>> cas.ticket.registry.hazelcast.crypto.enabled=true >>> cas.ticket.registry.hazelcast.cluster.core.async-backup-count=1 >>> cas.ticket.registry.hazelcast.cluster.core.backup-count=0 >>> cas.ticket.registry.hazelcast.crypto.signing.key=************** >>> cas.ticket.registry.hazelcast.crypto.signing.keySize=512 >>> cas.ticket.registry.hazelcast.crypto.encryption.key=*********** >>> cas.ticket.registry.hazelcast.crypto.encryption.keySize=16 >>> cas.ticket.registry.hazelcast.crypto.alg=AES >>> >>> >>> cas.ticket.tgt.remember-me.enabled=true >>> cas.ticket.tgt.remember-me.time-to-kill-in-seconds=28800 >>> cas.ticket.tgt.timeout.max-time-to-live-in-seconds=28800 >>> cas.ticket.tgt.hard-timeout.time-to-kill-in-seconds=28800 >>> cas.tgc.pin-to-session=true >>> cas.tgc.remember-me-max-age=P30D >>> cas.tgc.auto-configure-cookie-path=true >>> cas.tgc.max-age=-1 >>> cas.tgc.domain=sathishsekar.com >>> cas.tgc.name=CASTGC >>> cas.tgc.secure=true >>> cas.tgc.http-only=true >>> cas.tgc.crypto.enabled=false >>> >>> -- >>> - Website: https://apereo.github.io/cas >>> - Gitter Chatroom: https://gitter.im/apereo/cas >>> - List Guidelines: https://goo.gl/1VRrw7 >>> - Contributions: https://goo.gl/mh7qDG >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "CAS Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/apereo.org/d/msgid/cas-user/bfd02539-1e72-4e46-98d2-e98969ae8538n%40apereo.org >>> >>> <https://groups.google.com/a/apereo.org/d/msgid/cas-user/bfd02539-1e72-4e46-98d2-e98969ae8538n%40apereo.org?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Jonathon Taylor (he/him) >> Information Security Office >> [email protected] >> > -- - Website: https://apereo.github.io/cas - Gitter Chatroom: https://gitter.im/apereo/cas - List Guidelines: https://goo.gl/1VRrw7 - Contributions: https://goo.gl/mh7qDG --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/ea034ce5-7080-4611-a585-8d67b811edd0n%40apereo.org.
