Sathish,
Your async-backup-count and backup-count have non default values. Is it
possible these values are causing hazelcast to consume memory?
Try using default values to see if memory use improves.
You can monitor the JVM with JDK Misson Control or jConsole.
Ray
P.S. To keep your config file as simple as possible, only add properties that
are set to non default values.
P.P.S. The value for instance-name is usually something human readable and
applies to all hazelcast members, such as 'cas-preprod' rather than an ip
address.
On Tue, 2023-11-14 at 09:54 -0800, Sathish Sekar wrote:
Notice: This message was sent from outside the University of Victoria email
system. Please be cautious with links and sensitive information.
Hi Team,
Hazelcast is causing problem. Since we have 3 servers clustered. Earlier we
have tomcat memory of 512mb min and max. I posted in CAS community and they
suggested to increase memory by max 2gb. It is running fine for 12 days and
stopped showing heap memory issue. The following are capturing during heap error
1. Catalina.out.log size is more than 500 mb
2. facing above hazelcast errors in logs
KINDLY DO NEEDFUL
On Tuesday, November 14, 2023 at 4:11:54 PM UTC+5:30 Sathish Sekar wrote:
Hi Team,
Prod server stopped due to following issues. Kindly help. I'm facing heap
memory issue for long time :(
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "hz.10.34.196.43.MetricsRegistry.thread-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "NioReceiver[Catalina-Channel]"
cpa-cas.tomcat is down code=28 .
cpa-cas.tomcat is down code=28 .
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "hz.10.34.196.43.MetricsRegistry.thread-1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "Catalina-utility-1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "Catalina-utility-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "Tribes-MembershipReceiver[Catalina-Channel]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "idle-timeout-task"
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "Tribes-MembershipReceiver[Catalina-Channel]"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "hz.10.34.196.41.MetricsRegistry.thread-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "PooledConnectionFactory@1754546700"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "Tribes-MembershipSender[Catalina-Channel]"
Caused by: org.springframework.beans.BeanInstantiationException: Failed to
instantiate [com.hazelcast.core.HazelcastInstance]: Factory method
'casTicketRegistryHazelcastInstance' threw exception; nested exception is
java.lang.IllegalStateException: Node failed to start!
at
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at
org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
... 131 more
Caused by: java.lang.IllegalStateException: Node failed to start!
at
com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126)
at
com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217)
at
com.hazelcast.instance.impl.HazelcastInstanceFactory.getOrCreateHazelcastInstance(HazelcastInstanceFactory.java:114)
at
org.apereo.cas.config.HazelcastTicketRegistryConfiguration.casTicketRegistryHazelcastInstance(HazelcastTicketRegistryConfiguration.java:73)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
On Thursday, November 2, 2023 at 1:27:56 AM UTC+5:30 Jonathon Taylor wrote:
Not sure why it's complaining about an IP address that's not in your
configuration. I wonder if it's picking up a second IP that's configured? You
could try setting this:
cas.ticket.registry.hazelcast.cluster.core.instance-name=localhost
cas.ticket.registry.hazelcast.cluster.network.public-address=101.34.194.57:5701<http://101.34.194.57:5701>
On Mon, Oct 30, 2023 at 12:03 PM Sathish Sekar <[email protected]> wrote:
Hi Team,
I'm upgrading CAS 6.3.7.4 to 6.6.9. The below properties I have added for
Hazelcast and Ticket Registry in upgraded version. I'm seeing this error which
is printed in logs often. Kindly check below properties and guide me if any
modification/addition/deletion of properties.
ERROR [org.apereo.cas.ticket.registry.HazelcastTicketRegistry] - Unable to
locate ticket map for ticket definition
[DefaultTicketDefinition(implementationClass=class
org.apereo.cas.ticket.TicketGrantingTicketImpl, apiClass=interface
org.apereo.cas.ticket.TicketGrantingTicket, prefix=TGT,
properties=DefaultTicketDefinitionProperties(cascadeRemovals=false,
storageName=ticketGrantingTicketsCache, storageTimeout=9223372036854775807,
storagePassword=null, excludeFromCascade=false),
order=2147483647<tel:(214)%20748-3647>)]
2023-10-26 21:14:08,694 WARN
[org.apereo.cas.ticket.registry.HazelcastTicketRegistry] - No ticket definition
could be found in the catalog to match [TGT-16318-********]
2023-10-26 21:14:08,694 ERROR
[org.apereo.cas.ticket.registry.HazelcastTicketRegistry] -
HazelcastInstance[[101.34.196.41]:5701] is not active!
ProxyRegistry.java:getService:91
ProxyRegistry.java:<init>:73
ProxyServiceImpl.java:lambda$new$0:84
#hazelcast properties
cas.ticket.registry.hazelcast.page-size=500
cas.ticket.registry.hazelcast.cluster.network.members=101.34.194.57,101.34.194.58,101.34.194.59
cas.ticket.registry.hazelcast.cluster.core.instance-name=101.34.194.57
cas.ticket.registry.hazelcast.cluster.network.port=5701
cas.ticket.registry.hazelcast.core.enable-compression=false
cas.ticket.registry.hazelcast.core.enable-management-center-scripting=true
cas.ticket.registry.hazelcast.crypto.enabled=true
cas.ticket.registry.hazelcast.cluster.core.async-backup-count=1
cas.ticket.registry.hazelcast.cluster.core.backup-count=0
cas.ticket.registry.hazelcast.crypto.signing.key=**************
cas.ticket.registry.hazelcast.crypto.signing.keySize=512
cas.ticket.registry.hazelcast.crypto.encryption.key=***********
cas.ticket.registry.hazelcast.crypto.encryption.keySize=16
cas.ticket.registry.hazelcast.crypto.alg=AES
cas.ticket.tgt.remember-me.enabled=true
cas.ticket.tgt.remember-me.time-to-kill-in-seconds=28800
cas.ticket.tgt.timeout.max-time-to-live-in-seconds=28800
cas.ticket.tgt.hard-timeout.time-to-kill-in-seconds=28800
cas.tgc.pin-to-session=true
cas.tgc.remember-me-max-age=P30D
cas.tgc.auto-configure-cookie-path=true
cas.tgc.max-age=-1
cas.tgc.domain=sathishsekar.com<http://sathishsekar.com>
cas.tgc.name<http://cas.tgc.name>=CASTGC
cas.tgc.secure=true
cas.tgc.http-only=true
cas.tgc.crypto.enabled=false
--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS
Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email
[email protected].
To view this discussion on the web visit
https://groups.google.com/a/apereo.org/d/msgid/cas-user/bfd02539-1e72-4e46-98d2-e98969ae8538n%40apereo.org<https://groups.google.com/a/apereo.org/d/msgid/cas-user/bfd02539-1e72-4e46-98d2-e98969ae8538n%40apereo.org?utm_medium=email&utm_source=footer>.
--
Jonathon Taylor (he/him)
Information Security Office
[email protected]
--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS
Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/apereo.org/d/msgid/cas-user/f65e554232a7560ecb2143d9dfc074af84e4b824.camel%40uvic.ca.
