I believe, if you have 4 members, that asyncBackupCount should be 3. Because a node doesn't back itself up. (Hazelcast might be smart enough to fix that itself, but I don't know.)
Also, I'm curious as to why you have tcpIpEnabled set to true? You'd be much better off, from a performance standpoint, setting it to false and using UDP. TCP blocks, UDP doesn't. I've never run this in docker, so I don't know what, if anything, you need to do differently. But I would be at least a little suspicious of the interactions there -- it should work of course, but could you be missing something in the docker config, or the config on the host? -- DAVID A. CURRY, CISSP *DIRECTOR • INFORMATION SECURITY & PRIVACY* THE NEW SCHOOL • INFORMATION TECHNOLOGY 71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003 +1 646 909-4728 • [email protected] On Thu, Feb 6, 2020 at 4:36 PM 'Maksim Kopeyka' via CAS Community < [email protected]> wrote: > Hi David, > > I tried similar config with 4 nodes: > >> cas.ticket.registry.hazelcast.cluster.members=${HAZELCAST_CLUSTER_MEMBERS} >> cas.ticket.registry.hazelcast.cluster.asyncBackupCount=4 >> cas.ticket.registry.hazelcast.cluster.backupCount=0 >> cas.ticket.registry.hazelcast.cluster.port=5701 >> cas.ticket.registry.hazelcast.cluster.portAutoIncrement=false >> cas.ticket.registry.hazelcast.cluster.instanceName=localhost >> >> cas.ticket.registry.hazelcast.cluster.publicAddress=${HAZELCAST_PUBLIC_ADDRESS} >> cas.ticket.registry.hazelcast.cluster.tcpipEnabled=true >> cas.ticket.registry.hazelcast.crypto.enabled=false > > > I see this output on each node i.e. hazelcast creates a cluster and sees > all nodes: > > 2020-02-06 21:20:49,235 INFO >> [com.hazelcast.internal.cluster.ClusterService] - >> <[ecdc-rant-affiliateidp-dev-1]:5701 [dev] [3.12.4] >> Members {size:4, ver:4} [ >> Member [wcdc-rant-affiliateidp-dev-1]:5701 - >> a245c93b-beb0-4929-b831-e40a323cad8b >> Member [ecdc-rant-affiliateidp-dev-2]:5701 - >> bcbcd799-8cb8-4e5d-8802-5d95d4015ffd >> Member [wcdc-rant-affiliateidp-dev-2]:5701 - >> 9d3f52c9-1475-462e-844a-1b534efdca73 >> Member [ecdc-rant-affiliateidp-dev-1]:5701 - >> e9f81f52-7a99-4428-a402-5a2f48cba838 this >> ] >> > > > > However tickets distribution doesn't work. Nodes 1, 2, 3 don't know about > session on Node 4. > > I don't see any errors in the logs related to hazelcast but this one > appears time to time > >> 2020-02-06 17:31:56,248 ERROR >> [org.apereo.cas.web.flow.executor.EncryptedTranscoder] - <Null input buffer> >> java.lang.IllegalArgumentException: Null input buffer >> at javax.crypto.Cipher.doFinal(Unknown Source) ~[?:?] >> at >> org.apereo.cas.util.cipher.BaseBinaryCipherExecutor.decode(BaseBinaryCipherExecutor.java:92) >> ~[cas-server-core-util-api-6.1.3.jar!/:6.1.3] > > > > On Wednesday, February 5, 2020 at 9:28:43 PM UTC+2, David Curry wrote: >> >> Maksim, >> >> If you don't want to ever lose tickets, then you would want all nodes to >> back up all other nodes. So if you have 3 member nodes, you would want 2 >> async backup nodes (asyncBackupCount) and also you'd probably want to >> disable the default sync backup (backupCount) node since it will block. >> Here are the settings we're running with in production (although this is >> CAS 5): >> >> cas.ticket.registry.hazelcast.cluster.members: >> cas01.newschool.edu,cas02.newschool.edu,cas03.newschool.edu, >> cas04.newschool.edu,cas05.newschool.edu >> cas.ticket.registry.hazelcast.cluster.asyncBackupCount: 4 >> cas.ticket.registry.hazelcast.cluster.backupCount: 0 >> cas.ticket.registry.hazelcast.cluster.port: 5701 >> cas.ticket.registry.hazelcast.cluster.portAutoIncrement: false >> cas.ticket.registry.hazelcast.crypto.encryption.key: >> xxxIoXN6SBU5bF+iAVTKgw== >> cas.ticket.registry.hazelcast.crypto.signing.key: >> >> xxxmEbPGT_MXg0JWYLTe4oFaOaklocCqlY2VuHBdAHuh0V6-PdQxmgi4tTA3CZZos8TUbzg-L9nYHJpA5RqcvA >> cas.ticket.registry.hazelcast.crypto.enabled: true >> >> This works well for us behind an F5 load balancer; we do not use sticky >> sessions. We can (and do) reboot servers in the pool without anyone getting >> re-prompted to log in (just don't reboot them all at once). >> >> The crypto stuff (last three lines) is not needed for this to work, but >> you (arguably) might want it in production. You can leave it off while >> getting things to work and enable it later. >> >> One other thing -- did you remember to open 5701 in the firewall on all >> the servers? >> >> -- >> >> DAVID A. CURRY, CISSP >> *DIRECTOR • INFORMATION SECURITY & PRIVACY* >> THE NEW SCHOOL • INFORMATION TECHNOLOGY >> >> 71 FIFTH AVE., 9TH FL., NEW YORK, NY 10003 >> +1 646 909-4728 • [email protected] >> >> >> On Wed, Feb 5, 2020 at 1:40 PM Ray Bon <[email protected]> wrote: >> >>> Maksim, >>> >>> There is this config setting >>> cas.ticket.registry.hazelcast.cluster.members= >>> Add IPs of all members to the list. >>> https://apereo.github.io/cas/6.1.x/configuration/Configuration-Properties.html#hazelcast-ticket-registry >>> and the link to common settings. >>> >>> There are some hazelcast loggers in log4j2.xml >>> >>> <AsyncLogger name="com.hazelcast" >>> level="${sys:hazelcast.log.level}" includeLocation="true" /> >>> >>> Ray >>> >>> On Wed, 2020-02-05 at 09:16 -0800, 'Maksim Kopeyka' via CAS Community >>> wrote: >>> >>> Ray, >>> >>> I asked about CAS functionality to distribute tickets across nodes. I >>> need specific CAS functionality based on Hazelcast and seems to me this >>> functionality doesn't work as expected so I need to check it somehow. Maybe >>> with some debug logging. >>> I have a cluster with several nodes of CAS with hazelcast ticket >>> registry and I have a load balancer. If I turn off sticky sessions CAS asks >>> me about username/password every time. This cluster works in the same way >>> without hazelcast ticket registry. So seems to me this functionality >>> doesn't work and I don't see any errors in the logs. >>> >>> On Wednesday, February 5, 2020 at 7:03:55 PM UTC+2, rbon wrote: >>> >>> Maksim, >>> >>> I do not know if there is a stand alone client for accessing hazelcast >>> data. In the docs, >>> https://docs.hazelcast.org/docs/4.0/manual/html-single/index.html, >>> there is a section on clients and one on management. It looks like you >>> would have to create an application yourself but someone has probably done >>> that already. >>> >>> Ray >>> >>> On Wed, 2020-02-05 at 08:11 -0800, 'Maksim Kopeyka' via CAS Community >>> wrote: >>> >>> Hi Ray, >>> >>> Seems to me Hazelcast doesn't distribute data across all nodes because >>> each node doesn't have information about sessions on other nodes. >>> How to check data distribution? >>> >>> On Friday, January 31, 2020 at 11:02:05 PM UTC+2, rbon wrote: >>> >>> Maksim, >>> >>> Hazelcast is distributed but not replicated. Thus, when a server goes >>> down, the tickets on that server are lost. You have to relogin only if your >>> ticket was on that server. >>> Hazelcast has some mechanism of determining which node has which ticket. >>> It may also be possible to make hazelcast replicated but I have not tried. >>> >>> Ray >>> >>> P.S. you have two node3s in your config. >>> >>> On Fri, 2020-01-31 at 11:51 -0800, 'Maksim Kopeyka' via CAS Community >>> wrote: >>> >>> Hi Andy, >>> >>> Your example is very helpful. Thank you. >>> I see how hazelcast tickets registry works on my local env. I turned off >>> active container and another container continues to work with my active >>> session without relogin. >>> >>> However on my remote env. with CAS 6.1.3 it doesn't work in this way. >>> Load balancer ask me to re-login If I turn off active container. >>> Each node with hazelcast sees other nodes. I see such messages on all >>> nodes in case node1 is turned off: >>> >>> WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701 >>> [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => >>> java.net.SocketException {Connection refused to address node1/xx.xx.xx. >>> xx:5701}, Error-Count: 5> >>> WARN [com.hazelcast.internal.cluster.impl.MembershipManager] - <[node2]: >>> 5701 [dev] [3.12.4] Member [node1]:5701 - >>> b1fba639-dfff-4536-b5f4-a8681920594d >>> is suspected to be dead for reason: No connection> >>> WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701 >>> [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => >>> java.net.SocketException {Connection refused to address node1/xx.xx.xx. >>> xx:5701}, Error-Count: 6> >>> WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701 >>> [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => >>> java.net.SocketException {Connection refused to address node1/xx.xx.xx. >>> xx:5701}, Error-Count: 7> >>> WARN [com.hazelcast.nio.tcp.TcpIpConnectionErrorHandler] - <[node2]:5701 >>> [dev] [3.12.4] Removing connection to endpoint [node1]:5701 Cause => >>> java.net.SocketException {Connection refused to address node1/xx.xx.xx. >>> xx:57001}, Error-Count: 8> >>> >>> >>> >>> This is my hazelcast settings: >>> >>> cas.ticket.registry.hazelcast.cluster.members=node1:5701,node2:5701, >>> node3:5701,node3:5701 >>> cas.ticket.registry.hazelcast.cluster.asyncBackupCount=3 >>> cas.ticket.registry.hazelcast.cluster.port=5701 >>> cas.ticket.registry.hazelcast.cluster.portAutoIncrement=false >>> cas.ticket.registry.hazelcast.cluster.instanceName=localhost >>> cas.ticket.registry.hazelcast.cluster.publicAddress=node1:5701 >>> cas.ticket.registry.hazelcast.cluster.tcpipEnabled=true >>> >>> >>> Why hazelcast doesn't share data across the cluster? >>> >>> These messages I see on startup >>> >>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] >>> You configured your member address as host name. Please be aware of >>> that your dns can be spoofed. Make sure that your dns configurations >>> are correct.> >>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] >>> You configured your member address as host name. Please be aware of >>> that your dns can be spoofed. Make sure that your dns configurations >>> are correct.> >>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] >>> You configured your member address as host name. Please be aware of >>> that your dns can be spoofed. Make sure that your dns configurations >>> are correct.> >>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] >>> You configured your member address as host name. Please be aware of >>> that your dns can be spoofed. Make sure that your dns configurations >>> are correct.> >>> WARN [com.hazelcast.instance.AddressPicker] - <[LOCAL] [dev] [3.12.4] >>> Could not find a matching address to start with! Picking one of non-loopback >>> addresses.> >>> INFO [org.apereo.cas.util.CoreTicketUtils] - <Ticket registry encryption >>> /signing is turned off. This MAY NOT be safe in a clustered production >>> environment. Consider using other choices to handle encryption, signing >>> and verification of ticket registry tickets, and verify the chosen >>> ticket registry does support this behavior.> >>> >>> >>> On Wednesday, January 22, 2020 at 3:18:34 AM UTC+2, Andy Ng wrote: >>> >>> Hi Maksim, >>> >>> Pretty sure: >>> cas.ticket.registry.hazelcast.cluster.public-address >>> and >>> cas.ticket.registry.hazelcast.cluster.publicAddress >>> >>> Both works the same, since spring property allows both camelCase and >>> kebak-case. >>> >>> >>> And I did successfully use docker CAS and use Hazelcast as ticketing >>> system, however I am using it for demo so I just included a whole bunch of >>> private IP so it works...... >>> >>> here's my CAS properties: >>> >>> >>> cas.ticket.registry.hazelcast.cluster.members=172.20.0.1,172.20.0.2,172.20.0.3,172.20.0.4,172.20.0.5,172.20.0.6,172.20.0.7,172.20.0.8,172.20.0.9,172.20.0.10 >>> cas.ticket.registry.hazelcast.cluster.instanceName=localhost >>> my project link as well so you can reference if want to: >>> https://github.com/NgSekLong/SelectUrCAS/blob/master/source/ticket-registry/hazelcast/cas.yml >>> >>> >>> Cheers! >>> - Andy >>> >>> -- >>> >>> >>> Ray Bon >>> Programmer Analyst >>> Development Services, University Systems >>> 2507218831 | CLE 019 | [email protected] >>> >>> I respectfully acknowledge that my place of work is located within the >>> ancestral, traditional and unceded territory of the Songhees, Esquimalt and >>> WSÁNEĆ Nations. >>> >>> -- >>> >>> >>> Ray Bon >>> Programmer Analyst >>> Development Services, University Systems >>> 2507218831 | CLE 019 | [email protected] >>> >>> I respectfully acknowledge that my place of work is located within the >>> ancestral, traditional and unceded territory of the Songhees, Esquimalt and >>> WSÁNEĆ Nations. >>> >>> -- >>> >>> Ray Bon >>> Programmer Analyst >>> Development Services, University Systems >>> 2507218831 | CLE 019 | [email protected] >>> >>> I respectfully acknowledge that my place of work is located within the >>> ancestral, traditional and unceded territory of the Songhees, Esquimalt and >>> WSÁNEĆ Nations. >>> >>> -- >>> - Website: https://apereo.github.io/cas >>> - Gitter Chatroom: https://gitter.im/apereo/cas >>> - List Guidelines: https://goo.gl/1VRrw7 >>> - Contributions: https://goo.gl/mh7qDG >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "CAS Community" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/a/apereo.org/d/msgid/cas-user/cfe74ef4f3aeb0c1dfc506a7257e659418e41f0e.camel%40uvic.ca >>> <https://groups.google.com/a/apereo.org/d/msgid/cas-user/cfe74ef4f3aeb0c1dfc506a7257e659418e41f0e.camel%40uvic.ca?utm_medium=email&utm_source=footer> >>> . >>> >> -- > - Website: https://apereo.github.io/cas > - Gitter Chatroom: https://gitter.im/apereo/cas > - List Guidelines: https://goo.gl/1VRrw7 > - Contributions: https://goo.gl/mh7qDG > --- > You received this message because you are subscribed to the Google Groups > "CAS Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/apereo.org/d/msgid/cas-user/8d82a0af-84cb-4e9e-96e9-7e1e688eba1d%40apereo.org > <https://groups.google.com/a/apereo.org/d/msgid/cas-user/8d82a0af-84cb-4e9e-96e9-7e1e688eba1d%40apereo.org?utm_medium=email&utm_source=footer> > . > -- - Website: https://apereo.github.io/cas - Gitter Chatroom: https://gitter.im/apereo/cas - List Guidelines: https://goo.gl/1VRrw7 - Contributions: https://goo.gl/mh7qDG --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CA%2Bd9XAMEjrEkSXLhytw9Yb0WmVjJDjpm%3D4F4A%2B%3D%3DajDWZ52Ncg%40mail.gmail.com.
