Thanks for that clarification of replicated vs distributed. That was helpful. So if I understand correctly, CAS's use of Hazelcast, if distributed but not replicated, means that the loss of a node in a cluster means the remaining nodes may not actually have the desired content from the lost node.
Also, FWIW, I think I resolved our problem where we weren't seeing any of the tickets distributed to other nodes. I believe Hazelcast uses port 5701 by default for this. These were new VMs and they require each port to be explicitly added. (New host, who dis?) Seems to be working as expected now. Sorry, PEBKAC, mea culpa, etc... On Thu, Jul 8, 2021 at 6:03 AM Mark H. Wood <[email protected]> wrote: > On Wed, Jul 07, 2021 at 02:49:32PM -1000, Baron Fujimoto wrote: > > I'm also confused by this. What does distributed mean, if not replicated? > > I understand "replicated" to mean that each associated instance > contains the complete set of cache entries locally. There would be a > great deal of communication required to maintain consistency, but the > cost of cache queries is small. > > Another form of distribution is variously called "sharded,", > "partitioned," etc. Associated instances would hold subsets of the > complete cache content, and their association would mean that one > instance can ask the others if any have hits where it has a miss, and > to supply one. This isn't as fast as having a complete set in each > instance, but it reduces the communication load on insertion. It may > be appropriate where the cost of acquiring an uncached object is > sufficiently higher than the cost of asking for help from another > subset and awaiting a reply. > > If the cost of uncached objects is quite high, it can also make sense > to run multiple cache instances atop a shared backing store, which has > its own cost. > > > On Mon, Jul 5, 2021 at 7:42 AM Ray Bon <[email protected]> wrote: > > > > > Erik, > > > > > > Hazelcast is not a replicated cache by default, just distributed. I > > > understand there is a backup/restore system but you would need at least > > > three servers to test it. > > > > > > The only config I have are these: > > > cluster.members > > > cluster.instanceName > > > crypto.signing.key > > > crypto.encryption.key > > > crypto.enabled=true > > > > > > which is in a shared file. > > > > > > I have two cas'es running on my local and have not seen that error. You > > > can tell if the hazelcast servers are communicating if your tickets are > > > validated on a different server than they were created. Set the load > > > balancer to round robin. > > > > > > I do not see a cas property for replication. Is it an option for cas? > See > > > hazelcast docs, > > > > https://docs.hazelcast.com/imdg/4.1/data-structures/replicated-map.html > > > > > > Ray > > > > > > On Thu, 2021-07-01 at 19:41 +0000, 'Mallory, Erik' via CAS Community > wrote: > > > > > > Notice: This message was sent from outside the University of Victoria > email system. Please be cautious with links and sensitive information. > > > > > > > > > > > > All I'm having a problem implementing the Hazelcast ticket store in CAS > > > > > > 6.3.4 which uses hazelcast-4.1 > > > > > > > > > Currently I'm testing with a two node cluster fontended with a > > > > > > netscaler. Each node has it's own /etc/cas/config/cas.properties so > > > > > > each node as it's own hazelcast configuration. > > > > > > Here is the relevant hazelcast configuration parameters: > > > > > > cas.ticket.registry.hazelcast.page-size=500 > > > > > > cas.ticket.registry.hazelcast.cluster.tcpip-enabled=true > > > > > > cas.ticket.registry.hazelcast.cluster.map-merge-policy=PUT_IF_ABSENT > > > > > > cas.ticket.registry.hazelcast.cluster.instance-name=cas-dev > > > > > > cas.ticket.registry.hazelcast.cluster.members=10.0.79.38,10.0.79.37 > > > > > > cas.ticket.registry.hazelcast.cluster.eviction-policy=LRU > > > > > > cas.ticket.registry.hazelcast.cluster.max-no-heartbeat-seconds=300 > > > > > > cas.ticket.registry.hazelcast.cluster.logging-type=slf4j > > > > > > cas.ticket.registry.hazelcast.cluster.port=5701 > > > > > > cas.ticket.registry.hazelcast.cluster.max-size=85 > > > > > > cas.ticket.registry.hazelcast.cluster.backup-count=1 > > > > > > cas.ticket.registry.hazelcast.cluster.async-backup-count=0 > > > > > > cas.ticket.registry.hazelcast.cluster.max-size- > > > > > > policy=USED_HEAP_PERCENTAGE > > > > > > cas.ticket.registry.hazelcast.cluster.timeout=5 > > > > > > > > > IN my testing I found that the tickets were not being replicated the > > > > > > other host. I'd use the netscaler to switch between the backend CAS > > > > > > nodes, log in to one, fail over to the other node and attempt to access > > > > > > cas, and I was redirected to the login screen. > > > > > > > > > After restarting the cas services on both nodes and tailing out the cas > > > > > > log I noticed the following error: > > > > > > > > > Cannot add a dynamic configuration > > > > > > > > > 'MapConfig{name='serviceTicketsCache', inMemoryFormat=BINARY', > > > > > > metadataPolicy=CREATE_ON_ > > > > > > UPDATE, backupCount=1, asyncBackupCount=0, timeToLiveSeconds=0, > > > > > > maxIdleSeconds=500, readBackupData=false, evictionConfig=Evict > > > > > > ionConfig{size=85, maxSizePolicy=USED_HEAP_PERCENTAGE, > > > > > > evictionPolicy=LRU, comparatorClassName=null, comparator=null}, merkleT > > > > > > ree=MerkleTreeConfig{enabled=false, depth=10}, > > > > > > eventJournal=EventJournalConfig{enabled=false, capacity=10000, > > > > > > timeToLiveSecond > > > > > > s=0}, hotRestart=HotRestartConfig{enabled=false, fsync=false}, > > > > > > nearCacheConfig=null, mapStoreConfig=MapStoreConfig{enabled=fal > > > > > > se, className='null', factoryClassName='null', writeDelaySeconds=0, > > > > > > writeBatchSize=1, implementation=null, factoryImplementation=null, > > > > > > properties={}, initialLoadMode=LAZY, writeCoalescing=true}, > > > > > > mergePolicyConfig=MergePolicyConfig{policy='com.hazelcast.spi.merge.Lat > > > > > > estUpdateMergePolicy', batchSize=100}, wanReplicationRef=null, > > > > > > entryListenerConfigs=null, indexConfigs=null, attributeConfigs=null, > > > > > > splitBrainProtectionName=null, queryCacheConfigs=null, > > > > > > cacheDeserializedValues=INDEX_ONLY}' > > > > > > > > > as there is already a conflicting configuration > > > > > > > > > 'MapConfig{name='serviceTicketsCache', inMemoryFormat=BINARY', > > > > > > metadataPolicy=CREATE_ON_UPDATE, backupCount=1, asyncBackupCount=0, > > > > > > timeToLiveSeconds=0, maxIdleSeconds=10, readBackupData=false, > > > > > > evictionConfig=EvictionConfig{size=85, > > > > > > maxSizePolicy=USED_HEAP_PERCENTAGE, evictionPolicy=LRU, > > > > > > comparatorClassName=null, comparator=null}, > > > > > > merkleTree=MerkleTreeConfig{enabled=false, depth=10}, > > > > > > eventJournal=EventJournalConfig{enabled=false, capacity=10000, > > > > > > timeToLiveSeconds=0}, hotRestart=HotRestartConfig{enabled=false, > > > > > > fsync=false}, nearCacheConfig=null, > > > > > > mapStoreConfig=MapStoreConfig{enabled=false, className='null', > > > > > > factoryClassName='null', writeDelaySeconds=0, writeBatchSize=1, > > > > > > implementation=null, factoryImplementation=null, properties={}, > > > > > > initialLoadMode=LAZY, writeCoalescing=true}, > > > > > > mergePolicyConfig=MergePolicyConfig{policy='com.hazelcast.spi.merge.Lat > > > > > > estUpdateMergePolicy', batchSize=100}, wanReplicationRef=null, > > > > > > entryListenerConfigs=null, indexConfigs=null, attributeConfigs=null, > > > > > > splitBrainProtectionName=null, queryCacheConfigs=null, > > > > > > cacheDeserializedValues=INDEX_ONLY}'> > > > > > > > > > So off to google I go and I find > > > > > > https://github.com/hazelcast/hazelcast/issues/12222 > > > > > > > > > and I add -Dhazelcast.dynamicconfig.ignore.conflicts=true for giggles > > > > > > and to see something at least boot. > > > > > > > > > So now both services start up but I'm ignoring the dynamic config > > > > > > conflicts. My testing fails it would appear that hazelcast is not able > > > > > > to share the tgt between nodes. > > > > > > > > > Any help would be greatly appreciated. > > > > > > > > > > > > -- > > > > > > Erik Mallory > > > > > > Server Analyst > > > > > > Wichita State University > > > > > > > > > -- > > > > > > - Website: > > > > > > https://apereo.github.io/cas > > > > > > > > > - Gitter Chatroom: > > > > > > https://gitter.im/apereo/cas > > > > > > > > > - List Guidelines: > > > > > > https://goo.gl/1VRrw7 > > > > > > > > > - Contributions: > > > > > > https://goo.gl/mh7qDG > > > > > > > > > --- > > > > > > You received this message because you are subscribed to the Google > Groups "CAS Community" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > an email to > > > > > > [email protected] > > > > > > . > > > > > > To view this discussion on the web visit > > > > > > > https://groups.google.com/a/apereo.org/d/msgid/cas-user/eb34bc51bfd8f5db71a2b000f1b362491c243cbd.camel%40wichita.edu > > > > > > . > > > > > > -- > > > - Website: https://apereo.github.io/cas > > > - Gitter Chatroom: https://gitter.im/apereo/cas > > > - List Guidelines: https://goo.gl/1VRrw7 > > > - Contributions: https://goo.gl/mh7qDG > > > --- > > > You received this message because you are subscribed to the Google > Groups > > > "CAS Community" group. > > > To unsubscribe from this group and stop receiving emails from it, send > an > > > email to [email protected]. > > > To view this discussion on the web visit > > > > https://groups.google.com/a/apereo.org/d/msgid/cas-user/7f06f02da2b65ccfeb8f8bb1868ffd23eac32ef1.camel%40uvic.ca > > > < > https://groups.google.com/a/apereo.org/d/msgid/cas-user/7f06f02da2b65ccfeb8f8bb1868ffd23eac32ef1.camel%40uvic.ca?utm_medium=email&utm_source=footer > > > > > . > > > > > > > > > -- > > Baron Fujimoto <[email protected]> :: UH Information Technology Services > > minutas cantorum, minutas balorum, minutas carboratum desendus pantorum > > > > -- > > - Website: https://apereo.github.io/cas > > - Gitter Chatroom: https://gitter.im/apereo/cas > > - List Guidelines: https://goo.gl/1VRrw7 > > - Contributions: https://goo.gl/mh7qDG > > --- > > You received this message because you are subscribed to the Google > Groups "CAS Community" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAAjLUL3oSJ9Ho%3Dij4sFdr6E8%2BShxYRLf8oieSLgJEd-YZLiU8A%40mail.gmail.com > . > > -- > Mark H. Wood > Lead Technology Analyst > > University Library > Indiana University - Purdue University Indianapolis > 755 W. Michigan Street > Indianapolis, IN 46202 > 317-274-0749 > www.ulib.iupui.edu > > -- > - Website: https://apereo.github.io/cas > - Gitter Chatroom: https://gitter.im/apereo/cas > - List Guidelines: https://goo.gl/1VRrw7 > - Contributions: https://goo.gl/mh7qDG > --- > You received this message because you are subscribed to the Google Groups > "CAS Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/apereo.org/d/msgid/cas-user/YOcDSh7IzCPHaVo/%40IUPUI.Edu > . > -- Baron Fujimoto <[email protected]> :: UH Information Technology Services minutas cantorum, minutas balorum, minutas carboratum desendus pantorum -- - Website: https://apereo.github.io/cas - Gitter Chatroom: https://gitter.im/apereo/cas - List Guidelines: https://goo.gl/1VRrw7 - Contributions: https://goo.gl/mh7qDG --- You received this message because you are subscribed to the Google Groups "CAS Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/CAAjLUL28X_TKoXSipPd2gUf_aOr3R6O_WwVxrMOF2n_-5j1GNg%40mail.gmail.com.
