Folks, While configuring and testing product clusters in 3.2.0 release cycle, we came across several issues because of the multiple options we have with clustering setup configurations. At the basic level, we can configure a product cluster with mounted governance and configuration registries. Then, we can have multiple modes of distributed caching mechanisms associated with cluster. eg:- distributed, replicated, invalidation, local We also have Axis2 clustering (tribes) which can be configured through axis2.xml. Deployment synchronizer also plays a important role in a product cluster from 3.2.0 release onwards.
Now, with all these options, it will be extremely difficult for a user to figure out the most useful and best clustering configuration. Therefore, we should prepare a document or wiki with all these information. We came across errors such as "org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [org.wso2.carbon.caching.core.registry.RegistryCacheKey@ce8858d] for requestor [Thread[OOB-2,localhost-36388,5,Thread Pools]]! Lock held by [(another thread)]" when starting replicated caching enabled cluster nodes in parallel. As discussed with Senaka, we should start all nodes sequentially in such a cluster setup. We also came across several issues when running a cluster if the caching mode is set to "distributed". So, the recommended approach should be "replicated" mode. We also noticed that the cluster becomes inconsistent if the P2 features are not installed correctly in each node in a cluster. Most importantly, it is difficult (or impossible) to figure out the root cause if an error occurs in an inconsistent cluster setup. We should let users know which configurations are replicated among cluster nodes and which are not. For example, carbon datasources, message stores, processors etc.. are not replicated among child nodes of a cluster. We should prepare a clustering best practices and guideline doc and ship or host in OT with 3.2.0 release. Associated JIRA:- https://wso2.org/jira/browse/CARBON-10487 Regards Charitha
_______________________________________________ Carbon-dev mailing list [email protected] http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
