Mitchell, Can you share logs from all nodes?
Evgenii пн, 30 дек. 2019 г. в 14:42, Mitchell Rathbun (BLOOMBERG/ 731 LEX) < [email protected]>: > "On a disk, it's stored in meta storage inside DB folder." > > We also set the IgniteWorkDirectory to be different for every server node > id. However, we still will get "Node is not part of the baseline topology, > not using persistence." How would this still occur if the there is no > shared directory for the cluster? > > From: [email protected] At: 12/30/19 17:33:55 > To: Mitchell Rathbun (BLOOMBERG/ 731 LEX ) <[email protected]> > Cc: [email protected] > Subject: Re: Isolating IgniteCache instances across JVMs on same machine > by id > > Mitchell, > > -For your first response, I just want to make sure I am clear how this > works. Say I have two different server nodes with IgniteInstanceName A and > B. If both nodes are part of the same cluster and both nodes write to a > cache (in local mode) called "datacache", would this be treated as two > separate caches or would there be an issue? > *It will be more like the same cache, but all data will be local on each > node. It shouldn't be a problem. However, I would recommend using > partitioned caches with a NodeFilter instead.* > > -Where is the shared information stored for an IgniteCluster? Looking at > https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like > different clusters can be configured by using different port ranges. But > for each cluster, the nodes in the baseline topology must be stored. Where > does this information get stored? > *On a disk, it's stored in meta storage inside DB folder.* > > -If we need to specify different clusters on the same machine, do we need > to specify a range of ports for both TcpDiscoverySpi and > TcpCommunicationSpi? Or is one port for each fine? > *Next port from port range is used only if a first port is busy. If you > set only one port and it's busy at the moment, node won't be able to start. > So, it's up to you if you want to set the port range or one port only.* > > *Evgenii* > > пн, 30 дек. 2019 г. в 13:30, Mitchell Rathbun (BLOOMBERG/ 731 LEX) < > [email protected]>: > >> Makes sense. A couple of more related questions: >> >> -For your first response, I just want to make sure I am clear how this >> works. Say I have two different server nodes with IgniteInstanceName A and >> B. If both nodes are part of the same cluster and both nodes write to a >> cache (in local mode) called "datacache", would this be treated as two >> separate caches or would there be an issue? >> >> -Where is the shared information stored for an IgniteCluster? Looking at >> https://apacheignite.readme.io/v2.4/docs/cluster-config, it seems like >> different clusters can be configured by using different port ranges. But >> for each cluster, the nodes in the baseline topology must be stored. Where >> does this information get stored? >> >> -If we need to specify different clusters on the same machine, do we need >> to specify a range of ports for both TcpDiscoverySpi and >> TcpCommunicationSpi? Or is one port for each fine? >> >> From: [email protected] At: 12/30/19 15:25:40 >> To: Mitchell Rathbun (BLOOMBERG/ 731 LEX ) <[email protected]>, >> [email protected] >> Subject: Re: Isolating IgniteCache instances across JVMs on same machine >> by id >> >> -From your first response, it seems like the db id should be included in >> the cache name. Currently we just included it in the Ignite instance, but >> not the cache name, which I believe led to some of the issues we have seen. >> *No, it will be handled automatically, you don't need to do anything.* >> >> -What exactly would the node filter protect us from? Just the cache being >> explicitly restarted on a different node? Or any potential rebalancing as >> well? >> *It will tell cluster that only nodes, which returns true after applying >> a predicate will store data for this cache. Of course, data won't be >> rebalanced to the nodes, that shouldn't store data for this cache.* >> >> *>*-The cluster will be explicitly set to active the first time that it >> is created. All new nodes will then need to join the baseline topology to >> use persistence. It seems like the way to do this is to use >> IgniteCluster.setBaselineTopology. >> Is this method equivalent to adding the specified BaselineNode objects to >> the already existing BaselineTopology? Or does it completely reset the >> BaselineTopology with the specified BaselineNode objects? If the second, >> then to update the BaselineTopology, currentBaselineTopology would have to >> be called, the new node would have to be added, and then the collection >> would have to be passed back to setBaselineTopology. This would have a race >> condition if two nodes are trying to join the baseline topology at the same >> time. >> *Yes, it's equivalent to just adding the node, but using this method you >> can just use one method for several topology changes(for example, adding 2 >> nodes to the BT), so it will merge events on the cluster level.* >> >> пн, 30 дек. 2019 г. в 12:08, Mitchell Rathbun (BLOOMBERG/ 731 LEX) < >> [email protected]>: >> >>> Thank you for the response. A couple of things: >>> >>> -From your first response, it seems like the db id should be included in >>> the cache name. Currently we just included it in the Ignite instance, but >>> not the cache name, which I believe led to some of the issues we have seen. >>> >>> -What exactly would the node filter protect us from? Just the cache >>> being explicitly restarted on a different node? Or any potential >>> rebalancing as well? >>> >>> -The cluster will be explicitly set to active the first time that it is >>> created. All new nodes will then need to join the baseline topology to use >>> persistence. It seems like the way to do this is to use >>> IgniteCluster.setBaselineTopology. Is this method equivalent to adding the >>> specified BaselineNode objects to the already existing BaselineTopology? Or >>> does it completely reset the BaselineTopology with the specified >>> BaselineNode objects? If the second, then to update the BaselineTopology, >>> currentBaselineTopology would have to be called, the new node would have to >>> be added, and then the collection would have to be passed back to >>> setBaselineTopology. This would have a race condition if two nodes are >>> trying to join the baseline topology at the same time. >>> >>> >>> From: [email protected] At: 12/30/19 14:16:33 >>> To: Mitchell Rathbun (BLOOMBERG/ 731 LEX ) <[email protected]>, >>> [email protected] >>> Subject: Re: Isolating IgniteCache instances across JVMs on same machine >>> by id >>> >>> Hi, >>> >>> 1. If the cache mode is local, why does IgniteCluster even come into >>> play? All that we want to do is to read/write from the work directory >>> corresponding to our db id. Is there a way to get persistence without >>> activating the IgniteCluster? >>> *Cluster needs to make sure that all nodes in cluster know about this >>> cache to avoid situations in future when other nodes will try to create a >>> distributed cache with the same name, for example.* >>> >>> 2. How do we achieve the cache isolation we are looking for? Currently, >>> if I have db A and B, this would result in IgniteA and IgniteB in separate >>> JVM instances, each with their own directory. If I start IgniteA, and then >>> IgniteB, IgniteB will complain about not being part of the BaselineTopology >>> and will not use persistence. It seems like there are two options: >>> *You can use NodeFilter for the >>> cache: >>> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/CacheConfiguration.html#setNodeFilter-org.apache.ignite.lang.IgnitePredicate- >>> <https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/CacheConfiguration.html#setNodeFilter-org.apache.ignite.lang.IgnitePredicate->* >>> *It will define a subset of the nodes, which will store data for this >>> cache. So, you can set there only one node.* >>> *>*-Have every node join the baseline topology when it comes up (I >>> think there is one per machine by default). This seems like it could work, >>> but I am worried about how rebalancing would work. If all of the server >>> nodes are using LOCAL cache mode, then would rebalancing ever occur? If it >>> doesn't, then this seems like it would be an easy enough solution. Also, >>> where is the cluster state stored (Members of baseline topology, etc.)? >>> *No, rebalancing won't happen, it will be skipped. However, before that, >>> nodes should exchange an information regarding new topology ad partition >>> distribution. * >>> >>> *Baseline topology should contains all nodes that will store the data.* >>> >>> Evgenii >>> >>> >>> пн, 30 дек. 2019 г. в 10:59, Mitchell Rathbun (BLOOMBERG/ 731 LEX) < >>> [email protected]>: >>> >>>> Any thoughts on this? >>>> >>>> From: [email protected] At: 12/18/19 19:55:47 >>>> To: [email protected] >>>> Cc: Anant Narayan (BLOOMBERG/ 731 LEX ) <[email protected]>, >>>> Ranjith >>>> Lingamaneni (BLOOMBERG/ 731 LEX ) <[email protected]> >>>> Subject: Isolating IgniteCache instances across JVMs on same machine by >>>> id >>>> >>>> We have multiple different database instances for which we are looking >>>> into using Ignite as a local caching layer. Each db has a unique id that we >>>> are using as part of the IgniteInstanceName and for the WorkDirectory path. >>>> We are running IgniteCache in Local mode with persistence enabled, as we >>>> effectively want to completely separate the Ignite caches for each db id. >>>> However, this is not working due to the IgniteCluster being shared and the >>>> BaselineTopology concept. So a couple of questions: >>>> >>>> 1. If the cache mode is local, why does IgniteCluster even come into >>>> play? All that we want to do is to read/write from the work directory >>>> corresponding to our db id. Is there a way to get persistence without >>>> activating the IgniteCluster? >>>> >>>> 2. How do we achieve the cache isolation we are looking for? Currently, >>>> if I have db A and B, this would result in IgniteA and IgniteB in separate >>>> JVM instances, each with their own directory. If I start IgniteA, and then >>>> IgniteB, IgniteB will complain about not being part of the BaselineTopology >>>> and will not use persistence. It seems like there are two options: >>>> >>>> -Have every node join the baseline topology when it comes up (I think >>>> there is one per machine by default). This seems like it could work, but I >>>> am worried about how rebalancing would work. If all of the server nodes are >>>> using LOCAL cache mode, then would rebalancing ever occur? If it doesn't, >>>> then this seems like it would be an easy enough solution. Also, where is >>>> the cluster state stored (Members of baseline topology, etc.)? >>>> -If rebalancing does come into play even if all caches are local, then >>>> it seems like separate clusters would have to be specified when starting >>>> up. >>>> https://apacheignite.readme.io/docs/tcpip-discovery seems to provide a >>>> way to do that by making sure that TcpDiscoverySpi and TcpCommunicationSpi >>>> have different port ranges for every instance of the cluster. Again, this >>>> seems kind of like overkill if we know in advance that all of the caches >>>> will be local and will not need any of the cluster functionality. >>>> >>>> >>>> Are there any other simpler options? We are only interested in the >>>> IgniteCache functionality at a local per process level, at least for now. >>>> We might have 10-20 different caches on the same machine, and need these >>>> caches to be completely separated with no risk of any of the data being >>>> rebalanced/exposed to a different client instance. Ideally we would get >>>> persistence without having to activate IgniteCluster, but it does not seem >>>> like that is an option. >>>> >>>> >>>> >>> >> >
