Re: Understanding data store and partitioning
Hi, 1) You can somehow signal it from your application where you invoke loadCache method or from CacheStore implementation 2) loadCache is sync method and when it will be finished, all preloading will be finished on all nodes 3) You can listen to the topology changes and check count of the nodes in topology: https://apacheignite.readme.io/docs/events Evgenii -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Understanding data store and partitioning
Hi, 1) How does one know that cache pre-loading is already in process and now its time to call localLoadCache? 2) When a cache on all existing nodes is pre-loaded with all the data and if any new nodes joins a cluster then only some partitions are moved to newly joined node as a part of data rebalancing process. So in this case there is no need to invoke loadCache or localLoadCache method when a new node is started. But how can I find out that data is already pre-loaded and there is no need to call loadLocalCache? 3) Is there any way to find out that cluster is formed with desired number of nodes? Do I have to poll to check if all the nodes have joined the cluster? I want to start the cache loading only after all the nodes have joined the cluster. Thanks, Prasad -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Understanding data store and partitioning
Hi Kamal, All the data that has been already pre-loaded by the time the new node joined and has to be located on the new node according to the new topology version will be rebalanced there [1]. However if the pre-loading is still in progress then you need to call localLoadCache() on this node because the rest of the nodes, that was in the cluster before, will skip entries for which they are neither primary and backups and such entries won't be rebalanced to the new node. [1] https://apacheignite.readme.io/docs/rebalancing -- Denis -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264p6297.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Understanding data store and partitioning
Val, When a new node joins the cluster, should I have to call loadCache() or localLoadCache() method ? >From docs, 1. loadCache() - executes localLoadCache() on all the nodes 2. localLoadCache() - will trigger data loading only in the local node. --Kamal On Thu, Jul 14, 2016 at 4:12 AM, vkulichenko <valentin.kuliche...@gmail.com> wrote: > Hi, > > You don't need to load all person IDs when loading the data. The > loadCache() > implementation can use Affinity API to get the array of local partition IDs > and query the DB based on this IDs. With this approach each node will load > only those rows that has to be stored locally. See the second code example > in [1]. Also note the you can load different partitions in parallel. > > [1] > > https://apacheignite.readme.io/docs/data-loading#section-partition-aware-data-loading > > -Val > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264p6282.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
Re: Understanding data store and partitioning
Hi, You don't need to load all person IDs when loading the data. The loadCache() implementation can use Affinity API to get the array of local partition IDs and query the DB based on this IDs. With this approach each node will load only those rows that has to be stored locally. See the second code example in [1]. Also note the you can load different partitions in parallel. [1] https://apacheignite.readme.io/docs/data-loading#section-partition-aware-data-loading -Val -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264p6282.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Understanding data store and partitioning
Hi, On Wed, Jul 13, 2016 at 3:48 PM, pragmaticbigdata <amits...@gmail.com> wrote: > > You can find partition number using: affinity.partition(key) > > My question was - to get the partition id we need the cache key. When doing > the initial load into ignite we don't have the cache key. Does that mean we > cannot have an optimized data loading (i.e. partition aware data loading)? > Cache is a key-value store at first. Yes you can to use ID of record (from database), as key of cache. > > > Yes it does work. Each node hase own instance of CacheStore. > > Is the loadCache() method from the CacheStore invoked on a new server node > that joins the cluster? If yes, what is the reason behind it? Won't it be > copying the existing partitions from other server nodes? > If node joined after the method (loadCache) was executed, then partitions will by relocated to new node (with all data in it). > > Thanks > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264p6272.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Vladislav Pyatkov
Re: Understanding data store and partitioning
Hello, On Wed, Jul 13, 2016 at 2:26 PM, pragmaticbigdata <amits...@gmail.com> wrote: > Following the documentation on data loading > <https://apacheignite.readme.io/docs/data-loading> I have some questions > with regards to ignite version 1.6 > > 1. How does ignite derive the partition id from the cache key? What is the > relation between the partition id and the affinity key? > Affinity function reflect your key to partition. You can get the function using method ignite().affinity(cacheName) > > 2. Partition aware data loading suggests to persist the partition id along > with the data in the database. For this we would need to know the cache key > upfront (as the example indicates - personId) right? How could getting the > key be possible when doing the initial load? Did I misunderstood anything? > You can find partition number using: affinity.partition(key) > > 3. Is the data store implementation called on cluster rebalancing > especially > when a new server node joins the cluster? > Yes it does work. Each node hase own instance of CacheStore. > > Thanks! > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Vladislav Pyatkov