Re: Understanding data store and partitioning

[ezhuravlev]

Hi,

1) You can somehow signal it from your application where you invoke
loadCache method or from CacheStore implementation 

2) loadCache is sync method and when it will be finished, all preloading
will be finished on all nodes

3) You can listen to the topology changes and check count of the nodes in
topology: https://apacheignite.readme.io/docs/events

Evgenii



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Understanding data store and partitioning

[prasadbhalerao1983]

Hi,

1) How does one know that cache pre-loading is already in process and now
its time to call localLoadCache?

2) When a cache on all existing nodes is pre-loaded with all the data and if
any new nodes joins a cluster then only some partitions are moved to newly
joined node as a part of data rebalancing process. So in this case there is
no need to invoke loadCache or localLoadCache method when a new node is
started. 

But how can I find out that data is already pre-loaded and there is no need
to call loadLocalCache?

3) Is there any way to find out that cluster is formed with desired number
of nodes? Do I have to poll to check if all the nodes have joined the
cluster? I want to start the cache loading only after all the nodes have
joined the cluster.

Thanks,
Prasad



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Understanding data store and partitioning

[Denis Magda]

Hi Kamal,

All the data that has been already pre-loaded by the time the new node
joined and has to be located on the new node according to the new topology
version will be rebalanced there [1].

However if the pre-loading is still in progress then you need to call
localLoadCache() on this node because the rest of the nodes, that was in the
cluster before, will skip entries for which they are neither primary and
backups and such entries won't be rebalanced to the new node.

[1] https://apacheignite.readme.io/docs/rebalancing

--
Denis



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264p6297.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Understanding data store and partitioning

[Kamal C]

Val,

When a new node joins the cluster, should I have to call loadCache() or
localLoadCache() method ?

>From docs,

1. loadCache() - executes localLoadCache() on all the nodes
2. localLoadCache() - will trigger data loading only in the local node.

--Kamal

On Thu, Jul 14, 2016 at 4:12 AM, vkulichenko 
wrote:

> Hi,
>
> You don't need to load all person IDs when loading the data. The
> loadCache()
> implementation can use Affinity API to get the array of local partition IDs
> and query the DB based on this IDs. With this approach each node will load
> only those rows that has to be stored locally. See the second code example
> in [1]. Also note the you can load different partitions in parallel.
>
> [1]
>
> https://apacheignite.readme.io/docs/data-loading#section-partition-aware-data-loading
>
> -Val
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264p6282.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Re: Understanding data store and partitioning

[vkulichenko]

Hi,

You don't need to load all person IDs when loading the data. The loadCache()
implementation can use Affinity API to get the array of local partition IDs
and query the DB based on this IDs. With this approach each node will load
only those rows that has to be stored locally. See the second code example
in [1]. Also note the you can load different partitions in parallel.

[1]
https://apacheignite.readme.io/docs/data-loading#section-partition-aware-data-loading

-Val



--
View this message in context: 
http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264p6282.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Understanding data store and partitioning

[Vladislav Pyatkov]

Hi,

On Wed, Jul 13, 2016 at 3:48 PM, pragmaticbigdata 
wrote:

> > You can find partition number using: affinity.partition(key)
>
> My question was - to get the partition id we need the cache key. When doing
> the initial load into ignite we don't have the cache key. Does that mean we
> cannot have an optimized data loading (i.e. partition aware data loading)?
>

Cache is a key-value store at first. Yes you can to use ID of record (from
database), as key of cache.

>
> > Yes it does work. Each node hase own instance of CacheStore.
>
> Is the loadCache() method from the CacheStore invoked on a new server node
> that joins the cluster? If yes, what is the reason behind it? Won't it be
> copying the existing partitions from other server nodes?
>

If node joined after the method (loadCache) was executed, then partitions
will by relocated to new node (with all data in it).

>
> Thanks
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264p6272.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov

Re: Understanding data store and partitioning

[Vladislav Pyatkov]

Hello,



On Wed, Jul 13, 2016 at 2:26 PM, pragmaticbigdata 
wrote:

> Following the documentation on  data loading
>    I have some questions
> with regards to ignite version 1.6
>
> 1. How does ignite derive the partition id from the cache key? What is the
> relation between the partition id and the affinity key?
>

Affinity function reflect your key to partition. You can get the function
using method ignite().affinity(cacheName)

>
> 2. Partition aware data loading suggests to persist the partition id along
> with the data in the database. For this we would need to know the cache key
> upfront (as the example indicates - personId) right? How could getting the
> key be possible when doing the initial load? Did I misunderstood anything?
>

You can find partition number using: affinity.partition(key)

>
> 3. Is the data store implementation called on cluster rebalancing
> especially
> when a new server node joins the cluster?
>

Yes it does work. Each node hase own instance of CacheStore.

>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Understanding-data-store-and-partitioning-tp6264.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Vladislav Pyatkov