Thanks! This may still not help because my underlying datastore is a warehouse where only batch queries are efficient, Also, it looks like the key to partition mapping could change if the no of partitions in the aff function , which will result in unnecessary backfills. I think I might need to go one step forward in the ingestion process and use the datastreamer and consume from my event bus which is kinesis.
Sent from my iPhone > On Jan 15, 2016, at 5:11 AM, Alexey Goncharuk <[email protected]> > wrote: > > Btw, > > Even though the same query is executed on all nodes, Ignite will > automatically filter out the keys that does not belong to the local node upon > cache loading. When the number of nodes is high, this is not very effective > since only a small part of data (roughly K/N, where K is the number of > backups + 1, N is the number of nodes in a cluster) will loaded to the cache > for each node. > > This may be optimized if you store the partition ID alongside with the data > record in your database. Take a look at this topic in the documentation [1]. > In this case each node will select only those records that belong to this > node, which substantially decreases the database load. Applying this will > also allow you to workaround the bug related to running a cache query on > client nodes. > > Hope this helps, > AG > > ------- > [1] > https://apacheignite.readme.io/docs/data-loading#section-partition-aware-data-loading
