Alexey, What you are saying should already be possible by sending compute closures to nodes and calling localLoadCache(“myCustomSqlStatement”).
Isn’t it just a case of providing proper example and documentation? D. On Tue, May 3, 2016 at 7:22 AM, Alexey Kuznetsov <[email protected]> wrote: > I totally agree with Vladimir. > > From JdbcPojo store side we could introduce support of some kind load > descriptor > that will contains SQL to execute and node filter. > > On each node store will check node filter and execute SQL if node match the > filter. > > This will solve first problem - "do not load full database on each node" . > > As for second problem - "not ignore non-primary non-backup entries" - I > think this should be solved on a cache level, > because store does not know anything about primary / backup. > > Thoughts? > > > On Wed, Apr 27, 2016 at 9:27 PM, Vladimir Ozerov <[email protected]> > wrote: > > > Igniters, > > > > We receive more and more questions about the same problem: "I have big a > > database. How should I load it to Ignite?" > > > > Obviously, users try to use POJO store as a most convenient approach, but > > it cannot handle this case properly. > > 1) If user invoke *IgniteCache.loadCache()*, then the same request - and > > usually this is full table scan - will be invoked on every node leading > to > > very poor performance. For instance, we have a report of a load of 47M > > entries to cache on 16 nodes which took ... 8 hours!!! > > 2) If user invoke IgniteCache.localLoadCache(), then our internal cache > > logic will filter out non-primary and non-backup entries. So this > approach > > doesn't work either. > > 3) User could try using *IgniteDataStreamer*, but in this case he had to > > deal with all JDBC-related stuff on his own - not convenient. > > 4) Another approach I heard several times - "user should have an > attribute > > for affinity in the table ...". And the idea that this way user will be > > able to divide the whole data set into several disjoint sets with > specific > > affinity. Doesn't work. Consider the user with some legacy database - the > > most common use case. How is he going to work with affinity? > > > > Bottom line: Ignite has *no convenient way *to load millions of entries > > from a database. > > > > We need to start thinking of possible solutions. Several ideas from my > > side: > > > > 1) POJO store must be much more flexible. We should be able to pass > > different queries to different nodes when calling "loadCache". > > > > 2) Cache store could have additional mode when it will not ignore > > non-primary non-backup entries, but rather *distribute *it to other > nodes. > > E.g. with help of data streamer. > > > > Thoughts? > > > > Vladimir. > > > > > > -- > Alexey Kuznetsov > GridGain Systems > www.gridgain.com >
