Vladimir, As I see from the IEP, this data loading technique is supposed to be used for deployments with Ignite persistence enabled. Is it possible to generalize this solution and use for pure in-memory and in-memory + 3rd party DB scenarios?
-- Denis On Wed, Jun 20, 2018 at 8:08 AM Vladimir Ozerov <voze...@gridgain.com> wrote: > Igniters, > > Initial data load is one of the most important use cases for our product. > This is one the first things user try to do with Ignite. And if it takes > too much time, it is very likely that user will look for other solutions. > > We did good progress in this area recently. Specifically - a set of > internal improvements on our indexes, steaming mode for JDBC driver, COPY > command. But our internals are still not very efficient - every single > update goes through the whole set of Ignite components, such as page cache, > free-lists, BTrees, etc.. > > I created IEP-22 [1]. It's goal is to implement special direct data load > mode which will bypass our page cache and use alternative algorithm for > index updates. Together with COPY command and streaming this improvement > will allow Ignite to load data with very high speed. > > Please review the IEP and share your comments. > > Vladimir. > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-22%3A+Direct+Data+Load >
