hi
Initially this will be a single node demo but will be deployed as a 10 node
cluster. I am running 2.6 and it is loading data from SQL server. There are
2 issues i am finding at the moment
1) it takes an hour to load but only loads 2.5k out of 4.6 million records
(this seems serious)
2) it takes a long time to load (might be a better way of doing this)
running a trace on SQL server i see it is iterating the table key and for
each item it runs in one row (there should be 1800 per date).
The setup i have is simple it is a web console generated project that has
rmdbs integration. all the load cache has in the loader class is this
ignite.cache("IndexCache").loadCache(null);
if i run db beaver while loading it is doing like 5-10 records per second
which seems very slow. I have sold the advantage of ETL free data loading to
our management but if i have to now tell them by the way we have to rewrite
the load code to be more efficient it wont go down well. In my tests i was
using a small table and didn't recognise this as a potential issue at the
time. Any advice on both issues is welcome. I would genuinely like to be
able to just use the generated project to do the loading but this seems to
be not feasible.
some more info:
sql server is running on the same instance as ignite which is running out of
an intellij project from the web console.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
