I have the following load code for ignite (.net)
1) setup a data streamer ldr = igniteclient.GetDataStreamer<dynamic,
dynamic>(TableName);
2) open connection to database and read data
3) create class objects and pass to streamer
4) close db connection
I am observing the following behaviour
the row count for this sample data is 4.5 million rows. While the data is
loading i am running a count(*) from the underlying table and the value is
significantly behind the data reader. When the data reader stops reading at
4.5 million rows the ignite streamer is left open (i did this to see if it
is a case of it still being busy flushing) but the ignite table count stops
as soon as the database data reader stops. I end up with approx 180k rows in
ignite and even after waiting that count is unchanged. It looks to me like
the data reader is running away from ignite and ignite is not buffering
i have the following data streamer settings:
ldr = igniteclient.GetDataStreamer<dynamic, dynamic>(TableName);
ldr.AllowOverwrite = true;
ldr.AutoFlushFrequency = 10000;
ldr.PerNodeBufferSize = 10000000;
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
