Hi Naveen,
In our case biggest performance gain happened when we started adding data
to IgniteStreamer in parallel.
Earlier we are doing :
entryMapToBeStreamed.entrySet().*stream*().forEach(dataStreamer::addData);
Perf improved tremendously when we did something like this :
Hi Naveen,
There is no scheduled date for it yet. 2.4 release has just been voted for
and accepted.
Based on the version history (https://ignite.apache.org/download.cgi) one
could say that the average time between releases is about 2-3 months, so
it's quite possible that 2.5 will be released in
Hi Gaurav
Decoupling file reading and cache streaming requires kind of a messaging
layer in between right. Initially I was thinking since its a bulk activity
we will be doing, I did not want to have additional memory and system
resources consumed by the introduction of messaging layer.
But the
Thanks Alexey.
We are in the middle of the development, we may go live in next 3 to 4
months.
If we are done with Copy command by then, we are good.
When are we going to release 2.5 ??
Thanks
Naveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Hi Naveen,
I had similar situation. Two things you can do :
1. Decouple file reading from cache streaming, so that both can be handled
in separate threads asynchronously.
2. Once you have data from csv in collection, use use parallelStreams to
add data in streamer with multiple threads.
Thanks,
Hi DH
I am not using any custom streamReciever, my requirement is very simple.
Have huge data in CSV, reading line by line and parsing the line and
populating the POJO and using the DataStreamer to load data into cache.
while (sc.hasNextLine()) {
ct++;
HI
We are using Ignite 2.3
We have requirement to migrate the data from existing in-memory solution to
Ignite, its one time migration we should be doing
We have data available in CSV with a delimiter, we have split for the source
files into multiple chunks and each thread processing one file.
