HI

We are using Ignite 2.3
We have requirement to migrate the data from existing in-memory solution to
Ignite, its one time migration we should be doing 

We have data available in CSV with a delimiter, we have split for the source
files into multiple chunks and each thread processing one file. 
Here is the code which read the file line by line and call dataStreamer.add
method.


                    while (sc.hasNextLine()) {
                        ct++;
                        String line = sc.nextLine();
                        //System.out.println(line);
                        String[] tokens = line.split(Constants.Delimter, -1);
                       //System.out.println("No of tokens " + toekns.length);
                        //PARTY_ID~ASSOCIATED_PARTY_ID~UPDATEDDATETIME~UPDATEDBY
                        aASSOCIATED_PARTIES = new ASSOCIATED_PARTIES();
                        aASSOCIATED_PARTIES.setPARTY_ID(tokens[0]);
                        aASSOCIATED_PARTIES.setASSOCIATED_PARTY_ID(tokens[1]);
                        aASSOCIATED_PARTIES.setUPDATEDBY(tokens[3]);
                        aASSOCIATED_PARTIES.setUPDATEDDATETIME(new
Timestamp(System.currentTimeMillis()));
        
                        streamer.perNodeBufferSize(Constants.perNodeBufferSize);
                    
streamer.perNodeParallelOperations(Constants.perNodeParallelOperations);

                        streamer.addData(tokens[0], aASSOCIATED_PARTIES);
                        if (ct > 0 && ct % 10000 == 0) 
                             System.out.println("Done: " + ct);
         }

My question is, how do I call streamer.addData(tokens[0],
aASSOCIATED_PARTIES) for a batch instead of calling for every record. Guess
the way I have used is calling this for every line thats read from the file.
It should be called for lets say every 1000 records once, start accumulating
all the 1000 records and add it to cache 1000 records in batch. 
How could we do this, batch wise or time wise, like every minute it should
write.

Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Reply via email to