Re: Usage of DataStreamer for bulk loading

[Naveen]

Hi DH

I am not using any custom streamReciever, my requirement is very simple.
Have huge data in CSV, reading line by line and parsing the line and
populating the POJO and using the DataStreamer to load data into cache.

            while (sc.hasNextLine()) {
                        ct++;
                        String line = sc.nextLine();
                        String[] tokens = line.split(Constants.Delimter, -1);
                        aASSOCIATED_PARTIES = new ASSOCIATED_PARTIES();
                        
                        aASSOCIATED_PARTIES.setASSOCIATED_PARTY_ID(tokens[1]);
                        aASSOCIATED_PARTIES.setUPDATEDBY(tokens[3]);
                        
                       
streamer.perNodeBufferSize(Constants.perNodeBufferSize);
                    
streamer.perNodeParallelOperations(Constants.perNodeParallelOperations);

                        streamer.addData(tokens[0], aASSOCIATED_PARTIES);
           }

As you mentioned, I have made sure that one DataStreamer for each client,
after this change, it stopped failing. 
Any ways we can improve this performance.

Thanks
Naveen



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/