Since there are several approaches on solving this, I’m continue with your
code, a couple of suggestions:
- Create streamer instance out of loop, and close it on finally, not within the
loop.
- stmr.autoflushfrequency(0) as you do it every 2000 elements…
- Don’t forget remaining data (< 2000) from last iteration
IgniteDataStreamer<Long, PieCountryAllocation> stmr =
ignite.dataStreamer("PieCountryAllocationCache");
stmr.allowOverwrite(true);
// disable auto flush - we’ll do it manually
stmr.autoflushfrequency(0);
try{
int j=0;
for (Map.Entry<Long,PieCountryAllocation> entry :
PieCountryAllocationobjs.entrySet()) {
tempobjs.put(entry.getKey(),
entry.getValue());
//For ever 2000 rows i am callling stmr.addData(tempobjs) and then stmr.flush
and stmr.close(false)
if(j++ == 2000 ){
System.out.println(j);
stmr.addData(tempobjs);
// do flush every 2000 items
stmr.flush();
tempobjs.clear();
System.out.println(“Sent
Ended");
System.out.println(j);
j = 0;
}
}
// stream remaining data
if (!tempobjs.is <http://tempobjs.is/>Empty()){
stmr.addData(tempobjs);
}
} finally {
stmr.flush();
stmr.close(false);
}
> El 18 may 2020, a las 12:36, nithin91
> <nithinbharadwaj.govindar...@franklintempleton.com> escribió:
>
> int j=0;
> for (Map.Entry<Long,PieCountryAllocation> entry :
> PieCountryAllocationobjs.entrySet()) {
>
> tempobjs.put(entry.getKey(),
> entry.getValue());
> //For ever 2000 rows i am callling stmr.addData(tempobjs) and then
> stmr.flush and stmr.close(false)
> if((j%2000==0 && j!=0) ||
>
> (PieCountryAllocationobjs.keySet().size() < 2000 &&
> j==PieCountryAllocationobjs.keySet().size())
> ||
> j==PieCountryAllocationobjs.keySet().size()
> ){
> System.out.println(j);
> IgniteDataStreamer<Long,
> PieCountryAllocation> stmr =
> ignite.dataStreamer("PieCountryAllocationCache");
> stmr.allowOverwrite(true);
> stmr.addData(tempobjs);
> stmr.flush();
> stmr.close(false);
> tempobjs.clear();
> System.out.println("Stream
> Ended");
> System.out.println(j);
>
> }
> j++;
> }
