I understand from the code that there is no cursor from h2 db (or ignite
embed h2 db) internally and all mapper response consolidated at reducer. It
means when exporting large number of records, all data is in memory.
if (send(nodes,
oldStyle ?
new GridQueryRequest(qryReqId,
r.pageSize,
space,
mapQrys,
topVer,
extraSpaces(space, qry.spaces()),
null,
timeoutMillis) :
new GridH2QueryRequest()
.requestId(qryReqId)
.topologyVersion(topVer)
.pageSize(r.pageSize)
.caches(qry.caches())
.tables(distributedJoins ? qry.tables() : null)
.partitions(convert(partsMap))
.queries(mapQrys)
.flags(flags)
.timeout(timeoutMillis),
oldStyle && partsMap != null ? new
ExplicitPartitionsSpecializer(partsMap) : null,
false)) {
awaitAllReplies(r, nodes, cancel);
*// once the responses from all nodes for the query received.. proceed
further ?*
if (!retry) {
if (skipMergeTbl) {
List<List<?>> res = new ArrayList<>();
// Simple UNION ALL can have multiple indexes.
for (GridMergeIndex idx : r.idxs) {
Cursor cur = idx.findInStream(null, null);
while (cur.next()) {
Row row = cur.get();
int cols = row.getColumnCount();
List<Object> resRow = new ArrayList<>(cols);
for (int c = 0; c < cols; c++)
resRow.add(row.getValue(c).getObject());
res.add(resRow);
}
}
resIter = res.iterator();
}else {
// incase of split query scenario
}
}
return new GridQueryCacheObjectsIterator(resIter, cctx, keepPortable);
Query cursor is iterator which does column value mapping per page. But
still all records of query are still in memory. correct?
Please correct me if I am wrong. thanks.
Thanks
On 10 June 2017 at 15:53, Anil <[email protected]> wrote:
>
> jvm parameters used -
>
> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC
> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError
> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy
> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch
> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps/heapdump-client.hprof
>
> Thanks.
>
> On 10 June 2017 at 15:06, Anil <[email protected]> wrote:
>
>> HI,
>>
>> I have implemented export feature of ignite data using JDBC Interator
>>
>> ResultSet rs = statement.executeQuery();
>>
>> while (rs.next()){
>> // do operations
>>
>> }
>>
>> and fetch size is 200.
>>
>> when i run export operation twice for 4 L records whole 6B is filled up
>> and never getting released.
>>
>> Initially i thought that operations transforting result set to file
>> causing the memory full. But not.
>>
>> I just did follwoing and still the memory is growing and not getting
>> released
>>
>> while (rs.next()){
>> // nothing
>> }
>>
>> num #instances #bytes class name
>> ----------------------------------------------
>> 1: 55072353 2408335272 [C
>> 2: 54923606 1318166544 java.lang.String
>> 3: 779006 746187792 [B
>> 4: 903548 304746304 [Ljava.lang.Object;
>> 5: 773348 259844928 net.juniper.cs.entity.InstallBase
>> 6: 4745694 113896656 java.lang.Long
>> 7: 1111692 44467680 sun.nio.cs.UTF_8$Decoder
>> 8: 773348 30933920 org.apache.ignite.internal.bi
>> nary.BinaryObjectImpl
>> 9: 895627 21495048 java.util.ArrayList
>> 10: 12427 16517632 [I
>>
>>
>> Not sure why string objects are getting increased.
>>
>> Could you please help in understanding the issue ?
>>
>> Thanks
>>
>
>
