Thanks. I'll take a look and let you know about any findings.

Kind regards,
Alex

18 июня 2017 г. 3:33 PM пользователь "Anil" <anilk...@gmail.com> написал:

Hi Alex,

test program repository - https://github.com/adasari/test-ignite-jdbc.git

please let us if you have any suggestions/questions. thanks.

Thanks

On 15 June 2017 at 10:58, Anil <anilk...@gmail.com> wrote:

> Sure. thanks
>
> On 14 June 2017 at 19:51, afedotov <alexander.fedot...@gmail.com> wrote:
>
>> Hi, Anil.
>>
>> Could you please share your full code (class/method) you are using to
>> read data.
>>
>> Kind regards,
>> Alex
>>
>> 12 июня 2017 г. 4:07 PM пользователь "Anil [via Apache Ignite Users]" 
>> <[hidden
>> email] <http:///user/SendEmail.jtp?type=node&node=13706&i=0>> написал:
>>
>>> Do you have any advice on implementing large records export from ignite ?
>>>
>>> I could not use ScanQuery right as my whole application built around
>>> Jdbc driver and writing complex queries in scan query is very difficult.
>>>
>>> Thanks
>>>
>>> On 10 June 2017 at 18:48, Anil <[hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=0>> wrote:
>>>
>>>> I understand from the code that there is no cursor from h2 db (or
>>>> ignite embed h2 db) internally and all mapper response consolidated at
>>>> reducer. It means when exporting large number of records, all data is in
>>>> memory.
>>>>
>>>>              if (send(nodes,
>>>>                     oldStyle ?
>>>>                         new GridQueryRequest(qryReqId,
>>>>                             r.pageSize,
>>>>                             space,
>>>>                             mapQrys,
>>>>                             topVer,
>>>>                             extraSpaces(space, qry.spaces()),
>>>>                             null,
>>>>                             timeoutMillis) :
>>>>                         new GridH2QueryRequest()
>>>>                             .requestId(qryReqId)
>>>>                             .topologyVersion(topVer)
>>>>                             .pageSize(r.pageSize)
>>>>                             .caches(qry.caches())
>>>>                             .tables(distributedJoins ? qry.tables() :
>>>> null)
>>>>                             .partitions(convert(partsMap))
>>>>                             .queries(mapQrys)
>>>>                             .flags(flags)
>>>>                             .timeout(timeoutMillis),
>>>>                     oldStyle && partsMap != null ? new
>>>> ExplicitPartitionsSpecializer(partsMap) : null,
>>>>                     false)) {
>>>>
>>>>                     awaitAllReplies(r, nodes, cancel);
>>>>
>>>> *// once the responses from all nodes for the query received.. proceed
>>>> further ?*
>>>>
>>>>           if (!retry) {
>>>>                     if (skipMergeTbl) {
>>>>                         List<List<?>> res = new ArrayList<>();
>>>>
>>>>                         // Simple UNION ALL can have multiple indexes.
>>>>                         for (GridMergeIndex idx : r.idxs) {
>>>>                             Cursor cur = idx.findInStream(null, null);
>>>>
>>>>                             while (cur.next()) {
>>>>                                 Row row = cur.get();
>>>>
>>>>                                 int cols = row.getColumnCount();
>>>>
>>>>                                 List<Object> resRow = new
>>>> ArrayList<>(cols);
>>>>
>>>>                                 for (int c = 0; c < cols; c++)
>>>>                                     resRow.add(row.getValue(c).get
>>>> Object());
>>>>
>>>>                                 res.add(resRow);
>>>>                             }
>>>>                         }
>>>>
>>>>                         resIter = res.iterator();
>>>>                     }else {
>>>>                       // incase of split query scenario
>>>>                     }
>>>>
>>>>          }
>>>>
>>>>       return new GridQueryCacheObjectsIterator(resIter, cctx,
>>>> keepPortable);
>>>>
>>>>
>>>> Query cursor is iterator which does column value mapping per page. But
>>>> still all records of query are still in memory. correct?
>>>>
>>>> Please correct me if I am wrong. thanks.
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On 10 June 2017 at 15:53, Anil <[hidden email]
>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=1>> wrote:
>>>>
>>>>>
>>>>> jvm parameters used -
>>>>>
>>>>> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC
>>>>> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
>>>>> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError
>>>>> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy
>>>>> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC
>>>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch
>>>>> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps
>>>>> /heapdump-client.hprof
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On 10 June 2017 at 15:06, Anil <[hidden email]
>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=2>> wrote:
>>>>>
>>>>>> HI,
>>>>>>
>>>>>> I have implemented export feature of ignite data using JDBC Interator
>>>>>>
>>>>>> ResultSet rs = statement.executeQuery();
>>>>>>
>>>>>> while (rs.next()){
>>>>>> // do operations
>>>>>>
>>>>>> }
>>>>>>
>>>>>> and fetch size is 200.
>>>>>>
>>>>>> when i run export operation twice for 4 L records whole 6B is filled
>>>>>> up and never getting released.
>>>>>>
>>>>>> Initially i thought that operations transforting result set to file
>>>>>> causing the memory full. But not.
>>>>>>
>>>>>> I just did follwoing and still the memory is growing and not getting
>>>>>> released
>>>>>>
>>>>>> while (rs.next()){
>>>>>>  // nothing
>>>>>> }
>>>>>>
>>>>>> num     #instances         #bytes  class name
>>>>>> ----------------------------------------------
>>>>>>    1:      55072353     2408335272  [C
>>>>>>    2:      54923606     1318166544  java.lang.String
>>>>>>    3:        779006      746187792  [B
>>>>>>    4:        903548      304746304  [Ljava.lang.Object;
>>>>>>    5:        773348      259844928  net.juniper.cs.entity.InstallBase
>>>>>>    6:       4745694      113896656  java.lang.Long
>>>>>>    7:       1111692       44467680  sun.nio.cs.UTF_8$Decoder
>>>>>>    8:        773348       30933920  org.apache.ignite.internal.bi
>>>>>> nary.BinaryObjectImpl
>>>>>>    9:        895627       21495048  java.util.ArrayList
>>>>>>   10:         12427       16517632  [I
>>>>>>
>>>>>>
>>>>>> Not sure why string objects are getting increased.
>>>>>>
>>>>>> Could you please help in understanding the issue ?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> ------------------------------
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-
>>> ignite-client-tp13594p13626.html
>>> To start a new topic under Apache Ignite Users, email [hidden email]
>>> <http:///user/SendEmail.jtp?type=node&node=13706&i=1>
>>> To unsubscribe from Apache Ignite Users, click here.
>>> NAML
>>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>> ------------------------------
>> View this message in context: Re: High heap on ignite client
>> <http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13706.html>
>> Sent from the Apache Ignite Users mailing list archive
>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>>
>
>

Reply via email to