Actually, JDBC driver should extract data page by page. Need to take an in-depth look.
Kind regards, Alex. On Mon, Jun 19, 2017 at 3:08 PM, Anil [via Apache Ignite Users] < ml+s70518n13953...@n6.nabble.com> wrote: > HI Alex, > > I have attached the ignite client xml. 4L means 0.4 million records. > Sorry, I didn't generate JFR. But created heap dump. > > Do you agree that Jdbc driver loading everything in memory and next() just > for conversion ? > > Thanks > > On 19 June 2017 at 17:16, Alexander Fedotov <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=13953&i=0>> wrote: > >> Hi Anil. >> >> Could you please also share C:/Anil/ignite-client.xml ? As well, it >> would be useful if you took JFR reports for the case with allocation >> profiling enabled. >> Just to clarify, by 4L do you mean 4 million entries? >> >> Kind regards, >> Alex. >> >> On Mon, Jun 19, 2017 at 10:15 AM, Alexander Fedotov <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13953&i=1>> wrote: >> >>> Thanks. I'll take a look and let you know about any findings. >>> >>> Kind regards, >>> Alex >>> >>> 18 июня 2017 г. 3:33 PM пользователь "Anil" <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=13953&i=2>> написал: >>> >>> Hi Alex, >>> >>> test program repository - https://github.com/adasari/t >>> est-ignite-jdbc.git >>> >>> please let us if you have any suggestions/questions. thanks. >>> >>> Thanks >>> >>> On 15 June 2017 at 10:58, Anil <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=13953&i=3>> wrote: >>> >>>> Sure. thanks >>>> >>>> On 14 June 2017 at 19:51, afedotov <[hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=4>> wrote: >>>> >>>>> Hi, Anil. >>>>> >>>>> Could you please share your full code (class/method) you are using to >>>>> read data. >>>>> >>>>> Kind regards, >>>>> Alex >>>>> >>>>> 12 июня 2017 г. 4:07 PM пользователь "Anil [via Apache Ignite Users]" >>>>> <[hidden >>>>> email] <http:///user/SendEmail.jtp?type=node&node=13706&i=0>> написал: >>>>> >>>>>> Do you have any advice on implementing large records export from >>>>>> ignite ? >>>>>> >>>>>> I could not use ScanQuery right as my whole application built around >>>>>> Jdbc driver and writing complex queries in scan query is very difficult. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On 10 June 2017 at 18:48, Anil <[hidden email] >>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=0>> wrote: >>>>>> >>>>>>> I understand from the code that there is no cursor from h2 db (or >>>>>>> ignite embed h2 db) internally and all mapper response consolidated at >>>>>>> reducer. It means when exporting large number of records, all data is in >>>>>>> memory. >>>>>>> >>>>>>> if (send(nodes, >>>>>>> oldStyle ? >>>>>>> new GridQueryRequest(qryReqId, >>>>>>> r.pageSize, >>>>>>> space, >>>>>>> mapQrys, >>>>>>> topVer, >>>>>>> extraSpaces(space, qry.spaces()), >>>>>>> null, >>>>>>> timeoutMillis) : >>>>>>> new GridH2QueryRequest() >>>>>>> .requestId(qryReqId) >>>>>>> .topologyVersion(topVer) >>>>>>> .pageSize(r.pageSize) >>>>>>> .caches(qry.caches()) >>>>>>> .tables(distributedJoins ? qry.tables() >>>>>>> : null) >>>>>>> .partitions(convert(partsMap)) >>>>>>> .queries(mapQrys) >>>>>>> .flags(flags) >>>>>>> .timeout(timeoutMillis), >>>>>>> oldStyle && partsMap != null ? new >>>>>>> ExplicitPartitionsSpecializer(partsMap) : null, >>>>>>> false)) { >>>>>>> >>>>>>> awaitAllReplies(r, nodes, cancel); >>>>>>> >>>>>>> *// once the responses from all nodes for the query received.. >>>>>>> proceed further ?* >>>>>>> >>>>>>> if (!retry) { >>>>>>> if (skipMergeTbl) { >>>>>>> List<List<?>> res = new ArrayList<>(); >>>>>>> >>>>>>> // Simple UNION ALL can have multiple >>>>>>> indexes. >>>>>>> for (GridMergeIndex idx : r.idxs) { >>>>>>> Cursor cur = idx.findInStream(null, >>>>>>> null); >>>>>>> >>>>>>> while (cur.next()) { >>>>>>> Row row = cur.get(); >>>>>>> >>>>>>> int cols = row.getColumnCount(); >>>>>>> >>>>>>> List<Object> resRow = new >>>>>>> ArrayList<>(cols); >>>>>>> >>>>>>> for (int c = 0; c < cols; c++) >>>>>>> resRow.add(row.getValue(c).get >>>>>>> Object()); >>>>>>> >>>>>>> res.add(resRow); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> resIter = res.iterator(); >>>>>>> }else { >>>>>>> // incase of split query scenario >>>>>>> } >>>>>>> >>>>>>> } >>>>>>> >>>>>>> return new GridQueryCacheObjectsIterator(resIter, cctx, >>>>>>> keepPortable); >>>>>>> >>>>>>> >>>>>>> Query cursor is iterator which does column value mapping per page. >>>>>>> But still all records of query are still in memory. correct? >>>>>>> >>>>>>> Please correct me if I am wrong. thanks. >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> On 10 June 2017 at 15:53, Anil <[hidden email] >>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=1>> wrote: >>>>>>> >>>>>>>> >>>>>>>> jvm parameters used - >>>>>>>> >>>>>>>> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC >>>>>>>> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC >>>>>>>> -XX:+DisableExplicitGC >>>>>>>> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError >>>>>>>> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy >>>>>>>> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC >>>>>>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch >>>>>>>> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps >>>>>>>> /heapdump-client.hprof >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> On 10 June 2017 at 15:06, Anil <[hidden email] >>>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=2>> wrote: >>>>>>>> >>>>>>>>> HI, >>>>>>>>> >>>>>>>>> I have implemented export feature of ignite data using JDBC >>>>>>>>> Interator >>>>>>>>> >>>>>>>>> ResultSet rs = statement.executeQuery(); >>>>>>>>> >>>>>>>>> while (rs.next()){ >>>>>>>>> // do operations >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> and fetch size is 200. >>>>>>>>> >>>>>>>>> when i run export operation twice for 4 L records whole 6B is >>>>>>>>> filled up and never getting released. >>>>>>>>> >>>>>>>>> Initially i thought that operations transforting result set to >>>>>>>>> file causing the memory full. But not. >>>>>>>>> >>>>>>>>> I just did follwoing and still the memory is growing and not >>>>>>>>> getting released >>>>>>>>> >>>>>>>>> while (rs.next()){ >>>>>>>>> // nothing >>>>>>>>> } >>>>>>>>> >>>>>>>>> num #instances #bytes class name >>>>>>>>> ---------------------------------------------- >>>>>>>>> 1: 55072353 2408335272 [C >>>>>>>>> 2: 54923606 1318166544 java.lang.String >>>>>>>>> 3: 779006 746187792 [B >>>>>>>>> 4: 903548 304746304 [Ljava.lang.Object; >>>>>>>>> 5: 773348 259844928 net.juniper.cs.entity.Install >>>>>>>>> Base >>>>>>>>> 6: 4745694 113896656 java.lang.Long >>>>>>>>> 7: 1111692 44467680 sun.nio.cs.UTF_8$Decoder >>>>>>>>> 8: 773348 30933920 org.apache.ignite.internal.bi >>>>>>>>> nary.BinaryObjectImpl >>>>>>>>> 9: 895627 21495048 java.util.ArrayList >>>>>>>>> 10: 12427 16517632 [I >>>>>>>>> >>>>>>>>> >>>>>>>>> Not sure why string objects are getting increased. >>>>>>>>> >>>>>>>>> Could you please help in understanding the issue ? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> If you reply to this email, your message will be added to the >>>>>> discussion below: >>>>>> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on- >>>>>> ignite-client-tp13594p13626.html >>>>>> To start a new topic under Apache Ignite Users, email [hidden email] >>>>>> <http:///user/SendEmail.jtp?type=node&node=13706&i=1> >>>>>> To unsubscribe from Apache Ignite Users, click here. >>>>>> NAML >>>>>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>>> >>>>> >>>>> ------------------------------ >>>>> View this message in context: Re: High heap on ignite client >>>>> <http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13706.html> >>>>> Sent from the Apache Ignite Users mailing list archive >>>>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >>>>> >>>> >>>> >>> >>> >> > > *ignite-client.xml* (2K) Download Attachment > <http://apache-ignite-users.70518.x6.nabble.com/attachment/13953/0/ignite-client.xml> > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client- > tp13594p13953.html > To start a new topic under Apache Ignite Users, email > ml+s70518n1...@n6.nabble.com > To unsubscribe from Apache Ignite Users, click here > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=YWxleGFuZGVyLmZlZG90b2ZmQGdtYWlsLmNvbXwxfC0xMzYxNTU0NTg=> > . > NAML > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13956.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
