Sure. thanks On 14 June 2017 at 19:51, afedotov <[email protected]> wrote:
> Hi, Anil. > > Could you please share your full code (class/method) you are using to read > data. > > Kind regards, > Alex > > 12 июня 2017 г. 4:07 PM пользователь "Anil [via Apache Ignite Users]" <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=13706&i=0>> написал: > >> Do you have any advice on implementing large records export from ignite ? >> >> I could not use ScanQuery right as my whole application built around Jdbc >> driver and writing complex queries in scan query is very difficult. >> >> Thanks >> >> On 10 June 2017 at 18:48, Anil <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13626&i=0>> wrote: >> >>> I understand from the code that there is no cursor from h2 db (or ignite >>> embed h2 db) internally and all mapper response consolidated at reducer. It >>> means when exporting large number of records, all data is in memory. >>> >>> if (send(nodes, >>> oldStyle ? >>> new GridQueryRequest(qryReqId, >>> r.pageSize, >>> space, >>> mapQrys, >>> topVer, >>> extraSpaces(space, qry.spaces()), >>> null, >>> timeoutMillis) : >>> new GridH2QueryRequest() >>> .requestId(qryReqId) >>> .topologyVersion(topVer) >>> .pageSize(r.pageSize) >>> .caches(qry.caches()) >>> .tables(distributedJoins ? qry.tables() : >>> null) >>> .partitions(convert(partsMap)) >>> .queries(mapQrys) >>> .flags(flags) >>> .timeout(timeoutMillis), >>> oldStyle && partsMap != null ? new >>> ExplicitPartitionsSpecializer(partsMap) : null, >>> false)) { >>> >>> awaitAllReplies(r, nodes, cancel); >>> >>> *// once the responses from all nodes for the query received.. proceed >>> further ?* >>> >>> if (!retry) { >>> if (skipMergeTbl) { >>> List<List<?>> res = new ArrayList<>(); >>> >>> // Simple UNION ALL can have multiple indexes. >>> for (GridMergeIndex idx : r.idxs) { >>> Cursor cur = idx.findInStream(null, null); >>> >>> while (cur.next()) { >>> Row row = cur.get(); >>> >>> int cols = row.getColumnCount(); >>> >>> List<Object> resRow = new >>> ArrayList<>(cols); >>> >>> for (int c = 0; c < cols; c++) >>> resRow.add(row.getValue(c).get >>> Object()); >>> >>> res.add(resRow); >>> } >>> } >>> >>> resIter = res.iterator(); >>> }else { >>> // incase of split query scenario >>> } >>> >>> } >>> >>> return new GridQueryCacheObjectsIterator(resIter, cctx, >>> keepPortable); >>> >>> >>> Query cursor is iterator which does column value mapping per page. But >>> still all records of query are still in memory. correct? >>> >>> Please correct me if I am wrong. thanks. >>> >>> >>> Thanks >>> >>> >>> On 10 June 2017 at 15:53, Anil <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=13626&i=1>> wrote: >>> >>>> >>>> jvm parameters used - >>>> >>>> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC >>>> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC >>>> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError >>>> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy >>>> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC >>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch >>>> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps >>>> /heapdump-client.hprof >>>> >>>> Thanks. >>>> >>>> On 10 June 2017 at 15:06, Anil <[hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=2>> wrote: >>>> >>>>> HI, >>>>> >>>>> I have implemented export feature of ignite data using JDBC Interator >>>>> >>>>> ResultSet rs = statement.executeQuery(); >>>>> >>>>> while (rs.next()){ >>>>> // do operations >>>>> >>>>> } >>>>> >>>>> and fetch size is 200. >>>>> >>>>> when i run export operation twice for 4 L records whole 6B is filled >>>>> up and never getting released. >>>>> >>>>> Initially i thought that operations transforting result set to file >>>>> causing the memory full. But not. >>>>> >>>>> I just did follwoing and still the memory is growing and not getting >>>>> released >>>>> >>>>> while (rs.next()){ >>>>> // nothing >>>>> } >>>>> >>>>> num #instances #bytes class name >>>>> ---------------------------------------------- >>>>> 1: 55072353 2408335272 [C >>>>> 2: 54923606 1318166544 java.lang.String >>>>> 3: 779006 746187792 [B >>>>> 4: 903548 304746304 [Ljava.lang.Object; >>>>> 5: 773348 259844928 net.juniper.cs.entity.InstallBase >>>>> 6: 4745694 113896656 java.lang.Long >>>>> 7: 1111692 44467680 sun.nio.cs.UTF_8$Decoder >>>>> 8: 773348 30933920 org.apache.ignite.internal.bi >>>>> nary.BinaryObjectImpl >>>>> 9: 895627 21495048 java.util.ArrayList >>>>> 10: 12427 16517632 [I >>>>> >>>>> >>>>> Not sure why string objects are getting increased. >>>>> >>>>> Could you please help in understanding the issue ? >>>>> >>>>> Thanks >>>>> >>>> >>>> >>> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on- >> ignite-client-tp13594p13626.html >> To start a new topic under Apache Ignite Users, email [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13706&i=1> >> To unsubscribe from Apache Ignite Users, click here. >> NAML >> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > ------------------------------ > View this message in context: Re: High heap on ignite client > <http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13706.html> > Sent from the Apache Ignite Users mailing list archive > <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >
