Hi, Anil. Could you please share your full code (class/method) you are using to read data.
Kind regards, Alex 12 июня 2017 г. 4:07 PM пользователь "Anil [via Apache Ignite Users]" < [email protected]> написал: > Do you have any advice on implementing large records export from ignite ? > > I could not use ScanQuery right as my whole application built around Jdbc > driver and writing complex queries in scan query is very difficult. > > Thanks > > On 10 June 2017 at 18:48, Anil <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=13626&i=0>> wrote: > >> I understand from the code that there is no cursor from h2 db (or ignite >> embed h2 db) internally and all mapper response consolidated at reducer. It >> means when exporting large number of records, all data is in memory. >> >> if (send(nodes, >> oldStyle ? >> new GridQueryRequest(qryReqId, >> r.pageSize, >> space, >> mapQrys, >> topVer, >> extraSpaces(space, qry.spaces()), >> null, >> timeoutMillis) : >> new GridH2QueryRequest() >> .requestId(qryReqId) >> .topologyVersion(topVer) >> .pageSize(r.pageSize) >> .caches(qry.caches()) >> .tables(distributedJoins ? qry.tables() : >> null) >> .partitions(convert(partsMap)) >> .queries(mapQrys) >> .flags(flags) >> .timeout(timeoutMillis), >> oldStyle && partsMap != null ? new >> ExplicitPartitionsSpecializer(partsMap) : null, >> false)) { >> >> awaitAllReplies(r, nodes, cancel); >> >> *// once the responses from all nodes for the query received.. proceed >> further ?* >> >> if (!retry) { >> if (skipMergeTbl) { >> List<List<?>> res = new ArrayList<>(); >> >> // Simple UNION ALL can have multiple indexes. >> for (GridMergeIndex idx : r.idxs) { >> Cursor cur = idx.findInStream(null, null); >> >> while (cur.next()) { >> Row row = cur.get(); >> >> int cols = row.getColumnCount(); >> >> List<Object> resRow = new >> ArrayList<>(cols); >> >> for (int c = 0; c < cols; c++) >> resRow.add(row.getValue(c).get >> Object()); >> >> res.add(resRow); >> } >> } >> >> resIter = res.iterator(); >> }else { >> // incase of split query scenario >> } >> >> } >> >> return new GridQueryCacheObjectsIterator(resIter, cctx, >> keepPortable); >> >> >> Query cursor is iterator which does column value mapping per page. But >> still all records of query are still in memory. correct? >> >> Please correct me if I am wrong. thanks. >> >> >> Thanks >> >> >> On 10 June 2017 at 15:53, Anil <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13626&i=1>> wrote: >> >>> >>> jvm parameters used - >>> >>> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC >>> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC >>> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError >>> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy >>> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC >>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch >>> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps >>> /heapdump-client.hprof >>> >>> Thanks. >>> >>> On 10 June 2017 at 15:06, Anil <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=13626&i=2>> wrote: >>> >>>> HI, >>>> >>>> I have implemented export feature of ignite data using JDBC Interator >>>> >>>> ResultSet rs = statement.executeQuery(); >>>> >>>> while (rs.next()){ >>>> // do operations >>>> >>>> } >>>> >>>> and fetch size is 200. >>>> >>>> when i run export operation twice for 4 L records whole 6B is filled up >>>> and never getting released. >>>> >>>> Initially i thought that operations transforting result set to file >>>> causing the memory full. But not. >>>> >>>> I just did follwoing and still the memory is growing and not getting >>>> released >>>> >>>> while (rs.next()){ >>>> // nothing >>>> } >>>> >>>> num #instances #bytes class name >>>> ---------------------------------------------- >>>> 1: 55072353 2408335272 [C >>>> 2: 54923606 1318166544 java.lang.String >>>> 3: 779006 746187792 [B >>>> 4: 903548 304746304 [Ljava.lang.Object; >>>> 5: 773348 259844928 net.juniper.cs.entity.InstallBase >>>> 6: 4745694 113896656 java.lang.Long >>>> 7: 1111692 44467680 sun.nio.cs.UTF_8$Decoder >>>> 8: 773348 30933920 org.apache.ignite.internal.bi >>>> nary.BinaryObjectImpl >>>> 9: 895627 21495048 java.util.ArrayList >>>> 10: 12427 16517632 [I >>>> >>>> >>>> Not sure why string objects are getting increased. >>>> >>>> Could you please help in understanding the issue ? >>>> >>>> Thanks >>>> >>> >>> >> > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client- > tp13594p13626.html > To start a new topic under Apache Ignite Users, email > [email protected] > To unsubscribe from Apache Ignite Users, click here > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=YWxleGFuZGVyLmZlZG90b2ZmQGdtYWlsLmNvbXwxfC0xMzYxNTU0NTg=> > . > NAML > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13706.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
