I don't see anything wrong with your config. Could you please provide C:/Anil/dumps/gc-client.log There should be a reason for objects not being collected during GC.
Just one more thing, try replacing -XX:NewSize=512m with -XX:G1NewSizePercent=30 XX:NewSize won't let G1GC adjusting young gen size properly. Kind regards, Alex. On Mon, Jun 19, 2017 at 3:47 PM, afedotov <[email protected]> wrote: > Actually, JDBC driver should extract data page by page. > Need to take an in-depth look. > > Kind regards, > Alex. > > On Mon, Jun 19, 2017 at 3:08 PM, Anil [via Apache Ignite Users] <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=13956&i=0>> wrote: > >> HI Alex, >> >> I have attached the ignite client xml. 4L means 0.4 million records. >> Sorry, I didn't generate JFR. But created heap dump. >> >> Do you agree that Jdbc driver loading everything in memory and next() >> just for conversion ? >> >> Thanks >> >> On 19 June 2017 at 17:16, Alexander Fedotov <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13953&i=0>> wrote: >> >>> Hi Anil. >>> >>> Could you please also share C:/Anil/ignite-client.xml ? As well, it >>> would be useful if you took JFR reports for the case with allocation >>> profiling enabled. >>> Just to clarify, by 4L do you mean 4 million entries? >>> >>> Kind regards, >>> Alex. >>> >>> On Mon, Jun 19, 2017 at 10:15 AM, Alexander Fedotov <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=13953&i=1>> wrote: >>> >>>> Thanks. I'll take a look and let you know about any findings. >>>> >>>> Kind regards, >>>> Alex >>>> >>>> 18 июня 2017 г. 3:33 PM пользователь "Anil" <[hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=2>> написал: >>>> >>>> Hi Alex, >>>> >>>> test program repository - https://github.com/adasari/t >>>> est-ignite-jdbc.git >>>> >>>> please let us if you have any suggestions/questions. thanks. >>>> >>>> Thanks >>>> >>>> On 15 June 2017 at 10:58, Anil <[hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=3>> wrote: >>>> >>>>> Sure. thanks >>>>> >>>>> On 14 June 2017 at 19:51, afedotov <[hidden email] >>>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=4>> wrote: >>>>> >>>>>> Hi, Anil. >>>>>> >>>>>> Could you please share your full code (class/method) you are using to >>>>>> read data. >>>>>> >>>>>> Kind regards, >>>>>> Alex >>>>>> >>>>>> 12 июня 2017 г. 4:07 PM пользователь "Anil [via Apache Ignite Users]" >>>>>> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=13706&i=0>> >>>>>> написал: >>>>>> >>>>>>> Do you have any advice on implementing large records export from >>>>>>> ignite ? >>>>>>> >>>>>>> I could not use ScanQuery right as my whole application built around >>>>>>> Jdbc driver and writing complex queries in scan query is very difficult. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> On 10 June 2017 at 18:48, Anil <[hidden email] >>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=0>> wrote: >>>>>>> >>>>>>>> I understand from the code that there is no cursor from h2 db (or >>>>>>>> ignite embed h2 db) internally and all mapper response consolidated at >>>>>>>> reducer. It means when exporting large number of records, all data is >>>>>>>> in >>>>>>>> memory. >>>>>>>> >>>>>>>> if (send(nodes, >>>>>>>> oldStyle ? >>>>>>>> new GridQueryRequest(qryReqId, >>>>>>>> r.pageSize, >>>>>>>> space, >>>>>>>> mapQrys, >>>>>>>> topVer, >>>>>>>> extraSpaces(space, qry.spaces()), >>>>>>>> null, >>>>>>>> timeoutMillis) : >>>>>>>> new GridH2QueryRequest() >>>>>>>> .requestId(qryReqId) >>>>>>>> .topologyVersion(topVer) >>>>>>>> .pageSize(r.pageSize) >>>>>>>> .caches(qry.caches()) >>>>>>>> .tables(distributedJoins ? qry.tables() >>>>>>>> : null) >>>>>>>> .partitions(convert(partsMap)) >>>>>>>> .queries(mapQrys) >>>>>>>> .flags(flags) >>>>>>>> .timeout(timeoutMillis), >>>>>>>> oldStyle && partsMap != null ? new >>>>>>>> ExplicitPartitionsSpecializer(partsMap) : null, >>>>>>>> false)) { >>>>>>>> >>>>>>>> awaitAllReplies(r, nodes, cancel); >>>>>>>> >>>>>>>> *// once the responses from all nodes for the query received.. >>>>>>>> proceed further ?* >>>>>>>> >>>>>>>> if (!retry) { >>>>>>>> if (skipMergeTbl) { >>>>>>>> List<List<?>> res = new ArrayList<>(); >>>>>>>> >>>>>>>> // Simple UNION ALL can have multiple >>>>>>>> indexes. >>>>>>>> for (GridMergeIndex idx : r.idxs) { >>>>>>>> Cursor cur = idx.findInStream(null, >>>>>>>> null); >>>>>>>> >>>>>>>> while (cur.next()) { >>>>>>>> Row row = cur.get(); >>>>>>>> >>>>>>>> int cols = row.getColumnCount(); >>>>>>>> >>>>>>>> List<Object> resRow = new >>>>>>>> ArrayList<>(cols); >>>>>>>> >>>>>>>> for (int c = 0; c < cols; c++) >>>>>>>> resRow.add(row.getValue(c).get >>>>>>>> Object()); >>>>>>>> >>>>>>>> res.add(resRow); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> resIter = res.iterator(); >>>>>>>> }else { >>>>>>>> // incase of split query scenario >>>>>>>> } >>>>>>>> >>>>>>>> } >>>>>>>> >>>>>>>> return new GridQueryCacheObjectsIterator(resIter, cctx, >>>>>>>> keepPortable); >>>>>>>> >>>>>>>> >>>>>>>> Query cursor is iterator which does column value mapping per page. >>>>>>>> But still all records of query are still in memory. correct? >>>>>>>> >>>>>>>> Please correct me if I am wrong. thanks. >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> On 10 June 2017 at 15:53, Anil <[hidden email] >>>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=1>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> jvm parameters used - >>>>>>>>> >>>>>>>>> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC >>>>>>>>> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC >>>>>>>>> -XX:+DisableExplicitGC >>>>>>>>> -Xloggc:C:/Anil/dumps/gc-client.log -XX:+HeapDumpOnOutOfMemoryError >>>>>>>>> -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy >>>>>>>>> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC >>>>>>>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch >>>>>>>>> -XX:+PrintFlagsFinal -XX:HeapDumpPath=C:/Anil/dumps >>>>>>>>> /heapdump-client.hprof >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> >>>>>>>>> On 10 June 2017 at 15:06, Anil <[hidden email] >>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=2>> wrote: >>>>>>>>> >>>>>>>>>> HI, >>>>>>>>>> >>>>>>>>>> I have implemented export feature of ignite data using JDBC >>>>>>>>>> Interator >>>>>>>>>> >>>>>>>>>> ResultSet rs = statement.executeQuery(); >>>>>>>>>> >>>>>>>>>> while (rs.next()){ >>>>>>>>>> // do operations >>>>>>>>>> >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> and fetch size is 200. >>>>>>>>>> >>>>>>>>>> when i run export operation twice for 4 L records whole 6B is >>>>>>>>>> filled up and never getting released. >>>>>>>>>> >>>>>>>>>> Initially i thought that operations transforting result set to >>>>>>>>>> file causing the memory full. But not. >>>>>>>>>> >>>>>>>>>> I just did follwoing and still the memory is growing and not >>>>>>>>>> getting released >>>>>>>>>> >>>>>>>>>> while (rs.next()){ >>>>>>>>>> // nothing >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> num #instances #bytes class name >>>>>>>>>> ---------------------------------------------- >>>>>>>>>> 1: 55072353 2408335272 [C >>>>>>>>>> 2: 54923606 1318166544 java.lang.String >>>>>>>>>> 3: 779006 746187792 [B >>>>>>>>>> 4: 903548 304746304 [Ljava.lang.Object; >>>>>>>>>> 5: 773348 259844928 net.juniper.cs.entity.Install >>>>>>>>>> Base >>>>>>>>>> 6: 4745694 113896656 java.lang.Long >>>>>>>>>> 7: 1111692 44467680 sun.nio.cs.UTF_8$Decoder >>>>>>>>>> 8: 773348 30933920 org.apache.ignite.internal.bi >>>>>>>>>> nary.BinaryObjectImpl >>>>>>>>>> 9: 895627 21495048 java.util.ArrayList >>>>>>>>>> 10: 12427 16517632 [I >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Not sure why string objects are getting increased. >>>>>>>>>> >>>>>>>>>> Could you please help in understanding the issue ? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> ------------------------------ >>>>>>> If you reply to this email, your message will be added to the >>>>>>> discussion below: >>>>>>> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on- >>>>>>> ignite-client-tp13594p13626.html >>>>>>> To start a new topic under Apache Ignite Users, email [hidden email] >>>>>>> <http:///user/SendEmail.jtp?type=node&node=13706&i=1> >>>>>>> To unsubscribe from Apache Ignite Users, click here. >>>>>>> NAML >>>>>>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> View this message in context: Re: High heap on ignite client >>>>>> <http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13706.html> >>>>>> Sent from the Apache Ignite Users mailing list archive >>>>>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >>>>>> >>>>> >>>>> >>>> >>>> >>> >> >> *ignite-client.xml* (2K) Download Attachment >> <http://apache-ignite-users.70518.x6.nabble.com/attachment/13953/0/ignite-client.xml> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on- >> ignite-client-tp13594p13953.html >> To start a new topic under Apache Ignite Users, email [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13956&i=1> >> To unsubscribe from Apache Ignite Users, click here. >> NAML >> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > ------------------------------ > View this message in context: Re: High heap on ignite client > <http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13956.html> > Sent from the Apache Ignite Users mailing list archive > <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >
