Hi Anil, I have not been able to reproduce your case based on the code and config you provided. If you provide the corresponding JFR records I will check it for any problems.
PFA attached the code. You can run it on yourself and monitor the client's GC activity via, for example VisualVM I tried running a server node (ServerRunner) with the following VM settings: -Xmx1024m -Xmx3072m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+ScavengeBeforeFullGC -XX:+AlwaysPreTouch -XX:+PrintFlagsFinal The client node was run with: -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCCause -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:+PrintFlagsFinal Kind regards, Alex. On Tue, Jun 20, 2017 at 8:00 AM, Anil [via Apache Ignite Users] < [email protected]> wrote: > Hi Alex, > > Thanks for the suggestion Alex. I will try with new setting. thanks. > > I have attached the gc client file. > > Did you find anything around jdbc issue ? I have put a debug point in > GridReduceQueryExecutor at resIter = res.iterator(); and res object holding > all the records. > > Thanks > > On 19 June 2017 at 18:50, Alexander Fedotov <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=13980&i=0>> wrote: > >> I don't see anything wrong with your config. >> Could you please provide C:/Anil/dumps/gc-client.log >> There should be a reason for objects not being collected during GC. >> >> Just one more thing, try replacing -XX:NewSize=512m with >> -XX:G1NewSizePercent=30 >> XX:NewSize won't let G1GC adjusting young gen size properly. >> >> >> >> Kind regards, >> Alex. >> >> On Mon, Jun 19, 2017 at 3:47 PM, afedotov <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=13980&i=1>> wrote: >> >>> Actually, JDBC driver should extract data page by page. >>> Need to take an in-depth look. >>> >>> Kind regards, >>> Alex. >>> >>> On Mon, Jun 19, 2017 at 3:08 PM, Anil [via Apache Ignite Users] <[hidden >>> email] <http:///user/SendEmail.jtp?type=node&node=13956&i=0>> wrote: >>> >>>> HI Alex, >>>> >>>> I have attached the ignite client xml. 4L means 0.4 million records. >>>> Sorry, I didn't generate JFR. But created heap dump. >>>> >>>> Do you agree that Jdbc driver loading everything in memory and next() >>>> just for conversion ? >>>> >>>> Thanks >>>> >>>> On 19 June 2017 at 17:16, Alexander Fedotov <[hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=0>> wrote: >>>> >>>>> Hi Anil. >>>>> >>>>> Could you please also share C:/Anil/ignite-client.xml ? As well, it >>>>> would be useful if you took JFR reports for the case with allocation >>>>> profiling enabled. >>>>> Just to clarify, by 4L do you mean 4 million entries? >>>>> >>>>> Kind regards, >>>>> Alex. >>>>> >>>>> On Mon, Jun 19, 2017 at 10:15 AM, Alexander Fedotov <[hidden email] >>>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=1>> wrote: >>>>> >>>>>> Thanks. I'll take a look and let you know about any findings. >>>>>> >>>>>> Kind regards, >>>>>> Alex >>>>>> >>>>>> 18 июня 2017 г. 3:33 PM пользователь "Anil" <[hidden email] >>>>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=2>> написал: >>>>>> >>>>>> Hi Alex, >>>>>> >>>>>> test program repository - https://github.com/adasari/t >>>>>> est-ignite-jdbc.git >>>>>> >>>>>> please let us if you have any suggestions/questions. thanks. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On 15 June 2017 at 10:58, Anil <[hidden email] >>>>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=3>> wrote: >>>>>> >>>>>>> Sure. thanks >>>>>>> >>>>>>> On 14 June 2017 at 19:51, afedotov <[hidden email] >>>>>>> <http:///user/SendEmail.jtp?type=node&node=13953&i=4>> wrote: >>>>>>> >>>>>>>> Hi, Anil. >>>>>>>> >>>>>>>> Could you please share your full code (class/method) you are using >>>>>>>> to read data. >>>>>>>> >>>>>>>> Kind regards, >>>>>>>> Alex >>>>>>>> >>>>>>>> 12 июня 2017 г. 4:07 PM пользователь "Anil [via Apache Ignite >>>>>>>> Users]" <[hidden email] >>>>>>>> <http:///user/SendEmail.jtp?type=node&node=13706&i=0>> написал: >>>>>>>> >>>>>>>>> Do you have any advice on implementing large records export from >>>>>>>>> ignite ? >>>>>>>>> >>>>>>>>> I could not use ScanQuery right as my whole application built >>>>>>>>> around Jdbc driver and writing complex queries in scan query is very >>>>>>>>> difficult. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> On 10 June 2017 at 18:48, Anil <[hidden email] >>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=0>> wrote: >>>>>>>>> >>>>>>>>>> I understand from the code that there is no cursor from h2 db (or >>>>>>>>>> ignite embed h2 db) internally and all mapper response consolidated >>>>>>>>>> at >>>>>>>>>> reducer. It means when exporting large number of records, all data >>>>>>>>>> is in >>>>>>>>>> memory. >>>>>>>>>> >>>>>>>>>> if (send(nodes, >>>>>>>>>> oldStyle ? >>>>>>>>>> new GridQueryRequest(qryReqId, >>>>>>>>>> r.pageSize, >>>>>>>>>> space, >>>>>>>>>> mapQrys, >>>>>>>>>> topVer, >>>>>>>>>> extraSpaces(space, qry.spaces()), >>>>>>>>>> null, >>>>>>>>>> timeoutMillis) : >>>>>>>>>> new GridH2QueryRequest() >>>>>>>>>> .requestId(qryReqId) >>>>>>>>>> .topologyVersion(topVer) >>>>>>>>>> .pageSize(r.pageSize) >>>>>>>>>> .caches(qry.caches()) >>>>>>>>>> .tables(distributedJoins ? >>>>>>>>>> qry.tables() : null) >>>>>>>>>> .partitions(convert(partsMap)) >>>>>>>>>> .queries(mapQrys) >>>>>>>>>> .flags(flags) >>>>>>>>>> .timeout(timeoutMillis), >>>>>>>>>> oldStyle && partsMap != null ? new >>>>>>>>>> ExplicitPartitionsSpecializer(partsMap) : null, >>>>>>>>>> false)) { >>>>>>>>>> >>>>>>>>>> awaitAllReplies(r, nodes, cancel); >>>>>>>>>> >>>>>>>>>> *// once the responses from all nodes for the query received.. >>>>>>>>>> proceed further ?* >>>>>>>>>> >>>>>>>>>> if (!retry) { >>>>>>>>>> if (skipMergeTbl) { >>>>>>>>>> List<List<?>> res = new ArrayList<>(); >>>>>>>>>> >>>>>>>>>> // Simple UNION ALL can have multiple >>>>>>>>>> indexes. >>>>>>>>>> for (GridMergeIndex idx : r.idxs) { >>>>>>>>>> Cursor cur = idx.findInStream(null, >>>>>>>>>> null); >>>>>>>>>> >>>>>>>>>> while (cur.next()) { >>>>>>>>>> Row row = cur.get(); >>>>>>>>>> >>>>>>>>>> int cols = row.getColumnCount(); >>>>>>>>>> >>>>>>>>>> List<Object> resRow = new >>>>>>>>>> ArrayList<>(cols); >>>>>>>>>> >>>>>>>>>> for (int c = 0; c < cols; c++) >>>>>>>>>> resRow.add(row.getValue(c).get >>>>>>>>>> Object()); >>>>>>>>>> >>>>>>>>>> res.add(resRow); >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> resIter = res.iterator(); >>>>>>>>>> }else { >>>>>>>>>> // incase of split query scenario >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> return new GridQueryCacheObjectsIterator(resIter, cctx, >>>>>>>>>> keepPortable); >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Query cursor is iterator which does column value mapping per >>>>>>>>>> page. But still all records of query are still in memory. correct? >>>>>>>>>> >>>>>>>>>> Please correct me if I am wrong. thanks. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 10 June 2017 at 15:53, Anil <[hidden email] >>>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=1>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> jvm parameters used - >>>>>>>>>>> >>>>>>>>>>> -Xmx6144m -XX:NewSize=512m -XX:+UseTLAB -XX:+UseG1GC >>>>>>>>>>> -XX:MaxGCPauseMillis=500 -XX:+ScavengeBeforeFullGC >>>>>>>>>>> -XX:+DisableExplicitGC >>>>>>>>>>> -Xloggc:C:/Anil/dumps/gc-client.log >>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCCause >>>>>>>>>>> -XX:+PrintGCDetails -XX:+PrintAdaptiveSizePolicy >>>>>>>>>>> -XX:+PrintGCTimeStamps >>>>>>>>>>> -XX:+PrintGCDateStamps -XX:+HeapDumpAfterFullGC >>>>>>>>>>> -XX:+ScavengeBeforeFullGC >>>>>>>>>>> -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:+PrintFlagsFinal >>>>>>>>>>> -XX:HeapDumpPath=C:/Anil/dumps/heapdump-client.hprof >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> On 10 June 2017 at 15:06, Anil <[hidden email] >>>>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=13626&i=2>> wrote: >>>>>>>>>>> >>>>>>>>>>>> HI, >>>>>>>>>>>> >>>>>>>>>>>> I have implemented export feature of ignite data using JDBC >>>>>>>>>>>> Interator >>>>>>>>>>>> >>>>>>>>>>>> ResultSet rs = statement.executeQuery(); >>>>>>>>>>>> >>>>>>>>>>>> while (rs.next()){ >>>>>>>>>>>> // do operations >>>>>>>>>>>> >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> and fetch size is 200. >>>>>>>>>>>> >>>>>>>>>>>> when i run export operation twice for 4 L records whole 6B is >>>>>>>>>>>> filled up and never getting released. >>>>>>>>>>>> >>>>>>>>>>>> Initially i thought that operations transforting result set to >>>>>>>>>>>> file causing the memory full. But not. >>>>>>>>>>>> >>>>>>>>>>>> I just did follwoing and still the memory is growing and not >>>>>>>>>>>> getting released >>>>>>>>>>>> >>>>>>>>>>>> while (rs.next()){ >>>>>>>>>>>> // nothing >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> num #instances #bytes class name >>>>>>>>>>>> ---------------------------------------------- >>>>>>>>>>>> 1: 55072353 2408335272 [C >>>>>>>>>>>> 2: 54923606 1318166544 java.lang.String >>>>>>>>>>>> 3: 779006 746187792 [B >>>>>>>>>>>> 4: 903548 304746304 [Ljava.lang.Object; >>>>>>>>>>>> 5: 773348 259844928 >>>>>>>>>>>> net.juniper.cs.entity.InstallBase >>>>>>>>>>>> 6: 4745694 113896656 java.lang.Long >>>>>>>>>>>> 7: 1111692 44467680 sun.nio.cs.UTF_8$Decoder >>>>>>>>>>>> 8: 773348 30933920 >>>>>>>>>>>> org.apache.ignite.internal.binary.BinaryObjectImpl >>>>>>>>>>>> 9: 895627 21495048 java.util.ArrayList >>>>>>>>>>>> 10: 12427 16517632 [I >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Not sure why string objects are getting increased. >>>>>>>>>>>> >>>>>>>>>>>> Could you please help in understanding the issue ? >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ------------------------------ >>>>>>>>> If you reply to this email, your message will be added to the >>>>>>>>> discussion below: >>>>>>>>> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on- >>>>>>>>> ignite-client-tp13594p13626.html >>>>>>>>> To start a new topic under Apache Ignite Users, email [hidden >>>>>>>>> email] <http:///user/SendEmail.jtp?type=node&node=13706&i=1> >>>>>>>>> To unsubscribe from Apache Ignite Users, click here. >>>>>>>>> NAML >>>>>>>>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>>>>>> >>>>>>>> >>>>>>>> ------------------------------ >>>>>>>> View this message in context: Re: High heap on ignite client >>>>>>>> <http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13706.html> >>>>>>>> Sent from the Apache Ignite Users mailing list archive >>>>>>>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> *ignite-client.xml* (2K) Download Attachment >>>> <http://apache-ignite-users.70518.x6.nabble.com/attachment/13953/0/ignite-client.xml> >>>> >>>> >>>> ------------------------------ >>>> If you reply to this email, your message will be added to the >>>> discussion below: >>>> http://apache-ignite-users.70518.x6.nabble.com/High-heap-on- >>>> ignite-client-tp13594p13953.html >>>> To start a new topic under Apache Ignite Users, email [hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=13956&i=1> >>>> To unsubscribe from Apache Ignite Users, click here. >>>> NAML >>>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>> >>> >>> >>> ------------------------------ >>> View this message in context: Re: High heap on ignite client >>> <http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13956.html> >>> Sent from the Apache Ignite Users mailing list archive >>> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >>> >> >> > > *gc-client-old.log* (1M) Download Attachment > <http://apache-ignite-users.70518.x6.nabble.com/attachment/13980/0/gc-client-old.log> > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client- > tp13594p13980.html > To start a new topic under Apache Ignite Users, email > [email protected] > To unsubscribe from Apache Ignite Users, click here > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=YWxleGFuZGVyLmZlZG90b2ZmQGdtYWlsLmNvbXwxfC0xMzYxNTU0NTg=> > . > NAML > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > test-ignite-jdbc-reproducer.tar.gz (60K) <http://apache-ignite-users.70518.x6.nabble.com/attachment/13990/0/test-ignite-jdbc-reproducer.tar.gz> -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/High-heap-on-ignite-client-tp13594p13990.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
