I’m not sure if it’s possible to diagnose this without knowing what’s on the heap.
I don’t think there are any known issues around heap usage. Certainly a MERGE is going to use a lot of heap space, but it should be recovered once the statement is closed. Similarly, your select could use a lot of heap depending on the number of records scanned, but that should also be released once the statement is closed. > On 20 Sep 2021, at 07:34, Ibrahim Altun <ibrahim.al...@segmentify.com> wrote: > > Hi igniters, > > the situation getting frustrated keeps going really huge GCs. Here is the > last GC report; > > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjAvLS1nYy5sb2cuMC5jdXJyZW50IDIuemlwLS02LTMxLTg=&channel=WEB > > This is production environment we cannot get heap dump since the prodecure > stops the whole world. > > please advise. > > On 2021/09/13 07:47:16, Naveen Kumar <naveen.band...@gmail.com> wrote: >> Just to add what Ibrahim mentioned, I also have a similar issue but I am >> using 2.8.1 and we do have a good number of Insert/merge statements getting >> executed. >> We do get warnings for some of the MERGE statements, like "*The search row >> by explicit key isn't supported. The primary key is always used to used to >> search a row". *Does is have any impact on heap utilization >> >> In our case, heap memory more than the off heap memory and doing a >> frequent rolling-over restarts to avoid OOM. >> >> >> >> On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun <ibrahim.al...@segmentify.com> >> wrote: >> >>> Hi Ilya, >>> >>> since this is production environment i could not risk to take heap dump >>> for now, but i will try to convince my superiors to get one and analyze it. >>> >>> Queries are heavily used in our system but aren't they autoclosable >>> objects? do we have to close them anyway? >>> >>> here are some usage examples on our system; >>> --insert query is like this; MERGE INTO "ProductLabel" ("productId", >>> "label", "language") VALUES (?, ?, ?) >>> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery); >>> >>> another usage example; >>> --sqlFieldsQuery is like this; >>> String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)"; >>> SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql); >>> sqlFieldsQuery.setLazy(true); >>> sqlFieldsQuery.setArgs(emails.toArray()); >>> >>> try (QueryCursor<List<?>> ignored = igniteCacheService.getCache(ID, >>> IgniteCacheType.USER).query(sqlFieldsQuery)) {...} >>> >>> >>> >>> On 2021/09/12 20:28:09, Shishkov Ilya <shishkovi...@gmail.com> wrote: >>>> Hi, Ibrahim! >>>> Have you analyzed the heap dump of the server node JVMs? >>>> In case your application executes queries are their cursors closed? >>>> >>>> пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun < >>> ibrahim.al...@segmentify.com>: >>>> >>>>> Igniters any comment on this issue, we are facing huge GC problems on >>>>> production environment, please advise. >>>>> >>>>> On 2021/09/07 14:11:09, Ibrahim Altun <ibrahim.al...@segmentify.com> >>>>> wrote: >>>>>> Hi, >>>>>> >>>>>> totally 400 - 600K reads/writes/updates >>>>>> 12core >>>>>> 64GB RAM >>>>>> no iowait >>>>>> 10 nodes >>>>>> >>>>>> On 2021/09/07 12:51:28, Piotr Jagielski <p...@touk.pl> wrote: >>>>>>> Hi, >>>>>>> Can you provide some information on how you use the cluster? How >>> many >>>>> reads/writes/updates per second? Also CPU / RAM spec of cluster nodes? >>>>>>> >>>>>>> We observed full GC / CPU load / OOM killer when loading big >>> amount of >>>>> data (15 mln records, data streamer + allowOverwrite=true). We've seen >>>>> 200-400k updates per sec on JMX metrics, but load up to 10 on nodes, >>> iowait >>>>> to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, >>> 32GB >>>>> RAM). Ignite 2.10 >>>>>>> >>>>>>> Regards, >>>>>>> Piotr >>>>>>> >>>>>>> On 2021/09/02 08:36:07, Ibrahim Altun < >>> ibrahim.al...@segmentify.com> >>>>> wrote: >>>>>>>> After upgrading from 2.7.1 version to 2.10.0 version ignite nodes >>>>> facing >>>>>>>> huge full GC operations after 24-36 hours after node start. >>>>>>>> >>>>>>>> We try to increase heap size but no luck, here is the start >>>>> configuration >>>>>>>> for nodes; >>>>>>>> >>>>>>>> JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server >>>>>>>> >>>>> >>> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml >>>>>>>> -Dcom.sun.management.jmxremote >>>>>>>> -Dcom.sun.management.jmxremote.authenticate=false >>>>>>>> -Dcom.sun.management.jmxremote.port=49165 >>>>>>>> -Dcom.sun.management.jmxremote.host=localhost >>>>>>>> -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g >>>>>>>> -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true >>>>>>>> -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=100000 >>>>>>>> -Djava.net.preferIPv4Stack=true" >>>>>>>> >>>>>>>> JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC >>>>>>>> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC >>>>>>>> -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log >>>>>>>> -XX:+PrintGCDetails -XX:+PrintGCDateStamps >>>>>>>> -XX:+PrintTenuringDistribution -XX:+PrintGCCause >>>>>>>> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 >>>>>>>> -XX:GCLogFileSize=100M" >>>>>>>> >>>>>>>> here is the 80 hours of GC analyize report: >>>>>>>> >>>>> >>> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ==&channel=WEB >>>>>>>> >>>>>>>> do we need more heap size or is there a BUG that we need to be >>> aware? >>>>>>>> >>>>>>>> here is the node configuration: >>>>>>>> >>>>>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"; >>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>>>>>> xsi:schemaLocation=" >>>>>>>> http://www.springframework.org/schema/beans >>>>>>>> http://www.springframework.org/schema/beans/spring-beans.xsd >>> "> >>>>>>>> <bean id="ignite.cfg" >>>>>>>> class="org.apache.ignite.configuration.IgniteConfiguration"> >>>>>>>> <property name="gridLogger"> >>>>>>>> <bean >>> class="org.apache.ignite.logger.log4j2.Log4J2Logger"> >>>>>>>> <constructor-arg type="java.lang.String" >>>>>>>> value="/etc/apache-ignite/ignite-log4j2.xml"/> >>>>>>>> </bean> >>>>>>>> </property> >>>>>>>> <property name="communicationSpi"> >>>>>>>> <bean >>>>> class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"> >>>>>>>> <property name="usePairedConnections" value="true"/> >>>>>>>> </bean> >>>>>>>> </property> >>>>>>>> <property name="failureDetectionTimeout" value="60000"/> >>>>>>>> <property name="systemThreadPoolSize" value="128"/> >>>>>>>> <property name="publicThreadPoolSize" value="128"/> >>>>>>>> <property name="queryThreadPoolSize" value="128"/> >>>>>>>> <property name="serviceThreadPoolSize" value="128"/> >>>>>>>> <property name="stripedPoolSize" value="128"/> >>>>>>>> <property name="dataStreamerThreadPoolSize" value="4"/> >>>>>>>> <property name="rebalanceThreadPoolSize" value="16"/> >>>>>>>> >>>>>>>> <!-- Explicitly enable peer class loading. --> >>>>>>>> <property name="peerClassLoadingEnabled" value="true"/> >>>>>>>> >>>>>>>> <!-- Enable deploymentSpi, >>>>>>>> /usr/share/apache-ignite/libs/segmentify directory will be >>> checked >>>>>>>> every 5 seconds for changed files--> >>>>>>>> <property name="deploymentSpi"> >>>>>>>> <bean >>>>> class="org.apache.ignite.spi.deployment.uri.UriDeploymentSpi"> >>>>>>>> <property name="temporaryDirectoryPath" >>>>>>>> value="/tmp/temp_ignite_libs"/> >>>>>>>> <property name="uriList"> >>>>>>>> <list> >>>>>>>> >>>>>>>> <value>file://freq=5000@localhost >>>>> /usr/share/apache-ignite/libs/segmentify/</value> >>>>>>>> </list> >>>>>>>> </property> >>>>>>>> </bean> >>>>>>>> </property> >>>>>>>> >>>>>>>> <property name="cacheConfiguration"> >>>>>>>> <list> >>>>>>>> <!-- Partitioned cache example configuration (Atomic >>>>> mode). --> >>>>>>>> <bean >>>>> class="org.apache.ignite.configuration.CacheConfiguration"> >>>>>>>> <property name="name" value="default"/> >>>>>>>> <property name="atomicityMode" value="ATOMIC"/> >>>>>>>> <property name="backups" value="1"/> >>>>>>>> </bean> >>>>>>>> </list> >>>>>>>> </property> >>>>>>>> >>>>>>>> <!-- Explicitly configure TCP discovery SPI to provide list >>> of >>>>>>>> initial nodes. --> >>>>>>>> <property name="discoverySpi"> >>>>>>>> <bean >>>>> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> >>>>>>>> <property name="networkTimeout" value="60000"/> >>>>>>>> <property name="ipFinder"> >>>>>>>> <bean >>>>>>>> >>>>> >>> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> >>>>>>>> <property name="addresses"> >>>>>>>> <list> >>>>>>>> <!-- THERE ARE 10 NODES --> >>>>>>>> </list> >>>>>>>> </property> >>>>>>>> </bean> >>>>>>>> </property> >>>>>>>> </bean> >>>>>>>> </property> >>>>>>>> >>>>>>>> <!-- Enabling Apache Ignite native persistence. --> >>>>>>>> <property name="dataStorageConfiguration"> >>>>>>>> <bean >>>>> class="org.apache.ignite.configuration.DataStorageConfiguration"> >>>>>>>> <property name="defaultDataRegionConfiguration"> >>>>>>>> <bean >>>>>>>> class="org.apache.ignite.configuration.DataRegionConfiguration"> >>>>>>>> <property name="persistenceEnabled" >>>>> value="true"/> >>>>>>>> <property name="checkpointPageBufferSize" >>>>>>>> value="#{ 2L * 1024 * 1024 * 1024}"/> >>>>>>>> <property name="maxSize" value="#{ 40L * >>> 1024 * >>>>>>>> 1024 * 1024 }"/> >>>>>>>> </bean> >>>>>>>> </property> >>>>>>>> <property name="storagePath" >>>>> value="/srv/ignite/persist"/> >>>>>>>> <property name="walPath" value="/srv/ignite/wal"/> >>>>>>>> <property name="walArchivePath" >>> value="/srv/ignite/wal"/> >>>>>>>> <property name="walMode" value="LOG_ONLY"/> >>>>>>>> <property name="walSegmentSize" value="#{ 256L * >>> 1024 * >>>>> 1024 }"/> >>>>>>>> <property name="walFlushFrequency" value="5000"/> >>>>>>>> <property name="maxWalArchiveSize" value="#{ 512L * >>> 1024 >>>>> * 1024 }"/> >>>>>>>> <property name="writeThrottlingEnabled" >>> value="true"/> >>>>>>>> <property name="checkpointFrequency" value="300000"/> >>>>>>>> <property name="checkpointWriteOrder" >>> value="SEQUENTIAL" >>>>> /> >>>>>>>> </bean> >>>>>>>> </property> >>>>>>>> </bean> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> <https://www.segmentify.com/>İbrahim Halil AltunSenior Software >>>>> Engineer+90 >>>>>>>> 536 3327510 • segmentify.com → <https://www.segmentify.com/>UK • >>>>> Germany • >>>>>>>> Turkey <https://www.segmentify.com/ecommerce-growth-show> >>>>>>>> <https://www.g2.com/products/segmentify/reviews> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> -- >> Thanks & Regards, >> Naveen Bandaru >>
