It helps a lot. Thanks for the detailed example. I have a last question. Would it still be a potential issue to lower the queue size if we don't have any backup? Like if a new node pre-loads from the primary, any new changes will also be populated to the soon-to-be new primary?
Sent from my iPhone On Jun 30, 2017, at 11:27 AM, Ilya Lantukh [via Apache Ignite Users] <[email protected]<mailto:[email protected]>> wrote: Hi Charly, There is no special thread in charge of clearing the queue. Entry is cleared when you try to add new entry to the queue, but it is already full. So, once queue becomes full, it will always stay full, and lowering queue size obviously reduces heap consumption. If cache is destroyed, the queue should be cleared immediately. However, if queue size is too low, it may lead to data inconsistency on unstable topology. Here is an example: Node A just joined the cluster, became primary node for some partitions and started preloading from node B, which is backup. Node B sends value V for key K, but before it was received and processed on node A user executed cache.remove(K). - If queue size is 0, node A now doesn't have any information that entry was just removed and has to store (K, V) pair from node B. On the other hand, node B will remove K and now primary and backup nodes have different data. - If queue size is N > 0, node A will save a tombstone entry for K. It will have all necessary information to understand that (K, V) pair from node B is older than current value and don't store it. Tombstone will be cleared after N other keys are removed. Hope this helps. On Thu, Jun 22, 2017 at 2:11 AM, charly <[hidden email]</user/SendEmail.jtp?type=node&node=14182&i=0>> wrote: Hey everyone, We use Ignite 1.8 and see a difference in heap used when lowering the value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE. I'm not sure to understand what it's going to happen if we decrease that value on production. Could someone clear that up for me please? From what I understand, deleted entries go in that queue for cleaning, where one thread per cache is in charge of cleaning the queue. But what if the queue is full? I might have missed that information in the documentation, my bad if I did. Maybe on the same subject, is it going to slow down the free up of the heap after cache.destroy()? We have a use-case where we delete 4 caches at the same time containing ~500k entries heavily indexed. When doing so, it takes few minutes for the heap to decrease as you can see here http://imgur.com/a/kPsSc . We would not want to extend the time to free up memory even more. More information about our setup: - on heap (we'll eventually move to off heap but back then we could not make Ignite 1.8 to free up memory at all with our version of glibc. It would work with jmalloc but that was not permitted on production at that time). - no backup - atomicity mode: atomic - cache mode: partitioned - cluster of 4 nodes - jvm options: -J-DIGNITE_LONG_OPERATIONS_DUMP_TIMEOUT=300000 -J-Xms12g -J-Xmx12g -J-server -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC -J-XX:+UseTLAB -J-XX:NewSize=128m -J-XX:MaxNewSize=128m -J-XX:MaxTenuringThreshold=0 -J-XX:SurvivorRatio=1024 -J-XX:+UseCMSInitiatingOccupancyOnly -J-XX:CMSInitiatingOccupancyFraction=10 -J-XX:MaxGCPauseMillis=1000 -J-XX:InitiatingHeapOccupancyPercent=10 -J-XX:+UseCompressedOops -J-XX:ParallelGCThreads=8 -J-XX:ConcGCThreads=8 -J-XX:+PrintGCDetails -J-XX:+PrintGCTimeStamps -J-XX:+PrintGCDateStamps -J-XX:+UseGCLogFileRotation -J-XX:NumberOfGCLogFiles=10 -J-XX:GCLogFileSize=100M -J-Xloggc:/tmp/ignite-gc.log -J-Dcom.sun.management.jmxremote.port=49130 Thanks for your help, Charly -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037.html Sent from the Apache Ignite Users mailing list archive at Nabble.com<http://Nabble.com>. -- Best regards, Ilya ________________________________ If you reply to this email, your message will be added to the discussion below: http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14182.html To unsubscribe from Small value for IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE, click here<http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=14037&code=Y2Rlc3NhcnRAYXBwbmV4dXMuY29tfDE0MDM3fDE3Mzc3NzQ4OTA=>. NAML<http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Small-value-for-IGNITE-ATOMIC-CACHE-DELETE-HISTORY-SIZE-tp14037p14183.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
