Hi,
We had a 2 node cluster with persistence enabled with backup(1) enabled,
this morning we've added a new node to the cluster.
Although rebalancing is finished Ignite Persistence values are not evenly
distributed;
server-1:
[2021-11-08T14:31:29,371][INFO ][grid-timeout-worker-#13][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=d9a3fb2f, uptime=01:35:00.491]
^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36,
minorTopVer=3]
^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500,
commPort=47100]
^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=3.48%, GC=0%]
^-- Heap [used=5418MB, free=33.85%, comm=8192MB]
^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB]
^-- Page memory [pages=1524682]
^-- sysMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%,
allocRam=100MB, allocTotal=0MB]
^-- default region [type=default, persistence=true, lazyAlloc=true,
... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%,
allocRam=6144MB, allocTotal=80891MB]
^-- metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.81%,
allocRam=0MB, allocTotal=0MB]
^-- TxLog region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
^-- volatileDsMemPlc region [type=user, persistence=false,
lazyAlloc=true,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=0MB]
^-- Ignite persistence [used=80891MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=4, qSize=0]
server-2:
[2021-11-08T14:31:20,475][INFO ][grid-timeout-worker-#13][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=d001436d, uptime=00:46:00.231]
^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36,
minorTopVer=3]
^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500,
commPort=47100]
^-- CPU [CPUs=4, curLoad=4.43%, avgLoad=5.11%, GC=0%]
^-- Heap [used=6468MB, free=21.04%, comm=8192MB]
^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB]
^-- Page memory [pages=1524684]
^-- sysMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%,
allocRam=100MB, allocTotal=0MB]
^-- default region [type=default, persistence=true, lazyAlloc=true,
... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%,
allocRam=6144MB, allocTotal=82852MB]
^-- metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.8%,
allocRam=0MB, allocTotal=0MB]
^-- TxLog region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
^-- volatileDsMemPlc region [type=user, persistence=false,
lazyAlloc=true,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=0MB]
^-- Ignite persistence [used=82852MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=4, qSize=0]
server-3:
[2021-11-08T14:31:21,364][INFO ][grid-timeout-worker-#13][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=186395d1, uptime=03:36:01.279]
^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36,
minorTopVer=3]
^-- Network [addrs=[10.156.0.10, 127.0.0.1], discoPort=47500,
commPort=47100]
^-- CPU [CPUs=4, curLoad=0.4%, avgLoad=8.55%, GC=0%]
^-- Heap [used=6153MB, free=24.89%, comm=8192MB]
^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB]
^-- Page memory [pages=1524749]
^-- sysMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.98%,
allocRam=100MB, allocTotal=0MB]
^-- default region [type=default, persistence=true, lazyAlloc=true,
... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%,
allocRam=6144MB, allocTotal=16164MB]
^-- metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.56%,
allocRam=0MB, allocTotal=0MB]
^-- TxLog region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
^-- volatileDsMemPlc region [type=user, persistence=false,
lazyAlloc=true,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=0MB]
^-- Ignite persistence [used=16165MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=4, qSize=0]
Before new node added to cluster data was evenly distributed;
server-1:
[2021-11-08T06:00:52,779][INFO ][grid-timeout-worker-#13][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=cde7abcb, uptime=4 days, 03:38:05.557]
^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17,
minorTopVer=0]
^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500,
commPort=47100]
^-- CPU [CPUs=4, curLoad=100%, avgLoad=5.78%, GC=117%]
^-- Heap [used=7669MB, free=6.37%, comm=8192MB]
^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB]
^-- Page memory [pages=1524669]
^-- sysMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%,
allocRam=100MB, allocTotal=0MB]
^-- default region [type=default, persistence=true, lazyAlloc=true,
... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%,
allocRam=6144MB, allocTotal=121061MB]
^-- metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%,
allocRam=0MB, allocTotal=0MB]
^-- TxLog region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
^-- volatileDsMemPlc region [type=user, persistence=false,
lazyAlloc=true,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=0MB]
^-- Ignite persistence [used=121061MB]
^-- Outbound messages queue [size=1]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=1, idle=3, qSize=1]
server-2:
[2021-11-08T06:00:37,491][INFO ][grid-timeout-worker-#13][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=90f15a32, uptime=4 days, 03:37:54.391]
^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17,
minorTopVer=0]
^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500,
commPort=47100]
^-- CPU [CPUs=4, curLoad=2.6%, avgLoad=7.38%, GC=0%]
^-- Heap [used=7877MB, free=3.83%, comm=8192MB]
^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB]
^-- Page memory [pages=1524670]
^-- sysMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%,
allocRam=100MB, allocTotal=0MB]
^-- default region [type=default, persistence=true, lazyAlloc=true,
... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%,
allocRam=6144MB, allocTotal=121310MB]
^-- metastoreMemPlc region [type=internal, persistence=true,
lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%,
allocRam=0MB, allocTotal=0MB]
^-- TxLog region [type=internal, persistence=true, lazyAlloc=false,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=100MB, allocTotal=0MB]
^-- volatileDsMemPlc region [type=user, persistence=false,
lazyAlloc=true,
... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%,
allocRam=0MB]
^-- Ignite persistence [used=121310MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=4, qSize=0]
My expectation is data to be distributed evenly.
What am I missing?
Regardds.
--
<https://www.segmentify.com/>İbrahim Halil AltunSenior Software Engineer+90
536 3327510 • segmentify.com → <https://www.segmentify.com/>UK • Germany •
Turkey <https://www.segmentify.com/ecommerce-growth-show>
<https://www.g2.com/products/segmentify/reviews>
