Hi. We have a pair of Prometheus instances that consume significantly different amounts of storage. The instances are created by the same StatefulSet (created by Prometheus-operator), so they are using the same configuration.
Both instances have similar number of samples and series, but instance "0" consume up to ~50% more storage than instance "1". $ kubectl exec prometheus-prometheus-0 -- /bin/sh -c "promtool tsdb list ." BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE 01GWY0R4N3QG1QJS957XZ0SYP7 1680264000000 1680328800000 18h0m0s 3296299059 26900037 931315 7259935610 01GX05E79R6WNQ4F6MMB068WJ7 1680328800003 1680393600000 17h59m59.997s 3312300492 27012299 892364 7265602468 01GX237SWZYZ6X5XXMENGMQ1YM 1680393600002 1680458400000 17h59m59.998s 3315540127 27036907 894595 7247593445 01GX410BDAPBMPKP100300C7DW 1680458400001 1680523200000 17h59m59.999s 3320458065 27130364 987454 7328750825 01GX5YTZVD5W97D497JA11CATT 1680523200007 1680588000000 17h59m59.993s 3318443269 27135815 1007206 7380926789 01GX7WMF1FNJ6MGT0TJY2A5KEM 1680588000001 1680652800000 17h59m59.999s 3331999517 27259726 1028363 7364976990 01GX9TDYMKVWJ9WYYY7CD8BCWH 1680652800005 1680717600000 17h59m59.995s 3327868238 27186293 981912 7288127305 01GXBR7FYPSRWA6N6313MMR9BM 1680717600009 1680782400000 17h59m59.991s 3327937718 27125975 896286 7199443835 01GXDP01QKKXC137B7JJZ6706W 1680782400001 1680847200000 17h59m59.999s 3333037262 27172805 897459 7194002011 01GXFKTGVZN6RXRM74PXB5E61Q 1680847200001 1680912000000 17h59m59.999s 3329211104 27134065 879001 7202044230 01GXHHM1JST118SYQNX8Z5W8PX 1680912000004 1680976800000 17h59m59.996s 3329464442 27131788 876881 7192136400 01GXKFCM51YQFDAWTGP6BBXGQZ 1680976800003 1681041600000 17h59m59.997s 3329134675 27127804 875877 7197030123 01GXND71ZF62MK8M1DP5E7345M 1681041600011 1681106400000 17h59m59.989s 3327555787 27119184 887763 7216837469 01GXQB0QJX2T55FBFFSZC9PC4D 1681106400005 1681171200000 17h59m59.995s 3324035858 27084455 871653 7195109123 01GXS8T0EJXHH2C1B0CBAEPHQB 1681171200011 1681236000000 17h59m59.989s 3315573555 26493111 989655 6235040678 01GXSXCRNTNDJC359R7160ZEDX 1681236000001 1681257600000 5h59m59.999s 1107306526 9028997 828578 1830084344 01GXSPFKRCVXHRSD2WAFEFM0WD 1681257600005 1681264800000 1h59m59.995s 369706839 3015597 808854 671597409 01GXSXBRED7JT7WJY9318QYKKZ 1681264800002 1681272000000 1h59m59.998s 369661386 3012000 805553 668951473 01GXT47FQYC712E0M6XPSP41FF 1681272000001 1681279200000 1h59m59.999s 369740628 3021714 823966 673649781 $ kubectl exec prometheus-prometheus-1 -- /bin/sh -c "promtool tsdb list ." BLOCK ULID MIN TIME MAX TIME DURATION NUM SAMPLES NUM CHUNKS NUM SERIES SIZE 01GWY0RDK93D2RYJBHJRDMS100 1680264000000 1680328800000 18h0m0s 3296396516 26926127 957040 4831014683 01GX05ETVBDXMQH0KW9NX7RCPC 1680328800003 1680393600000 17h59m59.997s 3312324642 27036260 917296 4807892522 01GX2383F7YPDX400MN4DQ9CSX 1680393600002 1680458400000 17h59m59.998s 3315587751 27059963 918166 4832761551 01GX410PJXX52PKFVH1H205385 1680458400003 1680523200000 17h59m59.997s 3320397897 27157090 1014022 4890962085 01GX5YVKEAQ6D1NZM1AQW0YJ90 1680523200003 1680588000000 17h59m59.997s 3318472581 27171422 1042831 4854062752 01GX7WMWV41M3PW3BFV62P0M32 1680588000001 1680652800000 17h59m59.999s 3331918609 27288755 1056267 4861196239 01GX9TECS126QJM1A1F61GW0ZT 1680652800003 1680717600000 17h59m59.997s 3328065112 27214643 1008335 4831609465 01GXBR7NZ3RXVSP50V5J2QE4HQ 1680717600003 1680782400000 17h59m59.997s 3327954927 27159515 929150 4800273178 01GXDP1BCZF7THHQTSK4YGAYX7 1680782400001 1680847200000 17h59m59.999s 3333181469 27202041 926662 4805982300 01GXFKTTFMZ723VQ94BDY1H1T9 1680847200001 1680912000000 17h59m59.999s 3329426122 27163849 908757 4715850967 01GXHHMTBPDXX173MD9V1B24E7 1680912000003 1680976800000 17h59m59.997s 3329413976 27163579 909408 4791199693 01GXKFCZX5HX3KBR82Y5H8DS3D 1680976800003 1681041600000 17h59m59.997s 3329270278 27160688 908615 4789400558 01GXND7FRJ628BCK936DYH6BVV 1681041600003 1681106400000 17h59m59.997s 3327613909 27155028 922458 4784598143 01GXQB1BWNS0EDT4TX7HFACWBB 1681106400003 1681171200000 17h59m59.997s 3324104564 27121755 909230 4777589596 01GXS8TD8CA5E4BPWKY1VXEV71 1681171200003 1681236000000 17h59m59.997s 3320030145 26512567 1016662 4802788228 01GXSXCQE71EB940GMBPS4NVXV 1681236000001 1681257600000 5h59m59.999s 1107449702 9035526 835180 1609217929 01GXSPFXTNAJCYMSKABCF18YH9 1681257600005 1681264800000 1h59m59.995s 369710007 3018579 811751 607549046 01GXSXBN258J5VHMXR1FG9E84P 1681264800002 1681272000000 1h59m59.998s 369666717 3014789 808149 601091436 01GXT47CA6RV2QPZMVN45EH6CQ 1681272000000 1681279200000 2h0m0s 369734916 3022662 825536 607677006 What can cause this type of effects? As both instances scrape the same targets, we should achieve the same level of compression of the time series data (AFAICT). -- Pelle -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOURYnDWSYAWUCP-8sJGpvFvhscOC1h8xiYh31zVh9dCX7f4ng%40mail.gmail.com.
