Re: Zabbix 6.0.3 with Postgresql 14.2 on OpenBSD 7.1 stops graphing after two hours

Mark Patruck Tue, 19 Apr 2022 01:44:15 -0700

On 19.04.2022 10:30, Mark Patruck wrote:

On 16.04.2022 20:44, Noth wrote:

Hi,

Hi,

I built the new 7.1 packages for Zabbix and PostgreSQL, andupgraded my monitoring VM. To my horror the zabbix_server processstops graphing after a couple of hours of uptime, with thehousekeeper and history syncer processes at over 80% cpu usage.PostgreSQL shows INSERT processes stuck at 80% too, and restartingzabbix_server hangs, leaving zombie processes. I was using 2 vcpusand 2G of RAM. Feeling that the login.conf limits might be theproblem I uped them by quite a bit:


The zabbix_server process won't eat up your system normally...it's more
database related.

1) how big is your database? (important, as you don't have much memory
  for a busy PostgreSQL database)

2) did you make changes to /var/postgresql/data/postgresql.conf?
  (work_mem, shared_buffers, effective_cache_size, random_page_cost)

2) how many items do you monitor?

3) did you always follow the Zabbix upgrade guide (especially 5->6)?

4) how do your Zabbix server graphs "data gathering", "internal",
  "data handling" look like before the problem starts? Any high
  spikes?

5) have you already tried with a new database instance, i.e. move
  /var/postgresql/data to /var/postgresql/data_old and start with
  Zabbix 6.0.3 from scratch to see, if the database issues start
  immediately even when almost empty?

postgresql:\
-                :openfiles=1024:\
+                :openfiles=4096:\
               :tc=daemon:
                :datasize-max=2048M:\
                :datasize-cur=2048M:\


In almost all environments i've dealt with so far, you didn't need a
postgresql login class at all.

    ^^^^
Forget about that...i had the zabbix_server login class in mind.

        -Mark

I also uped sysctls:

-kern.seminfo.semmns=2048
-kern.shminfo.shmall=1024512
+kern.seminfo.semmns=4096
+kern.shminfo.shmall=1572864


This helped quite a bit, the graphing at least doesn't stop. Yet the housekeeper 
& history sync processes are still stuck continuously at over 80%. I even 
doubled the RAM to 4G and the vcpus to 4. Yet this is what top shows:

CPU0 states: 82.4% user,  0.0% nice, 14.7% sys,  1.0% spin,  0.0% intr,  2.0% 
idle
CPU1 states: 79.4% user,  0.0% nice,  9.8% sys,  0.0% spin,  0.0% intr, 10.8% 
idle
CPU2 states: 80.4% user,  0.0% nice,  9.8% sys,  0.0% spin,  0.0% intr,  9.8% 
idle
CPU3 states: 94.1% user,  0.0% nice,  4.9% sys,  0.0% spin,  0.0% intr,  1.0% 
idle
Memory: Real: 738M/2131M act/tot Free: 1796M Cache: 1091M Swap: 0K/2055M

PID USERNAME PRI NICE  SIZE   RES STATE     WAIT      TIME    CPU COMMAND
48835 _postgre  64    0  307M  247M onproc/0  -        23.4H 98.24% postgres: 
zabbix zabbix [local] SELECT
42079 _postgre  64    0  307M  245M run/2     -        25.3H 98.10% postgres: 
zabbix zabbix [local] SELECT
3630 _postgre  63    0  308M  282M onproc/1  -        21.2H 82.42% postgres: 
zabbix zabbix [local] DELETE
12622 _postgre  60    0  307M  127M onproc/2  -        20.8H 79.10% postgres: 
zabbix zabbix [local] INSERT

None of this behaviour happened with versions 2.x through to 5.x.Yes I've followed the pkg-readme for both PostgreSQL and Zabbix.Maybe these need adjusting? I'm at a loss on what I need to tune tomake everything go back to being running with low CPU usage. I amfully aware 6.x now collects data points of a gazillion more things(so many entries with just $1 as the name now, I'm not sure what'sgoing on there).


Hopefully some of you have an idea, cheers,

Noth

OpenBSD 7.1 (GENERIC.MP) #1: Wed Apr  6 18:48:24 CEST 2022
r...@builder2.nineinchnetworks.ch:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4255670272 (4058MB)
avail mem = 4109398016 (3919MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xbfbcb000 (13 entries)
bios0: vendor BHYVE version "13.0" date 11/10/2020
bios0: FreeBSD BHYVE
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S5
acpi0: tables DSDT FACP HPET APIC MCFG SPCR
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 16777216 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3408.57 MHz, 06-9e-0d
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: CPU supports MTRRs but not enabled by BIOS
cpu0: apic clock running at 134MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3413.28 MHz, 06-9e-0d
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 0, package 1
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3411.94 MHz, 06-9e-0d
cpu2: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 0, package 2
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz, 3419.69 MHz, 06-9e-0d
cpu3: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,RDSEED,MD_CLEAR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 0, package 3
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 11, 32 pins, remapped
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PC00)
acpipci0 at acpi0 PC00
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
acpicmos0 at acpi0
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: bhyve
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 unknown vendor 0x1275 product 0x1275 rev 0x00
ahci0 at pci0 dev 3 function 0 "Intel 82801H AHCI" rev 0x00: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
cd0 at scsibus1 targ 0 lun 0: <BHYVE, BHYVE DVD-ROM, 001> removable
virtio0 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio0
scsibus2 at vioblk0: 1 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 204800MB, 512 bytes/sector, 419430400 sectors
virtio0: msix shared
virtio1 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk1 at virtio1
scsibus3 at vioblk1: 1 targets
sd1 at scsibus3 targ 0 lun 0: <VirtIO, Block Device, >
sd1: 664MB, 512 bytes/sector, 1360832 sectors
virtio1: msix shared
virtio2 at pci0 dev 6 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio2: address 00:a0:98:16:bb:2c
virtio2: msix shared
pcib0 at pci0 dev 31 function 0 "Intel 82371SB ISA" rev 0x00
isa0 at pcib0
isadma0 at isa0
com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
efifb at mainbus0 not configured
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
root on sd0a (2b7e2c838e3db383.a) swap on sd0b dump on sd0b


--
Mark Patruck ( mark at wrapped.cx )
GPG key 0xF2865E51 / 187F F6D3 EE04 1DCE 1C74  F644 0D3C F66F F286 5E51
https://www.wrapped.cx


--
Mark Patruck ( mark at wrapped.cx )
GPG key 0xF2865E51 / 187F F6D3 EE04 1DCE 1C74  F644 0D3C F66F F286 5E51

https://www.wrapped.cx

Re: Zabbix 6.0.3 with Postgresql 14.2 on OpenBSD 7.1 stops graphing after two hours

Reply via email to