I'm also hit by this, but I'm not sure if this is a bug or rather simplified implementation. My scenario is to use relayd as a tls-accelerator for Nexus. I've needed to allocate 12GB RAM to OpenBSD VM in order to be able to push 2.5GB file to Nexus. Please note that although relayd basically buffers everything from PUT first and then pushes this to Nexus, it still behaves correctly as after successful transfer it correctly unallocates RAM it consumed for the transfer. In my 2.5GB file case relayd goes up to 4-6GB RAM consumption but after tranfer it goes correctly down to few MBs consumtion.
So the point is, it may also be simplified PUT implementation instead of a real memory leak. On Fri, Oct 20, 2017 at 12:38 PM, Ronan VIEL <[email protected]> wrote: > Hi, > > > > I have found what I guess is a bug in relayd. I noticed the issue when > transfering a large file (over 1GB) which killed my system. > > Here is my sendbug report, my relayd.conf (nothing special here) and output > of "top" where you can see that the memory used by the relayd process (30946) > is doubling every 5 seconds. > > > > Regards, > > Ronan > > > > > > > > > >>Synopsis: Relayd memory leak crashes system with large file transfer > >>Category: System > >>Environment: > > System : OpenBSD 6.2 > > Details : OpenBSD 6.2 (GENERIC) #0: Thu Oct 12 19:16:36 CEST > 2017 > > > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC > > > > Architecture: OpenBSD.amd64 > > Machine : amd64 > >>Description: > > The system is running relayd configured as a relay for TLS > acceleration. > > When using this relay to transfer large files (i.e. 10GB) with PUT > > method, the relayd process is doubling its memory usage regularly > until > > the system crashes. > > > >>How-To-Repeat: > > Setup a simple HTTP relay, start a file transfer and monitor > memory > > usage with top. I use NextCloud as target service on which I > upload > > the large file. > >>Fix: > > Unknown > > > > dmesg: > > OpenBSD 6.2 (GENERIC) #0: Thu Oct 12 19:16:36 CEST 2017 > > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC > > real mem = 1056899072 (1007MB) > > avail mem = 1017995264 (970MB) > > mpath0 at root > > scsibus0 at mpath0: 256 targets > > mainbus0 at root > > bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (364 entries) > > bios0: vendor Phoenix Technologies LTD version "6.00" date 04/14/2014 > > bios0: VMware, Inc. VMware Virtual Platform > > acpi0 at bios0: rev 2 > > acpi0: sleep states S0 S1 S4 S5 > > acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET > > acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) > S16F(S3) S17F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) > S1F0(S3) PE50(S3) [...] > > acpitimer0 at acpi0: 3579545 Hz, 24 bits > > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > > cpu0 at mainbus0: apid 0 (boot processor) > > cpu0: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz, 2394.07 MHz > > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > > cpu0: 256KB 64b/line 8-way L2 cache > > cpu0: TSC frequency 2394070950 Hz > > cpu0: smt 0, core 0, package 0 > > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges > > cpu0: apic clock running at 65MHz > > ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 11, 24 pins > > acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127 > > acpihpet0 at acpi0: 14318179 Hz > > acpiprt0 at acpi0: bus 0 (PCI0) > > acpicpu0 at acpi0: C1(@1 halt!) > > "PNP0001" at acpi0 not configured > > "PNP0F13" at acpi0 not configured > > "PNP0A05" at acpi0 not configured > > "PNP0400" at acpi0 not configured > > "PNP0700" at acpi0 not configured > > acpiac0 at acpi0: AC unit online > > pvbus0 at mainbus0: VMware > > vmt0 at pvbus0 > > pci0 at mainbus0 bus 0 > > pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01 > > ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01 > > pci1 at ppb0 bus 1 > > pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08 > > pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 > configured to compatibility, channel 1 configured to compatibility > > atapiscsi0 at pciide0 channel 0 drive 0 > > scsibus1 at atapiscsi0: 2 targets > > cd0 at scsibus1 targ 0 lun 0: ATAPI 5/cdrom removable > > cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2 > > pciide0: channel 1 disabled (no drives) > > piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus > disabled > > "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured > > vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00 > > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > > mpi0 at pci0 dev 16 function 0 "Symbios Logic 53c1030" rev 0x01: apic 1 int 17 > > mpi0: 0, firmware 1.3.41.32 > > scsibus2 at mpi0: 16 targets, initiator 7 > > sd0 at scsibus2 targ 0 lun 0: SCSI2 0/direct fixed > > sd0: 8192MB, 512 bytes/sector, 16777216 sectors > > mpi0: target 0 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1 > > ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02 > > pci2 at ppb1 bus 2 > > ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01 > > pci3 at ppb2 bus 3 > > vmx0 at pci3 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 18, > address 00:50:56:01:42:33 > > ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01 > > pci4 at ppb3 bus 4 > > ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01 > > pci5 at ppb4 bus 5 > > ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01 > > pci6 at ppb5 bus 6 > > ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01 > > pci7 at ppb6 bus 7 > > ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01 > > pci8 at ppb7 bus 8 > > ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01 > > pci9 at ppb8 bus 9 > > ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01 > > pci10 at ppb9 bus 10 > > ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01 > > pci11 at ppb10 bus 11 > > vmx1 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 19, > address 00:50:56:10:14:7b > > ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01 > > pci12 at ppb11 bus 12 > > ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01 > > pci13 at ppb12 bus 13 > > ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01 > > pci14 at ppb13 bus 14 > > ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01 > > pci15 at ppb14 bus 15 > > ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01 > > pci16 at ppb15 bus 16 > > ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01 > > pci17 at ppb16 bus 17 > > ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01 > > pci18 at ppb17 bus 18 > > ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01 > > pci19 at ppb18 bus 19 > > vmx2 at pci19 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 16, > address 00:50:56:10:14:cc > > ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01 > > pci20 at ppb19 bus 20 > > ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01 > > pci21 at ppb20 bus 21 > > ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01 > > pci22 at ppb21 bus 22 > > ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01 > > pci23 at ppb22 bus 23 > > ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01 > > pci24 at ppb23 bus 24 > > ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01 > > pci25 at ppb24 bus 25 > > ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01 > > pci26 at ppb25 bus 26 > > ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01 > > pci27 at ppb26 bus 27 > > ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01 > > pci28 at ppb27 bus 28 > > ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01 > > pci29 at ppb28 bus 29 > > ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01 > > pci30 at ppb29 bus 30 > > ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01 > > pci31 at ppb30 bus 31 > > ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01 > > pci32 at ppb31 bus 32 > > ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01 > > pci33 at ppb32 bus 33 > > ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01 > > pci34 at ppb33 bus 34 > > isa0 at pcib0 > > isadma0 at isa0 > > fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 > > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > > com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo > > pckbc0 at isa0 port 0x60/5 irq 1 irq 12 > > pckbd0 at pckbc0 (kbd slot) > > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > > pms0 at pckbc0 (aux slot) > > wsmouse0 at pms0 mux 0 > > pcppi0 at isa0 port 0x61 > > spkr0 at pcppi0 > > lpt0 at isa0 port 0x378/4 irq 7 > > vscsi0 at root > > scsibus3 at vscsi0: 256 targets > > softraid0 at root > > scsibus4 at softraid0: 256 targets > > root on sd0a (a9a47698e70bebe6.a) swap on sd0b dump on sd0b > > fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec > > > > > > > > relayd.conf: > > log updates > > prefork 10 > > > > cloud_fqdn = "cloud.mydomain.tld" > > table { 172.16.0.1 } > > > > http protocol tls-webproxy { > > match request header append "X-Forwarded-For" value "$REMOTE_ADDR" > > match request header append "X-Forwarded-By" \ > > value > "$SERVER_ADDR:SERVER_PORT" > > match header set "Connection" value "close" > > > > pass request quick header "Host" value \ > > $cloud_fqdn forward to > > block > > } > > > > relay "myRelay" { > > listen on a.b.c.d port 443 tls > > protocol tls-webproxy > > > > forward to port 80 > > } > > > > > > top output: > > > > > > load averages: 0.00, 0.00, 0.00 gw-1.orange-projectbooster.co 11:32:50 > > 58 processes: 57 idle, 1 on processor up 0:16 CPU states: 0.8% user, 0.0% > nice, 0.8% system, 0.2% interrupt, 98.2% idle > > Memory: Real: 114M/394M act/tot Free: 585M Cache: 196M Swap: 0K/182M > > > > PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND > > 1 root 10 0 388K 432K idle wait 0:01 0.00% init > > 24756 _relayd 2 0 1584K 4276K sleep kqread 0:00 0.00% relayd > > 66338 _relayd 2 0 1596K 4276K idle kqread 0:00 0.00% relayd > > 83319 _relayd 2 0 1588K 4268K idle kqread 0:00 0.00% relayd > > 64814 _relayd 2 0 1600K 4280K idle kqread 0:00 0.00% relayd > > 30946 _relayd 2 0 2368K 5348K sleep kqread 0:00 0.00% relayd > > 24050 _relayd 2 0 1588K 4264K idle kqread 0:00 0.00% relayd > > 52580 _relayd 2 0 1576K 4228K idle kqread 0:00 0.00% relayd > > 3884 _relayd 2 0 1588K 4284K sleep kqread 0:00 0.00% relayd > > 90046 _relayd 2 0 1588K 4244K idle kqread 0:00 0.00% relayd > > 75377 root 18 0 752K 860K sleep pause 0:00 0.00% ksh > > 24898 _relayd 2 0 1592K 4260K idle kqread 0:00 0.00% relayd > > 13327 _relayd 2 0 1588K 4252K idle kqread 0:00 0.00% relayd > > 21783 _relayd 2 0 2024K 4956K sleep kqread 0:00 0.00% relayd > > 23277 _syslogd 2 0 912K 1600K sleep kqread 0:00 0.00% syslogd > > 78899 _relayd 2 0 2024K 4968K sleep kqread 0:00 0.00% relayd > > 50448 _pflogd 4 0 712K 604K sleep bpf 0:00 0.00% pflogd > > 55475 root 2 0 3096K 5320K idle kqread 0:00 0.00% relayd > > > > > > load averages: 0.00, 0.00, 0.00 gw-1.orange-projectbooster.co 11:32:55 > > 58 processes: 57 idle, 1 on processor up 0:16 CPU states: 1.6% user, 0.0% > nice, 2.0% system, 1.8% interrupt, 94.6% idle > > Memory: Real: 130M/412M act/tot Free: 567M Cache: 196M Swap: 0K/182M > > > > PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND > > 30946 _relayd 2 0 18M 21M sleep kqread 0:00 0.59% relayd > > 1 root 10 0 388K 432K idle wait 0:01 0.00% init > > 24756 _relayd 2 0 1584K 4276K sleep kqread 0:00 0.00% relayd > > 66338 _relayd 2 0 1596K 4276K idle kqread 0:00 0.00% relayd > > 83319 _relayd 2 0 1588K 4268K idle kqread 0:00 0.00% relayd > > 64814 _relayd 2 0 1600K 4280K idle kqread 0:00 0.00% relayd > > 24050 _relayd 2 0 1588K 4264K idle kqread 0:00 0.00% relayd > > 3884 _relayd 2 0 1588K 4284K sleep kqread 0:00 0.00% relayd > > 52580 _relayd 2 0 1576K 4228K idle kqread 0:00 0.00% relayd > > 90046 _relayd 2 0 1588K 4244K idle kqread 0:00 0.00% relayd > > 75377 root 18 0 752K 860K sleep pause 0:00 0.00% ksh > > 24898 _relayd 2 0 1592K 4260K idle kqread 0:00 0.00% relayd > > 13327 _relayd 2 0 1588K 4252K idle kqread 0:00 0.00% relayd > > 21783 _relayd 2 0 2024K 4956K sleep kqread 0:00 0.00% relayd > > 23277 _syslogd 2 0 912K 1600K sleep kqread 0:00 0.00% syslogd > > 78899 _relayd 2 0 2024K 4968K sleep kqread 0:00 0.00% relayd > > 50448 _pflogd 4 0 712K 604K sleep bpf 0:00 0.00% pflogd > > 189 _relayd 2 0 2344K 5304K sleep kqread 0:00 0.00% relayd > > > > > > load averages: 0.00, 0.00, 0.00 gw-1.orange-projectbooster.co 11:33:00 > > 58 processes: 57 idle, 1 on processor up 0:16 CPU states: 2.6% user, 0.0% > nice, 4.0% system, 1.2% interrupt, 92.2% idle > > Memory: Real: 146M/428M act/tot Free: 551M Cache: 196M Swap: 0K/182M > > > > PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND > > 30946 _relayd 2 0 34M 37M sleep kqread 0:00 1.37% relayd > > 1 root 10 0 388K 432K idle wait 0:01 0.00% init > > 24756 _relayd 2 0 1584K 4276K sleep kqread 0:00 0.00% relayd > > 66338 _relayd 2 0 1596K 4276K idle kqread 0:00 0.00% relayd > > 83319 _relayd 2 0 1588K 4268K idle kqread 0:00 0.00% relayd > > 64814 _relayd 2 0 1600K 4280K idle kqread 0:00 0.00% relayd > > 24050 _relayd 2 0 1588K 4264K idle kqread 0:00 0.00% relayd > > 3884 _relayd 2 0 1588K 4284K sleep kqread 0:00 0.00% relayd > > 52580 _relayd 2 0 1576K 4228K idle kqread 0:00 0.00% relayd > > 90046 _relayd 2 0 1588K 4244K idle kqread 0:00 0.00% relayd > > 75377 root 18 0 752K 860K sleep pause 0:00 0.00% ksh > > 24898 _relayd 2 0 1592K 4260K idle kqread 0:00 0.00% relayd > > 13327 _relayd 2 0 1588K 4252K idle kqread 0:00 0.00% relayd > > 21783 _relayd 2 0 2024K 4956K sleep kqread 0:00 0.00% relayd > > 23277 _syslogd 2 0 912K 1600K sleep kqread 0:00 0.00% syslogd > > 189 _relayd 2 0 2344K 5304K sleep kqread 0:00 0.00% relayd > > 78899 _relayd 2 0 2024K 4968K sleep kqread 0:00 0.00% relayd > > 50448 _pflogd 4 0 712K 604K sleep bpf 0:00 0.00% pflogd > > > > > > load averages: 0.00, 0.00, 0.00 gw-1.orange-projectbooster.co 11:33:05 > > 58 processes: 57 idle, 1 on processor up 0:16 CPU states: 2.8% user, 0.0% > nice, 4.2% system, 1.4% interrupt, 91.6% idle > > Memory: Real: 178M/461M act/tot Free: 517M Cache: 196M Swap: 0K/182M > > > > PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND > > 30946 _relayd 2 0 66M 69M sleep kqread 0:01 1.90% relayd > > 1 root 10 0 388K 432K idle wait 0:01 0.00% init > > 24756 _relayd 2 0 1584K 4276K sleep kqread 0:00 0.00% relayd > > 66338 _relayd 2 0 1596K 4276K idle kqread 0:00 0.00% relayd > > 83319 _relayd 2 0 1588K 4268K idle kqread 0:00 0.00% relayd > > 64814 _relayd 2 0 1600K 4280K idle kqread 0:00 0.00% relayd > > 24050 _relayd 2 0 1588K 4264K idle kqread 0:00 0.00% relayd > > 3884 _relayd 2 0 1588K 4284K sleep kqread 0:00 0.00% relayd > > 52580 _relayd 2 0 1576K 4228K idle kqread 0:00 0.00% relayd > > 90046 _relayd 2 0 1588K 4244K idle kqread 0:00 0.00% relayd > > 75377 root 18 0 752K 860K sleep pause 0:00 0.00% ksh > > 24898 _relayd 2 0 1592K 4260K idle kqread 0:00 0.00% relayd > > 13327 _relayd 2 0 1588K 4252K idle kqread 0:00 0.00% relayd > > 21783 _relayd 2 0 2024K 4956K sleep kqread 0:00 0.00% relayd > > 23277 _syslogd 2 0 912K 1600K sleep kqread 0:00 0.00% syslogd > > 189 _relayd 2 0 2344K 5304K sleep kqread 0:00 0.00% relayd > > 50448 _pflogd 4 0 712K 604K sleep bpf 0:00 0.00% pflogd > > 78899 _relayd 2 0 2024K 4968K sleep kqread 0:00 0.00% relayd > > > > > > load averages: 0.00, 0.00, 0.00 gw-1.orange-projectbooster.co 11:33:10 > > 58 processes: 1 running, 56 idle, 1 on processor up 0:16 CPU states: 3.0% > user, 0.0% nice, 4.6% system, 1.4% interrupt, 91.0% idle > > Memory: Real: 242M/528M act/tot Free: 451M Cache: 196M Swap: 0K/182M > > > > PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND > > 30946 _relayd 28 0 130M 134M run - 0:01 2.59% relayd > > 1 root 10 0 388K 432K idle wait 0:01 0.00% init > > 24756 _relayd 2 0 1584K 4276K sleep kqread 0:00 0.00% relayd > > 66338 _relayd 2 0 1596K 4276K idle kqread 0:00 0.00% relayd > > 83319 _relayd 2 0 1588K 4268K idle kqread 0:00 0.00% relayd > > 24050 _relayd 2 0 1588K 4288K sleep kqread 0:00 0.00% relayd > > 64814 _relayd 2 0 1600K 4280K idle kqread 0:00 0.00% relayd > > 3884 _relayd 2 0 1588K 4284K sleep kqread 0:00 0.00% relayd > > 52580 _relayd 2 0 1576K 4228K idle kqread 0:00 0.00% relayd > > 90046 _relayd 2 0 1588K 4244K idle kqread 0:00 0.00% relayd > > 75377 root 18 0 752K 860K sleep pause 0:00 0.00% ksh > > 24898 _relayd 2 0 1592K 4260K idle kqread 0:00 0.00% relayd > > 13327 _relayd 2 0 1588K 4252K idle kqread 0:00 0.00% relayd > > 21783 _relayd 2 0 2024K 4956K sleep kqread 0:00 0.00% relayd > > 23277 _syslogd 2 0 912K 1600K sleep kqread 0:00 0.00% syslogd > > 93747 _relayd 2 0 2272K 5248K sleep kqread 0:00 0.00% relayd > > 189 _relayd 2 0 2344K 5304K sleep kqread 0:00 0.00% relayd > > 50448 _pflogd 4 0 712K 604K sleep bpf 0:00 0.00% pflogd > > > > > >
