Re: Looking for DMVPN implementation

[Jens Sauer]

Hi Renato,

i'm excited and cant wait to give it a try - thx so much

cheers

Jens Sauer




- Ursprüngliche Message -
Von: Renato Westphal 
An: Jens Sauer 
CC: "misc@openbsd.org" 
Gesendet: 17:27 Montag, 3.Oktober 2016
Betreff: Re: Looking for DMVPN implementation

2016-10-01 19:44 GMT-03:00 Jens Sauer :

> Hi OpenBSD community,
>
> i'm looking for an OpenSource implementation of DMVPN (Dynamic Multipoint
Virtual private network).
>
> Currently i just found the draft (from 2013) :
> https://tools.ietf.org/html/draft-detienne-dmvpn-00
>
> Comming from Cisco and would be pleased to see it under OpenBSD.
>
http://www.cisco.com/c/dam/en/us/products/collateral/security/dynamic-multipo
int-vpn-dmvpn/DMVPN_Overview.pdf
>
> Hope i could get an advice in how to implement (use) it under OpenDSD.

Hi Jens,

I already started working on this in g2k16 and I should have something
to show in a few months. In the hackathon, claudio@ gave me some
pointers on how to add multipoint support in gre(4) and right now I'm
evaluating how to design nhrpd(8) in the best way possible (including
the integration with iked(8) - only IKEv2 will be supported).

I'll let you know when I have something ready.

Cheers,
--
Renato Westphal

Re: fw_update stops with Fatal error: Unsigned package ...

[Josh Grosse]

On 2016-10-03 14:11, Mihai Popescu wrote:

I've installed a snapshot somewhile ago, then I needed to update the
firmware for athn device. I get this error:

# fw_update
UNSIGNED PACKAGES: athn-firmware-1.1p1
Fatal error: Unsigned package
http://firmware.openbsd.org/firmware/snapshots/athn-firmware-1.1p1.tgz
 at /usr/libdata/perl5/OpenBSD/PkgAdd.pm line 717.

As you can see from dmesg, I have other firmare needed hardware
installed, but theirs firmware was loaded at first boot with no
problem then.

What is a way to get the proper firmware installed, please?


OpenBSD 6.0-current (GENERIC.MP) #2432: Sat Sep 10 14:06:57 MDT 2016

dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

[snip]

Update your snapshot.  Packages (including firmware) use a new
signing methodology.

http://marc.info/?l=openbsd-tech=147283361813517=2

fw_update stops with Fatal error: Unsigned package ...

[Mihai Popescu]

I've installed a snapshot somewhile ago, then I needed to update the
firmware for athn device. I get this error:

# fw_update
UNSIGNED PACKAGES: athn-firmware-1.1p1
Fatal error: Unsigned package
http://firmware.openbsd.org/firmware/snapshots/athn-firmware-1.1p1.tgz
 at /usr/libdata/perl5/OpenBSD/PkgAdd.pm line 717.

As you can see from dmesg, I have other firmare needed hardware
installed, but theirs firmware was loaded at first boot with no
problem then.

What is a way to get the proper firmware installed, please?


OpenBSD 6.0-current (GENERIC.MP) #2432: Sat Sep 10 14:06:57 MDT 2016
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8029429760 (7657MB)
avail mem = 7781588992 (7421MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xeebc0 (57 entries)
bios0: vendor LENOVO version "9VKT33AUS" date 09/11/2013
bios0: LENOVO 1990RZ2
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC TCPA MCFG SLIC MCFG HPET SSDT
acpi0: wakeup devices PCE2(S4) PCE3(S4) PCE4(S4) PCE5(S4) PCE6(S4)
PCE7(S4) PCE9(S4) PCEA(S4) PCEB(S4) PCEC(S4) SBAZ(S4) P0PC(S4)
PE20(S4) PE21(S4) PE22(S4) PE23(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Athlon(tm) II X2 B26 Processor, 3193.48 MHz
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,NODEID,ITSC
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 16 4MB entries fully associative
cpu0: DTLB 48 4KB entries fully associative, 48 4MB entries fully associative
cpu0: AMD erratum 721 detected and fixed
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 199MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD Athlon(tm) II X2 B26 Processor, 3192.02 MHz
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,NODEID,ITSC
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 16 4MB entries fully associative
cpu1: DTLB 48 4KB entries fully associative, 48 4MB entries fully associative
cpu1: AMD erratum 721 detected and fixed
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 3 pa 0xfec0, version 21, 24 pins
acpimcfg0 at acpi0 addr 0xe000, bus 0-255
acpimcfg1 at acpi0 addr 0xe000, bus 0-255
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus -1 (PCE2)
acpiprt3 at acpi0: bus -1 (PCE3)
acpiprt4 at acpi0: bus -1 (PCE4)
acpiprt5 at acpi0: bus -1 (PCE5)
acpiprt6 at acpi0: bus -1 (PCE6)
acpiprt7 at acpi0: bus -1 (PCE7)
acpiprt8 at acpi0: bus -1 (PCE9)
acpiprt9 at acpi0: bus -1 (PCEA)
acpiprt10 at acpi0: bus 2 (P0PC)
acpiprt11 at acpi0: bus 3 (PE20)
acpiprt12 at acpi0: bus -1 (PE21)
acpiprt13 at acpi0: bus -1 (PE22)
acpiprt14 at acpi0: bus 4 (PE23)
acpicpu0 at acpi0: C1(@1 halt!), PSS
acpicpu1 at acpi0: C1(@1 halt!), PSS
"PNP0501" at acpi0 not configured
tpm0 at acpi0: TPM_ addr 0xfed4/0x5000: device 0x104a rev 0x4e
acpibtn0 at acpi0: PWRB
cpu0: 3193 MHz: speeds: 3200 2500 1900 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD RS880 Host" rev 0x00
ppb0 at pci0 dev 1 function 0 unknown vendor 0x17aa product 0x9602 rev 0x00
pci1 at ppb0 bus 1
radeondrm0 at pci1 dev 5 function 0 "ATI Radeon HD 4250" rev 0x00
drm0 at radeondrm0
radeondrm0: apic 3 int 18
ahci0 at pci0 dev 17 function 0 "ATI SBx00 SATA" rev 0x00: apic 3 int
19, AHCI 1.2
ahci0: port 0: 3.0Gb/s
ahci0: port 1: 1.5Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0:  SCSI3
0/direct fixed naa.50014ee1018094dc
sd0: 305245MB, 512 bytes/sector, 625142448 sectors
cd0 at scsibus1 targ 1 lun 0:  ATAPI
5/cdrom removable
ohci0 at pci0 dev 18 function 0 "ATI SB700 USB" rev 0x00: apic 3 int
18, version 1.0, legacy support
ehci0 at pci0 dev 18 function 2 "ATI SB700 USB2" rev 0x00: apic 3 int 17
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "ATI EHCI root hub" rev
2.00/1.00 addr 1
ohci1 at pci0 dev 19 function 0 "ATI SB700 USB" rev 0x00: apic 3 int
18, version 1.0, legacy support
ehci1 at pci0 dev 19 function 2 "ATI SB700 USB2" rev 0x00: apic 3 int 17
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "ATI EHCI root hub" rev
2.00/1.00 addr 1
piixpm0 at pci0 dev 20 

Re: iked(8) OpenBSD road warrior setup anybody?

[Michael Hekeler]

> Does anybody use iked(8) for remote access (aka Road Warrior setup)
> from OpenBSD clients?

Yes. I do.


> There's a lot of info on setting it up for
> Windows/Android/iOS clients, but I didn't find anything about
> OpenBSD clients setup.

The Client Setup is the same for all platforms  (AFAIK)
You can build the GUI Client just for create the configfile if you like.

After the creation you can start the client without GUI


> I have such setup but with recent changes to iked my VPN connection is
> somewhat unstable.

For me it works stable.

Re: signify: write to stdout: Broken pipe

[Ted Unangst]

lvdd wrote:
> Hi,
> 
> with some help from Alex Greif offlist helping me reproducing the
> issue, I decided to reinstall the system using a different mirror and
> different approaches.

pkg_add was switched to a new file format, and there are some bugs
that result in bad error messages when working with the old format.
this should eventually clear up.

iked(8) OpenBSD road warrior setup anybody?

[Pavel Korovin]

Dead all,

Does anybody use iked(8) for remote access (aka Road Warrior setup)
from OpenBSD clients? There's a lot of info on setting it up for
Windows/Android/iOS clients, but I didn't find anything about
OpenBSD clients setup.

I have such setup but with recent changes to iked my VPN connection is
somewhat unstable.

-- 
With best regards,
Pavel Korovin

Re: Looking for DMVPN implementation

[Renato Westphal]

2016-10-01 19:44 GMT-03:00 Jens Sauer :
> Hi OpenBSD community,
>
> i'm looking for an OpenSource implementation of DMVPN (Dynamic Multipoint 
> Virtual private network).
>
> Currently i just found the draft (from 2013) :
> https://tools.ietf.org/html/draft-detienne-dmvpn-00
>
> Comming from Cisco and would be pleased to see it under OpenBSD.
> http://www.cisco.com/c/dam/en/us/products/collateral/security/dynamic-multipoint-vpn-dmvpn/DMVPN_Overview.pdf
>
> Hope i could get an advice in how to implement (use) it under OpenDSD.

Hi Jens,

I already started working on this in g2k16 and I should have something
to show in a few months. In the hackathon, claudio@ gave me some
pointers on how to add multipoint support in gre(4) and right now I'm
evaluating how to design nhrpd(8) in the best way possible (including
the integration with iked(8) - only IKEv2 will be supported).

I'll let you know when I have something ready.

Cheers,
-- 
Renato Westphal

Re: Cron logs in /var/cron/log instead of /var/log/cron?

[Kenneth Gober]

On Mon, Oct 3, 2016 at 12:27 AM,   wrote:
> Is there any harm or issue with setting the log location
> of cron logs to /var/log/cron instead, or is it best to leave it
> in /var/cron/log?

I've moved cron logs to /var/log/cron on some of my own systems, and
while cron does work just fine, there are a bunch of changes you need
to make:

0. wait until no cron jobs are running or starting soon, that keeps this
procedure simple.

1. move the cron log(s):
# cd /var/cron
# mv log /var/log/cron
# mv log.0.gz /var/log/cron.0.gz
(continue with log.1.gz, etc. as desired)

2. edit /etc/syslog.conf, changing /var/cron/log to /var/log/cron

3. edit /etc/newsyslog.conf, changing /var/cron/log to /var/log/cron

4. edit /etc/mtree/special, moving the 'log' entry from the /var/cron section
to the /var/log section (and renaming from log to cron)

Without #4 you will get spurious warnings from security(8) when it
can't find cron logs where it expects them.

-ken

Re: Large datasize - how to limit physical memory?

[Otto Moerbeek]

On Mon, Oct 03, 2016 at 02:56:05PM +0200, Raimo Niskanen wrote:

> On Fri, Sep 30, 2016 at 01:02:10PM +0200, Otto Moerbeek wrote:
> > On Fri, Sep 30, 2016 at 09:10:21AM +0200, Raimo Niskanen wrote:
> > 
> > > On Wed, Sep 28, 2016 at 09:19:51AM +0200, Raimo Niskanen wrote:
> > > > Dear misc@
> > > > 
> > > > I have searched the archives and read the documentation of 
> > > > login.conf(5),
> > > > ksh(1):ulimit and can not find how to limit the amount of physical 
> > > > memory a
> > > > process may use.
> > > > 
> > > > I have the following limits where I have set down ulimit -m and ulimit 
> > > > -l
> > > > to 1 kbytes in an attempt to limit the process I spawn which is
> > > > the Erlang VM.
> > > > 
> > > > $ ulimit -a
> > > > time(cpu-seconds)unlimited
> > > > file(blocks) unlimited
> > > > coredump(blocks) unlimited
> > > > data(kbytes) 33554432
> > > > stack(kbytes)8192
> > > > lockedmem(kbytes)1
> > > > memory(kbytes)   1
> > > > nofiles(descriptors) 1024
> > > > processes1024
> > > > 
> > > > Note that the machine has got 8 GB of physical memory and 8 GB of swap 
> > > > and
> > > > that I have set datasize=infinity in /etc/login.conf. I got
> > > > datasize=33554432 which seems to be the same as kern.shminfo.shmmax.
> > > > The datasize is twice the physical memory + swap.
> > > > 
> > > > Then I start the Erlang VM and tell it to allocate an address block of 
> > > > 3
> > > > MByte for future use where it will store all literal data in the same 
> > > > block
> > > > (this is a garbage collector optimization).  Not much of this data is
> > > > actually used.
> > > > 
> > > >  68196 beam CALL  
> > > > mmap(0,0x75300,0,0x1002,-1,0)
> > > >  68196 beam RET   mmap 11871265173504/0xacbfe8b3000
> > > > 
> > > > Note the protection flags on the block.  No access is allowed.  This 
> > > > trick
> > > > works just fine; here is what top says:
> > > > 
> > > > load averages:  0.15,  0.13,  0.09 frerin.otp.ericsson.se 
> > > > 08:49:46
> > > > 48 processes: 47 idle, 1 on processor up 
> > > > 13:49
> > > > CPU0 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  
> > > > 100% idle
> > > > CPU1 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  
> > > > 100% idle
> > > > Memory: Real: 43M/636M act/tot Free: 7028M Cache: 508M Swap: 0K/8155M
> > > > 
> > > >   PID USERNAME PRI NICE  SIZE   RES STATE WAIT  TIMECPU 
> > > > COMMAND
> > > > 68196 raimo  20   29G   15M sleep poll  0:00  1.42% beam
> > > > 
> > > > So I have a process with a data size of 29 GB on a machine with 16 GB
> > > > memory + swap.  I have also tried to start an additional Erlang VM that
> > > > also allocates 29 GB of virtual memory which also works.
> > > > 
> > > > That this is allowed is just fine for me - this trick of allocating a
> > > > "large enough" PROT_NONE memory to get one address range for some 
> > > > special
> > > > data type is very useful for the Erlang VM.  But I wonder how to limit 
> > > > the
> > > > actual memory use?  Setting down ulimit -m and ulimit -l to 1 kbytes
> > > > did not prevent this process from getting 15 MByte of "RES" memory...
> > > > 
> > > > Is there some way to limit the actual amount of memory for a process 
> > > > when I
> > > > need to set up the datasize to allow for large unused virtual memory
> > > > blocks?
> > > 
> > > I have found clues in getrlimit,setrlimit(2):
> > > 
> > >  RLIMIT_DATA The maximum size (in bytes) of the data segment for a
> > >  process; this includes memory allocated via malloc(3)
> > >  and all other anonymous memory mapped via mmap(2).
> > > :
> > >  RLIMIT_RSS  The maximum size (in bytes) to which a process's
> > >  resident set size may grow.  This imposes a limit
> > >  on the amount of physical memory to be given to a
> > >  process; if memory is tight, the system will prefer
> > >  to take memory from processes that are exceeding
> > >  their declared resident set size.
> > > 
> > > Now I try to figure out the implications of this...  If I set the data 
> > > size
> > > so the sum of the data sizes for all processes in the system is larger 
> > > than
> > > physical memory + swap, then any process may allocate the last block of
> > > memory in the system so a more important process later will fail to
> > > allocate?
> > 
> > yes.
> > 
> > > 
> > > And the memoryuse limit is rather toothless since there is no immediate
> > > check of this limit.  When the system gets low on memory; is all that
> > > happens that processes that exceed their memoryuse limit probably will get
> > > blocks swapped out?
> > 
> > RLIMIT_DATA *is* enforced, but it could be that PROT_NONE memory is
> > not counted. I don;t know atm.
> 

Re: Fix paxtest output on OpenBSD 6.0?

[Peter Janos]

 It went out twice, sorry. First I sent the below mail, but after even
hours it didn't showed up, I thought maybe length restriction, so I sent
the mail again without the below "RAW" part, with that it was displayed
in a few minutes. Whatever, the paxtest compares are here in a picture
too (mirror urls), more readable to the human eye:
https://s22.postimg.org/f169vbabl/paxtest_openbsd.pnghttps://i.imgsafe.org/22cb7604d4.pnghttps://lut.im/C3F0KIhF6O/GPjZ5bRQrTK8fLpg.png
Is W^X causing the "Vulnerable" lines? Is it still ok, because of "bad
test"? or is it really a security problem?? install60.iso
Executable anonymous mapping (mprotect) : Vulnerable
Executable bss (mprotect) : Vulnerable
Executable data (mprotect) : Vulnerable
Executable heap (mprotect) : Vulnerable
Executable shared library bss (mprotect) : Vulnerable
Executable shared library data (mprotect) : Vulnerable
Return to function (strcpy) : paxtest : return address contains a NULL
byte.
Return to function (strcpy, PIE) : paxtest : return address contains a
NULL byte.
Return to function (memcpy) : Vulnerable
Return to function (memcpy, PIE) : Vulnerable Increasing
kern.stackgap_random=262144 to kern.stackgap_random=16777216 increases
the: Stack randomization test (SEGMEXEC) : 14 quality bits (guessed)
Stack randomization test (PAGEEXEC) : 14 quality bits (guessed)
Arg/env randomization test (SEGMEXEC) : 14 quality bits (guessed)
Arg/env randomization test (PAGEEXEC) : 14 quality bits (guessed) "to 20
quality bits". Thanks! Sent: Sunday, October 02, 2016 at 12:12 PM
From: "Peter Janos" 
To: misc@openbsd.org
Subject: Fix paxtest output on OpenBSD 6.0?Fix paxtest output on OpenBSD
6.0?

Hallo :)

Also I included a few other OS.

Mirror for the post is here:
https://pastebin.com/raw/y9qHwZxi

Tests are after a default/fresh install (not livecd), using
https://www.grsecurity.net/~spender/paxtest-0.9.15.tar.gz


All OS were installed/tested in VirtualBox-5.1.6_110634_el7-1.x86_64 on a
RHEL 7.2 / T450.



When I used 'paxtest-0.9.15' on OpenBSD, had to ADD two lines:

$ grep -n 'randarg1: randbody.o randarg1.o' Makefile.OpenBSD
157:randarg1: randbody.o randarg1.o
$ grep -n 'randarg2: randbody.o randarg2.o' Makefile.OpenBSD
159:randarg2: randbody.o randarg2.o
$

or else compile would fail, thx for the hint from Pinter Oliver!



On FreeBSD/HBSD I had to use paxtest-0.9.14-freebsd.tar compiled on FBSD9
from
https://github.com/HardenedBSD/tools/blob/master/tests/paxtest-freebsd/paxtest-0.9.14-freebsd.tgz



If anyone has outputs for NetBSD and DragonFlyBSD, please post.


Always used blackhat mode.

##
SUM (copy it to a simple editor, ex.: gedit, then from there to
LibreOffice Calc):

###
CentOS-7-x86_64-Everything-1511.txt Executable anonymous mapping Killed
debian-8.6.0-amd64-CD-1.txt Executable anonymous mapping Killed
Fedora-Server-dvd-x86_64-24-1.2.txt Executable anonymous mapping Killed
Fedora-Workstation-netinst-x86_64-24-1.2.txt Executable anonymous mapping
Killed
FreeBSD-10.3-RELEASE-amd64-dvd1.txt Executable anonymous mapping Killed
FreeBSD-11.0-RC3-amd64-dvd1.txt Executable anonymous mapping Killed
FreeBSD-9.3-RELEASE-amd64-dvd1.txt Executable anonymous mapping Killed
HardenedBSD-11-STABLE-v46.5-amd64-disc1.txt Executable anonymous mapping
Killed
install60.txt Executable anonymous mapping Killed
linuxmint-18-cinnamon-64bit.txt Executable anonymous mapping Killed
openSUSE-Leap-42.1-DVD-x86_64.txt Executable anonymous mapping Killed
SLE-12-SP1-Server-DVD-x86_64-GM-DVD1.txt Executable anonymous mapping
Killed
ubuntu-16.04.1-desktop-amd64.txt Executable anonymous mapping Killed
ubuntu-16.04.1-server-amd64.txt Executable anonymous mapping Killed
###
CentOS-7-x86_64-Everything-1511.txt Executable bss Killed
debian-8.6.0-amd64-CD-1.txt Executable bss Killed
Fedora-Server-dvd-x86_64-24-1.2.txt Executable bss Killed
Fedora-Workstation-netinst-x86_64-24-1.2.txt Executable bss Killed
FreeBSD-10.3-RELEASE-amd64-dvd1.txt Executable bss Killed
FreeBSD-11.0-RC3-amd64-dvd1.txt Executable bss Killed
FreeBSD-9.3-RELEASE-amd64-dvd1.txt Executable bss Killed
HardenedBSD-11-STABLE-v46.5-amd64-disc1.txt Executable bss Killed
install60.txt Executable bss Killed
linuxmint-18-cinnamon-64bit.txt Executable bss Killed
openSUSE-Leap-42.1-DVD-x86_64.txt Executable bss Killed
SLE-12-SP1-Server-DVD-x86_64-GM-DVD1.txt Executable bss Killed
ubuntu-16.04.1-desktop-amd64.txt Executable bss Killed
ubuntu-16.04.1-server-amd64.txt Executable bss Killed
###
CentOS-7-x86_64-Everything-1511.txt Executable data Killed
debian-8.6.0-amd64-CD-1.txt Executable data Killed
Fedora-Server-dvd-x86_64-24-1.2.txt Executable data Killed
Fedora-Workstation-netinst-x86_64-24-1.2.txt Executable data Killed

Re: Large datasize - how to limit physical memory?

[Raimo Niskanen]

On Fri, Sep 30, 2016 at 01:02:10PM +0200, Otto Moerbeek wrote:
> On Fri, Sep 30, 2016 at 09:10:21AM +0200, Raimo Niskanen wrote:
> 
> > On Wed, Sep 28, 2016 at 09:19:51AM +0200, Raimo Niskanen wrote:
> > > Dear misc@
> > > 
> > > I have searched the archives and read the documentation of login.conf(5),
> > > ksh(1):ulimit and can not find how to limit the amount of physical memory 
> > > a
> > > process may use.
> > > 
> > > I have the following limits where I have set down ulimit -m and ulimit -l
> > > to 1 kbytes in an attempt to limit the process I spawn which is
> > > the Erlang VM.
> > > 
> > > $ ulimit -a
> > > time(cpu-seconds)unlimited
> > > file(blocks) unlimited
> > > coredump(blocks) unlimited
> > > data(kbytes) 33554432
> > > stack(kbytes)8192
> > > lockedmem(kbytes)1
> > > memory(kbytes)   1
> > > nofiles(descriptors) 1024
> > > processes1024
> > > 
> > > Note that the machine has got 8 GB of physical memory and 8 GB of swap and
> > > that I have set datasize=infinity in /etc/login.conf. I got
> > > datasize=33554432 which seems to be the same as kern.shminfo.shmmax.
> > > The datasize is twice the physical memory + swap.
> > > 
> > > Then I start the Erlang VM and tell it to allocate an address block of 
> > > 3
> > > MByte for future use where it will store all literal data in the same 
> > > block
> > > (this is a garbage collector optimization).  Not much of this data is
> > > actually used.
> > > 
> > >  68196 beam CALL  
> > > mmap(0,0x75300,0,0x1002,-1,0)
> > >  68196 beam RET   mmap 11871265173504/0xacbfe8b3000
> > > 
> > > Note the protection flags on the block.  No access is allowed.  This trick
> > > works just fine; here is what top says:
> > > 
> > > load averages:  0.15,  0.13,  0.09 frerin.otp.ericsson.se 08:49:46
> > > 48 processes: 47 idle, 1 on processor up 13:49
> > > CPU0 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% 
> > > idle
> > > CPU1 states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% 
> > > idle
> > > Memory: Real: 43M/636M act/tot Free: 7028M Cache: 508M Swap: 0K/8155M
> > > 
> > >   PID USERNAME PRI NICE  SIZE   RES STATE WAIT  TIMECPU 
> > > COMMAND
> > > 68196 raimo  20   29G   15M sleep poll  0:00  1.42% beam
> > > 
> > > So I have a process with a data size of 29 GB on a machine with 16 GB
> > > memory + swap.  I have also tried to start an additional Erlang VM that
> > > also allocates 29 GB of virtual memory which also works.
> > > 
> > > That this is allowed is just fine for me - this trick of allocating a
> > > "large enough" PROT_NONE memory to get one address range for some special
> > > data type is very useful for the Erlang VM.  But I wonder how to limit the
> > > actual memory use?  Setting down ulimit -m and ulimit -l to 1 kbytes
> > > did not prevent this process from getting 15 MByte of "RES" memory...
> > > 
> > > Is there some way to limit the actual amount of memory for a process when 
> > > I
> > > need to set up the datasize to allow for large unused virtual memory
> > > blocks?
> > 
> > I have found clues in getrlimit,setrlimit(2):
> > 
> >  RLIMIT_DATA The maximum size (in bytes) of the data segment for a
> >  process; this includes memory allocated via malloc(3)
> >  and all other anonymous memory mapped via mmap(2).
> > :
> >  RLIMIT_RSS  The maximum size (in bytes) to which a process's
> >  resident set size may grow.  This imposes a limit
> >  on the amount of physical memory to be given to a
> >  process; if memory is tight, the system will prefer
> >  to take memory from processes that are exceeding
> >  their declared resident set size.
> > 
> > Now I try to figure out the implications of this...  If I set the data size
> > so the sum of the data sizes for all processes in the system is larger than
> > physical memory + swap, then any process may allocate the last block of
> > memory in the system so a more important process later will fail to
> > allocate?
> 
> yes.
> 
> > 
> > And the memoryuse limit is rather toothless since there is no immediate
> > check of this limit.  When the system gets low on memory; is all that
> > happens that processes that exceed their memoryuse limit probably will get
> > blocks swapped out?
> 
> RLIMIT_DATA *is* enforced, but it could be that PROT_NONE memory is
> not counted. I don;t know atm.

That PROT_NONE is not counted sounds just as we want it to be...

That RLIMIT_DATA *is* enforced does not rhyme with what I saw, or I do not
know what I saw...  As you can se above I had set ulimit -m 1 (kbytes)
and yet top reports RES 15M.  Is that not over the limit?  The PROT_NONE
memory is reported in the 29GB entry by top.  I can easily 

Re: Large datasize - how to limit physical memory?

[Raimo Niskanen]

On Fri, Sep 30, 2016 at 01:10:45PM +0200, Otto Moerbeek wrote:
> On Fri, Sep 30, 2016 at 01:02:10PM +0200, Otto Moerbeek wrote:
> 
> 
> > > > Note that the machine has got 8 GB of physical memory and 8 GB of swap 
> > > > and
> > > > that I have set datasize=infinity in /etc/login.conf. I got
> > > > datasize=33554432 which seems to be the same as kern.shminfo.shmmax.
> 
> The number you are looking for is MAXDSIZ, whichs is 32G on amd64,

Ok.  A different entity with the same value.  Thank you!


> 
>   -Otto

-- 

/ Raimo Niskanen, Erlang/OTP, Ericsson AB

error: [drm:pid10679:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... render ring idle

[Mark Kettenis]

Please send bug reports, using the sendbug(1), to b...@openbsd.org.

Anyway, with:

> inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 5500" rev 0x09
> drm0 at inteldrm0
> inteldrm0: msi
> inteldrm0: 1920x1080

and

> [22.013] (II) LoadModule: "intel"
> [22.013] (II) Loading /usr/X11R6/lib/modules/drivers/intel_drv.so
> [22.016] (II) Module intel: vendor="X.Org Foundation"
> [22.016] compiled for 1.18.4, module version = 2.99.916
> [22.016] Module class: X.Org Video Driver
> [22.016] ABI class: X.Org Video Driver, version 20.0
> [22.017] (II) intel: Driver for Intel(R) Integrated Graphics Chipsets:
>  i810, i810-dc100, i810e, i815, i830M, 845G, 854, 852GM/855GM, 865G,
>  915G, E7221 (i915), 915GM, 945G, 945GM, 945GME, Pineview GM,
>  Pineview G, 965G, G35, 965Q, 946GZ, 965GM, 965GME/GLE, G33, Q35, Q33,
>  GM45, 4 Series, G45/G43, Q45/Q43, G41, B43
> [22.017] (II) intel: Driver for Intel(R) HD Graphics: 2000-6000
> [22.017] (II) intel: Driver for Intel(R) Iris(TM) Graphics: 5100, 6100
> [22.017] (II) intel: Driver for Intel(R) Iris(TM) Pro Graphics: 5200, 
> 6200, P6300
> [22.019] (II) intel(0): Using Kernel Mode Setting driver: i915, version 
> 1.6.0 20080730

This is expected.  You should use the "modesetting" driver instead of
the "intel" driver.  And if you don't have an /etc/X11/xorg.conf file,
that would be the default.

If you really need an /etc/X11/xorg.conf file, change the driver
there.  Otherwise, just delete the file.

Cheers,

Mark