Re: Understanding memory statistics

[Anindya Mukherjee]

Hi,

Thank you very much for taking the time to answer these. I think I now
have a better understanding and also got rid of a couple of
misconceptions. I'm switching gears slightly to work on porting a
couple of my favourite packages; once done I'll resume my research.

Regards,
Anindya

From: Otto Moerbeek 
Sent: January 23, 2021 12:17 AM
To: Anindya Mukherjee 
Cc: misc@openbsd.org 
Subject: Re: Understanding memory statistics 
 
On Sat, Jan 23, 2021 at 04:40:06AM +, Anindya Mukherjee wrote:

> Thanks for the explanation! I noticed during my earlier investigation
> that the source of the data size is struct vmspace::vm_dused. This is
> updated mostly in uvm_mapanon and uvm_map. The second function seems to
> be a more general case. I think during my file mapping the second
> function is called and the part where vm_dused is updated is skipped.
> I'm setting up a VM on my machine with a debug kernel (this should be an
> interesting exercise) to do some exploratory kernel debugging, just to
> understand the process. So far this seems to make sense. I never doubted
> the system is doing the "right" thing :)
> 
> I also had some questions about the page and the buffer caches. So far I
> gathered the following facts (please correct me if I am wrong):
> 
> OpenBSD has separate page and buffer caches, i.e., no UBC.
> Q: Is this done for security reasons or due to some complication? Just
> curious.

page cache is a misnomer, it does not exist.

UBC is the concept that file i/o via mmapped files and the read/write
systemc calls use the same mechanism for caching the files in memory.
One of the consequences is that if a file is both mmapped and being
written to via write(2), that change is also visible in the mmapped
view immediately. Vice-versa the same.

We do not have that unification, a long time ago an attempt was done
(you can still find that in cvs) but that was reverted since it caused
all kinds of probems the author wasn't able to solve in reasonable
time. So My guess it would be we do not have it because of
complexity/nobody did and finished the work.

So the machanism we have is a buffer cache (which in the end uses
pages) used by the read and write system calls. Plus we have the
mmapping mechanism used for both file mapping and anonymous mappings
(mappings not corresponding to any file, like program data).

> 
> The Inactive pages seem to back the page cache. I ran my file mapping
> code a few times mapping/releasing a large file (about 300 MB) with
> systat running in the uvm view, and saw the page counts for Active and
> Inactive swing back and forth, keeping the total fixed.
> 
> Then I ran md5 on another 100 MB file, and this time the Cache number in
> top grew by about 100 MB, with some brief disk activity (I'm on SSD so
> things are zippy). I next ran my file mapping program on it. This time
> the Active pages grew by about 100 MB, raising the total by the same
> amount. When the program ended, those pages moved to Inactive, keeping
> the total fixed. There was no disk activity during this and Cache
> remained unchanged.
> 
> This seems to indicate that the data for the new file was copied from
> the buffer cache to the page cache during the mapping, and both copies
> were maintained.

Mmapped file activity would indeed show in Active or Inactive pages
(which one depends on how much access is done to the pages) while I/O
via read or write shows up in Cache indeed. But I would not know from
the top of my head if pages in the Buffer Cache from a file are
explicitly copied to new pages when the same files is mmapped or just
cause some changes on the page admin level so that the mapping refers
to pages that are already in mem via the Buffer Cache. Thinking about
it it would involve some form of unification, so likely some mem to
mem copying is going on. It shows my experience is mostly userland
memory management

    -Otto

> 
> Regards,
> Anindya
> 
> From: Otto Moerbeek 
> Sent: January 22, 2021 12:01 AM
> To: Anindya Mukherjee 
> Cc: misc@openbsd.org 
> Subject: Re: Understanding memory statistics 
>  
> On Thu, Jan 21, 2021 at 10:38:59PM +, Anindya Mukherjee wrote:
> 
> > Hi,
> > 
> > Just to follow up, I was playing with allocating memory from a test
> > program in various ways in order to produce a situation when SIZE is
> > less than RES. The following program causes this to happen. If I mmap a
> > large file, the SIZE remains tiny, indicating that the mapped region is
> > not counted as part of text + data + stack. Then when I go ahead and
> > touch all the memory, SIZE remains tiny but RES grows to the size of the
> > file. Very interesting.
> 
> So SIZE does not include mappings backed by a file system object, but
> RES does. RES only grows once the pages are touched, this is demand
> paging in action (anon pages act the same way).
> 
> Nice. I already suspected would be something like that, but never took
> the time to find out by experimenting or code study.
> 
> Now the 

Re: relay email from users to per-user smtp servers

[Rudolf Sykora]

Dear Chris,


thank you for your ideas.

Chris Bennett  writes:
> It sounds to me that need to configure your email clients to do the
> switcheroo about from addresses, etc.
> I use neomutt, which might not be suitable since it's a text only.
> But I login to my shell. Then I start neomutt and bind keys within my
> neomuttrc, to then login to another IMAP server
> (from a list I have setup) and then reply with the correct From address,
> using either the local smtpd or one on another server.
>
> Using smtpd to send really just means sending the right email
> and credentials, so there is no real problem having many email addresses
> for one user. I use Dovecot for IMAP and base smtpd.

You might be right, however... I hoped that using smtpd would ensure
such things like solution for possible errors during sending, queueing
of mail when offline, etc (and the user mail agent would just pass the
mail itself, the name of the server to relay it to, and where to get
credentials from [or just pass them, too]). But, reading the answers,
it probably cannot be easily arranged like that.

(Btw., I am not against text-only user interfaces. I am writing this in
mu4e, though I am thinking even about mblaze tools [but that has its own
problems], or some plan9 [9front] tools. Unfortunately, a good solution
for email is still something out of my reach.)


Thanks
Ruda

boot_config(8) man page issue; and possibly openbsd.org/report.html

[Andrew Easton]

Hello everyone, 

it appears that there may be a way to misinterpret
the man page for boot_config(8) in the state of
commit 0b249e2164be2385bc6a5e82814435649b2b06e0
Date:   Sun Jan 24 10:21:43 2021 +
on the github openbsd src mirror.

The problem description is under the headline
"Issue".

There is a realistic chance that I am
misunderstanding something. If this is the case, what
further measures can I take to narrow down the problem?

=== Issue ===

The boot_config(8) man page reads:

[quote]
The Ethernet card is not detected at boot because the
kernel configuration does not match the physical
hardware configuration, e.g.  wrong IRQ in
OpenBSD/i386.  [...]
UKC> find ne
[...]
25 ne1 at isa0 port 0x300 size 0 iomem -1 iosiz 0 irq 10 drq -1 drq2 -1 flags 
0x0
[...]
ne1 seems to match the configuration except it uses
IRQ 5 instead of IRQ 10. 
[...]
UKC> change ne1
[...]
irq [10] ? 5
[\quote]

Note, that the line that is longer than 72 characters
in this email is also longer than 72 characters in
the original manual page. I selfishly chose a more
accurate quote of the manual page over breaking the
line. Plese let me know, in case this is a serious
problem. Maybe https://www.openbsd.org/report.html
should be augmented to refer to this specific
quotation problem, in case it is important enough to
document properly.

The sentence "ne1 seems to match the configuration
except it uses IRQ 5 instead of IRQ 10" has two
ways of being interpreted: (1) the kernel
configuration is using IRQ 5 and (2) the hardware
configuration is using IRQ 5.

After having misunderstood the sentence by assuming
meaning (1) and noticing the inconsintency with the
other relevant lines in the above quote of the man
page, I suggest clarifying the sentence like so
(change marked with square brackets):
"ne1 seems to match the configuration except it [is
supposed to use] IRQ 5 instead of IRQ 10" .

One even more explicit option is:
"ne1 [in the kernel configuration] seems to match
[the hardware] configuration except [for the IRQ
being used. The kernel currently uses IRQ 10 and the
hardware uses IRQ 5, so the kernel configuration
needs to be adjusted to match the hardware
configuration.]"

I want to thank the tireless efforts which have gone
and are currently going into maintaining the
exceptionally high documentation standards of
OpenBSD. In case my suggestions above are overly
pedantic, I sincerely apologize.



=== Affected Man Page File ===
/share/man/man8/boot_config.8
in 
commit 0b249e2164be2385bc6a5e82814435649b2b06e0
Date:   Sun Jan 24 10:21:43 2021 +



Thank you very much for your time.


Kind regards,
Andrew Easton

Re: relay email from users to per-user smtp servers

[Edgar Pettijohn]

You may want to look at table(5) specifically the credentials section.

Not sure how ugly it would get with multiple relay rules, but I think
it should be possible.

However, I think it would just be easier to teach your mua to do it
for you.

Edgar

Re: pkg_info(1) manpage issue

[Edgar Pettijohn]

On Sun, Jan 24, 2021 at 09:35:46PM +0100, Andrew Easton wrote:
> Hello everyone, 
> 
> it appears that there may be a way to misinterpret
> the man page for pkg_info(1) in the state of
> commit 0b249e2164be2385bc6a5e82814435649b2b06e0
> Date:   Sun Jan 24 10:21:43 2021 +
> on the github openbsd src mirror.
> 
> The problem description is under the headline
> "Issue".
> 
> There is a realistic chance that I am
> misunderstanding something. If this is the case, what
> further measures can I take to narrow down the problem?
> 
> === Issue ===
> 
> The uninstalled packages are supposed to be listed
> together with their one-line summary. The manual
> page accessible with $ man 1 pkg_info; reads:
> 
> [quote]
> When browsing through uninstalled packages, running
> pkg_info -I *.tgz will report a summary line for each
> package, so that it is possible to run
> pkg_info pkgname.tgz to obtain a longer package
> description, and pkg_add -n pkgname.tgz to check that
> the installation would proceed cleanly, including
> dependencies.
> [\quote]
> 
> Copy-pasting the command "pkg_info -I *.tgz" from the
> manual page into the terminal yields the following
> error:
> 
> computer$ pkg_info -I *.tgz
> Invalid spec: *.tgz
> Invalid spec: *.tgz
> computer$
>

Are there any packages in the directory? Try:

$ cd /usr/ports/packages/`uname -m`/all
$ pkg_info -I *.tgz

Obviously you're results will vary, but I get:

file:./autoconf-2.67p1.tgz: unsigned package
autoconf-2.67p1.tgz automatically configure source code on many Un*x platforms
file:./autoconf-2.69p3.tgz: unsigned package
autoconf-2.69p3.tgz automatically configure source code on many Un*x platforms
file:./automake-1.16.2.tgz: unsigned package
automake-1.16.2.tgz GNU Standards-compliant Makefile generator
file:./bison-3.3.2p1.tgz: unsigned package
bison-3.3.2p1.tgz   GNU parser generator
file:./blas-3.8.0p0.tgz: unsigned package

followed by many more...

Or an absolute path like:

pkg_info -I https://cnd.openbsd.org/pub/OpenBSD/6.8/packages-stable/`uname 
-m`/curl-7.72.0p0.tgz
curl-7.72.0p0.tgz   transfer files with FTP, HTTP, HTTPS, etc.

However, pkg_add -n 
https://cnd.openbsd.org/pub/OpenBSD/6.8/packages-stable/`uname 
-m`/curl-7.72.0p0.tgz
Does give weird output:
quirks-3.440 signed on 2021-01-22T19:53:09Z
Can't find transfer
Can't find files
Can't find with
Can't find FTP,
Can't find HTTP,
Can't find HTTPS,
Can't find etc.

 
Edgar

> 
> Even calling pkg_info(1) with doas(1) yields the same
> behaviour:
> 
> computer$ doas pkg_info -I *.tgz
> Invalid spec: *.tgz
> Invalid spec: *.tgz
> computer$
> 
> 
> Note that there is a relatively long pause between
> the two "Invalid spec" messages both with and without
> doas(1).
> 
> 
> In which way am I misunderstanding the manual page
> for pkg_info(1) for the specific operation of
> browsing uninstalled packages and showing a summary
> line?
> 
> 
> === Affected Man-Page File ===
> /usr.sbin/pkg_add/pkg_info.1
> in 
> commit 0b249e2164be2385bc6a5e82814435649b2b06e0
> Date:   Sun Jan 24 10:21:43 2021 +
> 
> 
> 
> Kind regards,
> Andrew Easton
> 

Re: relay email from users to per-user smtp servers

[Chris Bennett]

On Sun, Jan 24, 2021 at 04:52:13PM +0100, Rudolf Sykora wrote:
> 
> 
> In my case, my computer gathers mail from various mail services using
> mbsync. I want to be able to reply the mail, but have the reply use the
> mail server that is most suited for the reply. E.g., if I get an email
> from school (downloaded from the school service), I need that my answer
> is as if sent using the school service; similarly with my work mail,
> etc. (The people at work want me to use their address when dealing with
> work-related matters; it's then easily recognizable.) I thought the
> program used to create a message does not have to know how to send email
> (speak SMTP) but would just pass it to something that can (smtpd), to
> relay the mail further. But for this I'd need that the server to be used
> for sending be easily modified by user, as well as have some easy means
> to supply the needed credentials.
> 

It sounds to me that need to configure your email clients to do the
switcheroo about from addresses, etc.
I use neomutt, which might not be suitable since it's a text only.
But I login to my shell. Then I start neomutt and bind keys within my
neomuttrc, to then login to another IMAP server
(from a list I have setup) and then reply with the correct From address,
using either the local smtpd or one on another server.

Using smtpd to send really just means sending the right email
and credentials, so there is no real problem having many email addresses
for one user. I use Dovecot for IMAP and base smtpd.

This is assuming that I have understood your problem correctly.

Chris Bennett

pkg_info(1) manpage issue

[Andrew Easton]

Hello everyone, 

it appears that there may be a way to misinterpret
the man page for pkg_info(1) in the state of
commit 0b249e2164be2385bc6a5e82814435649b2b06e0
Date:   Sun Jan 24 10:21:43 2021 +
on the github openbsd src mirror.

The problem description is under the headline
"Issue".

There is a realistic chance that I am
misunderstanding something. If this is the case, what
further measures can I take to narrow down the problem?

=== Issue ===

The uninstalled packages are supposed to be listed
together with their one-line summary. The manual
page accessible with $ man 1 pkg_info; reads:

[quote]
When browsing through uninstalled packages, running
pkg_info -I *.tgz will report a summary line for each
package, so that it is possible to run
pkg_info pkgname.tgz to obtain a longer package
description, and pkg_add -n pkgname.tgz to check that
the installation would proceed cleanly, including
dependencies.
[\quote]

Copy-pasting the command "pkg_info -I *.tgz" from the
manual page into the terminal yields the following
error:

computer$ pkg_info -I *.tgz
Invalid spec: *.tgz
Invalid spec: *.tgz
computer$


Even calling pkg_info(1) with doas(1) yields the same
behaviour:

computer$ doas pkg_info -I *.tgz
Invalid spec: *.tgz
Invalid spec: *.tgz
computer$


Note that there is a relatively long pause between
the two "Invalid spec" messages both with and without
doas(1).


In which way am I misunderstanding the manual page
for pkg_info(1) for the specific operation of
browsing uninstalled packages and showing a summary
line?


=== Affected Man-Page File ===
/usr.sbin/pkg_add/pkg_info.1
in 
commit 0b249e2164be2385bc6a5e82814435649b2b06e0
Date:   Sun Jan 24 10:21:43 2021 +



Kind regards,
Andrew Easton

Re: relay email from users to per-user smtp servers

[Rudolf Sykora]

Martijn van Duren  writes:

> I have no clue on what you're trying to do, so here's my best guess.
> Your users have something like a shell account and want to submit mail
> via the sendmail command. Your users want you to forward the mail to
> their "standard" mailserver before it is relayed further.
>
> Afaik there is no option to configure a specific runtime determined
> relay host option, so you'd have to create a different "relay host"
> action per match rule.

thank you, as far as I can tell you understand me right.

In my case, my computer gathers mail from various mail services using
mbsync. I want to be able to reply the mail, but have the reply use the
mail server that is most suited for the reply. E.g., if I get an email
from school (downloaded from the school service), I need that my answer
is as if sent using the school service; similarly with my work mail,
etc. (The people at work want me to use their address when dealing with
work-related matters; it's then easily recognizable.) I thought the
program used to create a message does not have to know how to send email
(speak SMTP) but would just pass it to something that can (smtpd), to
relay the mail further. But for this I'd need that the server to be used
for sending be easily modified by user, as well as have some easy means
to supply the needed credentials.

Thanks
Ruda

Re: FDE disk setup instructions are misleading when installing from USB

[tetrahedra]

On Sat, Jan 23, 2021 at 04:25:39PM -0800, Bryan Wright wrote:

Perhaps someone will make some changes to the installer or documentation. But, 
I can tell you, a diff,  or at least a proposed specific solution, will always 
go a lot further than pointing out a potential problem, simply because there 
are relatively few developers and they all have things they are busy with.


It's hard for me to propose a solution since I don't know the situation. 
However, the solution I originally had in mind was the solution that 
someone else proposed already:


add language to the FDE section of the FAQ explaining how to verify 
which drive is which, before overwriting the first 1M of that drive with 
zeroes.


The other solution which was proposed (update the installer to support 
FDE, since everyone does it these days) is also great.

Re: Provide a code example

[Otto Moerbeek]

On Sun, Jan 24, 2021 at 09:12:16AM +0100, Ivan wrote:

> Can you send a C code example of obtaining sysctl vm.loadavg via sysctl(2) 
> function?
> -- 
> Regards,
> Ivan
> 

Youn have the source: grepping for VM_LOADAVG turns up 

usr.bin/top/machine.c:  static int sysload_mib[] = {CTL_VM, VM_LOADAVG};

amongst others.

-Otto

Provide a code example

[Ivan]

Can you send a C code example of obtaining sysctl vm.loadavg via sysctl(2) 
function?
-- 
Regards,
Ivan

Re: FDE disk setup instructions are misleading when installing from USB

[Mihai Popescu]

> I am therefore extremely grateful to Cullum Smith and the other "OpenBSD
on a
> laptop" howtos for making it feasible to get this far.

Go get grateful on that blog's address and stop spamming here, please.
Your point is totally wrong. Totally!

Re: relay email from users to per-user smtp servers

[Martijn van Duren]

I have no clue on what you're trying to do, so here's my best guess.
Your users have something like a shell account and want to submit mail
via the sendmail command. Your users want you to forward the mail to
their "standard" mailserver before it is relayed further.

Afaik there is no option to configure a specific runtime determined
relay host option, so you'd have to create a different "relay host"
action per match rule.

Another option is to let your users submit the mail to their mail
provider directly through port 587 or something similar, or why not
just let you handle the mail and let them add your server's IP to
their SPF-record and install filter-dkimsign, which supports multiple
domains for DMARC from-domain matching.

But like I said, no idea what you're trying to achieve, so my answer
might be way off.

martijn@

On Fri, 2021-01-22 at 23:45 +0100, Rudolf Sykora wrote:
> Dear list,
> 
> 
> if I have several users, each of which wants to be able to send email to
> generally different smtp servers with their own credentials, how does
> one arrange such a thing? Can it be done easily without having a
> separate rule in the system-wide /etc/mail/smtp.conf for each individual
> user? Can it be done that they themselves can easily modify the smtp
> servers information/credentials?
> 
> 
> Thanks for comments
> Ruda
>