Re: netbsd-10 linux emulation issue

[Thomas Klausner]

On Tue, Mar 05, 2024 at 04:12:15PM +0100, Hauke Fath (SPG) wrote:
> Hi,
> 
> I have been running the IBM Spectrum Protect (AKA Tivoli Storage Manager,
> AKA2 ADSM) backup client* in linux emulation on NetBSD for almost twenty
> years. It has generally been a very well-behaved piece of software.
> 
> On netbsd-10 now, the client will establish a session, transfer a few files
> (or at least it says so), then hang until killed.
> 
> Besides gettimeofday calls, the ktrace has a lot of
> 
>  17467  17467 dsmc RET   futex -1 errno -110 Operation timed out
> 
> Is there anything substantial that has changed in the linux emul that could
> have caused the breakage?

I also see problems with futex in Linux emulation, see
http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=56828

Perhaps futex support is new compared to what you were previously using?
 Thomas

Re: erlang -> asmjit -> mremap questions/bugs

[Thomas Klausner]

On Wed, Mar 01, 2023 at 05:48:08PM +0300, Valeriy E. Ushakov wrote:
> On Wed, Mar 01, 2023 at 15:29:27 +0100, Thomas Klausner wrote:
> 
> > It seems the problem is that mmap() in the mremap(2) man page example
> > (which was used to implement the asmjit version) is not using
> > MAP_SHARED.
> > 
> > - I'd like to add MAP_SHARED in the mmap() call in the mremap(2) man
> >   page example. Is that fine?
> 
> Not really.  There are no other processes involved in the example, so
> MAP_SHARED makes no sense.
> 
> 
> > - Reading mmap(2) it seems that one of MAP_SHARED or MAP_PRIVATE is
> >   required, but there is no error if none is provided. Should we change
> >   mmap() to return an error in that case?
> 
> sys/kern/vfs_vnops.c:
> 
> /*
>  * Old programs may not select a specific sharing type, so
>  * default to an appropriate one.
>  */

So this looks like it would default to MAP_PRIVATE.

> > - Why does MAP_PRIVATE (instead of MAP_SHARED) not work?
> 
> Are there multiple processes involved in the erlang case?

I don't think so - it's a VM running a program. Multiple threads
perhaps. I'll ask.
 Thomas

erlang -> asmjit -> mremap questions/bugs

[Thomas Klausner]

Hi!

Erlang comes with a JIT: asmjit.

The current version doesn't work with PaX MPROTECT. I filed a bug
report and asmjit added code using mremap().

https://github.com/asmjit/asmjit/blob/master/src/asmjit/core/virtmem.cpp#L619

The asmjit self-tests work, but in erlang with the new version there
are segfaults.

https://github.com/erlang/otp/issues/6909#issuecomment-1446606936

An Erlang developer helped debugging and found weird behaviour:

https://github.com/asmjit/asmjit/issues/399#issuecomment-1448640096

It seems the problem is that mmap() in the mremap(2) man page example
(which was used to implement the asmjit version) is not using
MAP_SHARED.

- I'd like to add MAP_SHARED in the mmap() call in the mremap(2) man
  page example. Is that fine?

- Reading mmap(2) it seems that one of MAP_SHARED or MAP_PRIVATE is
  required, but there is no error if none is provided. Should we change
  mmap() to return an error in that case?

- Why does MAP_PRIVATE (instead of MAP_SHARED) not work?

- Does anyone have an idea why the old asmjit code (shm_open() instead
  of mremap()) might still be working on NetBSD 9 but not on NetBSD
  10.99.2? Both have PaX MPROTECT enabled by default, and the mremap()
  code.

Thanks,
 Thomas

Re: Module autounload proposal: opt-in, not opt-out

[Thomas Klausner]

On Mon, Aug 08, 2022 at 07:18:05AM -0700, Paul Goyette wrote:
> It really seems to me that the current module sub-systems is at
> best a second-class capability.  I often get the feeling that
> others don't really care about modules, until it's the only way
> to provide something else (dtrace).

To reply to this point - I'm very interested in modules, but it
doesn't meet a basic requirement for me - easily going back to a
previous kernel when the new one is broken for some reason - if they
share the same kernel version, they share the same
/stand/amd64/9.99.99/modules directory, and if the problem is there,
I'm stuck.

Also, I'm not clear on what the new workflow is for my old one:

Update kernel:

A.
   build.sh kernel=NAME
   ln -f /netbsd /netbsd.old
   install /netbsd /netbsd
   reboot

B.
   build.sh what?
   ln -f /netbsd /netbsd.old
   ? something for old modules ?
   install .../netbsd /netbsd
   ? something for new modules, are they even built? ?
   reboot

If the kernel is broken somehow

A. drop to boot prompt
   boot /netbsd.old

B. drop to bootprompt
   ? What do I do to tell the old kernel where its old modules are ?
   boot /netbsd.old

I think /netbsd/ as a directory including a kernel and modules would
be a solution. I think christos? proposed it at some point and but it
never happened.
 Thomas

ETOOMANYZLIBS

[Thomas Klausner]

riastradh pointed out that this probably needs to be applied to

src/sys/net/zlib.c

as well, but that code seems to be from an older zlib version and the
patch doesn't apply cleanly.

Can it be changed to use common/dist/zlib instead?

If not, someone(TM) please merge the change.
 Thomas


- Forwarded message from Thomas Klausner  -

Module Name:src
Committed By:   wiz
Date:   Thu Mar 24 10:13:01 UTC 2022

Modified Files:
src/common/dist/zlib: deflate.c deflate.h trees.c

Log Message:
zlib: Fix a bug that can crash deflate on some input when using Z_FIXED.

https://github.com/madler/zlib/commit/5c44459c3b28a9bd3283aaceab7c615f8020c531

This bug was reported by Danilo Ramos of Eideticom, Inc. It has
lain in wait 13 years before being found! The bug was introduced
in zlib 1.2.2.2, with the addition of the Z_FIXED option. That
option forces the use of fixed Huffman codes. For rare inputs with
a large number of distant matches, the pending buffer into which
the compressed data is written can overwrite the distance symbol
table which it overlays. That results in corrupted output due to
invalid distances, and can result in out-of-bound accesses,
crashing the application.

The fix here combines the distance buffer and literal/length
buffers into a single symbol buffer. Now three bytes of pending
buffer space are opened up for each literal or length/distance
pair consumed, instead of the previous two bytes. This assures
that the pending buffer cannot overwrite the symbol table, since
the maximum fixed code compressed length/distance is 31 bits, and
since there are four bytes of pending space for every three bytes
of symbol space.


To generate a diff of this commit:
cvs rdiff -u -r1.4 -r1.5 src/common/dist/zlib/deflate.c \
src/common/dist/zlib/trees.c
cvs rdiff -u -r1.3 -r1.4 src/common/dist/zlib/deflate.h

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

- End forwarded message -

Re: python vs. semaphores

[Thomas Klausner]

On Tue, Dec 14, 2021 at 02:08:39PM +0100, Thomas Klausner wrote:
> Is anyone aware of problems with semaphores on NetBSD, or has looked
> at this particular problem before?

Ok, my first test already gave me a bug.

The man page says that multiple consecutive sem_open calls (without
sem_close in between) should return the same address. They don't.

I've written an atf test. I don't know how to mark it as
currently-broken, so I'll attach it here (and to the bug report).

I don't think fixing this one will fix semaphores in Python though :)
 Thomas
? .gdbinit
? Atffile
? atf-run.log
? t_sched
? t_sem
Index: t_sem.c
===
RCS file: /cvsroot/src/tests/lib/librt/t_sem.c,v
retrieving revision 1.5
diff -u -r1.5 t_sem.c
--- t_sem.c 14 May 2020 08:34:19 -  1.5
+++ t_sem.c 14 Dec 2021 15:38:47 -
@@ -313,6 +313,31 @@
(void)sem_unlink("/sem_c");
 }
 
+ATF_TC_WITH_CLEANUP(sem_open_address);
+ATF_TC_HEAD(sem_open_address, tc)
+{
+   atf_tc_set_md_var(tc, "descr", "Validate that multiple sem_open calls "
+   "return the same address");
+}
+ATF_TC_BODY(sem_open_address, tc)
+{
+   sem_t *sem, *sem2, *sem3;
+   sem = sem_open("/sem_d", O_CREAT | O_EXCL, 0777, 0);
+   ATF_REQUIRE(sem != SEM_FAILED);
+   sem2 = sem_open("/sem_d", O_CREAT | O_EXCL, 0777, 0);
+   ATF_REQUIRE(sem2 == SEM_FAILED && errno == EEXIST);
+   sem3 = sem_open("/sem_d", 0);
+   ATF_REQUIRE(sem3 != SEM_FAILED);
+   ATF_REQUIRE(sem == sem3);
+   ATF_REQUIRE_EQ(sem_close(sem3), 0);
+   ATF_REQUIRE_EQ(sem_close(sem), 0);
+   ATF_REQUIRE_EQ(sem_unlink("/sem_d"), 0);
+}
+ATF_TC_CLEANUP(sem_open_address, tc)
+{
+   (void)sem_unlink("/sem_d");
+}
+
 ATF_TP_ADD_TCS(tp)
 {
 
@@ -320,6 +345,7 @@
ATF_TP_ADD_TC(tp, child);
ATF_TP_ADD_TC(tp, pshared);
ATF_TP_ADD_TC(tp, invalid_ops);
+   ATF_TP_ADD_TC(tp, sem_open_address);
 
return atf_no_error();
 }

python vs. semaphores

[Thomas Klausner]

Hi!

Since at least python 2.7 (2011, probably longer, I haven't dug
deeper) pkgsrc comes with a patch for Python that disables POSIX
semaphore use (pthread_mutex* and pthread_cond* are used instead).

I don't have a small test case yet, but when you build python without
the pkgsrc patches and run

./python ../Tools/scripts/run_tests.py test_compileall 
test_multiprocessing_fork test_concurrent_futures

these tests will not finish and I can see lots of pythons in state 'psem'

 2791 wiz   77096M   11M psem/31 0:00  0.00%  0.00% python

Is anyone aware of problems with semaphores on NetBSD, or has looked
at this particular problem before?

 Thomas

Re: General device properties API

[Thomas Klausner]

Hi Jason!

Some minor comments:


Should we use ssize_t instead of int for the size of objects (or -1 if
not existing)? I think it's the more appropriate type; on the other
hand, we don't seem to be using this type much in the kernel yet at
all.


Does device_getprop_string always NUL-terminate the buffer, even if
the provided one is too small?


I read device_getpropencoding as device_getpr_open_coding, so perhaps
we can add a couple more "_" as word separators. I.e.

device_getprop_len
device_getprop_encoding
device_getprop_type

(and similarly for the devhandle_* versions of those); that makes them
(in my eyes) more consistent with device_getprop_string,
device_getprop_bool, etc.

Cheers,
 Thomas

Re: regarding the changes to kernel entropy gathering

[Thomas Klausner]

On Sun, Apr 04, 2021 at 11:14:31AM -0700, John Nemeth wrote:
>  I understand the need for good random sources, and won't argue
> it.  My question is, how can we tell what random sources a system
> actually has, i.e. is there some flag that cpuctl identify shows
> when a system has RDRAND/RDSEED?  Are there other sources that can
> be positively identified as providing randomness?

That should do it:

# cpuctl identify 0 | grep -e RDRAND -e RDSEED 
cpu0: features1 0x7ed8320b
cpu0: features5 0x209c01a9

 Thomas

MAXTSIZ removal?

[Thomas Klausner]

There was a commit by christos that made MAXTSIZ optional, but
at least the amd64 vmparam.h still defines it.

Any reason not to remove it?

(I still can't start emulators/mame with a GENERIC without that change)
 Thomas
 

Re: bug help needed

[Thomas Klausner]

Hi!

I'm seeing at least the first issue as well, see the thread "recurring
tstile hangs on -current" on current-users.

The second one has similar lines in the backtrace, but I didn't see
the concrete symptoms yet.
 Thomas

On Mon, Jun 03, 2019 at 09:26:18AM +0200, Frank Kardel wrote:
> Hi,
> 
> while moving to NetBSD-9 I'd like to point out some bugs that
> may be relevant for NetBSD-8/-9. These bugs relate to
> lockups/stalls and unexpected performance. If someone with
> deeper locking- or uvm-wisdom could peek at these we might get
> more stability,
> 
> Frank
> 
> Lockups -current:
> kern/54207 [serious/high]:
> 
> -current locks up solidly when pkgsrc building
> adapta-gtk-theme-3.95.0.11
> 
> 
> Responsible:kern-bug-people
> State:  open
> Class:  sw-bug
> Originator: Frank Kardel
> Release:NetBSD 8.0_STABLE
> Arrival-Date:   Thu May 16 14:40:00 + 2019
> 
> 
>kern/54210 [serious/high]:
>
>NetBSD-8 processes presumably not exiting
>
> 
> 
> Responsible:kern-bug-people
> State:  open
> Class:  sw-bug
> Originator: Frank Kardel
> Release:NetBSD 8.99.39
> Arrival-Date:   Wed May 15 10:00:00 + 2019
> 
> 
> mm driver/uvm interaction (-current):
> 
>kern/54223 [serious/high]:
>
>kernel diagnostic assertion "umap->refcount != 0" failed: file
>"/src/NetBSD/cur/src/sys/uvm/uvm_bio.c", line 330 (/dev/kmem access)
>
> 
> 
> Responsible:kern-bug-people
> State:  open
> Class:  sw-bug
> Originator: Frank Kardel
> Release:NetBSD 8.99.41
> Arrival-Date:   Wed May 22 20:15:00 + 2019
> 
> 
> Performance anomaly:
> 
>kern/54209 [serious/medium]:
>
>NetBSD 8 large memory performance extremely low
>
> 
> Responsible:kern-bug-people
> State:  open
> Class:  sw-bug
> Originator: Frank Kardel
> Release:NetBSD 8.0_STABLE
> Arrival-Date:   Thu May 16 14:40:00 + 2019
> 

firefox sandboxing

[Thomas Klausner]

Hi!

Here's an interesting article about firefox sandboxing improvements
on Linux:

https://www.morbo.org/2018/05/linux-sandboxing-improvements-in_10.html

We already support chroot(2). Are user namespaces
(http://man7.org/linux/man-pages/man7/user_namespaces.7.html - looks
like capabilities) something that would be good to have for NetBSD?

Cheers,
 Thomas

Re: amd64: kernel aslr support

[Thomas Klausner]

 Thomas
# $NetBSD: Makefile,v 1.1 2016/11/16 00:49:27 pgoyette Exp $

.include "../Makefile.inc"

.PATH:  ${S}/kern

KMOD=   aslr_test
SRCS=   aslr_test.c

.include 
/*  $NetBSD: bufq_priocscan.c,v 1.21 2017/05/04 11:03:27 kamil Exp $    
*/

/*-
 * Copyright (c) 2018 Thomas Klausner
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 * 1. Redistributions of source code must retain the above copyright
 *notice, this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright
 *notice, this list of conditions and the following disclaimer in the
 *documentation and/or other materials provided with the distribution.
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
 * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 * SUCH DAMAGE.
 */

#include 
__KERNEL_RCSID(0, "$NetBSD$");

#include 
#include 
#include 
#include 
#include 

#define BTSEG_NONE  0
#define BTSEG_TEXT  1
#define BTSEG_RODATA2
#define BTSEG_DATA  3

static int
aslr_test_init(void)
{
extern struct bootspace bootspace;
size_t i;

const char *names[4] = {
[BTSEG_NONE] = "none",
[BTSEG_TEXT] = "text",
[BTSEG_RODATA] = "rodata",
[BTSEG_DATA] = "data"
};

for (i = 0; i < BTSPACE_NSEGS; i++) {
if (bootspace.segs[i].type == BTSEG_NONE) {
continue;
}
printf("Segment %zu (%s): va=%p size=%zu\n", i,
   names[bootspace.segs[i].type],
   (void *)bootspace.segs[i].va,
   bootspace.segs[i].sz);
}
return 0;
}

static int
aslr_test_fini(void)
{
return 0;
}

MODULE(MODULE_CLASS_MISC, aslr_test, NULL);

static int
aslr_test_modcmd(modcmd_t cmd, void *opaque)
{

switch (cmd) {
case MODULE_CMD_INIT:
return aslr_test_init();
case MODULE_CMD_FINI:
return aslr_test_fini();
default:
return ENOTTY;
}
}

Re: amd64: kernel aslr support

[Thomas Klausner]

Hi!

On Wed, Nov 15, 2017 at 07:40:55PM +0100, Maxime Villard wrote:
> Le 14/11/2017 à 15:43, Maxime Villard a écrit :
> > The size and number of these blocks is controlled by the split-by-file
> > parameter in Makefile.amd64. Right now it is set to 2MB, which produces a
> > kernel with ~23 allocatable (ie useful at runtime) sections, which is a 
> > third
> > of the total number supported (BTSPACE_NSEGS = 64). I will probably reduce
> > this parameter a bit in the future, to 1.5MB, or even 1MB.
> 
> Actually I just did it. So now it's 1MB (better security), physically shifted
> by the prekern (better entropy), and mapped with large pages (better
> performance). And along the way it mostly mitigates TLB cache attacks.
> 
> This is still wip but feel free to test, as always,

I've tried out the instructions at
http://m00nbsd.net/542a5cfd448aaf7db7adcadce74123d2.html and they
worked fine for me. Thank you!

I have a couple questions:

How can I check (after booting) if the kernel is using ASLR properly?

Why does GENERIC_KASLR disable KDTRACE_HOOKS? Is this necessary, or
are KDTRACE_HOOKS lowering the security somehow?

Thanks,
 Thomas

mount_apfs?

[Thomas Klausner]

In case someone is looking for a challenge, here's a description of
the APFS (Apple File System) format:

https://blog.cugu.eu/post/apfs/

Cheers,
 Thomas

Re: how to tell if a process is 64-bit

[Thomas Klausner]

On Sat, Sep 09, 2017 at 05:58:53AM +1000, Matthew Green wrote:
> > In a cross-platform process utility tool the question came up how to
> > decide if a process is 64-bit.
> > 
> > https://github.com/giampaolo/psutil/issues/1102
> > 
> > What's the best answer for NetBSD?
> 
> in C:
> 
> internally, just check #ifdef _LP64.
> externally, kvm_getprocs() with KERN_PROC_PID.
> 
> in other languages?  don't ask me.

I'm asking for a python frontend to the C one.

I came up with this fragment:

#include 
#include 
#include 

int main(int argc, char *argv[]) {
int pid, res;
struct kinfo_proc2 *kp;
kvm_t *kvmp = kvm_open(NULL, NULL, NULL, KVM_NO_FILES, NULL);
if (kvmp == NULL)
exit(1);
pid = atoi(argv[1]);
kp = kvm_getproc2(kvmp, KERN_PROC_PID, pid, sizeof(*kp), );
if (res != 1)
exit(1);
kvm_close(kvmp);
}


Which part of the structure can I check to find out 32-bitness?

Thanks,
 Thomas

Re: how to tell if a process is 64-bit

[Thomas Klausner]

On Fri, Sep 08, 2017 at 07:38:24AM -0400, Mouse wrote:
> First, I have to ask: what does it mean to say that a particular
> process is - or isn't - 64-bit?

Many 64-bit ports support running 32-bit applications
(compat_netbsd32, compat_linux32).
 Thomas

how to tell if a process is 64-bit

[Thomas Klausner]

Hi!

In a cross-platform process utility tool the question came up how to
decide if a process is 64-bit.

https://github.com/giampaolo/psutil/issues/1102

What's the best answer for NetBSD?
 Thomas

Re: Restricting rdtsc [was: kernel aslr]

[Thomas Klausner]

On Tue, Mar 28, 2017 at 10:36:52PM +0200, Maxime Villard wrote:
> I already thought about this a few months ago, and my conclusion back then
> was that it is very difficult to achieve if we want both good performance
> and good security. This is a little off-topic, but the idea would consist in
> having two identical kernel text segments mapped at different addresses. Only
> one kernel is active at a time. Every once in a while we randomize the other
> kernel, wait for interrupts to happen in the currently running lwps, and
> migrate these lwps to the new kernel, dropping refcounts along the way. When
> it reaches zero, everybody uses the new kernel, and we unmap the previous
> one. And we keep jumping between kernels this way regularly. I also had other
> magic tricks for .data and .rodata, but that's another debate.

This would be a step in the direction of allowing updating running
kernels, wouldn't it?
 Thomas

Fixed modular kernel path and different kernels

[Thomas Klausner]

Hi!

One issue that made me avoid modular kernels is that updating is
finickier.

When you switch from one modular kernel to another, you have to
replace /netbsd and /stamd/amd64/${VERSION}/modules.

So for example, when testing two different kernels of the same
-current major version, you can not just boot /netbsd.old, but you
have to replace /stamd/amd64/${VERSION}/modules with the old modules
before rebooting. That can be a problem if a module is completely
broken due to a bug. So you need to always have a reliable kernel of a
different NetBSD version available.

Other systems handle this differently.

One example is having a directory that contains the kernel _and its
modules_. Then you can easily switch between two different kernels
even of the same NetBSD version by just choosing the other directory.

Perhaps there are other, even better solutions. My point is, we should
switch away from our current method.

Am I overlooking something?
 Thomas

Re: RFC: softint-based if_input

[Thomas Klausner]

On Thu, Feb 04, 2016 at 05:53:45AM +, Taylor R Campbell wrote:
>Date: Thu, 4 Feb 2016 14:35:46 +0900
>From: Ryota Ozaki 
> 
>On Thu, Feb 4, 2016 at 11:35 AM, Thor Lancelot Simon  
> wrote:
>> On Wed, Feb 03, 2016 at 01:44:58PM +0900, Ryota Ozaki wrote:
>>>
>>> It's probably simple but time-consuming because there are more than
>>> one and a half hundreds drivers. I just want to avoid spending time
>>> for this task and go back to the main task (L3 MP-ification)...
>>
>> If you can get spatch to build, it may be smart enough to do this for 
> you.
> 
>What do you mean? spatch?
> 
> http://coccinelle.lip6.fr/

Also known as pkgsrc/devel/coccinelle.
 Thomas

Re: sys/clock.h finalization

[Thomas Klausner]

On Mon, Dec 22, 2014 at 01:44:02AM +0100, Kamil Rytarowski wrote:
 Please let finalize src/sys/sys/clock.h:
 
 1. Please review and add man page to src/share/man/man9
 Thomas K. is responsible for man pages according to src/doc/RESPONSIBLE

Committed with slight changes, thanks!
 Thomas

Re: Does options P1003_1B_SEMAPHORE still exist?

[Thomas Klausner]

On Tue, May 06, 2014 at 07:34:44PM +0200, Matthias Drochner wrote:
 As has been mentioned before, the (kernel) semaphore implementation
 in -current doesn't work well. In pkgsrc Python, use of semaphores
 was disabled because threaded Python scripts didn't work. I also found
 that disabling semaphore use in gcc/libgomp fixes crashes of programs
 using OMP (eg darktable).

I didn't know about this. Does this look similar to

http://www.imagemagick.org/discourse-server/viewtopic.php?f=2t=25475

?

 Thomas

Re: Patch: cprng_fast performance - please review.

[Thomas Klausner]

On Fri, Apr 18, 2014 at 01:39:18PM -0400, Thor Lancelot Simon wrote:
 How do you count to 9K?  I see:
 
   2K for p
   2K for q
   1280 bytes for w

Are you talking about this w?
+   uint32_t w[1280], *p = state-p, *q = state-q;

This looks like 1280x4 bytes to me.
 Thomas

Re: Vnode API change: VOP_LOCK

[Thomas Klausner]

On Fri, Feb 21, 2014 at 12:35:40PM +0100, J. Hannken-Illjes wrote:
 The current implementation of vn_lock() is racy.  Modification of
 the vnode oprations vector for active vnodes is unsafe because it
 is not known whether deadfs or the original file system will be
 called.
 
 - Pass down LK_RETRY to the lock operation (hint for deadfs only).
 
 - Split deadfs lock operations from genfs_XXXlock and change
   deadfs lock operation to return ENOENT if LK_RETRY is unset.
 
 - Change all other lock operations to check for dead vnode once
   the vnode is locked and unlock and return ENOENT in this case.
 
 - Add flag LK_INTERLOCK (requiring LK_NOWAIT) to make it possible
   for vrelel() to try a lock with v_interlock held.
 
 With these changes in place vnode lock operations will never succeed
 after vclean() has marked the vnode as VI_XLOCK and before vclean()
 has changed the operations vector.
 
 Diff available at http://www.netbsd.org/~hannken/vnode-pass3-1.diff
 
 Comments or objections anyone?

I wanted to understand what you're describing here, but I don't know
what deadfs is, and apropos deadfs only pointed to hier(7). Could
someone please describe it, at best in a manpage? (I'm happy to help
with formatting.)
 Thomas

dtrace support level?

[Thomas Klausner]

Hi!

Just out of curiosity, what percentage, approximately, of the stuff on
http://www.brendangregg.com/dtrace.html
would work on NetBSD-current right now?
 Thomas

Re: ~5 percent kernel performance loss in the last 3 weeks

[Thomas Klausner]

On Wed, Dec 04, 2013 at 02:09:49PM +0100, Christoph Badura wrote:
 I was able to pin this down on the pcu/fpu changes.  A kernel from just
 before that change doesn't have the performance loss.

Those have been reverted.
 Thomas

Re: pbulk using tmpfs

[Thomas Klausner]

On Sat, Apr 20, 2013 at 11:47:59PM +0200, iMil wrote:
 I'm running bulk builds for various archs using pbulk for quite
 some time now, and one of the domUs used for this is hosted by
 a server with a LSI MPT Fusion Raid controller. In short, that
 controller sucks, poor write performances, it is a well known
 fact. In order to speed up builds, I would like to know if I
 could benefit from tmpfs by mounting the WRKOBJDIR as a tmpfs
 directory.
 
 In particular, while most packages will fit into a couple of
 megabytes, some of them (firefox, libreoffice, openoffice,
 koffice and such) take a couple of gigabytes to get built.
 
 My question is: how will tmpfs behave? If I mount the WRKOBJDIR
 without any argument, it will use, if I understand correctly
 the manpage, all of the available memory. Does this mean I will
 end up in a situation were the system memory will be filled by
 the build or is there some kind of smart mechanism within tmpfs
 which will overflow somewhere?

I have this setup (mostly created with mksandbox):
tmpfs on /home/wiz/sandbox type tmpfs (local)
/bin on /home/wiz/sandbox/bin type null (read-only, local)
/sbin on /home/wiz/sandbox/sbin type null (read-only, local)
/lib on /home/wiz/sandbox/lib type null (read-only, local)
/libexec on /home/wiz/sandbox/libexec type null (read-only, local)
/usr/X11R7 on /home/wiz/sandbox/usr/X11R7 type null (read-only, local)
/usr/bin on /home/wiz/sandbox/usr/bin type null (read-only, local)
/usr/games on /home/wiz/sandbox/usr/games type null (read-only, local)
/usr/include on /home/wiz/sandbox/usr/include type null (read-only, local)
/usr/lib on /home/wiz/sandbox/usr/lib type null (read-only, local)
/usr/libdata on /home/wiz/sandbox/usr/libdata type null (read-only, local)
/usr/libexec on /home/wiz/sandbox/usr/libexec type null (read-only, local)
/usr/share on /home/wiz/sandbox/usr/share type null (read-only, local)
/usr/sbin on /home/wiz/sandbox/usr/sbin type null (read-only, local)
/var/mail on /home/wiz/sandbox/var/mail type null (read-only, local)
/archive/foreign/src on /home/wiz/sandbox/usr/src type null (read-only, local)
/archive/foreign/pkgsrc on /home/wiz/sandbox/usr/pkgsrc type null (local)
/archive/foreign/xsrc on /home/wiz/sandbox/usr/xsrc type null (read-only, local)
/archive/packages/6.99.19 on /home/wiz/sandbox/packages type null (local)
/archive/distfiles on /home/wiz/sandbox/distfiles type null (local)

and have been building all my packages this way for years now, using
pbulk. Sometimes I trigger some tmpfs panics (see e.g. kern/42484,
kern/47480, kern/47739) but otherwise it works very well, is fast and
usually stable. The machine I used this on regularly had 12GB RAM
though (now even more), so I'm not RAM-limited.
 Thomas

Re: pinning down dk? assignment

[Thomas Klausner]

On Mon, Jul 23, 2012 at 04:21:55PM +0200, Martin Husemann wrote:
 You can mount them by name. I have (on my non-GPT root disk):
 
 NAME=sb2k5Root/a/   ffs rw,log   1 1
 NAME=sb2k5Root/bnoneswapsw,dp0 0
 
 in /etc/fstab. With gpt you can label the partitions with arbitrary names
 (instead of the disklabel volume name/partition that I use above).

Thanks for the example. I've added it to fstab(5), together with an example for 
gpt-disks.
 Thomas

Re: Quota on tmpfs

[Thomas Klausner]

On Thu, Jul 12, 2012 at 09:33:42PM -0400, Matthew Mondor wrote:
 If I remember there is some optional support for symbolic links to
 resolve to user-specific targets, but I forgot the details.  With
 that /tmp/ could potentially be a symbolic link pointing to
 say, /tmpfs/user/ I think.

I think you mean the MAGIC SYMLINKS section in man 7 symlink.
 Thomas

Re: Rump FS throughput

[Thomas Klausner]

On Thu, May 31, 2012 at 01:45:53PM -0400, Matthew Mondor wrote:
 Although it's useful to mount random media more safely than it would be
 using kernel-space, I noticed that using 64KB reads, the kernel cd9660
 will gladly read ~20MB/s from a DVD, but that rump_cd9660 using
 64KB reads is limited to aproximately 4MB/s at most, even if the system
 is mostly idle during those transfers (on netbsd-6/amd64 and 4 3.3GHz
 cores).

Some suggestions from Antti via email proxy:
Maybe he is using the block device (/dev/cd0a) instead of the raw device
(/dev/rcd0a).  IIRC the former has some pretty serious performance
problems for userspace I/O.  Also in the maybe department, libp2k
should probably detect and autoadjust a block device to raw device.
Or, someone could just fix the bdev stuff.


Cheers,
 Thomas

Re: watching dynamic device attachment?

[Thomas Klausner]

On Fri, May 04, 2012 at 09:52:36AM -0500, Eric Haszlakiewicz wrote:
 hmm... I noticed that both devpubd(8) and drvctl(8) refer to a drvctl(4), but
 I can't find any such man page.  Can someone point me at where it might be?

AFAIK noone wrote it yet :(
 Thomas

Re: O-A loan

[Thomas Klausner]

On Mon, Feb 20, 2012 at 03:52:39AM +, YAMAMOTO Takashi wrote:
  y...@mwd.biglobe.ne.jp (YAMAMOTO Takashi) wrote:
  i'd like to to either fix or remove O-A page loaning.
  
  the biggest problem is the lack of users.  thus no way to test.
  any ideas?
  
  After looking at O-A loaning code in yamt-pagecache branch, looking at
  the benchmark results by Jaime (some data was sent privately, I think)
  and contemplating about this mechanism - I tend to think that O-A loaning
  should be removed.  It is a significant code complexity in UVM with some
  special casing and benchmarks indicate that the benefit is somewhat small.
  
  Does not seem to be worth, does it?
 
 with this particular use alone, i tend to agree.
 
 fwiw, the following is probably one of the best cases for loaned read.
 
 YAMAMOTO Takashi
 
 without loaned read:
 
   kuma% dd if=/netbsd bs=1m|wc -c
   14+1 records in
   14+1 records out
   14768691 bytes transferred in 0.038 secs (388649763 bytes/sec)
14768691
 
 with loaned read:
 
   kuma% dd if=/netbsd bs=1m|wc -c
   14+1 records in
   14+1 records out
   14768691 bytes transferred in 0.023 secs (642117000 bytes/sec)
14768691

Can you please tell in a few words what O-A loaning is, and why it speeds up 
this use case?

Thanks,
 Thomas

Re: Respawn crashed PUFFS filesystems?

[Thomas Klausner]

On Sun, Feb 12, 2012 at 07:12:25AM +0100, Emmanuel Dreyfus wrote:
 I thought that we could respawn a crashed userland filesystem, lookup
 all active vnodes again, and redo all operations failed at crash time.
 That way a crashed filesystem would  just cause a delay in ongoing
 operations, but it would not even cause a failure. Does it makes sense?

Please also consider the problem that it might be a persistent problem, i.e. 
crash immediately or quite soon again.
 Thomas

USB contigmalloc()

[Thomas Klausner]

Matthew Dillon implemented a solution for USB/graphics cards problem
stemming from the fact that some time after the boot, they can't
acquire enough contiguous memory.

http://thread.gmane.org/gmane.os.dragonfly-bsd.kernel/14431

Anyone interested in porting it or implementing something similar? :)
 Thomas

Re: USB contigmalloc()

[Thomas Klausner]

On Fri, Dec 02, 2011 at 01:51:42AM +0900, Izumi Tsutsui wrote:
 Anyway, currently our udl(4) doesn't require such allocation,

It's also a problem for radeondrm (no USB involved).
 Thomas

Re: bug #44412

[Thomas Klausner]

On Fri, Jul 22, 2011 at 02:55:21PM -0400, David Riley wrote:
 I filed this report a while back.  Someone else has tested my fix on non-PPC 
 systems (x86, x86_64) and reported that it seems to work as well.  I'm 
 attaching the patch against -current here; could someone give it a look and 
 include it if it seems to be OK?  Or provide feedback if it doesn't.

Thanks for the bug report and patch.
I'm not using netatalk and can't judge the patch; until someone comes
along that can, please add the patch to the bug report as well, so
it's not lost.

Thank you!
 Thomas

Re: Getting rid of standard boiler plate in kernel man pages CODE REFERENCES

[Thomas Klausner]

Hi Iain!

 I'm not sure what you are proposing.. can you show an example diff (eg
 using bluetooth.9) ?

Sure, attached.
 Thomas
Index: bluetooth.9
===
RCS file: /cvsroot/src/share/man/man9/bluetooth.9,v
retrieving revision 1.4
diff -u -r1.4 bluetooth.9
--- bluetooth.9 2 Dec 2007 20:03:11 -   1.4
+++ bluetooth.9 8 Nov 2010 09:33:42 -
@@ -348,13 +348,6 @@
 .Ar ref .
 .El
 .Sh CODE REFERENCES
-This section describes places in the
-.Nx
-source tree where actual code implementing or using the
-Bluetooth Protocol Stack can be found.
-All pathnames are relative to
-.Pa /usr/src .
-.Pp
 The Bluetooth Protocol Stack is contained in the
 .Pa sys/netbt
 directory.
Index: intro.9
===
RCS file: /cvsroot/src/share/man/man9/intro.9,v
retrieving revision 1.13
diff -u -r1.13 intro.9
--- intro.9 15 Apr 2010 08:40:46 -  1.13
+++ intro.9 8 Nov 2010 09:33:42 -
@@ -45,6 +45,8 @@
 .It CODE REFERENCES
 Contains the pathname(s) of the source file(s) which contain the definition
 and/or source code of the variables or functions being documented.
+Any paths are relative to
+.Pa /usr/src .
 .El
 .Sh MEMORY MANAGEMENT
 Machine-dependent swap interface.

Getting rid of standard boiler plate in kernel man pages CODE REFERENCES

[Thomas Klausner]

Hi!

Many many many kernel (section 9) man pages contain the following
sentence in their CODE REFERENCES section:

This section describes places within the
.Nx 
source tree where actual code implementing
.Nm 
can be found.
All pathnames are relative to
.Pa /usr/src .
.Pp 

I suggest we remove it and adapt the DESCRIPTION section in intro(9)
to mention that the pathnames are relative to /usr/src.

Any disagreement?
 Thomas

Re: something really screwed up with mmap+ffs on 5.0_STABLE

[Thomas Klausner]

On Tue, Aug 17, 2010 at 11:52:31PM +0300, Antti Kantee wrote:
 It would be great if someone could confirm or debunk this on -current
 and for archs beyond i386.  Just get the latest sources, go to
 sys/rump/net/lib/libshmif, comment out line 61 (the one with PREFAULT_RW)
 from if_shmif.c, make  make install, and run tests/net/icmp/t_ping
 floodping in a loop.  You should see a coredump within a few thousand
 iteratios (few minutes) if the problem is there.

I think you mean if_shmem.c.

I just tested this on 5.99.39/amd64.
# ./t_ping floodping
got 0/1
passed
# while true; do ./t_ping floodping; done
panic: kernel diagnostic assertion busmem-shm_magic == SHMIF_MAGIC failed: 
file if_shmem.c, line 287
panic: kernel diagnostic assertion busmem-shm_magic == SHMIF_MAGIC failed: 
file if_shmem.c, line 287
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
panic: bus is not magical
...

Cheers,
 Thomas

Re: CVS commit: src/tests/net/icmp

[Thomas Klausner]

[redirected from source-changes-d to a hopefully more suitable mailing
list]

On Mon, Jul 05, 2010 at 12:26:17AM +0300, Antti Kantee wrote:
 I'm happy to give a more detailed explanation on how it works, but I need
 one or two questions to determine the place where I should start from.
 I'm planning a short article on the unique advantages of rump in kernel
 testing (four advantages by my counts so far), and some questions now
 might even help me write that one about what people want to read instead
 of what I guess they'd want to read.

I looked at the tests some more (tmpfs race, and the interface one
from above). I think I can read them, but am unclear on some of the
basic properties of a rump kernel.

For example:
1. Where is '/'? Does it have any relation to the host systems '/'? Is
it completely virtual in the memory of the rump kernel?
2. Do I understand correctly that for e.g. copying a file from the
host file system into a rump kernel file system, I would use read and
rump_sys_write?
3. Similarly for network interfaces -- open a socket with socket(2) or
rump_socket(or so) and copy bytes with read/rump_sys_write?
4. Could you NFS export the rump kernel file system to the host?
(Probably better to a second rump kernel...)
5. I think I read that rump (or some other part) can now talk USB. How
would one attach a USB device on the host system to a USB controller
inside the rump kernel?

Is that a good start for questions? :)
 Thomas