First bits of a Haiku compatibility layer

[Stephan]

Hello,

Many people here may know that Haiku is an open-source remake of BeOS,
a 90s era desktop OS developed by Be Inc. After ~20 years of
development, it does indeed look like a modernized version of Beos.
What makes it so special is that it is extremely integrated, intuitive
and enjoyable to use. It has a zero-latency, pixel-perfect looking GUI
and follows the design and usability concepts of its commercial
predecessor. That is why I always found it interesting to rebase it on
a mature unixish operating system, especially NetBSD.

While the Be engineers may have incorporated some concepts of Unix
into their OS, it is not so much a Unix or POSIX system. For example,
it has its own error codes, IPC and thread API, data types and generic
lower-level C functions (details later).

Am I the first person having this idea?

No, for example there was a project called cosmoe with the aim of
making the Haiku userland run on top of a POSIX OS. It does so by
emulating BeOS (and thus, Haiku) IPC on SysV IPC, BeOS threads on
pthreads, and other functions on common POSIX or C functions. I tried
this on (I think) NetBSD 6 and got it working to some extent but it
was very unstable.

https://github.com/Ithamar/cosmoe

Secondly, I saw in his personal files that the NetBSD developer kamil@
did some research on this topic. I tried to reach out to him to hear
his findings, but unfortunately none of the E-Mail addresses I found
were working anymore.

http://www.netbsd.org/~kamil/haiku/

Some years ago I already started to work on a compatibility layer for
NetBSD and resumed working on it recently. The original architecture
looks like this

Services / Apps
libbe.so -> C++ Kits
libroot.so -> C runtime, Syscall access, ...
Kernel

I think a compatibility layer would mostly consist of kernel
components and a custom libroot.so.

I have created a libroot that provides functionality missing in libc
and it should behave like the original one. It makes use of libc and
libpthread at the moment as well as syscalls of the kernel components.
The source can be found on Github:

https://github.com/stephanwib/src/tree/trunk/lib/libroot

At the moment I have the Ports IPC mechanism finished as a kernel
driver. It is here:

https://github.com/stephanwib/src/blob/trunk/sys/kern/uipc_port.c

The kernel part follows NetBSD concepts, it uses NetBSD data types
(which were chosen to be very compatible, though) and POSIX errors.
The custom libroot.so translates these as required by higher layers of
the Haiku userland.

More information on these interfaces are described here:

https://www.haiku-os.org/legacy-docs/bebook/TheKernelKit_Overview.html

All of this should be considered a concept or prototype at the moment.
Anyhow, I was able to compile the original Haiku port IPC test
programs and they all ran successfully.

I would like to hear feedback on this topic as well as if other
developers are interested. There is work to do in the kernel, the
library and probably pkgsrc to obtain and build the userland.

Stephan

NetBSD and ChatGPT

[Stephan]

Hello,

I think most people have heard about the ChatGTP AI system and its
ability e.g. to write texts or accomplish programming tasks. I was
wondering if it would be able to implement something in the NetBSD
kernel and I would like to share my experiences, in case someone is
interested.

I asked if it knows semaphores and if it could show a sample API,
which it did. I then asked for an implementation for the NetBSD kernel
(syscalls were meant here for a userspace API). I received pretty much
nothing but an empty skeleton with some basic header files, an empty
"struct sem" and some function definitions without any content. It
said that the implementation is a complex task and I should consult
with an experienced specialist.

I needed to advise over and over again and the code slowly became how
it (approximately) should be. Here is what I said:


-Please add a list that will contain all semaphore structures.
-Perfect. Can you add neccessary variables to struct sem, like the
name, the id and a condvar to handle waits?
-Great. Can you add an appropriate interlock to struct sem? Also,
waits should be interruptible.
-I think for all functions except create_sem_impl, you need to walk
through the list of semaphores to look up the right struct sem.
-Regarding the lookup of struct sem, wouldn´t it be better to use
LIST_FOREACH? Also could you implement the lookup in a separate
function?
-Right, can you change the implementation functions to meet the way
they are natively implemented in NetBSD?
-That is not yet the way syscalls are implemented under NetBSD. Look
in sys/syscallargs.h how it must be done.
-Pretty good. But did you miss the SCARG macro for accessing system
call arguments?

I was impressed that this is possible. ChatGTP was able to understand
this and perform the implementation of these steps using NetBSD kernel
API. At the end I think it got a little confused and sometimes refused
to do things (too complex) or parts of the code were wrong that were
already good previously.

As I said, the last version of the code got messy. The only thing to
note is that it was able to apply the system call interface like this:


int
create_sem_impl(struct lwp *l, const netbsd_create_sem_args *uap,
register_t *retval)
{
  struct sem *sem;
  char *name;
  int error;

  error = kauth_authorize_system(l->l_cred, KAUTH_SYSTEM_SEMAPHORE,
  KAUTH_REQ_SYSTEM_SEMAPHORE_CREATE, NULL, NULL, NULL);
  if (error)
return (error);

  sem = kmem_zalloc(sizeof(*sem), KM_SLEEP);
  name = kmem_alloc(MAXCOMLEN + 1, KM_SLEEP);

  error = copyinstr(SCARG(uap, name), name, MAXCOMLEN + 1, NULL);
...

Below is the version of the whole code which I think is best.

Stephan


#include 
#include 
#include 
#include 
#include 
#include 
#include 

struct sem {
  int id;
  char name[MAXCOMLEN + 1];
  int count;
  kmutex_t lock;
  kcondvar_t cv;
  LIST_ENTRY(sem) sem_list;
};

static LIST_HEAD(, sem) sem_head = LIST_HEAD_INITIALIZER(sem_head);

struct sem *lookup_sem(int id) {
  struct sem *sem;

  LIST_FOREACH(sem, _head, sem_list) {
if (sem->id == id)
  return sem;
  }

  return NULL;
}

int create_sem_impl(const char *name, int count) {
  struct sem *sem;

  /* Allocate memory for the semaphore */
  sem = kmem_alloc(sizeof(*sem), KM_SLEEP);
  if (sem == NULL)
return ENOMEM;

  /* Initialize the semaphore */
  sem->id = next_id++;
  strlcpy(sem->name, name, sizeof(sem->name));
  sem->count = count;
  mutex_init(>lock, MUTEX_DEFAULT, IPL_NONE);
  cv_init(>cv, name);

  /* Add the semaphore to the list of semaphores */
  LIST_INSERT_HEAD(_head, sem, sem_list);
  return sem->id;
}

int destroy_sem_impl(int sem_id) {
  struct sem *sem;

  /* Look up the semaphore in the list of semaphores */
  sem = lookup_sem(sem_id);
  if (sem == NULL)
return ENOENT;

  /* Remove the semaphore from the list of semaphores */
  LIST_REMOVE(sem, sem_list);

  /* Destroy the semaphore */
  mutex_destroy(>lock);
  cv_destroy(>cv);
  kmem_free(sem, sizeof(*sem));
  return 0;
}

int acquire_sem_impl(int sem_id) {
  struct sem *sem;
  int error = 0;

  /* Look up the semaphore in the list of semaphores */
  sem = lookup_sem(sem_id);
  if (sem == NULL)
return ENOENT;

  /* Acquire the lock for the semaphore */
  mutex_enter(>lock);

  /* Wait for the semaphore to become available */
  while (sem->count == 0) {
error = cv_wait_sig(>cv, >lock);
if (error)
  break;
  }

  /* Decrement the count if the semaphore is available */

if (!error) sem->count--;

/* Release the lock for the semaphore */

mutex_exit(>lock);

return error;
}

int release_sem_impl(int sem_id) {

struct sem *sem;

/* Look up the semaphore in the list of semaphores */

sem = lookup_sem(sem_id);

if (sem == NULL)

return ENOENT;

/* Acquire the lock for the semaphore */

mutex_enter(>lock);

/* Increment the count for the semaphore */

sem->count++;

/* Signal the semaphore */

cv_signal(>cv);

/* Release the lock for the semaphore */

mutex_exit(>lock);

return 0;
 }

Re: Adding files to the NetBSD source tree

[Stephan]

Thank you all, these set lists were a blind spot to me.

Adding my file fixed the build process.

Surprisingly I was not able to log in using the live image, because
some messages appeared about some PAM libraries missing or had wrong
permissions.

I suspect something got inconsistent in DESTDIR, I hope I can
republish it without compiling the whole system again.

Am Mi., 9. Nov. 2022 um 10:54 Uhr schrieb Michael van Elst :
>
> stephan...@googlemail.com (Stephan) writes:
>
> >Hallo,
>
> >1) Put the header file into sys/sys in the source tree:
> >2) Made sure it contains an (I think) RCS tag.
>
> RCS tags are maintained by the version control system. For most
> parts these are just documentation and you either leave them out or
> provide an "empty" template like "$NetBSD: $"
>
>
>
> >./build.sh -u -U -m amd64 -a x86_64 -j4 release
> ...
> >cd /home/stephan/src/distrib/sets &&
> ...
> >Files in DESTDIR but missing from flist.
>
>
> The release build uses explict file lists stored in src/distrib/sets/lists/
> for checking the result of the build. You need to keep the file lists in
> sync for everything that ends in the installation sets.
>
> Machine independent system headers that should be installed go into the
> 'comp' set and the file list is therefore src/distrib/sets/lists/comp/mi.
>
>

Adding files to the NetBSD source tree

[Stephan]

Hallo,

I am doing some research / development on NetBSD for which I want to
set up a proper build environment first. I am using VS Code and WSL on
Windows 10 with an Ubuntu 22.04 userland. I obtained the NetBSD
sources from Github and configured tasks to build everything using
build.sh, including release and live-image (the latter I can boot in
qemu).

I can repeatedly pull in changes and refresh the build and everything
works fine. However, adding new files to the source tree seems to be a
little tricky. As a first step I tried to include a C header file
called port.h which is to land in /usr/include/sys as follows:

1) Put the header file into sys/sys in the source tree:

~/src/sys/sys$ ls -l port.h
-rw-r--r-- 1 stephan stephan 4712 Nov  9 09:07 port.h


2) Made sure it contains an (I think) RCS tag. I feel these are
important but I could not find information on how these are handled
exactly.

$ head -1 port.h
/*  $NetBSD: port.h,v 1.00 2022/11/08 16:57:28 stephanwib Exp $*/


3) Added port.h to sys/sys/Makefile under INCS=

param.h pcu.h pipe.h pmf.h poll.h pool.h port.h power.h proc.h \


4) Updated the RCS tag of sys/sys/Makefile (although I don´t know if
this is needed)

~/src/sys/sys$ head -1 Makefile
#   $NetBSD: Makefile,v 1.181 2022/11/08 13:03:10 stephanwib Exp $



Building is done with this command:

./build.sh -u -U -m amd64 -a x86_64 -j4 release


With my changes in the tree it fails with these messages:


  ===
checkflist ===> distrib/sets
--- check_DESTDIR ---
--- checkflist ---
cd /home/stephan/src/distrib/sets &&
DESTDIR=/home/stephan/src/obj/destdir.amd64  MACHINE=amd64
MACHINE_ARCH=x86_64
AWK=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbawk
 
CKSUM=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbcksum
 DB=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbdb
 
EGREP=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbgrep\
-E  HOST_SH=/usr/bin/sh
MAKE=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbmake
 
MKTEMP=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbmktemp
 
MTREE=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbmtree
 PAX=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbpax
 
COMPRESS_PROGRAM=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbxz
 GZIP=-n  XZ_OPT=-9  TAR_SUFF=tar.xz
PKG_CREATE=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbpkg_create
 SED=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbsed
 
TSORT=/home/stephan/src/obj/tooldir.Linux-4.4.0-19041-Microsoft-x86_64/bin/nbtsort\
-q  /usr/bin/sh /home/stephan/src/distrib/sets/checkflist  -L base  -M
/home/stephan/src/obj/destdir.amd64/METALOG.sanitised

===  1 extra files in DESTDIR  =
Files in DESTDIR but missing from flist.
File is obsolete or flist is out of date ?
--
./usr/include/sys/port.h
=  end of 1 extra files  ===



Can somebody say what it missing in this case?


Many thanks,

Stephan

Re: Debugging CUPS binaries

[Stephan]

> http://gnats.netbsd.org/48250

Is that fix included in 7.0.1?

bozohttp/Lua: Max POST data size

[Stephan]

Hi!

I´m trying to send base64 encoded data to a REST service based on
bozohttpd and Lua. The content type is
application/x-www-form-urlencoded and there is only one field called
"DATA". The request begins as below:


POST /print/test HTTP/1.1
User-Agent: curl/7.27.0
Host: myhost
Accept: */*
Content-Length: 981769
Content-Type: application/x-www-form-urlencoded

DATA=JVBERi0xL
-


The Lua script looks like this:
-
local httpd = require("httpd")

function test(env, hdr, qry)
  file = io.open("/tmp/request", "w+")

  httpd.print("ENV:")
  for i, j in pairs(env) do
  httpd.print(i .. "\t\t" .. j)
  end

  httpd.print("HDR:")
  for i, j in pairs(hdr) do
  httpd.print(i .. "\t\t" .. j)
  end

  httpd.print("QRY:")
  for i, j in pairs(qry) do
  httpd.print(i .. "\t\t" .. j)
  file:write(j)
  end

  file:close()
end
httpd.register_handler('test', test)
---

In this case, the data can not be transferred entirely to the server.
Insted, the server sends the reply too early and only 16KB are written
to the file /tmp/request. So it looks like the POST request is
truncated after 16KB of data for some reason.

I hope someone can help me as to why that is.


Thanks and regards,

Stephan

Re: Would Raspberry Pi 3's wifi be supported by NetBSD arm port?

[Stephan]

The first question is if the device itself will work. As with its
predecessor, I´m not able to find a datasheet of the BCM2837.

I wonder what has changed, e.g. with respect to timers and the
interrupt controller. Anyone having a clue?

2016-03-03 6:33 GMT+01:00 Mayuresh :
> The specs say that pi 3 maintains a backward compatibility.
>
> https://www.raspberrypi.org/blog/raspberry-pi-3-on-sale/
>
> So I am hoping that arm port might work out of the box.
>
> Just curious about the new bits - wifi and bluetooth. Above link says the
> chip is BCM43438.
>
> [I haven't yet ordered a pi 3, and I am content with pi 2 for my needs
> right now, though just curious.]
>
> Mayuresh

Re: Last version of of citrix Client core dumps (Was Re: How to run Microsoft Internet Explorer on NetBSD?)

[Stephan]

The assembly looks like junk and considering the adresses, you have
tried to disassemble some memory on the stack. Is this a 64-bit wfica
binary?

Either way, this seems to be a completely different case. How does the
backtrace look like?

2016-02-26 13:01 GMT+01:00 Patrick Welche <pr...@cam.ac.uk>:
> On Fri, Feb 26, 2016 at 09:22:38AM +0100, Stephan wrote:
>> I still recommend Receiver for HTML5 in this case.
>>
>> The dump looks like a mess and eventually gdb is unable to process
>> this dump of a Linux binary on NetBSD correctly. It would be
>> interesting to know what is mapped at 0xba90004d. You could break at
>> that adress (b *0xba90004d) and check with pmap. Also, what´s the
>> corresponding instruction (x/i 0xba90004d)?
>
> Getting to stop while running for the pmap is proving interesting (need
> authentication via pnbrowse before dump in wfica).
>
> The instruction seems to always be the same though:
>
> Core was generated by `wfica'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  0x7f7fee73a9f4 in ?? ()
> [Current thread is 1 (process 27767)]
> (gdb) x/20i 0x7f7fee73a9e8
>0x7f7fee73a9e8:  add%al,(%rax)
>0x7f7fee73a9ea:  add%al,(%rax)
>0x7f7fee73a9ec:  add%al,(%rax)
>0x7f7fee73a9ee:  add%al,(%rax)
>0x7f7fee73a9f0:  push   %rsi
>0x7f7fee73a9f1:  rex.R
>0x7f7fee73a9f2:  rex.RB push %r13
> => 0x7f7fee73a9f4:  rex.RB
>0x7f7fee73a9f5:  rex.WRB
>0x7f7fee73a9f6:  cs rex.R
>0x7f7fee73a9f8:  rex.WR
>0x7f7fee73a9f9:  rex.WR add %r8b,(%rax)
>0x7f7fee73a9fc:  add%dl,0x75(%rcx)
>0x7f7fee73a9ff:  gs jne 0x7f7fee73aa67
>0x7f7fee73aa02:  add%al,(%rax)
>0x7f7fee73aa04:  add%al,(%rax)
>0x7f7fee73aa06:  add%al,(%rax)
>0x7f7fee73aa08:  stc
>0x7f7fee73aa09:  mov%al,%al
>0x7f7fee73aa0b:  idivl  0x7f(%rdi)
>
>
> Cheers,
>
> Patrick

Re: Last version of of citrix Client core dumps (Was Re: How to run Microsoft Internet Explorer on NetBSD?)

[Stephan]

I still recommend Receiver for HTML5 in this case.

The dump looks like a mess and eventually gdb is unable to process
this dump of a Linux binary on NetBSD correctly. It would be
interesting to know what is mapped at 0xba90004d. You could break at
that adress (b *0xba90004d) and check with pmap. Also, what´s the
corresponding instruction (x/i 0xba90004d)?

2016-02-25 15:42 GMT+01:00 Patrick Welche <pr...@cam.ac.uk>:
> On Thu, Feb 25, 2016 at 02:44:32PM +0100, Stephan wrote:
>> Why don´t you just use the Receiver for HTML5? With regard to your
>> crash, do you have a backtrace handy?
>
> Have a look at Jose's from earlier in this thread:
>
> http://mail-index.netbsd.org/netbsd-users/2016/02/03/msg017788.html
>
> P

Re: Last version of of citrix Client core dumps (Was Re: How to run Microsoft Internet Explorer on NetBSD?)

[Stephan]

Why don´t you just use the Receiver for HTML5? With regard to your
crash, do you have a backtrace handy?

2016-02-25 14:03 GMT+01:00 Patrick Welche :
> On Wed, Feb 03, 2016 at 07:31:35PM +0100, Jose Luis Rodriguez Garcia wrote:
>> On Tue, Feb 2, 2016 at 4:17 AM, Eric Haszlakiewicz  wrote:
>> > On 2/1/2016 3:51 PM, Jose Luis Rodriguez Garcia wrote:
>>
>>
>> > Have you tried downloading a newer version of the client from Citrix's 
>> > site?
>>
>> I have just tried the last version of citrix as you told me.
>>
>> It solves the problem of the certificate (the old version didn't
>> understand the CA from Godaddy, but it coredumps at the startup.
>>
>> I copy backtrace/registers and output from ktruss:
> ...
>>   2412   2412 wficaRET   writev 324/0x144
>>   2412   2412 wficaCALL  poll(0xbfbff418,1,0x)
>>   2412   2412 wficaRET   poll 1
>>   2412   2412 wficaCALL  read(3,0x8264a38,0x1000)
>>   2412   2412 wficaGIO   fd 3 read 32 bytes
>>"\^A\^B8\^B\0\0\0\0002\0`\^A\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
>>   2412   2412 wficaRET   read 32/0x20
>>   2412   2412 wficaCALL  read(3,0x8264a38,0x1000)
>>   2412   2412 wficaRET   read -1 unknown errno 35
>>   2412   2412 wficaCALL  read(3,0x8264a38,0x1000)
>>   2412   2412 wficaRET   read -1 unknown errno 35
>>   2412   2412 wficaCALL  gettimeofday(0xbfbff8bc,0)
>>   2412   2412 wficaRET   gettimeofday 0
>>   2412   2412 wficaCALL  gettimeofday(0xbfbff83c,0)
>>   2412   2412 wficaRET   gettimeofday 0
>>   2412   2412 wficaCALL  gettimeofday(0xbfbff7dc,0)
>>   2412   2412 wficaRET   gettimeofday 0
>>   2412   2412 wficaCALL  socketcall(9,0xbfbff5f0)
>>   2412   2412 wficaMISC  send: 16,
>> 0600ffe87cffdcffb92200
>>   2412   2412 wficaMISC  msghdr: [name=0x0, namelen=0,
>> iov=0xf323bc84, iovlen=1, control=0x0, controllen=3226819742, flags=0]
>>   2412   2412 wficaGIO   fd 6 wrote 34 bytes
>>
>> "\^W\^C\^A\0\^]\\\M-f`WIH*\M^F\M-%\^X\M-y\M^@9\M-t({In39$\M-!\^[\M^TNc\M^N/\M-_"
>>   2412   2412 wficaRET   socketcall 34/0x22
>>   2412   2412 wficaCALL  gettimeofday(0xbfbff6fc,0)
>>   2412   2412 wficaRET   gettimeofday 0
>>   2412   2412 wficaCALL  gettimeofday(0xbfbff68c,0)
>>   2412   2412 wficaRET   gettimeofday 0
>>   2412   2412 wficaCALL  gettimeofday(0xbfbff6fc,0)
>>   2412   2412 wficaRET   gettimeofday 0
>>   2412   2412 wficaPSIG  SIGSEGV SIG_DFL: code=SEGV_MAPERR,
>> addr=0x92ee48c6, trap=14)
>>   2412   2459 wficaRET   select -1 unknown errno 4
>>   2412   2412 wficaNAMI  "wfica.core"
>
> I am seeing this too. Essentially there are a load of calls to
> linux_sys_recvmsg which returns 35 (EAGAIN). The last round looks like
>
>  got here linux_sys_recvmsg 834 (do_sys_recvmsg=35)
>  got here linux_select1 894 selcommon=0
>  got here linux_sys_select 844
>  got here linux_sys_recvmsg 815
>  got here linux_to_bsd_msghdr 455
>  got here linux_sys_recvmsg 820
>  got here linux_to_bsd_msg_flags 279
>  msg_name=0x0
>  msg_namelen=0
>  msg_iov=0x7f7febd0
>  msg_iov=0x7f
>  msg_control=0x7f7fec20
>  msg_controllen=80
>  msg_flags=400
>  control=0x80586483
>  from=0xfe804534be58
>  got here linux_sys_recvmsg 834 (do_sys_recvmsg=35)
>  got here linux_select1 894 selcommon=4
>  pid 3382 (wfica), uid 1000: exited on signal 11 (core dumped)
>
> So it seems that after too many retries, a timer fires and kills the
> process (linux_select1()). I suppose the underlying issue is with
> linux_sys_recvmsg, but how can you find out what?
>
> Cheers,
>
> Patrick

Re: NetBSD 7.0_RC1 on Raspberry Pi 2 not SMP?

[Stephan]

What known bugs are left in terms of SMP?

2015-07-10 14:54 GMT+02:00 Nick Hudson sk...@netbsd.org:

 On 07/10/15 13:48, Mayuresh wrote:

 #uname -a
 NetBSD pinet 7.0_RC1 NetBSD 7.0_RC1 (RPI2.201506190427Z) evbarm

 sysctl shows me the following:

 hw.ncpu = 1

 cat /proc/cpuinfo shows no output.

 Is the default kernel in this release not SMP or is it that SMP is not yet
 supported for pi?


 There are changes in HEAD to add support, but they have issues so they've
 not been pulled up.

 Nick

Re: SCP file transfer speed

[Stephan]

I just tried with OpenSSH from pkgsrc (openssh-5.8.2nb5) on the
receiver side and it gives full speed:

file100M.dat 22%   23MB  11.2MB/s  11.3MB/s   00:06 ETA

2015-03-20 9:38 GMT+01:00 Stephan stephan...@googlemail.com:
 That gives me about 9,3 MB/s in my test, contratry to 8,5 MB/s without
 specifying the cipher.

 I don´t know, however, what ciphers are used if I don´t specify one.

 2015-03-20 7:21 GMT+01:00 Michael van Elst mlel...@serpens.de:
 stephan...@googlemail.com (Stephan) writes:

When I copy large files through scp to a NetBSD box, I get transfer
speeds of only 7 MB/s on a 100 MBit connection. This should be around
11 MB/s. I´ve seen this on different x86/amd64 hardware with NetBSD 5,
6 and 7-BETA. The NICs are largely wm, fxp and bnx.

 scp speed most often is limited by the CPU doing the encoding (or worse:
 compression).

 You could try something like:

 scp -oCiphers=arcfour 

 On a modern computer that's more than 4 times as fast as the default
 AES cipher. If you don't trust arcfour, maybe blowfish-cbc is good
 enough for you and still about 3 times as fast.


 --
 --
 Michael van Elst
 Internet: mlel...@serpens.de
 A potential Snark may lurk in every tree.

Re: SCP file transfer speed

[Stephan]

I´m actually having three boxes here at work:

One is a single-core Pentium 4 server @ 2,8 GHz on NetBSD 5.1 / FFSv1.
Two is an 8-core Xeon E5440 server @ 2,8 GHz on NetBSD 6.0 / FFSv2 WAPBL.
Three is a 4-core Xeon L5410 @ 2,33 GHz on NetBSD 7 BETA / FFSv2 WAPBL.

All of these have the same issue; transfer speeds are between 7,5 and
8,5 MB/s. That is why I suspected that this is a very basic issue.

Copying to /dev/null makes no significant difference.

2015-03-19 17:11 GMT+01:00 Greg Troxel g...@ir.bbn.com:

 Stephan stephan...@googlemail.com writes:

 I performed a quick benchmark with netio and it showed up the best
 possible speed on TCP.

 NETIO - Network Throughput Benchmark, Version 1.26
 (C) 1997-2005 Kai Uwe Rommel

 TCP connection established.
 Packet size  1k bytes:  11506 KByte/s Tx,  6 KByte/s Rx.
 Packet size  2k bytes:  11512 KByte/s Tx,  11469 KByte/s Rx.
 Packet size  4k bytes:  11513 KByte/s Tx,  11469 KByte/s Rx.
 Packet size  8k bytes:  11513 KByte/s Tx,  11100 KByte/s Rx.
 Packet size 16k bytes:  11513 KByte/s Tx,  11469 KByte/s Rx.
 Packet size 32k bytes:  11513 KByte/s Tx,  11470 KByte/s Rx.
 Done.

 That looks good; 91.7 Mb/s and I dimly recall 93 as max theoretically
 possible with framing, IP, TCP headers.

 UDP, however, does not work out of the box. It continuously displays
 these errors:

 sendto(): No buffer space available

 Well, that's not really not working.  A modern cpu can call sendto()
 faster than the interface can send packets, and without any sort of flow
 control or congestion control, any buffer will be overwhelmed.  But as
 you say it's not relevant.

 During an (uncompressed) scp copy, CPU usage on the receiver side is
 largely below 10 % and disk utilization about 50% (which makes me
 wonder because this disk can make about 100MB/s on sequential
 write). I can copy to other Linux and Solaris boxes in this network
 with about 10 - 11 MB/s from the NetBSD boxes - just receiving is
 slow.

 Is it just one NetBSD box that is slow?  How many did you try?
 Do you have wapbl enabled?  Which fs?

 Try copying to /dev/null instead, to take the disk out of the experiment.
 I just copied to the disk (I had used /dev/null before) on one of my
 normal machines, and still got 10 MB/s.

SCP file transfer speed

[Stephan]

Hi

When I copy large files through scp to a NetBSD box, I get transfer
speeds of only 7 MB/s on a 100 MBit connection. This should be around
11 MB/s. I´ve seen this on different x86/amd64 hardware with NetBSD 5,
6 and 7-BETA. The NICs are largely wm, fxp and bnx.

Can others reproduce this?

Re: SCP file transfer speed

[Stephan]

I performed a quick benchmark with netio and it showed up the best
possible speed on TCP.

NETIO - Network Throughput Benchmark, Version 1.26
(C) 1997-2005 Kai Uwe Rommel

TCP connection established.
Packet size  1k bytes:  11506 KByte/s Tx,  6 KByte/s Rx.
Packet size  2k bytes:  11512 KByte/s Tx,  11469 KByte/s Rx.
Packet size  4k bytes:  11513 KByte/s Tx,  11469 KByte/s Rx.
Packet size  8k bytes:  11513 KByte/s Tx,  11100 KByte/s Rx.
Packet size 16k bytes:  11513 KByte/s Tx,  11469 KByte/s Rx.
Packet size 32k bytes:  11513 KByte/s Tx,  11470 KByte/s Rx.
Done.


UDP, however, does not work out of the box. It continuously displays
these errors:

sendto(): No buffer space available

This might be tweakable through sysctl, but is however irrelevant in
this case. During an (uncompressed) scp copy, CPU usage on the
receiver side is largely below 10 % and disk utilization about 50%
(which makes me wonder because this disk can make about 100MB/s on
sequential write). I can copy to other Linux and Solaris boxes in this
network with about 10 - 11 MB/s from the NetBSD boxes - just receiving
is slow.

2015-03-19 14:40 GMT+01:00 Greg Troxel g...@ir.bbn.com:

 Stephan stephan...@googlemail.com writes:

 When I copy large files through scp to a NetBSD box, I get transfer
 speeds of only 7 MB/s on a 100 MBit connection. This should be around
 11 MB/s. I´ve seen this on different x86/amd64 hardware with NetBSD 5,
 6 and 7-BETA. The NICs are largely wm, fxp and bnx.

 I tested with the most bloated distfile I had handy -- qt4, at 230 MB.
 With compression off, I got 10 MB/s to three different machines
 (including a Xen DOM0), 9.0 MB/s to a xen domU, and 6.7 MB/s to another
 dom0 that was bought in 2005.  These are all in use with a network with
 other traffic, all 100 Mb/s interfaces.   I also saw 5 MB/s to another
 machine (many switches away in a different building), but it's a build
 server that's usually very heavily loaded.

 There are multiple things going on; you're testing packet delivery and
 how TCP reacts to it, ssh overhead (CPU time,b expansion), and slowdowns
 From writing to the filesystem.  I agree that one would expect to be
 limited by the 100 Mb/s Ethernet itself with modern hardware.

 You could use ttcp to look at just TCP throughput.

 Check cpu usage with top during all this.

 Note that ssh is encrypting/macing the plaintext, which results in
 larger ciphertext plus ssh headers, plus there are TCP and IP headers,
 plus ethernet headers and inter-frame spacing.  So you will not see 100
 Mb/s of payload.

 gson's results of 16 MB/s is interesting; that's 128 Mb/s of payload,
 which definitely seems limited by the Ethernet.   manu@ has reported
 speeds that are most (800ish) of GbE with gluster.

 I wonder if ssh's internal flow control could be having an effect.

Re: NetBSD for the dekstop

[Stephan]

Hi folks

I´m not sure if everybody has gotten my point I described in mail #9
to this discussion.

What I certainly will not do is anything with one of the X based
desktop environments. These are the problem and not the solution.

I performed some experimentation and I decided to focus on Haiku
becuase its in a pretty good shape and under active development. They
have a fantastic UI which is a pretty exact copy of Be Inc´s BeOS. It
has also nothing to do with X.

I´m not the first one with this idea and there was a project called
cosmoe which had the goal of porting the Haiku userland to a Posix
kernel (well, actually, Linux). BeOS/Haiku has some own IPC primitives
(ports and messages for instance) which are unknown to Posix. Cosmoe
is trying to emulate these through shared memory and SysV semaphores.
I think this is a good base for a prototype.

I had to make a couple of changes to the code but it does compile now
and I can start the appserver, which renders to wscons through SDL
(I´m seeing a white rectangle and a mouse pointer). The Haiku to Posix
wrapper is, however, broken on NetBSD so I can´t do anything further
yet.


2015-02-18 21:24 GMT+01:00 Riccardo Mottola riccardo.mott...@libero.it:
 Hi,

 Stephan wrote:

 Hi!

 Is there anyone still interested in bringing NetBSD to the desktop?

 Why still? I use it as a desktop. I have it running on two laptops,
 sparcstations, mini-pcs, etc

 I use for everything GNUstep applications (GWorkspace, GSPdf, Zipper, Ink,
 LaternaMagica... etc etc)

 On one laptop I use for Mail and Web SeaMonkey, on the other one I use
 Firefox and Thunderbird.

 I must say the Mozilla applications work absolutely fine, exactly as on
 Linux. No stability problems compared to Linux or other BSDs.

 I am in the meanwhile trying to improve GNUMail and TalkSoup. Clearly, for
 the Browser it will be a long way.


 To be true, not everything is so smooth compared to Windows or MacOS, I'm
 sorry to say that, but not inferior to other unix-like systems.

 What are your problems?

 Riccardo

semtimedop()

[Stephan]

Hi!

There is no semtimedop(), right? Is somebody working on it?

Re: NetBSD for the dekstop

[Stephan]

I think there are basically 2 types of open source OSses, those that
are designed for the desktop and those that are not.

The first case is for example Syllable and Haiku OS. Syllable is not
actively maintained anymore and it is far from stable. Haiku is in a
pretty good shape but but it is not yet completely stable and lacks
support for ARM for example, which we have.

The second case is Linux and the BSDs. NetBSD, among others, is very
mature, fast and supports a lot of hardware. There is however no GUI
stack, but you have the choise between 1000 of different crazy
desktop environments, which themselves rely on many different
toolkits and environmental services. You can´t speak of a desktop OS
because everything looks and behaves completely different. There is
also often no good vertical integration with the OS underneath. That
is exactly why Linux on the desktop never got any relevance, because
neither administrators nor users can handle it if no Linux
installation is equal to another.

What I´m thinking about is to steal the UI layer (appserver,
toolkit, basic apps) from Syllable or Haiku and make it work on top of
NetBSD. I grabbed Syllable´s source and applied many C++ fixes to
libsyllable, registrar and the appserver in order to make these
compile. That´s already working. I can´t link anything though because
some interfaces differ (threding, semaphores) and some stuff that we
don´t have (message passing, among others). Haiku is probably better,
because of its license (MIT vs. GPL) and it is actively maintained -
but I just wanted to see how far I could get investing a few hours.

2015-02-17 14:24 GMT+01:00 Mayuresh mayur...@acm.org:
 On Tue, Feb 17, 2015 at 01:29:11PM +0100, Stephan wrote:
 Is there anyone still interested in bringing NetBSD to the desktop?

 I use NetBSD on desktops and laptops.

 What specific characteristics do you think NetBSD needs to bring it to
 desktop? (Also, I didn't get still part of it.)

 Mayuresh

NetBSD for the dekstop

[Stephan]

Hi!

Is there anyone still interested in bringing NetBSD to the desktop?

Re: Using CARP with dhcpd?

[Stephan]

I have no answer to your question, but if you need a high availibility
solution, I can offer you a WIP pkgsrc package of the heartbeat
cluster. It´s pretty old and basic but it should provide what you
need.

I could also manually compile corosync + pacemaker on NetBSD but that
doesn´t work reliably because of libqb´s IPC mechanism.

Re: ixg(4) performances

[Stephan]

Correction: msecs. - not secs.

2014-07-17 11:00 GMT+02:00 Stephan stephan...@googlemail.com:
 I have 2 boxes here at hand with wm interfaces and they reply in
 between 0,5 and 0,8 secs. Likewise, when I ping another host (Solaris)
 from these boxes it gives me a response time of 0,5 - 0,8 secs. When I
 ping the same Solaris host from another source it tells me a time of
 ~0,1 secs.

 2014-07-17 8:31 GMT+02:00 Emmanuel Dreyfus m...@netbsd.org:
 On Thu, Jul 17, 2014 at 07:28:31AM +0200, Stephan wrote:
 I saw these ping latencys of 0,5 seconds on very different NetBSD
 Servers with completely different hardware. There might be a
 regression somewhere. I don´t know, if so, wheather it is specific to
 ICMP or networking in general.


 It must be specific to the driver and/or the hardware. I have bge 
 interfaceon the same machine that have lattency between 0.100 and 0.250 ms.
 --
 Emmanuel Dreyfus
 m...@netbsd.org

Re: ixg(4) performances

[Stephan]

I saw these ping latencys of 0,5 seconds on very different NetBSD
Servers with completely different hardware. There might be a
regression somewhere. I don´t know, if so, wheather it is specific to
ICMP or networking in general.

2014-07-17 6:19 GMT+02:00 Emmanuel Dreyfus m...@netbsd.org:
 Emmanuel Dreyfus m...@netbsd.org wrote:

 I experiment 10 GE link with ixb(4), but the result is really weak: The
 two machines have a direct link through a SFTP+ câble, and copying a file
 over NFS I get a throughput of 1.8 Mb/s, which is less than 2% of the
 link capacity. Any idea of where to look for imrovement?

 I measured TCP performance with netperf and I was able to increase
 throughput up to 2.2 Gb/s using the following settings:
 ifconfig ixb0 mtu 9000 tso4 ip4csum tcp4csum-tx udp4csum-tx
 sysctl:
 kern.sbmax = 67108864
 kern.somaxkva = 67108864
 net.inet.tcp.rfc1323 = 1
 net.inet.tcp.sendspace = 2097152
 net.inet.tcp.recvspace = 2097152
 net.inet.tcp.recvbuf_auto = 0
 net.inet.tcp.sendbuf_auto = 0

 tcp4csum-rx and udp4csum-rx produces errors despite being advertised by
 ifconfig capabilities line.

 I do not find any way to improve further. But there is a very high
 latency that may be responsible for the problem:
 PING 10.103.101.113 (10.103.101.113): 48 data bytes
 64 bytes from 10.103.101.113: icmp_seq=0 ttl=255 time=0.582 ms
 64 bytes from 10.103.101.113: icmp_seq=1 ttl=255 time=0.583 ms
 64 bytes from 10.103.101.113: icmp_seq=2 ttl=255 time=0.568 ms
 64 bytes from 10.103.101.113: icmp_seq=3 ttl=255 time=0.565 ms

 This is twice what I get on gregular gigabit ethernet. Any idea of why
 it is so high? = 500 ms seems wrong on a 10GE direct link...

 --
 Emmanuel Dreyfus
 http://hcpnet.free.fr/pubz
 m...@netbsd.org

Re: ixg(4) performances

[Stephan]

Sorry, I meant netio, not netperf - my bad. It should give you
something like this:

# netio -t 127.0.0.1

NETIO - Network Throughput Benchmark, Version 1.26
(C) 1997-2005 Kai Uwe Rommel

TCP connection established.
Packet size  1k bytes:  297270 KByte/s Tx,  276700 KByte/s Rx.
Packet size  2k bytes:  497285 KByte/s Tx,  24667 Byte/s Rx.
Packet size  4k bytes:  535448 KByte/s Tx,  495991 KByte/s Rx.
Packet size  8k bytes:  25511 Byte/s Tx,  472072 KByte/s Rx.
Packet size 16k bytes:  26267 Byte/s Tx,  511965 KByte/s Rx.
Packet size 32k bytes:  1234733 KByte/s Tx,  30356 Byte/s Rx.
Done.

(-u for UDP)



2014-07-02 8:09 GMT+02:00 Emmanuel Dreyfus m...@netbsd.org:
 Stephan stephan...@googlemail.com wrote:

 I would recommend using netperf for measuring TCP and UDP performance.
 Besides that, it measures different block/segment sizes.

 TCP STREAM
 Recv   SendSend
 Socket Socket  Message  Elapsed
 Size   SizeSize Time Throughput
 bytes  bytes   bytessecs.10^6bits/sec

  32768  32768  3276810.001162.17

 UDP STREAM
 Socket  Message  Elapsed  Messages
 SizeSize Time Okay Errors   Throughputbytes
 bytessecs#  #   10^6bits/sec
   92169216   10.01  366386  02699.60
  41600   10.01   78157575.87
 --
 Emmanuel Dreyfus
 http://hcpnet.free.fr/pubz
 m...@netbsd.org

Re: ixg(4) performances

[Stephan]

Hi,

did you measure raw TCP and UDP throughput using iperf or netperf?


Regards,

Stephan

2014-07-01 8:56 GMT+02:00 Emmanuel Dreyfus m...@netbsd.org:
 Hello

 I experiment 10 GE link with ixb(4), but the result is really weak: The
 two machines have a direct link through a SFTP+ câble, and copying a file
 over NFS I get a throughput of 1.8 Mb/s, which is less than 2% of the
 link capacity. Any idea of where to look for imrovement?

 Here is dmesg output for the boards:
 ixg0 at pci5 dev 0 function 0: Intel(R) PRO/10GbE PCI-Express Network Driver, 
 Version - 2.3.10
 ixg0: clearing prefetchable bit
 ixg0: interrupting at ioapic0 pin 18
 ixg0: PCI Express Bus: Speed 2.5Gb/s Width x8
 ixg1 at pci5 dev 0 function 1: Intel(R) PRO/10GbE PCI-Express Network Driver, 
 Version - 2.3.10
 ixg1: clearing prefetchable bit
 ixg1: interrupting at ioapic0 pin 19
 ixg1: PCI Express Bus: Speed 2.5Gb/s Width x8



 --
 Emmanuel Dreyfus
 m...@netbsd.org

Re: ixg(4) performances

[Stephan]

I would recommend using netperf for measuring TCP and UDP performance.
Besides that, it measures different block/segment sizes.

2014-07-02 5:26 GMT+02:00 Emmanuel Dreyfus m...@netbsd.org:
 Stephan stephan...@googlemail.com wrote:

 did you measure raw TCP and UDP throughput using iperf or netperf?

 ttcp reports 1,39 Gb/s with TCP, which is still not impressing. Oddly,
 it does not report anything for UDP (-u flag), while netstat -i shows
 data transmitted.

 --
 Emmanuel Dreyfus
 http://hcpnet.free.fr/pubz
 m...@netbsd.org

Re: install fails at newfs

[Stephan]

I have had exactly the same issue in the past. My solution was to
clear the first 512 Bytes of the physical disk* and rebooting.

* dd if=/dev/zero of=/dev/rsd0d bs=512 count=1


Regards,

Stephan

2014-06-22 13:43 GMT+02:00 Martin Husemann mar...@duskware.de:
 On Sun, Jun 22, 2014 at 05:11:45PM +0530, Mayuresh Kathe wrote:
 a wedge! i'll have to read up on it.
 in the meanwhile, is it possible to eliminate the wedge and do a normal
 install
 on the wd0 device?

 Just use the dk0 device for the installation, it basically just is an alias
 for wd0a.

 Martin

Re: NetBSD+Xen with short-lived throw-away guests

[Stephan]

What we need is something like Citrix Provisioning Services.


Regards,

Stephan

2014-03-05 13:16 GMT+01:00 Torbjorn Granlund t...@gmplib.org:
 I am developing a compiler teaching/judging system based on NetBSD+Xen
 (as Dom0) where all execution of student compilers is done in Xen guests
 (DomU).  For executing the code the student compilers generate from our
 test suite, we use either Xen (for x86 and JVM targets) or qemu (for
 ARM, SPARC, MIPS targets).

 Before booting a Xen guest, a prepared master disk image is copied to a
 throw-away image.  The throw-away image is used for one judging round,
 then discarded.

 The Xen master disk images are about 4 GB each.  They could be trimmed,
 but not to a really small size.

 My first attempt was to create a tmpfs for the throw-away image, then cp
 from a master image to the tmpfs.  This copying is fast, more so since
 the master image tend to live in buffer cache.

 But to my surprise the NetBSD guest then runs poorly.  It creates 100%
 load the Dom0 for several minutes while booting.  This DomU uses PV
 virtualisation (in fact it uses the kernel made available for this
 purpose by the NetBSD project, version 6.1.3).

 Attempt 2 is more plain.  I just cp (or dd) the master image to a
 throw-away image in the same filesystem.  The disk is an Intel S3500 SSD
 (or actually two mirrored S3500).  This copy procedure takes about two
 minutes, and also wears on the SSD (4 GB is written every 5 minutes
 under full load).  But this naive solution is faster than the tmpfs
 trick.

 I haven't tried replacing tmpfs with mfs.  I haven't tried using HVM
 instead of PV.

 Do you have any advice on how to do this cleverly?

 A copy-on-write on a per-file basis would be awesome...

 The system is operational and used already, I just want to improve its
 speed and decrease SSD wear.


 Torbjörn

Re: Xorg slow start

[Stephan]

Hi all,

I have kind of a strange MSI Wind PC Box at home which I think also
has that type of graphics card. I saw very similar issues when
starting X (from the console) - a blank screen for one minute or so. I
didn´t look at that closer because I don´t care much about the GUI
mess in the Unix world. However, I remember seeing some keyboard
related timeout messages in dmesg. Can you see something silimar?


Regards,

Stephan

2014-02-17 11:25 GMT+01:00 Martin Husemann mar...@duskware.de:
 On Mon, Feb 17, 2014 at 12:42:21PM +0400, Konstantin H wrote:
 15.02.2014 00:51, Michel Behr ?:
 Isn't there a way to look at the timestamps on log and/or dmesg to see what
 is causing the delay?
 I checked logs, but it did not make anything clear to me.

 Is that a 25s pause between line 377 and 378 during mouse probing
 and another 17s one for the keyboard between line 396 and 397?

 Sounds suspicious to me. Can you try some other usb mouse/keyboard and
 unplug the ones you use now, just for testing? Not sure if that would
 cause any changes, so just try if it is easy to do.

 Worth a PR, I think (please include the Xorg.log and full dmesg).

 Martin