fakeroot-hurd bug or not?

[Svante Signell]

Hi,

The attached test program is extracted from one of p11-kit tests (the
only failing test) in trust/test_token.c:
test_not_writable (): token = p11_token_new (333, "/non-existant",
"Label");
main(): p11_test (test_not_writable, "/token/not-writable");

This test fails with fakeroot-hurd on Hurd due to that /non-existant is
writable for faked nodes according to:
error_t
netfs_report_access (struct iouser *cred, struct node *np, int *types)
{
  struct netnode *nn = netfs_node_netnode (np);
  if (!(nn->faked & FAKE_MODE))
return file_check_access (nn->file, types);
  else
*types = O_RDWR|O_EXEC;
  return 0;
}

which is reasonable if fakeroot is to mimic real root accesses. Return
values are *is_writable=1 and errno="No such file or directory".

However, running this program with fakeroot-tcp on Hurd or fakeroot-sysv
on Linux returns *is_writable=0 and errno="Permission denied" while
running it as real root of course returns *is_writable=1 and errno="No
such file or directory" on both Linux and Hurd. Maybe even the tests
should not be run under fakeroot at all as is done now with
fakeroot debian/rules binary?

The attached patch makes the behaviour the same as on Linux and
fakeroot-tcp. The question is which behaviour is the expected one.

Index: hurd-0.6.git20150704/trans/fakeroot.c
===
--- hurd-0.6.git20150704.orig/trans/fakeroot.c
+++ hurd-0.6.git20150704/trans/fakeroot.c
@@ -785,11 +785,7 @@ error_t
 netfs_report_access (struct iouser *cred, struct node *np, int *types)
 {
   struct netnode *nn = netfs_node_netnode (np);
-  if (!(nn->faked & FAKE_MODE))
-return file_check_access (nn->file, types);
-  else
-*types = O_RDWR|O_EXEC;
-  return 0;
+  return = file_check_access (nn->file, types);
 }
 
 error_t

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#define bool int
#define true 1
#define false 0

static bool
check_directory (const char *path,
 bool *make_directory,
 bool *is_writable)
{
  struct stat sb;
  char *parent, *path1;
  bool dummy = 0;
  bool ret;

  fprintf(stderr,"\ntrust/token.c (check_directory): ENTERING stat() path=%s, *make_directory=%d, *is_writable=%d\n", path, *make_directory, *is_writable);
  fflush(stderr);
  if (stat (path, ) == 0) {
fprintf(stderr,"\ntrust/token.c (check_directory): stat(path, ) == 0 BEFORE *is_writable\n");
fflush(stderr);
*make_directory = false;
int err = access (path, W_OK);
*is_writable = S_ISDIR (sb.st_mode) && (err == 0);
fprintf(stderr,"\ntrust/token.c (check_directory): stat(path, ) == 0 AFTER *is_writable, access(2) sets errno\n");
fprintf(stderr,"\ntrust/token.c (check_directory): access(path, W_OK)=%d, errno=%s, *is_writable=%d\n", err, strerror(errno), *is_writable);
fflush(stderr);
return true;
  }

  switch (errno) {

  case EACCES:
fprintf(stderr,"\ntrust/token.c (check_directory): case EACCES\n");
fflush(stderr);
*is_writable = false;
*make_directory = false;
return true;

  case ENOENT:
fprintf(stderr,"\ntrust/token.c (check_directory): case ENOENT\n");
fflush(stderr);
*make_directory = true;
//parent = p11_path_parent (path);
path1 = strdup(path);
parent = dirname (path1);
if (parent == NULL)
  ret = false;
else
  ret = check_directory (parent, , is_writable);
return ret;

  default:
fprintf(stderr,"\ntrust/token.c (check_directory): case default\n");
fflush(stderr);
error (1, errno, "couldn't access: %s", path);
return false;
  }
}

int main(void)
{
  const char *path ="/non-existant";
  bool ret = false, make_directory = false;
  int is_writable = 0;

  ret = check_directory (path, _directory, _writable);
  return ret;
}

Re: fakeroot-hurd bug or not?

[Samuel Thibault]

Svante Signell, le Wed 23 Sep 2015 15:45:13 +0200, a écrit :
> This test fails with fakeroot-hurd on Hurd due to that /non-existant is
> writable for faked nodes according to:

No.  Check what happens in the testcase again.  It fails due to /
being announced as writable by fakeroot (see the recursive call to
check_directory in the ENOENT case).

> running it as real root of course returns *is_writable=1 and errno="No
> such file or directory" on both Linux and Hurd.

Take care that the errno value should not be looked at when the function
succeeded. It may be just a leftover from a previous call, or even
something else, it's really undefined.

> Maybe even the tests should not be run under fakeroot at all as is
> done now with fakeroot debian/rules binary?

That is very questionable, yes.  Checks should be done in the build
target, not in the binary target.  Running tests in fakeroot does not
make much sense to me.

All that being said, we should probably not let the programs inside
fakeroot believe they can write to / (because they may then try to,
while they can't actually).

> The attached patch makes the behaviour the same as on Linux and
> fakeroot-tcp. The question is which behaviour is the expected one.

It is indeed tempting do do this change.

> Index: hurd-0.6.git20150704/trans/fakeroot.c
> ===
> --- hurd-0.6.git20150704.orig/trans/fakeroot.c
> +++ hurd-0.6.git20150704/trans/fakeroot.c
> @@ -785,11 +785,7 @@ error_t
>  netfs_report_access (struct iouser *cred, struct node *np, int *types)
>  {
>struct netnode *nn = netfs_node_netnode (np);
> -  if (!(nn->faked & FAKE_MODE))
> -return file_check_access (nn->file, types);
> -  else
> -*types = O_RDWR|O_EXEC;
> -  return 0;
> +  return = file_check_access (nn->file, types);
>  }
>  
>  error_t

I'm almost believing it's the right thing to do, I'm just afraid of
corner cases.  What reassures me is that in both attempt_mkfile and
attempt_chmod, we always add RUSR and WUSR permissions on the real
node (and XUSR when it makes sense), and so for files created inside
the fakeroot session (i.e. those which have FAKE_MODE, for which we
do change the code here), the behavior will be the same in the end.
The only exception is /, which does have FAKE_MODE set in main(), but
already exists and shouldn't be made writable.  AIUI, this is exactly
where your proposed change actually change a behavior, and is precisely
what you intended to do.

Could somebody else double-check my reasoning is correct?
(we have to be careful with these permissions, we need to converge to
something that just does has the right behavior in all cases)

Perhaps for some reason it'd even fix the issues I reported on Mon, 7
Sep 2015 22:03:43 +0200, for instance the gpsd experimental package,
Svante, could you check?

Samuel

GSoC report: Physical memory management

[Justus Winter]

Hello :)

this is my report on my GSoC project, "Physical memory management".

Tldr: There are some remaining issues to fix, but here is a flashy
screenshot of a patched kernel not quite booting Debian/Hurd:

GNU Mach 1.5
biosmem: physical memory map:
biosmem: 00:09fc00, available
biosmem: 09fc00:0a, reserved
biosmem: 0f:10, reserved
biosmem: 10:007fffe000, available
biosmem: 007fffe000:008000, reserved
biosmem: 00feffc000:00ff00, reserved
biosmem: 00fffc:01, reserved
biosmem: heap: 406000-3800
vm_page: page table size: 524270 entries (26624k)
vm_page: DMA: pages: 4080 (15M), free: 3066 (11M)
vm_page: DIRECTMAP: pages: 225280 (880M), free: 218367 (852M)
vm_page: HIGHMEM: pages: 294910 (1151M), free: 294910 (1151M)
pcibios_init : BIOS32 Service Directory structure at 0xf6ac0
pcibios_init : BIOS32 Service Directory entry at 0xfd4be
pcibios_init : PCI BIOS revision 2.10 entry at 0xfd456
Probing PCI hardware.
ide: Intel 82371 PIIX3 (dual FIFO) DMA Bus Mastering IDE 
Controller on PCI bus 0 function 9
ide: BM-DMA feature is not enabled (BIOS), enabling
ide0: BM-DMA at 0xc100-0xc107
ide1: BM-DMA at 0xc108-0xc10f
hd0: got CHS=762/128/63 CTL=c8 from BIOS
hd0: QEMU HARDDISK, 3001MB w/256kB Cache, CHS=762/128/63, DMA
hd2: QEMU DVD-ROM, ATAPI CDROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
Floppy drive(s): fd0 is 1.44M
FDC 0 is a S82078B
eth0: RealTek RTL-8029 found at 0xc000, IRQ 11, 52:54: 0:12:34:56.
ne2k-pci.c:v1.05 6/13/2002 D. Becker/P. Gortmaker
  http://www.scyld.com/network/ne2k-pci.html
rtl8139.c:v1.23a 8/24/2003 Donald Becker, bec...@scyld.com.
 http://www.scyld.com/network/rtl8139.html
via-rhine.c:v1.16 7/22/2003  Written by Donald Becker 
  http://www.scyld.com/network/via-rhine.html
0 3c515 cards found.
eth1: D-Link DE-600 pocket adapter: not at I/O 0x378.
D-Link DE-620 pocket adapter not identified in the printer port
Partition check (DOS partitions):
 hd0: hd0s1 hd0s2 < hd0s5 >
lpr0: at atbus0, port = 378, spl = 6, pic = 7.
module 0: ext2fs.static --multiboot-command-line=${kernel-command-line} 
--host-priv-port=${host-port} --device-master-port=${device-port} 
--exec-server-task=${exec-task} -T typed device:hd0s1 $(task-create) 
$(task-resume)
module 1: ld.so.1 /hurd/exec $(exec-task=task-create)
2 multiboot modules
task loaded: ext2fs.static --multiboot-command-line=gnumach console=com0 
init=/bin/bash --host-priv-port=1 --device-master-port=2 --exec-server-task=3 
-T typed device:hd0s1
task loaded: ld.so.1 /hurd/exec

start ext2fs.static: Hurd server bootstrap: ext2fs.static[device:hd0s1] exec 
startup proc auth.

wired MEMORY_OBJECT_LOCK_RESULT_MUST_BLOCK

At the top you see the "biosmem" module.  It is imported from Richard
Brauns x15 kernel.  It interprets the memory layout as provided by the
multiboot-compliant bootloader.

Then the vm_page module is initialized.  It is also from x15, slightly
adapted and glued into the environment.  This module contains the
buddy allocator which provides efficient allocation of continuous
chunks of memory at the cost of high internal fragmentation.  To avoid
the fragmentation overhead, we do not use it directly, but through the
slab allocator that is already used for most of the allocations inside
the kernel.

The module provides three different segments, DMA, DIRECTMAP, and
HIGHMEM.  The DMA segment provides memory at low addresses for drivers
that need buffers that devices can read from and write into using DMA.

The DIRECTMAP segment is directly mapped into the kernels address
space.  We can use this segment for kernel objects and can directly
access it in inter-task memory copies.  You can see its size capped at
880M because the kernel is using a 1G kernel / 3G userspace split (the
stock Debian kernel uses a 2/2 split to maximize the amount of
mappable physical memory).

We cannot use the HIGHMEM segment for kernel objects, but we can give
it out to userspace tasks.

You can then see the drivers probing for devices, modules being loaded
and the Hurd server bootstrap running.  The system then deadlocks,
with a custom message being emitted right before that.

And this is the current state of affairs.  It is basically done,
modulo finding all the remaining issues.  I know I've been saying that
a lot lately...

From a high-level point of view what I did was:

* Replace the per-task IPC tables with a per-task radix tree, and also
  the reverse lookup (hash) table with a radix tree.

* Inserting nodes into the radix tree sometimes requires memory
  allocations, which may block when memory is tight.  Therefore, I had
  to convert a lot of simple locks with sleep locks, a manual process
  that required fixes to the locking here and there.

* Replace the current memory allocator with a buddy allocator.

Upon completion 

Re: fakeroot-hurd bug or not?

[Samuel Thibault]

Svante Signell, le Wed 23 Sep 2015 20:07:56 +0200, a écrit :
> Seems like dh does this by default now, see debian/rules. From the
> build log
>   debian/rules build
> make: 'build' is up to date.
>  fakeroot debian/rules binary
> dh binary --with autoreconf
> ...
> dh_auto_test
>  make -j1 check

That's probably worth discussing on debian-devel, it's really odd to run
auto_test in fakeroot.

Samuel

Re: misc/50166

[Robert Millan]

El 21/09/15 a les 23:53, Antti Kantee ha escrit:

On 21/09/15 20:37, Robert Millan wrote:

The result is much smaller than I expected. In fact, other than make
(because of the
bootstrap issue) and some Rump components, all remaining MAXPATHLEN/etc
issues are
handled by nbtool_config.h.

Please consider attached patch.


Ok I committed a variant of it (I moved the rump kernel userspace bits into 
rumpuser_port.h).  Please check if NetBSD HEAD now works for your use case.


Well I'm glad to say I just tested, and current CVS HEAD builds correctly on 
GNU/Hurd now.


I'm also happy to apply a patch where MAXPATHLEN/PATH_MAX/MAXHOSTNAMELEN is 
removed from the rump kernel userspace code, if you (or someone else) want to 
send one in another PR.


I think that would be nice having, but unfortunately I lack the spare time at 
the moment. CCing bug-hurd in case someone is interested.

Thanks!

--
Robert Millan

Re: fakeroot-hurd bug or not?

[Samuel Thibault]

Svante Signell, le Wed 23 Sep 2015 23:24:16 +0200, a écrit :
> You are a DD, please raise this question on debian-devel. I'm just
> nobody :(

One doesn't have to be a DD to raise an issue on debian-devel. People on
debian-devel don't care who you are, but what topic you bring there. Of
course, if you have a record of bad topics, people will tend not to read
the mail, but that's just a tendency. Personally, I never really read
who sends the mail, but the subject and the content, and then who sent
it.

Samuel

Re: fakeroot-hurd bug or not?

[Svante Signell]

On Wed, 2015-09-23 at 19:08 +0200, Samuel Thibault wrote:
> Svante Signell, le Wed 23 Sep 2015 15:45:13 +0200, a écrit :
> > This test fails with fakeroot-hurd on Hurd due to that /non
> > -existant is
> > writable for faked nodes according to:
> 
> No.  Check what happens in the testcase again.  It fails due to /
> being announced as writable by fakeroot (see the recursive call to
> check_directory in the ENOENT case).

Sorry, I meant / is writable, the file attempted to create is /non
-existant

> > running it as real root of course returns *is_writable=1 and
> > errno="No
> > such file or directory" on both Linux and Hurd.
> 
> Take care that the errno value should not be looked at when the
> function
> succeeded. It may be just a leftover from a previous call, or even
> something else, it's really undefined.

Yes, errno is not interesting when access(2) succeeds. Sorry again for
not being clear to omit errno in the text for that case. Of course I
meant what you just wrote :)

> > Maybe even the tests should not be run under fakeroot at all as is
> > done now with fakeroot debian/rules binary?
> 
> That is very questionable, yes.  Checks should be done in the build
> target, not in the binary target.  Running tests in fakeroot does not
> make much sense to me.

Seems like dh does this by default now, see debian/rules. From the
build log
  debian/rules build
make: 'build' is up to date.
 fakeroot debian/rules binary
dh binary --with autoreconf
...
dh_auto_test
 make -j1 check

> All that being said, we should probably not let the programs inside
> fakeroot believe they can write to / (because they may then try to,
> while they can't actually).
> 
> > The attached patch makes the behaviour the same as on Linux and
> > fakeroot-tcp. The question is which behaviour is the expected one.
> 
> It is indeed tempting do do this change.
...
> I'm almost believing it's the right thing to do, I'm just afraid of
> corner cases.  What reassures me is that in both attempt_mkfile and
> attempt_chmod, we always add RUSR and WUSR permissions on the real
> node (and XUSR when it makes sense), and so for files created inside
> the fakeroot session (i.e. those which have FAKE_MODE, for which we
> do change the code here), the behavior will be the same in the end.
> The only exception is /, which does have FAKE_MODE set in main(), but
> already exists and shouldn't be made writable.  AIUI, this is exactly
> where your proposed change actually change a behavior, and is
> precisely
> what you intended to do.
> 
> Could somebody else double-check my reasoning is correct?
> (we have to be careful with these permissions, we need to converge to
> something that just does has the right behavior in all cases)
> 
> Perhaps for some reason it'd even fix the issues I reported on Mon, 7
> Sep 2015 22:03:43 +0200, for instance the gpsd experimental package,
> Svante, could you check?

Will do, bbl!

Re: fakeroot-hurd bug or not?

[Svante Signell]

On Wed, 2015-09-23 at 20:09 +0200, Samuel Thibault wrote:
> Svante Signell, le Wed 23 Sep 2015 20:07:56 +0200, a écrit :
> > Seems like dh does this by default now, see debian/rules. From the
> > build log
> >   debian/rules build
> > make: 'build' is up to date.
> >  fakeroot debian/rules binary
> > dh binary --with autoreconf
> > ...
> > dh_auto_test
> >  make -j1 check
> 
> That's probably worth discussing on debian-devel, it's really odd to
> run
> auto_test in fakeroot.

You are a DD, please raise this question on debian-devel. I'm just
nobody :(

Re: GSoC report: Physical memory management

[Samuel Thibault]

Nice work :)

Samuel

Re: misc/50166

[Antti Kantee]

On 23/09/15 21:25, Robert Millan wrote:

I'm also happy to apply a patch where
MAXPATHLEN/PATH_MAX/MAXHOSTNAMELEN is removed from the rump kernel
userspace code, if you (or someone else) want to send one in another PR.


I think that would be nice having, but unfortunately I lack the spare
time at the moment. CCing bug-hurd in case someone is interested.


Right, probably not highest priority now, since the code does work ... 
unless someone wants to exercise their possibility to put it in a 8k 
long subdirectory path on Hurd ;)

Re: USB Mass Storage with rump

[Olaf Buddenhagen]

Hi,

On Sat, Sep 19, 2015 at 11:57:03PM +0200, Robert Millan wrote:

> single Rump instance inside a single translator which exposes all of
> /dev in Rump namespace somewhere under host /dev hierarchy (e.g.
> /dev/rump/*).

This is certainly tempting, but also dangerous -- once a somewhat
working solution is in place, there is less motivation to do the "right"
thing...

Note that besides going fully monolithic (as described above), or fully
modular (as you suggested originally), a compromise solution that is
monolitic on the vertical axis (all layers in one server), but modular
on the horizontal one (seperate instance for each device) is also a
possibility. I'm not saying it's necessarily the best choice -- but it's
an option to consider. Ultimately it depends on how much effort the
person implementing it is willing to invest...

-antrik-

Re: USB Mass Storage with rump

[Olaf Buddenhagen]

Hi,

On Sat, Sep 19, 2015 at 10:52:13AM +0200, Robert Millan wrote:

> Since you most likely want to provide multiplexing, authorisation,
> etc, to any application who wants to access USB, I wouldn't recommend
> to lump USB mass storage and *HCI in the same Rump instance.

Quite frankly, I wouldn't either :-) It just sounded like Bruno wanted
to go with such a simplistic approach for the start...

> Instead, you could run a Rump instance with USB mass storage only
> which uses libusb as backend rather than its own *HCI driver (but that
> requires some coding work as it's currently not implemented ;-))

Yeah, I guess that's the price to pay if we want a properly layered
driver stack based on an originally monolithic implementation :-) As
long as we don't need to jump through hoops to achieve that (and it gets
upstream support), I'd say it's worth the effort...

-antrik-

Re: USB Mass Storage with rump

[Olaf Buddenhagen]

Hi,

On Sat, Sep 19, 2015 at 10:59:39AM +0200, Samuel Thibault wrote:

> It'd probably be easy to make ext2fs open a device node, just like we
> made pfinet do it.

As I already mentioned on IRC, I don't think we should emulate Mach
device nodes at all here. Rather, the USB mass storage server(s) would
export UNIX device nodes, which ext2fs/libstore can already deal with
AFAIK.

-antrik-

Re: Sound translator / PCI device handling

[Olaf Buddenhagen]

Hi,

On Sat, Sep 19, 2015 at 01:24:17PM +, Antti Kantee wrote:

> IMO the right way to do device drivers in a hosted environment is to
> have one entity which decides which servers see which devices and just
> let the servers attach to all devices they see.

>From what I gathered from Robert's and your explanations, it sounds to
me like the PCI server we are envisioning would be perfectly in line
with this design. The PCI server would expose one UNIX device node for
each PCI device present; and the Hurd backend in libpciaccess would open
these nodes to access the config space bits of any particular device.

The device node(s) to use would probably be passed as command line
parameters to each Rump translator instance -- so it would only see the
devices explicitly specified. No need for any other configuration :-)

-antrik-

Re: Sound translator

[Olaf Buddenhagen]

Hi,

On Sat, Sep 19, 2015 at 10:52:01AM +0200, Robert Millan wrote:
> El 19/09/15 a les 01:06, Olaf Buddenhagen ha escrit:

> >Is there any actual logic that could be split out into the audio and
> >mixer translators?
[...]
> That depends on the sound API you want to provide to applications.
> 
> Rump doesn't implement OSS, but rather "Sun audio", which seems to be
> an ancestor of OSS, with very similar design.
> 
> So if you wanted Sun audio, then yes it's a 1:1 wrapper. Otherwise you
> have to convert.

That's a bummer. I was assuming that all BSDs -- and by extension, Rump
-- use OSS...

> I find OSS more useful than Sun audio myself, so I hacked a modified
> version of libossaudio (a conversion library included with NetBSD).

How does that work in NetBSD?

I guess whether it makes sense to split the translation for audio and
mixer devices depends on how this is factored in libossaudio...

-antrik-