Re: [blfs-dev] Python multiprocessing checks in chroot
On 3/26/20 8:38 PM, Xi Ruoyao via blfs-dev wrote: On 2020-03-26 23:22 +, Ken Moffat via blfs-dev wrote: On Thu, Mar 26, 2020 at 10:57:03PM +0100, Pierre Labastie via blfs-dev wrote: (asking about only one item) If you do not do this, configuring will fail with a python traceback report referencing a /usr/lib/pythonN.N/multiprocessing/synchronize.py file and ending 'OSError: [Errno 38] Function not implemented'. (this explanation possibly in italics, i.e. emphasis, except for the filename markup) As the starter of this thread, I do not see exactly this error, but rather: File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 59, in " function, see issue 3770.") ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770. from tmpfs (5): A tmpfs filesystem mounted at /dev/shm is used for the implementation of POSIX shared memory (shm_overview(7)) and POSIX semaphores (sem_overview(7)). In kernel doc (Documentation/filesystems/tmpfs.txt): 2) glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for POSIX shared memory (shm_open, shm_unlink). Adding the following line to /etc/fstab should take care of this: tmpfs /dev/shmtmpfs defaults0 0 Remember to create the directory that you intend to mount tmpfs on if necessary. This mount is _not_ needed for SYSV shared memory. The internal mount is used for that. (In the 2.3 kernel versions it was necessary to mount the predecessor of tmpfs (shm fs) to use SYSV shared memory) So I think we should mount a tmpfs to /dev/shm, anyway. In lfs-systemd it's automatically done by systemd. In lfs-sysv book the mountvirtfs bootscript makes /run/shm and link it to /dev/shm. I think it's a "hack" which is deprecated, and shall be replaced with a line mounting a tmpfs to /dev/shm, in /etc/fstab. If /dev/shm is a symlink to /run/shm and /run is a tmpfs and shm is a subdirectory of /run, what is the difference? There is one level of indirection with the symlink, but any operations end up on a tmpfs now. -- Bruce -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
On 2020-03-26 23:22 +, Ken Moffat via blfs-dev wrote: > On Thu, Mar 26, 2020 at 10:57:03PM +0100, Pierre Labastie via blfs-dev wrote: > > (asking about only one item) > > > If you do not do this, configuring will fail with a python traceback > > > report referencing a /usr/lib/pythonN.N/multiprocessing/synchronize.py > > > file and ending 'OSError: [Errno 38] Function not implemented'. > > > (this explanation possibly in italics, i.e. emphasis, except for the > > > filename markup) > > > > As the starter of this thread, I do not see exactly this error, but rather: > > > > File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 59, in > > > > " function, see issue 3770.") > > ImportError: This platform lacks a functioning sem_open implementation, > > therefore, the required synchronization primitives needed will not function, > > see issue 3770. > > from tmpfs (5): >A tmpfs filesystem mounted at /dev/shm is used for the implementation >of POSIX shared memory (shm_overview(7)) and POSIX semaphores >(sem_overview(7)). In kernel doc (Documentation/filesystems/tmpfs.txt): > 2) glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for >POSIX shared memory (shm_open, shm_unlink). Adding the following >line to /etc/fstab should take care of this: > > tmpfs /dev/shmtmpfs defaults0 0 > >Remember to create the directory that you intend to mount tmpfs on >if necessary. > >This mount is _not_ needed for SYSV shared memory. The internal >mount is used for that. (In the 2.3 kernel versions it was >necessary to mount the predecessor of tmpfs (shm fs) to use SYSV >shared memory) So I think we should mount a tmpfs to /dev/shm, anyway. In lfs-systemd it's automatically done by systemd. In lfs-sysv book the mountvirtfs bootscript makes /run/shm and link it to /dev/shm. I think it's a "hack" which is deprecated, and shall be replaced with a line mounting a tmpfs to /dev/shm, in /etc/fstab. > Which package is this which does not mention OSError, please ? > > ĸen > -- > When alle is ſayed and all is done, ye must chooſe your faces wisely, > for soon enouff ye will be playing with fyre." > The Nice and Accurate Prophecies of Agnes Nutter, Prophecy 5004 -- Xi Ruoyao School of Aerospace Science and Technology, Xidian University -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
On Thu, Mar 26, 2020 at 10:57:03PM +0100, Pierre Labastie via blfs-dev wrote: (asking about only one item) > > > > > If you do not do this, configuring will fail with a python traceback > > report referencing a /usr/lib/pythonN.N/multiprocessing/synchronize.py > > file and ending 'OSError: [Errno 38] Function not implemented'. > > (this explanation possibly in italics, i.e. emphasis, except for the > > filename markup) > > As the starter of this thread, I do not see exactly this error, but rather: > > File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 59, in > " function, see issue 3770.") > ImportError: This platform lacks a functioning sem_open implementation, > therefore, the required synchronization primitives needed will not function, > see issue 3770. > Which package is this which does not mention OSError, please ? ĸen -- When alle is ſayed and all is done, ye must chooſe your faces wisely, for soon enouff ye will be playing with fyre." The Nice and Accurate Prophecies of Agnes Nutter, Prophecy 5004 -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
Le 26/03/2020 à 18:15, Ken Moffat via blfs-dev a écrit : > On Thu, Mar 26, 2020 at 04:51:59PM +, Ken Moffat via blfs-dev wrote: >> On Thu, Mar 26, 2020 at 01:41:40PM +, Ken Moffat via blfs-dev wrote: >> >> [snip how, and what, I tested] >> > > And now the proposal: > > Replace the Note about $SHELL and /dev/shm in firefox with a standard > copy member. The mount part comes first because OSError 38 comes > before any eventual failure to determine the shell. > > > Note: > > If you are compiling this package in chroot you must do two things: > > First, as the root user, ensure that /run is mounted: > > mount -t tmpfs /run But this _should_ already be done if the commands on the "Preparing the Virtual Kernel File Systems" page have been run. Mounting before or after entering chroot does not change anything (it is possible I am wrong with this, but I need to be able to reproduce this on my systems to be convinced) And this is not enough if the host does not have /dev/shm->/run/shm, as is the case on debian and lfs systemd. On those systems, the note about /dev/shm _is_ accurate (well, not quite, it should be "ensure that a tempfs is mounted on /dev/shm") > > If you do not do this, configuring will fail with a python traceback > report referencing a /usr/lib/pythonN.N/multiprocessing/synchronize.py > file and ending 'OSError: [Errno 38] Function not implemented'. > (this explanation possibly in italics, i.e. emphasis, except for the > filename markup) As the starter of this thread, I do not see exactly this error, but rather: File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 59, in " function, see issue 3770.") ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770. > > Second, as your normal user either set the $SHELL environment > variable or prepend SHELL=/bin/sh. Actually, the SHELL variable _is_ set (at least for my systems) for root in chroot, but not exported. So "export SHELL" is enough. Then when doing su - , the SHELL is set and exported for the user (note that I have the bash shell startup files as in the blfs book). > > > I suppose a name like xincludes/mozilla-chroot.xml > > Use in firefox, mozjs60, mozjs68, seamonkey, thunderbird. Actually, lfs sysv is somewhat weird among distros: it seems to be the only distro with /dev/shm->/run/shm. All systemd distros have the other way around, and gentoo and artix do not even have /run/shm. I know it used to be common that distros had /dev/shm->/run/shm, but nowadays, it seems to be not used anymore. Pierre -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
On 3/26/20 3:29 PM, Pierre Labastie via blfs-dev wrote: Le 26/03/2020 à 21:19, Pierre Labastie via blfs-dev a écrit : Le 26/03/2020 à 18:32, Bruce Dubbs via blfs-dev a écrit : On 3/26/20 10:22 AM, Xi Ruoyao via blfs-dev wrote: Now the book is handling the case "/dev/shm is a link to /run/shm" correctly and explicitly. Unfortunately it seems for the case "/dev/shm is *not* a link to /run/shm" the book is defective. I'm surprised that it didn't cause any problem building LFS... I have not tested completely, but what if we did this in LFS instead of a bind mount of $LFS/dev: mount -vt devtmpfs devtmpfs $LFS/dev On my LFS system I then have: /mnt/lfs/dev/shm -> /run/shm but if the host does not have /mnt/lfs/dev/shm, then a simple mkdir -pv $LFS/dev/shm should suffice. -- Bruce Hmm, I have a problem with /dev/shm->/run/shm... It is not true on lfs systemd! All I've said is with lfs systemd or debian (which is systemd too). If you just do 'mount -vt devtmpfs devtmpfs $LFS/dev', do you have any $LFS/dev/shm ? -- Bruce -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
Le 26/03/2020 à 21:19, Pierre Labastie via blfs-dev a écrit : > Le 26/03/2020 à 18:32, Bruce Dubbs via blfs-dev a écrit : >> On 3/26/20 10:22 AM, Xi Ruoyao via blfs-dev wrote: >> >>> Now the book is handling the case "/dev/shm is a link to /run/shm" correctly >>> and >>> explicitly. Unfortunately it seems for the case "/dev/shm is *not* a link >>> to >>> /run/shm" the book is defective. I'm surprised that it didn't cause any >>> problem >>> building LFS... >> >> I have not tested completely, but what if we did this in LFS instead of a >> bind >> mount of $LFS/dev: >> >> mount -vt devtmpfs devtmpfs $LFS/dev >> >> On my LFS system I then have: >> /mnt/lfs/dev/shm -> /run/shm >> >> but if the host does not have /mnt/lfs/dev/shm, then a simple >> mkdir -pv $LFS/dev/shm >> >> should suffice. >> >> -- Bruce >> > > Hmm, I have a problem with /dev/shm->/run/shm... > It is not true on lfs systemd! > > All I've said is with lfs systemd > or debian (which is systemd too). Pierre -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
Le 26/03/2020 à 18:32, Bruce Dubbs via blfs-dev a écrit : > On 3/26/20 10:22 AM, Xi Ruoyao via blfs-dev wrote: > >> Now the book is handling the case "/dev/shm is a link to /run/shm" correctly >> and >> explicitly. Unfortunately it seems for the case "/dev/shm is *not* a link to >> /run/shm" the book is defective. I'm surprised that it didn't cause any >> problem >> building LFS... > > I have not tested completely, but what if we did this in LFS instead of a bind > mount of $LFS/dev: > > mount -vt devtmpfs devtmpfs $LFS/dev > > On my LFS system I then have: > /mnt/lfs/dev/shm -> /run/shm > > but if the host does not have /mnt/lfs/dev/shm, then a simple > mkdir -pv $LFS/dev/shm > > should suffice. > > -- Bruce > Hmm, I have a problem with /dev/shm->/run/shm... It is not true on lfs systemd! All I've said is with lfs systemd Pierre -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
On 3/26/20 10:22 AM, Xi Ruoyao via blfs-dev wrote: Now the book is handling the case "/dev/shm is a link to /run/shm" correctly and explicitly. Unfortunately it seems for the case "/dev/shm is *not* a link to /run/shm" the book is defective. I'm surprised that it didn't cause any problem building LFS... I have not tested completely, but what if we did this in LFS instead of a bind mount of $LFS/dev: mount -vt devtmpfs devtmpfs $LFS/dev On my LFS system I then have: /mnt/lfs/dev/shm -> /run/shm but if the host does not have /mnt/lfs/dev/shm, then a simple mkdir -pv $LFS/dev/shm should suffice. -- Bruce -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
On Thu, Mar 26, 2020 at 04:51:59PM +, Ken Moffat via blfs-dev wrote: > On Thu, Mar 26, 2020 at 01:41:40PM +, Ken Moffat via blfs-dev wrote: > > [snip how, and what, I tested] > And now the proposal: Replace the Note about $SHELL and /dev/shm in firefox with a standard copy member. The mount part comes first because OSError 38 comes before any eventual failure to determine the shell. Note: If you are compiling this package in chroot you must do two things: First, as the root user, ensure that /run is mounted: mount -t tmpfs /run If you do not do this, configuring will fail with a python traceback report referencing a /usr/lib/pythonN.N/multiprocessing/synchronize.py file and ending 'OSError: [Errno 38] Function not implemented'. (this explanation possibly in italics, i.e. emphasis, except for the filename markup) Second, as your normal user either set the $SHELL environment variable or prepend SHELL=/bin/sh. I suppose a name like xincludes/mozilla-chroot.xml Use in firefox, mozjs60, mozjs68, seamonkey, thunderbird. ĸen -- When alle is ſayed and all is done, ye must chooſe your faces wisely, for soon enouff ye will be playing with fyre." The Nice and Accurate Prophecies of Agnes Nutter, Prophecy 5004 -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
On Thu, Mar 26, 2020 at 01:41:40PM +, Ken Moffat via blfs-dev wrote:
[snip who replied to whom]
Here is what I did for testing.
> I'll reply in a while with what I have now tested (after I've
> assembled my notes). I've gone with Bruce's suggestion to (only)
> mount a tmpfs on /run when building in chroot - I think xry111 has
> identified the issue (bind mounts are not recursive).
>
More likely, it is the fact that in BLFS it is a symlink to
/run/shm. Anyway, here is what I did and the errors. Sorry, it is
long. I'll post my proposed changes as a follow-up.
I think that in the past I noted that /dev/shm existed in chroot,
and since I don't often build these packages in chroot I assumed
that part was OK.
The testing:
===
For each package: mount a system at /mnt/lfs (host is 9.1, chroot is
LFS from 20200127 with BLFS from then and later)
bind /dev /proc /sys /sources /home
chroot
su - lfs
Attempt to build as user lfs.
ALL of these mozilla-derived packages failed.
On failure:
note the message
exit back to root (cannot su, not a tty)
mount -t tmpfs /run
su - lfs
Successful retry with DESTDIR install
review contents of /run
exit back to root
umount /run
review contents of /run
exit chroot
umount /mnt/lfs and everything mounted on it :
mount | grep lfs | awk '{ print $3 }' | tac | xargs umount
repeat for next package.
mozjs60:
---
(I initially tried strace during the configure, but I could not find
anything useful in the enormous output.)
Failed (as they all did) while creating config.status -
Traceback (most recent call last):
File "../js/src/../../configure.py", line 127, in
sys.exit(main(sys.argv))
File "../js/src/../../configure.py", line 34, in main
return config_status(config)
File "../js/src/../../configure.py", line 122, in config_status
return config_status(args=[], **encode(sanitized_config, encoding))
File "/home/lfs/mozjs-60.8.0/python/mozbuild/mozbuild/config_status.py", line
133, in config_status
reader = BuildReader(env)
File "/home/lfs/mozjs-60.8.0/python/mozbuild/mozbuild/frontend/reader.py",
line 868, in __init__
self._gyp_worker_pool = ProcessPoolExecutor(max_workers=max_workers)
File
"/home/lfs/mozjs-60.8.0/third_party/python/futures/concurrent/futures/process.py",
line 285, in __init__
EXTRA_QUEUED_CALLS)
File "/usr/lib/python2.7/multiprocessing/__init__.py", line 218, in Queue
return Queue(maxsize)
File "/usr/lib/python2.7/multiprocessing/queues.py", line 63, in __init__
self._rlock = Lock()
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 147, in
__init__
SemLock.__init__(self, SEMAPHORE, 1, 1)
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented
After successful build, /run was empty, but after I had umounted it,
while still in chroot I looked again : /run/mount and /run/shm
existed.
seamonkey:
-
The traceback ends:
File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75, in __init__
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented
*** Fix above errors and then restart with\
"make -f client.mk build"
make[1]: *** [/home/lfs/seamonkey-2.53.1/client.mk:362: configure] Error 1
make[1]: Leaving directory '/home/lfs/seamonkey-2.53.1'
make: *** [client.mk:374:
/home/lfs/seamonkey-2.53.1/obj-x86_64-pc-linux-gnu/Makefile] Error 2
Looking at /run, empty both before umounting and afterwards
thunderbird:
---
0:18.73 File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75,
in __init__
0:18.73 sl = self._semlock = _multiprocessing.SemLock(kind, value,
maxvalue)
0:18.73 OSError: [Errno 38] Function not implemented
0:18.79 *** Fix above errors and then restart with\
0:18.79"./mach build"
0:18.79 make: *** [client.mk:115: configure] Error 1
NB, client.mk is invoked from ./mach.
Again. /run seemed to be empty even after unmounting.
firefox:
---
0:12.67 File "/usr/lib/python2.7/multiprocessing/synchronize.py", line 75,
in __init__
0:12.67 sl = self._semlock = _multiprocessing.SemLock(kind, value,
maxvalue)
0:12.67 OSError: [Errno 38] Function not implemented
with message about fixing, restart with ./mach build
[snipped, messages like thunderbird ]
After umount, /run contained /run/mount (only).
JS68:
File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 38] Function not implemented
Again, after umount /run contained /run/mount (only).
Tested, but does NOT require this - node-js:
===
When I first encountered this problem (years ago, in firefox) for
some reason I thought it would be a good idea to rebuild node-js
although
Re: [blfs-dev] Python multiprocessing checks in chroot
On Thu, Mar 26, 2020 at 04:06:46PM +0100, Pierre Labastie via blfs-dev wrote:
> Le 26/03/2020 à 14:41, Ken Moffat via blfs-dev a écrit :
> > On Thu, Mar 26, 2020 at 11:40:59AM +0100, Pierre Labastie via
> > blfs-dev wrote:
> >> Le 23/03/2020 à 03:49, Bruce Dubbs via blfs-dev a écrit :
> >>> On 3/22/20 9:25 PM, Xi Ruoyao via blfs-dev wrote:
> On 2020-03-23 01:44 +, Ken Moffat via blfs-dev wrote:
> > On Mon, Mar 23, 2020 at 09:12:18AM +0800, Xi Ruoyao via
> > blfs-dev wrote:
> >> On 2020-03-22 21:34 +, Ken Moffat via blfs-dev wrote:
> >>
[...]
> > I'll reply in a while with what I have now tested (after I've
> > assembled my notes). I've gone with Bruce's suggestion to (only)
> > mount a tmpfs on /run when building in chroot - I think xry111 has
> > identified the issue (bind mounts are not recursive).
> >
> > Meanwhile, if you do not see that OSError when building any of firefox,
> > mozjs{60,68}, seamonkey, thunderbird *in chroot* then you are doing
> > something different. This issue was raised in
> > http://lists.linuxfromscratch.org/pipermail/blfs-support/2020-March/081897.html
Replying to this before I post my test results, and before I've
ready xry111's latest post.
Since I wrote that, I actually read the Note in firefox about
ensuring that /dev/shm is mounted. ISTR that you cannot mount
/dev/shm in LFS because it is a broken symlink to /run/shm.
Therefore, I think the /dev/shm part of the note is old and
no-longer correct.
>
> The error reported by the OP ("This platform lacks a functioning sem_open
> implementation...") is the one I see when /dev/shm is not world writable, and
> the build is done as a regular user.
Note that I am NOT touching /dev/shm directly.
> When building as root, the error is different (something about an unknown
> shell).
>
In my tests I initially (mozjs60) made sure SHELL was exported. As
you will see when I post the details, I ended up exiting from chroot
and unmounting /mnt/lfs after successfully completing the tests for
each package. Later it occurred to me that I had not exported SHELL
while testign the subsequent packages (second package was seahorse).
So I started another test, and was surprised to see that SHELL was
still in my lfs user's enviromnet. I had mounted /run (known to be
necessary), so I unset SHELL. Configure again failed while creating
config.log, this time because it could not determine the shell - and
that was as a regular user.
> Those are the only errors I see. Note that chroot is entered with the
> following commands (Makefile syntax):
> sudo mount -v --bind /dev $(MOUNT_PT)/dev; \
> sudo mount -vt devpts devpts $(MOUNT_PT)/dev/pts -o gid=5,mode=620; \
> sudo mount -vt proc proc $(MOUNT_PT)/proc; \
> sudo mount -vt sysfs sysfs $(MOUNT_PT)/sys; \
> sudo mount -vt tmpfs tmpfs $(MOUNT_PT)/run; \
> if [ -h $(MOUNT_PT)/dev/shm ]; then :; \
> sudo mkdir -pv $(MOUNT_PT)/$$(readlink $(MOUNT_PT)/dev/shm); \
> fi;
Yeah, I can see that I have not been mounting /dev/shm when building
these packages in chroot (before these, I always boot the
LFS+essentials (git, nfs, etc) build to check it boots, and to
recalculate the SBU.
>
> #Since on debian, /dev/shm is not a symlink, the last mkdir is not executed.
>
> #Note that a tmpfs is mounted on /run.
>
> # Then
> /usr/sbin/chroot $(MOUNT_PT) \
> /usr/bin/env -i \
> HOME=/root TERM="$$TERM" \
> PS1='(lfs chroot) \u:\w\$$ ' \
> PATH=/bin:/usr/bin:/sbin:/usr/sbin \
> /bin/bash --login
> (again Makefile syntax).
>
> Do I understand correctly that you and xry111 suggest that a tmpfs be mounted
> on /dev/shm? Without any perm change? Let me try...
No, I'm following Bruce's suggestion to just mount a tmpfs on /run,
as root *inside_chroot*.
> Yesss:
> root [ / ]# ls -al /dev/shm
> total 0
> drwxrwxrwt 2 root root 40 Mar 26 15:59 .
> drwxr-xr-x 21 root root 3900 Mar 26 09:48 ..
>
> I just added a line:
> sudo mount -vt tmpfs tmpfs $(MOUNT_PT)/dev/shm; \
>
> while still on host
>
> Pierre
Interesting, there are clearly multiple workarounds.
ĸen
--
When alle is ſayed and all is done, ye must chooſe your faces wisely,
for soon enouff ye will be playing with fyre."
The Nice and Accurate Prophecies of Agnes Nutter, Prophecy 5004
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
On 2020-03-26 16:06 +0100, Pierre Labastie via blfs-dev wrote:
> Le 26/03/2020 à 14:41, Ken Moffat via blfs-dev a écrit :
> > On Thu, Mar 26, 2020 at 11:40:59AM +0100, Pierre Labastie via
> > blfs-dev wrote:
> > > Le 23/03/2020 à 03:49, Bruce Dubbs via blfs-dev a écrit :
> > > > On 3/22/20 9:25 PM, Xi Ruoyao via blfs-dev wrote:
> > > > > On 2020-03-23 01:44 +, Ken Moffat via blfs-dev wrote:
> > > > > > On Mon, Mar 23, 2020 at 09:12:18AM +0800, Xi Ruoyao via
> > > > > > blfs-dev wrote:
> > > > > > > On 2020-03-22 21:34 +, Ken Moffat via blfs-dev wrote:
> > > > > > >
> > > > > > > > # mount --bind /run /mnt/lfs/run
> > > > > > >
> > > > > > > I think it's dangerous: potentially harmful to the host.
> > > > > > > Some service running in the LFS chroot may overwrite the
> > > > > > > runtime directory of the service running on the host.
> > > > > >
> > > > > > So, you are saying that packages from mozilla should not now
> > > > > > be built in chroot ?
> > > > >
> > > > > No. I think we need a better way.
> > > > >
> > > > > > But, what do you mean by a service running in chroot ? I
> > > > > > assume we are specifically talking of systemd here ? Do
> > > > > > services not get started during the boot process ? The
> > > > > > systemd instance in chroot has never started, so I assume it
> > > > > > will be ineffective and systemd on the host will only care
> > > > > > about services described in /etc/systemd ?
> > > > >
> > > > > Maybe that's not a issue. But still, /run contains lots of
> > > > > sockets of running services. That means now we can connect
> > > > > host services from the chroot environment. Even if it's not
> > > > > dangerous to host, it's polluting the new LFS system.
> > > > >
> > > > > Consider /run/initctl. We don't want something in chroot to
> > > > > switch the *host* to runlevel 1 :).
> > > >
> > > > What about
> > > >
> > > > mount -t tmpfs /run
> > > >
> > > > from within chroot?
> > > >
> > > > -- Bruce
> > >
> > > FWIIW, here are my findings: On a debian host, we have:
> > >
> > > lrwxrwxrwx 1 root root 8 mars 26 09:48 /run/shm -> /dev/shm
> > > drwxrwxrwt 2 root root 40 mars 26 11:02 /dev/shm
> > >
> > > But when mounting /dev to $LFS with mount --bind, we have
> > >
> > > drwxr-xr-x 2 root root 40 mars 26 11:01 /mnt/lfs/dev/shm
> > >
> > > that is, the sticky bit and the write perm are lost for an average
> > > user.
> > >
> > > Now trying to build FF in chroot (creating kernel virtual
> > > filesystems and chrooting with the lfs book instructions): - as
> > > root: needs to issue "export SHELL", but otherwise succeeds
> > > without anything else (/run exists as a tmpfs , but /run/shm does
> > > not exist) - as user: no need to run "export SHELL" (I guess "su -
> > > " does the export), but the perms to /dev/shm need to be
> > > changed to rwt
> > >
> > > In both cases, the build goes to completion (not tried to install)
> > > after the indicated changes.
> > >
> > > Another check I've done: on a lfs system (9.1-rc1 Sys V) as per
> > > the book, /dev/shm is drwxrwxrwt, and /run/shm does not exist. But
> > > everybody knows it is possible to build firefox Actually, if you
> > > just mount a devtmpfs:
> > >
> > > mkdir tempdir-for-trying-something sudo mount -t devtmpfs devtmpfs
> > > tempdir-for-trying-something ls -ld
> > > tempdir-for-trying-something/shm
> > >
> > > then the shm perms are drwxrwxrwt.
> > >
> > > So I'd suggest our instructions be changed to mount a devtmpfs in
> > > $LFS/dev
> > >
> > > But what makes me think I do something different from ĸen is that
> > > I never see: OSError: [Errno 38] Function not implemented
> > >
> > >
> > > Pierre
> > >
> > I'll reply in a while with what I have now tested (after I've
> > assembled my notes). I've gone with Bruce's suggestion to (only)
> > mount a tmpfs on /run when building in chroot - I think xry111 has
> > identified the issue (bind mounts are not recursive).
> >
> > Meanwhile, if you do not see that OSError when building any of firefox,
> > mozjs{60,68}, seamonkey, thunderbird *in chroot* then you are doing
> > something different. This issue was raised in
> > http://lists.linuxfromscratch.org/pipermail/blfs-support/2020-March/081897.html
>
> The error reported by the OP ("This platform lacks a functioning sem_open
> implementation...") is the one I see when /dev/shm is not world writable, and
> the build is done as a regular user.
> When building as root, the error is different (something about an unknown
> shell).
>
> Those are the only errors I see. Note that chroot is entered with the
> following commands (Makefile syntax):
> sudo mount -v --bind /dev $(MOUNT_PT)/dev; \
> sudo mount -vt devpts devpts $(MOUNT_PT)/dev/pts -o gid=5,mode=620; \
> sudo mount -vt proc proc $(MOUNT_PT)/proc; \
> sudo mount -vt sysfs sysfs $(MOUNT_PT)/sys; \
> sudo mount -vt tmpfs tmpfs $(MOUNT_PT)/run; \
> if [ -h $(MOUNT_PT)/dev/shm
Re: [blfs-dev] Python multiprocessing checks in chroot
Le 26/03/2020 à 14:41, Ken Moffat via blfs-dev a écrit :
> On Thu, Mar 26, 2020 at 11:40:59AM +0100, Pierre Labastie via
> blfs-dev wrote:
>> Le 23/03/2020 à 03:49, Bruce Dubbs via blfs-dev a écrit :
>>> On 3/22/20 9:25 PM, Xi Ruoyao via blfs-dev wrote:
On 2020-03-23 01:44 +, Ken Moffat via blfs-dev wrote:
> On Mon, Mar 23, 2020 at 09:12:18AM +0800, Xi Ruoyao via
> blfs-dev wrote:
>> On 2020-03-22 21:34 +, Ken Moffat via blfs-dev wrote:
>>
>>> # mount --bind /run /mnt/lfs/run
>>
>> I think it's dangerous: potentially harmful to the host.
>> Some service running in the LFS chroot may overwrite the
>> runtime directory of the service running on the host.
>
> So, you are saying that packages from mozilla should not now
> be built in chroot ?
No. I think we need a better way.
> But, what do you mean by a service running in chroot ? I
> assume we are specifically talking of systemd here ? Do
> services not get started during the boot process ? The
> systemd instance in chroot has never started, so I assume it
> will be ineffective and systemd on the host will only care
> about services described in /etc/systemd ?
Maybe that's not a issue. But still, /run contains lots of
sockets of running services. That means now we can connect
host services from the chroot environment. Even if it's not
dangerous to host, it's polluting the new LFS system.
Consider /run/initctl. We don't want something in chroot to
switch the *host* to runlevel 1 :).
>>>
>>> What about
>>>
>>> mount -t tmpfs /run
>>>
>>> from within chroot?
>>>
>>> -- Bruce
>>
>> FWIIW, here are my findings: On a debian host, we have:
>>
>> lrwxrwxrwx 1 root root 8 mars 26 09:48 /run/shm -> /dev/shm
>> drwxrwxrwt 2 root root 40 mars 26 11:02 /dev/shm
>>
>> But when mounting /dev to $LFS with mount --bind, we have
>>
>> drwxr-xr-x 2 root root 40 mars 26 11:01 /mnt/lfs/dev/shm
>>
>> that is, the sticky bit and the write perm are lost for an average
>> user.
>>
>> Now trying to build FF in chroot (creating kernel virtual
>> filesystems and chrooting with the lfs book instructions): - as
>> root: needs to issue "export SHELL", but otherwise succeeds
>> without anything else (/run exists as a tmpfs , but /run/shm does
>> not exist) - as user: no need to run "export SHELL" (I guess "su -
>> " does the export), but the perms to /dev/shm need to be
>> changed to rwt
>>
>> In both cases, the build goes to completion (not tried to install)
>> after the indicated changes.
>>
>> Another check I've done: on a lfs system (9.1-rc1 Sys V) as per
>> the book, /dev/shm is drwxrwxrwt, and /run/shm does not exist. But
>> everybody knows it is possible to build firefox Actually, if you
>> just mount a devtmpfs:
>>
>> mkdir tempdir-for-trying-something sudo mount -t devtmpfs devtmpfs
>> tempdir-for-trying-something ls -ld
>> tempdir-for-trying-something/shm
>>
>> then the shm perms are drwxrwxrwt.
>>
>> So I'd suggest our instructions be changed to mount a devtmpfs in
>> $LFS/dev
>>
>> But what makes me think I do something different from ĸen is that
>> I never see: OSError: [Errno 38] Function not implemented
>>
>>
>> Pierre
>>
> I'll reply in a while with what I have now tested (after I've
> assembled my notes). I've gone with Bruce's suggestion to (only)
> mount a tmpfs on /run when building in chroot - I think xry111 has
> identified the issue (bind mounts are not recursive).
>
> Meanwhile, if you do not see that OSError when building any of firefox,
> mozjs{60,68}, seamonkey, thunderbird *in chroot* then you are doing
> something different. This issue was raised in
> http://lists.linuxfromscratch.org/pipermail/blfs-support/2020-March/081897.html
The error reported by the OP ("This platform lacks a functioning sem_open
implementation...") is the one I see when /dev/shm is not world writable, and
the build is done as a regular user.
When building as root, the error is different (something about an unknown
shell).
Those are the only errors I see. Note that chroot is entered with the
following commands (Makefile syntax):
sudo mount -v --bind /dev $(MOUNT_PT)/dev; \
sudo mount -vt devpts devpts $(MOUNT_PT)/dev/pts -o gid=5,mode=620; \
sudo mount -vt proc proc $(MOUNT_PT)/proc; \
sudo mount -vt sysfs sysfs $(MOUNT_PT)/sys; \
sudo mount -vt tmpfs tmpfs $(MOUNT_PT)/run; \
if [ -h $(MOUNT_PT)/dev/shm ]; then :; \
sudo mkdir -pv $(MOUNT_PT)/$$(readlink $(MOUNT_PT)/dev/shm); \
fi;
#Since on debian, /dev/shm is not a symlink, the last mkdir is not executed.
#Note that a tmpfs is mounted on /run.
# Then
/usr/sbin/chroot $(MOUNT_PT) \
/usr/bin/env -i \
HOME=/root TERM="$$TERM" \
PS1='(lfs chroot) \u:\w\$$ ' \
PATH=/bin:/usr/bin:/sbin:/usr/sbin \
/bin/bash --login
(again Makefile
Re: [blfs-dev] Python multiprocessing checks in chroot
On Thu, Mar 26, 2020 at 11:40:59AM +0100, Pierre Labastie via
blfs-dev wrote:
> Le 23/03/2020 à 03:49, Bruce Dubbs via blfs-dev a écrit :
> > On 3/22/20 9:25 PM, Xi Ruoyao via blfs-dev wrote:
> >> On 2020-03-23 01:44 +, Ken Moffat via blfs-dev wrote:
> >>> On Mon, Mar 23, 2020 at 09:12:18AM +0800, Xi Ruoyao via
> >>> blfs-dev wrote:
> On 2020-03-22 21:34 +, Ken Moffat via blfs-dev wrote:
>
> > # mount --bind /run /mnt/lfs/run
>
> I think it's dangerous: potentially harmful to the host.
> Some service running in the LFS chroot may overwrite the
> runtime directory of the service running on the host.
> >>>
> >>> So, you are saying that packages from mozilla should not now
> >>> be built in chroot ?
> >>
> >> No. I think we need a better way.
> >>
> >>> But, what do you mean by a service running in chroot ? I
> >>> assume we are specifically talking of systemd here ? Do
> >>> services not get started during the boot process ? The
> >>> systemd instance in chroot has never started, so I assume it
> >>> will be ineffective and systemd on the host will only care
> >>> about services described in /etc/systemd ?
> >>
> >> Maybe that's not a issue. But still, /run contains lots of
> >> sockets of running services. That means now we can connect
> >> host services from the chroot environment. Even if it's not
> >> dangerous to host, it's polluting the new LFS system.
> >>
> >> Consider /run/initctl. We don't want something in chroot to
> >> switch the *host* to runlevel 1 :).
> >
> > What about
> >
> > mount -t tmpfs /run
> >
> > from within chroot?
> >
> > -- Bruce
>
> FWIIW, here are my findings: On a debian host, we have:
>
> lrwxrwxrwx 1 root root 8 mars 26 09:48 /run/shm -> /dev/shm
> drwxrwxrwt 2 root root 40 mars 26 11:02 /dev/shm
>
> But when mounting /dev to $LFS with mount --bind, we have
>
> drwxr-xr-x 2 root root 40 mars 26 11:01 /mnt/lfs/dev/shm
>
> that is, the sticky bit and the write perm are lost for an average
> user.
>
> Now trying to build FF in chroot (creating kernel virtual
> filesystems and chrooting with the lfs book instructions): - as
> root: needs to issue "export SHELL", but otherwise succeeds
> without anything else (/run exists as a tmpfs , but /run/shm does
> not exist) - as user: no need to run "export SHELL" (I guess "su -
> " does the export), but the perms to /dev/shm need to be
> changed to rwt
>
> In both cases, the build goes to completion (not tried to install)
> after the indicated changes.
>
> Another check I've done: on a lfs system (9.1-rc1 Sys V) as per
> the book, /dev/shm is drwxrwxrwt, and /run/shm does not exist. But
> everybody knows it is possible to build firefox Actually, if you
> just mount a devtmpfs:
>
> mkdir tempdir-for-trying-something sudo mount -t devtmpfs devtmpfs
> tempdir-for-trying-something ls -ld
> tempdir-for-trying-something/shm
>
> then the shm perms are drwxrwxrwt.
>
> So I'd suggest our instructions be changed to mount a devtmpfs in
> $LFS/dev
>
> But what makes me think I do something different from ĸen is that
> I never see: OSError: [Errno 38] Function not implemented
>
>
> Pierre
>
I'll reply in a while with what I have now tested (after I've
assembled my notes). I've gone with Bruce's suggestion to (only)
mount a tmpfs on /run when building in chroot - I think xry111 has
identified the issue (bind mounts are not recursive).
Meanwhile, if you do not see that OSError when building any of firefox,
mozjs{60,68}, seamonkey, thunderbird *in chroot* then you are doing
something different. This issue was raised in
http://lists.linuxfromscratch.org/pipermail/blfs-support/2020-March/081897.html
I'll document *how* I tested so that people can pick it apart if
they wish :)
ĸen
--
When alle is ſayed and all is done, ye must chooſe your faces wisely,
for soon enouff ye will be playing with fyre."
The Nice and Accurate Prophecies of Agnes Nutter, Prophecy 5004
--
http://lists.linuxfromscratch.org/listinfo/blfs-dev
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
On 2020-03-26 09:41 +0100, Pierre Labastie via blfs-dev wrote: > So my suspicion is that when we mount --bind /dev, we lose some permissions, > and those permissions are needed for FF. I'll have to try again without > /run/shm The problem is /dev/shm is a seperated tmpfs, instead of a normal directory in /dev. mount --bind is not recursive so in the bind mount /dev/shm is a normal directory, which is not writable. I suggest to simply mount a tmpfs on /dev/shm: mount --bind /dev $LFS/dev mount -v -t tmpfs tmpfs $LFS/dev/shm > Will report > Pierre -- Xi Ruoyao School of Aerospace Science and Technology, Xidian University -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
Le 23/03/2020 à 03:49, Bruce Dubbs via blfs-dev a écrit : > On 3/22/20 9:25 PM, Xi Ruoyao via blfs-dev wrote: >> On 2020-03-23 01:44 +, Ken Moffat via blfs-dev wrote: >>> On Mon, Mar 23, 2020 at 09:12:18AM +0800, Xi Ruoyao via blfs-dev wrote: On 2020-03-22 21:34 +, Ken Moffat via blfs-dev wrote: > # mount --bind /run /mnt/lfs/run I think it's dangerous: potentially harmful to the host. Some service running in the LFS chroot may overwrite the runtime directory of the service running on the host. >>> >>> So, you are saying that packages from mozilla should not now be >>> built in chroot ? >> >> No. I think we need a better way. >> >>> But, what do you mean by a service running in chroot ? I assume we >>> are specifically talking of systemd here ? Do services not get >>> started during the boot process ? The systemd instance in chroot >>> has never started, so I assume it will be ineffective and systemd on >>> the host will only care about services described in /etc/systemd ? >> >> Maybe that's not a issue. But still, /run contains lots of sockets of >> running >> services. That means now we can connect host services from the chroot >> environment. Even if it's not dangerous to host, it's polluting the new LFS >> system. >> >> Consider /run/initctl. We don't want something in chroot to switch the >> *host* >> to runlevel 1 :). > > What about > > mount -t tmpfs /run > > from within chroot? > > -- Bruce FWIIW, here are my findings: On a debian host, we have: lrwxrwxrwx 1 root root 8 mars 26 09:48 /run/shm -> /dev/shm drwxrwxrwt 2 root root 40 mars 26 11:02 /dev/shm But when mounting /dev to $LFS with mount --bind, we have drwxr-xr-x 2 root root 40 mars 26 11:01 /mnt/lfs/dev/shm that is, the sticky bit and the write perm are lost for an average user. Now trying to build FF in chroot (creating kernel virtual filesystems and chrooting with the lfs book instructions): - as root: needs to issue "export SHELL", but otherwise succeeds without anything else (/run exists as a tmpfs , but /run/shm does not exist) - as user: no need to run "export SHELL" (I guess "su - " does the export), but the perms to /dev/shm need to be changed to rwt In both cases, the build goes to completion (not tried to install) after the indicated changes. Another check I've done: on a lfs system (9.1-rc1 Sys V) as per the book, /dev/shm is drwxrwxrwt, and /run/shm does not exist. But everybody knows it is possible to build firefox Actually, if you just mount a devtmpfs: mkdir tempdir-for-trying-something sudo mount -t devtmpfs devtmpfs tempdir-for-trying-something ls -ld tempdir-for-trying-something/shm then the shm perms are drwxrwxrwt. So I'd suggest our instructions be changed to mount a devtmpfs in $LFS/dev But what makes me think I do something different from ĸen is that I never see: OSError: [Errno 38] Function not implemented Pierre -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
Re: [blfs-dev] Python multiprocessing checks in chroot
Le 23/03/2020 à 15:29, Ken Moffat via blfs-dev a écrit : > On Sun, Mar 22, 2020 at 09:49:02PM -0500, Bruce Dubbs via blfs-dev wrote: >> On 3/22/20 9:25 PM, Xi Ruoyao via blfs-dev wrote: >>> On 2020-03-23 01:44 +, Ken Moffat via blfs-dev wrote: On Mon, Mar 23, 2020 at 09:12:18AM +0800, Xi Ruoyao via blfs-dev wrote: > On 2020-03-22 21:34 +, Ken Moffat via blfs-dev wrote: So, you are saying that packages from mozilla should not now be built in chroot ? >>> >>> No. I think we need a better way. >>> But, what do you mean by a service running in chroot ? I assume we are specifically talking of systemd here ? Do services not get started during the boot process ? The systemd instance in chroot has never started, so I assume it will be ineffective and systemd on the host will only care about services described in /etc/systemd ? >>> >>> Maybe that's not a issue. But still, /run contains lots of sockets of >>> running >>> services. That means now we can connect host services from the chroot >>> environment. Even if it's not dangerous to host, it's polluting the new LFS >>> system. >>> >>> Consider /run/initctl. We don't want something in chroot to switch the >>> *host* >>> to runlevel 1 :). >> > > As a developer, I take the view that I'll do what I want with my > machine, including shooting myself in the foot :) For the book, > yes, I take your point. > >> What about >> >> mount -t tmpfs /run >> >> from within chroot? >> >> -- Bruce > > I suspect that what is actually needed is /run/shm. Will need to do > some testing and stracing. ... > FWIIW, I've succesfully built FF with these two instructions added to what is in lfs on the "Preparing Virtual Kernel File Systems" page: ln -s /dev/shm /run chmod a+wt /dev/shm Note that: The host is debian. Debian has: drwxrwxrwt 2 root root 40 mars 26 08:48 /dev/shm and lrwxrwxrwx 1 root root 8 mars 23 21:29 /run/shm -> /dev/shm but when /dev is (mount --bind)ed, /dev/shm becomes drwxr-xr-x 21 root root 3900 mars 23 21:29 /mnt/lfs/dev The change in permission on /dev/shm _is_ needed to allow the configuration to run. Note that I have not tried to just change the permission on /dev/shm and not symlinking /run/shm to /dev. Ah, I just launched a VM with lfs (9.1-rc1), and /dev/shm is exactly the same as above (a directory with rwt permissions for all). And there is no /run/shm... So my suspicion is that when we mount --bind /dev, we lose some permissions, and those permissions are needed for FF. I'll have to try again without /run/shm Will report Pierre -- http://lists.linuxfromscratch.org/listinfo/blfs-dev FAQ: http://www.linuxfromscratch.org/blfs/faq.html Unsubscribe: See the above information page
