Re: pkgsrc build hang of python38
On Tue, Mar 02, 2021 at 11:14:01AM +0100, Riccardo Mottola wrote: > oh... I use reboot... since 25 years or more... why not? It will not shut down all server processes gracefully. > If an unclean shutdown happens like a crash or powerloss? Entropy will be restored to the last state saved on reboot - that is either the state of last boot, or the last nightly run - or whenever you saved entropy state manually in between. Martin
Re: pkgsrc build hang of python38
Hi! Lloyd Parkes wrote: >> I don't think it is that user friendly to recularly "feed" this >> entropy... and it gets lost at the next reboot? Could we have a script >> on boot? > > The entropy is preserved across reboot by the shutdown hooks, so you > normally only need to do this once. If you have the bad habit (like I > do) of rebooting some systems with the "reboot" command instead of > "shutdown -r", then your entropy will not be preserved. Once I setup > some entropy and started using "shutdown -r" everything worked fine. oh... I use reboot... since 25 years or more... why not? If an unclean shutdown happens like a crash or powerloss? Riccardo
Re: pkgsrc build hang of python38
On 2/03/21 6:51 am, Riccardo Mottola wrote: I thouht your cited entropy fix was a "once time" not to be reissued again (when? at every reboot? kernel update?) I don't think it is that user friendly to recularly "feed" this entropy... and it gets lost at the next reboot? Could we have a script on boot? The entropy is preserved across reboot by the shutdown hooks, so you normally only need to do this once. If you have the bad habit (like I do) of rebooting some systems with the "reboot" command instead of "shutdown -r", then your entropy will not be preserved. Once I setup some entropy and started using "shutdown -r" everything worked fine. Cheers, Lloyd
Re: pkgsrc build hang of python38
Hi Lloyd and Chavdar, Lloyd Parkes wrote: > > On 28/02/21 9:50 am, Riccardo Mottola wrote: >> python38 hangs during build. I tried several times, it hangs in >> exactly the same place again. I don't know of course where the issue >> comes from, since all is new now :) > > Do you have any console messages? Maybe messages about blocking the > python process because of not enough entropy? > > That's a problem I encountered when I was building Python on current. > If you are working on the console, then it's all obvious, but if you > aren't on the console then it's all mysterious. > > If that is your problem, then generate some entropy with something > like "ls -lR /" and then run > > dd if=/dev/urandom of=/dev/random bs=32 count=1 > sysctl -w kern.entropy.consolidate=1 > > See > https://mail-index.netbsd.org/current-users/2020/05/01/msg038495.html > for all the details. yes, I have an entropy issue, console shows: [ 300.511173] entropy: pid 3383 (python) blocking due to lack of entropy I wasn't running on console because it is "spammed" with cron message pam_vprompt(): no conversation functions - dont know where they are coming from I remember this discussion past year, now that you tell me. But I it was related to the fact that random wasn't initialized well for certain specific CPU architectures, but then the technical discussion died there. I don't see any warnings in dmesg during boot that my CPU/Architecture is missing some support, so I wasn't thinking of it anymore. I thouht your cited entropy fix was a "once time" not to be reissued again (when? at every reboot? kernel update?) I don't think it is that user friendly to recularly "feed" this entropy... and it gets lost at the next reboot? Could we have a script on boot? Riccardo
Re: pkgsrc build hang of python38
On Sat, Feb 27, 2021 at 21:50:35 +0100, Riccardo Mottola wrote: > python38 hangs during build. I tried several times, it hangs in > exactly the same place again. I don't know of course where the issue > comes from, since all is new now :) I vaguely remember running into something like this b/c the system was out of randomness. I don't remember if there should be something obvious in dmesg when that happens, I think it should. As a workaround: https://mail-index.netbsd.org/current-users/2020/05/01/msg038495.html | you can fool it into thinking there is more entropy by running | | dd if=/dev/urandom of=/dev/random bs=32 count=1 | sysctl -w kern.entropy.consolidate=1 | | as root. The kernel will assume that any data written by root to | /dev/u?random have full entropy -uwe
Re: pkgsrc build hang of python38
On 28/02/21 9:50 am, Riccardo Mottola wrote: python38 hangs during build. I tried several times, it hangs in exactly the same place again. I don't know of course where the issue comes from, since all is new now :) Do you have any console messages? Maybe messages about blocking the python process because of not enough entropy? That's a problem I encountered when I was building Python on current. If you are working on the console, then it's all obvious, but if you aren't on the console then it's all mysterious. If that is your problem, then generate some entropy with something like "ls -lR /" and then run dd if=/dev/urandom of=/dev/random bs=32 count=1 sysctl -w kern.entropy.consolidate=1 See https://mail-index.netbsd.org/current-users/2020/05/01/msg038495.html for all the details. Cheers, Lloyd
Re: pkgsrc build hang of python38
On Sat, 27 Feb 2021 at 21:25, Chavdar Ivanov wrote: > > On Sat, 27 Feb 2021 at 20:51, Riccardo Mottola > wrote: > > > > Hi, > > > > after succesfully upgrading to 9.99.80 with a full distribution > > rebuild, I am also rebuilding pkgsrc!I did follow the pkgdb migration. > > > > python38 hangs during build. I tried several times, it hangs in exactly > > the same place again. I don't know of course where the issue comes from, > > since all is new now :) > > > > > > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/__init__.py > > /usr/pkg/lib/python3.8/pydoc_data > > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/_pydoc.css > > /usr/pkg/lib/python3.8/pydoc_data > > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/topics.py > > /usr/pkg/lib/python3.8/pydoc_data > > /usr/bin/install -c -o root -g wheel -m 644 `cat > > pybuilddir.txt`/_sysconfigdata_netbsd9.py > > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8; > > /usr/bin/install -c -o root -g wheel -m 644 ./LICENSE > > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/LICENSE.txt > > if test -d > > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/distutils/tests; > > then /usr/bin/install -c -o root -g wheel -m 644 ./Modules/xxmodule.c > > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/distutils/tests > > ; fi > > PYTHONPATH=/usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8 > > LD_LIBRARY_PATH=/usr/pkgsrc/lang/python38/work/Python-3.8.8 ./python -E > > -Wi > > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/compileall.py > > -j0 -d /usr/pkg/lib/python3.8 -f -x > > 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' > > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8 > > > > > > > > It hangs here with almost 0 cpu consumption, top shows a 99% idle > > system. I clean, retry and hang there again! > > > > When it happens again, try finding the exact process which is hanging > at the moment ( best is with htop in tree mode ), then attach to it > with gdb and run 'thread apply all bt' - I bet all threads (bar one) > are waiting... When you quit gdb, the process magically continues... > > That has been happening with my builds, mainly when I do > 'pkg_rolling-replace', all the time the last several months, I am sure > you can find the mail archives. i still don't know if it is only a > problem with the entropy or something with the thread processing. In > my case it happens most often when running cmake and rebuilding some > kde packages, but this happens elsewhere as well. > > > > > > I tried upgrading python27 but that pulls in python38, so I hang > > again and if I try updating other packages I will end up tehre, > > consistent hang- See https://mail-index.netbsd.org/current-users/2020/05/24/msg038692.html thread for example. > > > > Riccardo > > > > > -- > --
Re: pkgsrc build hang of python38
On Sat, 27 Feb 2021 at 20:51, Riccardo Mottola wrote: > > Hi, > > after succesfully upgrading to 9.99.80 with a full distribution > rebuild, I am also rebuilding pkgsrc!I did follow the pkgdb migration. > > python38 hangs during build. I tried several times, it hangs in exactly > the same place again. I don't know of course where the issue comes from, > since all is new now :) > > > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/__init__.py > /usr/pkg/lib/python3.8/pydoc_data > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/_pydoc.css > /usr/pkg/lib/python3.8/pydoc_data > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/topics.py > /usr/pkg/lib/python3.8/pydoc_data > /usr/bin/install -c -o root -g wheel -m 644 `cat > pybuilddir.txt`/_sysconfigdata_netbsd9.py > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8; > /usr/bin/install -c -o root -g wheel -m 644 ./LICENSE > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/LICENSE.txt > if test -d > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/distutils/tests; > then /usr/bin/install -c -o root -g wheel -m 644 ./Modules/xxmodule.c > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/distutils/tests > ; fi > PYTHONPATH=/usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8 > LD_LIBRARY_PATH=/usr/pkgsrc/lang/python38/work/Python-3.8.8 ./python -E > -Wi > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/compileall.py > -j0 -d /usr/pkg/lib/python3.8 -f -x > 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' > /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8 > > > > It hangs here with almost 0 cpu consumption, top shows a 99% idle > system. I clean, retry and hang there again! > When it happens again, try finding the exact process which is hanging at the moment ( best is with htop in tree mode ), then attach to it with gdb and run 'thread apply all bt' - I bet all threads (bar one) are waiting... When you quit gdb, the process magically continues... That has been happening with my builds, mainly when I do 'pkg_rolling-replace', all the time the last several months, I am sure you can find the mail archives. i still don't know if it is only a problem with the entropy or something with the thread processing. In my case it happens most often when running cmake and rebuilding some kde packages, but this happens elsewhere as well. > I tried upgrading python27 but that pulls in python38, so I hang > again and if I try updating other packages I will end up tehre, > consistent hang- > > Riccardo > --
pkgsrc build hang of python38
Hi, after succesfully upgrading to 9.99.80 with a full distribution rebuild, I am also rebuilding pkgsrc!I did follow the pkgdb migration. python38 hangs during build. I tried several times, it hangs in exactly the same place again. I don't know of course where the issue comes from, since all is new now :) /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/__init__.py /usr/pkg/lib/python3.8/pydoc_data /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/_pydoc.css /usr/pkg/lib/python3.8/pydoc_data /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/topics.py /usr/pkg/lib/python3.8/pydoc_data /usr/bin/install -c -o root -g wheel -m 644 `cat pybuilddir.txt`/_sysconfigdata_netbsd9.py /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8; /usr/bin/install -c -o root -g wheel -m 644 ./LICENSE /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/LICENSE.txt if test -d /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/distutils/tests; then /usr/bin/install -c -o root -g wheel -m 644 ./Modules/xxmodule.c /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/distutils/tests ; fi PYTHONPATH=/usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8 LD_LIBRARY_PATH=/usr/pkgsrc/lang/python38/work/Python-3.8.8 ./python -E -Wi /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8/compileall.py -j0 -d /usr/pkg/lib/python3.8 -f -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' /usr/pkgsrc/lang/python38/work/.destdir/usr/pkg/lib/python3.8 It hangs here with almost 0 cpu consumption, top shows a 99% idle system. I clean, retry and hang there again! I tried upgrading python27 but that pulls in python38, so I hang again and if I try updating other packages I will end up tehre, consistent hang- Riccardo
