Bug#738575: pthread: segfault in libpthread on Intel Galileo board
Followup-For: Bug #738575 X-Debbugs-Cc: raykinsell...@gmail.com On Thu, 16 Nov 2023 12:47:40 +, James wrote: > I've been thinking more about how to improve the chances that the > package could be accepted into Debian -- my suggestion would be to > rebuild it and upload it to the mentors[1] portal, where it can > (hopefully) receive review. I've considered uploading it myself, but > I don't have hardware to test the results on, so I'd be navigating > without a way to test the results. From personal experience > attempting packaging: the mentoring guidelines and making sure to run > lintian checks are both worthwhile. I now have an X1000 Quark board to test this on (thanks, Ray), and am hoping to find some time over the next week or two to try that out in combination with the libx1000.git source-and-packaging repo.
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
On Thu, 16 Nov 2023 at 09:57, Ray Kinsella wrote: > > On Wed, 15 Nov 2023 at 22:30, James Addison wrote: >> >> On Wed, 15 Nov 2023 at 21:57, Ray Kinsella wrote: > [...] > I spent a not insignificant amount of time devising this solution, to get > "Debian Support" > After a few false starts and missteps, eventually I came up with LibX1000 > which was a pretty effective fix IMHO. > > When I started sharing it around with the Debian & Kernel folks - the > response was pretty clear. > "This is a mess, Intel should just fix the bug ... " - which honestly in > retrospect was the right thing to do. Yep; frustrating though it can be, pushing back to figure out the origins of bugs and resolve them there is likely the way to free up enough developer and maintainer time, and to improve hardware and software quality enough, to reach Reliable Technology Utopia.. should be any day now :) >> > It's been a long time since Intel manufactured the X1000 / Quark, I am not >> > sure how many are left in the wild. >> > Do you think this is something that Debian might want to package and ship? >> >> I should admit that I'm not a Debian maintainer or developer, just a >> passerby who attempts to make progress on bugs that interest to me >> (possibly to the annoyance of actual DMs/DDs), so my opinion is >> somewhat external (i.e.: take with a grain of salt). > > > Thrilled that you reached out about it. I've been thinking more about how to improve the chances that the package could be accepted into Debian -- my suggestion would be to rebuild it and upload it to the mentors[1] portal, where it can (hopefully) receive review. I've considered uploading it myself, but I don't have hardware to test the results on, so I'd be navigating without a way to test the results. From personal experience attempting packaging: the mentoring guidelines and making sure to run lintian checks are both worthwhile. Even so there'd be no guarantee of review or upload acceptance -- and unfortunately the same test-hardware limitation probably applies to most reviewers -- so I don't know whether it'd be worth your time, but in my mind it's possible that someone attempts to install Debian on an X1000 platform in future, learns of this bug, and then in a hypothetical future _might_ find libx1000 in the archive, and then be grateful for the availalble fix. (after re-reading your blog-post, I think that there could technically still be rare cases where the bug appears despite the package being installed -- the mention of 98% of cases handled -- but even so, a mostly-usable system compared to a completely useless one seems like a big improvement) [1] - https://mentors.debian.net/
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
On Wed, 15 Nov 2023 at 22:30, James Addison wrote: > On Wed, 15 Nov 2023 at 21:57, Ray Kinsella > wrote: > > > > Thanks for this - you are kind to look at this issue. > > You're welcome - I enjoyed learning a bit about the Quark hardware, > and the esoteric lock bug. A shame I didn't learn about it five years > ago I suppose, but there we are. > > I spent a not insignificant amount of time devising this solution, to get "Debian Support" After a few false starts and missteps, eventually I came up with LibX1000 which was a pretty effective fix IMHO. When I started sharing it around with the Debian & Kernel folks - the response was pretty clear. "This is a mess, Intel should just fix the bug ... " - which honestly in retrospect was the right thing to do. > > It's been a long time since Intel manufactured the X1000 / Quark, I am > not sure how many are left in the wild. > > Do you think this is something that Debian might want to package and > ship? > > I should admit that I'm not a Debian maintainer or developer, just a > passerby who attempts to make progress on bugs that interest to me > (possibly to the annoyance of actual DMs/DDs), so my opinion is > somewhat external (i.e.: take with a grain of salt). Thrilled that you reached out about it. > It's entirely > possible that the maintenance for an additional package wouldn't be > worthwhile -- and in general, 32-bit x86 support in Debian does tend > to be dwindling. Basically: someone has to be motivated about it > enough to be responsible for the package. > [SNIP] > Do you know whether Intel shipped many Quark units? I see that > manufacturing stopped in Y2019, which isn't too long ago, but I don't > know much about how widely-adopted they were. There were a few micro[processors,controllers] shipped under Quark. My memory is that the X1000 didn't last long beyond 2017. I remember seeing stacks of them (Galileo Boards) sitting gathering dust in Frys and Maplin. So I couldn't say how many are in the wild being used. It was the > energy-efficiency focus of them that gathered my interest in the first > place, FWIW. > > Boards like the Up-board (https://up-board.org/) and its successors really filled this gap more effectively > > On Wed, 15 Nov 2023 at 12:27, James Addison wrote: > >> > >> Followup-For: Bug #738575 > >> X-Debbugs-Cc: raykinsell...@gmail.com > >> > >> If I understand correctly, then Ray's libx1000 library[1] provides a > way to > >> work around this in software. It uses some LD_PRELOAD magic, and from > what I > >> remember, it's worth being careful when using that approach. > >> > >> I opened an RFP[2] for libx1000 earlier this year, and took another > look at the > >> Debian packaging metadata in the codebase today, resulting in a few > suggested > >> edits as a pull request on GitHub - cc'ing you to notify you about > that, Ray. > >> I'm unsure whether some of the proposed postinst steps are required, > and will > >> ask you about those upstream too. > >> > >> [1] - > http://ashroe.eu/x1000/2016/10/21/fixing-lock-prefix-on-x1000.html > >> > >> [2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037070 >
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
On Wed, 15 Nov 2023 at 21:57, Ray Kinsella wrote: > > Thanks for this - you are kind to look at this issue. You're welcome - I enjoyed learning a bit about the Quark hardware, and the esoteric lock bug. A shame I didn't learn about it five years ago I suppose, but there we are. > It's been a long time since Intel manufactured the X1000 / Quark, I am not > sure how many are left in the wild. > Do you think this is something that Debian might want to package and ship? I should admit that I'm not a Debian maintainer or developer, just a passerby who attempts to make progress on bugs that interest to me (possibly to the annoyance of actual DMs/DDs), so my opinion is somewhat external (i.e.: take with a grain of salt). It's entirely possible that the maintenance for an additional package wouldn't be worthwhile -- and in general, 32-bit x86 support in Debian does tend to be dwindling. Basically: someone has to be motivated about it enough to be responsible for the package. On the other hand: it seemed to me based on a quick look that much of the packaging work is already in place, and that this package would be opt-in for anyone who wanted to use it. The typical use case would be people preparing root filesystems on removable/replicable storage for later (re)attachment to Quark systems, I'd guess. Even so: the LD_PRELOAD and GRUB commandline stuff does make me a bit wary - generally we shouldn't make any unnecessary or unexpected modifications to the target system, because those should be the responsibility of the sysadmin and not of the maintainer. Do you know whether Intel shipped many Quark units? I see that manufacturing stopped in Y2019, which isn't too long ago, but I don't know much about how widely-adopted they were. It was the energy-efficiency focus of them that gathered my interest in the first place, FWIW. > On Wed, 15 Nov 2023 at 12:27, James Addison wrote: >> >> Followup-For: Bug #738575 >> X-Debbugs-Cc: raykinsell...@gmail.com >> >> If I understand correctly, then Ray's libx1000 library[1] provides a way to >> work around this in software. It uses some LD_PRELOAD magic, and from what I >> remember, it's worth being careful when using that approach. >> >> I opened an RFP[2] for libx1000 earlier this year, and took another look at >> the >> Debian packaging metadata in the codebase today, resulting in a few suggested >> edits as a pull request on GitHub - cc'ing you to notify you about that, Ray. >> I'm unsure whether some of the proposed postinst steps are required, and will >> ask you about those upstream too. >> >> [1] - http://ashroe.eu/x1000/2016/10/21/fixing-lock-prefix-on-x1000.html >> >> [2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037070
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
Hi James, Thanks for this - you are kind to look at this issue. It's been a long time since Intel manufactured the X1000 / Quark, I am not sure how many are left in the wild. Do you think this is something that Debian might want to package and ship? Thanks, Ray K On Wed, 15 Nov 2023 at 12:27, James Addison wrote: > Followup-For: Bug #738575 > X-Debbugs-Cc: raykinsell...@gmail.com > > If I understand correctly, then Ray's libx1000 library[1] provides a way to > work around this in software. It uses some LD_PRELOAD magic, and from > what I > remember, it's worth being careful when using that approach. > > I opened an RFP[2] for libx1000 earlier this year, and took another look > at the > Debian packaging metadata in the codebase today, resulting in a few > suggested > edits as a pull request on GitHub - cc'ing you to notify you about that, > Ray. > I'm unsure whether some of the proposed postinst steps are required, and > will > ask you about those upstream too. > > [1] - http://ashroe.eu/x1000/2016/10/21/fixing-lock-prefix-on-x1000.html > > [2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037070 >
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
Followup-For: Bug #738575 X-Debbugs-Cc: raykinsell...@gmail.com If I understand correctly, then Ray's libx1000 library[1] provides a way to work around this in software. It uses some LD_PRELOAD magic, and from what I remember, it's worth being careful when using that approach. I opened an RFP[2] for libx1000 earlier this year, and took another look at the Debian packaging metadata in the codebase today, resulting in a few suggested edits as a pull request on GitHub - cc'ing you to notify you about that, Ray. I'm unsure whether some of the proposed postinst steps are required, and will ask you about those upstream too. [1] - http://ashroe.eu/x1000/2016/10/21/fixing-lock-prefix-on-x1000.html [2] - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037070
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
What version of glibc are you compiling? Can you offer some more assistance as to how you compiled it? Are you cross-compiling or do have a working set of tools on your galileo to compile natively? I am attempting to compile from my Ubuntu 14.04 (64-bit) machine with ../glibc-2.20/configure -with-cpu=i386 And am receiving checking sysdep dirs... configure: error: The i386 subspecies of x86_64 is not supported. Thanks, David On Fri, 09 May 2014 00:04:30 +0200 Jan Just Keijser janj...@nikhef.nl wrote: I did find a workaround, however: if I configure glibc with the extra parameter -with-cpu=i386
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
Hi, FWIW: I've got a Galileo board and am running into similar issuse (while running CentOS 5 on it ;)) Exact same error at the exact same location; it's not the instruction that is not supported , a very small piece of test code that does lock cmpxchgl %edx, (%eax) works just fine - and this instruction is used left and right in the vmlinux kernel itself anyway. So it must be the memory offset that is triggering error ... JJK / Jan Just Keijser -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
On Thu, May 08, 2014 at 04:45:00PM +0200, Jan Just Keijser wrote: Hi, FWIW: I've got a Galileo board and am running into similar issuse (while running CentOS 5 on it ;)) Exact same error at the exact same location; it's not the instruction that is not supported , a very small piece of test code that does lock cmpxchgl %edx, (%eax) This instruction might work in some cases, but not in all cases. works just fine - and this instruction is used left and right in the vmlinux kernel itself anyway. So it must be the memory offset that is triggering error ... No it's not. It is a CPU bug, as stated by Intel in their release notes (p 15), which only happens in specific circumstances. http://downloadmirror.intel.com/23197/eng/Quark_SW_RelNotes_330232_001.pdf -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
Hi, On 08/05/14 22:11, Aurelien Jarno wrote: On Thu, May 08, 2014 at 04:45:00PM +0200, Jan Just Keijser wrote: Hi, FWIW: I've got a Galileo board and am running into similar issuse (while running CentOS 5 on it ;)) Exact same error at the exact same location; it's not the instruction that is not supported , a very small piece of test code that does lock cmpxchgl %edx, (%eax) This instruction might work in some cases, but not in all cases. works just fine - and this instruction is used left and right in the vmlinux kernel itself anyway. So it must be the memory offset that is triggering error ... No it's not. It is a CPU bug, as stated by Intel in their release notes (p 15), which only happens in specific circumstances. http://downloadmirror.intel.com/23197/eng/Quark_SW_RelNotes_330232_001.pdf OK I hadn't thought of reading the Quark release notes ;) I did find a workaround, however: if I configure glibc with the extra parameter -with-cpu=i386 then the assembly code generated becomes __nptl_setxid: .LFB75: .loc 1 1046 0 .LVL87: pushl %ebp .LCFI27: movl%esp, %ebp .LCFI28: pushl %edi .LCFI29: pushl %esi .LCFI30: pushl %ebx .LCFI31: subl$8, %esp .LCFI32: .LBB79: .loc 1 1049 0 xorl%eax, %eax movl$1, %ecx #APP lock;cmpxchgl %ecx, stack_cache_lock jnz _L_lock_698 .subsection 1 .type _L_lock_698,@function and the segfault does not occur - I can now happily run a 'stock' centos 5 sshd on my galileo board. Here's my setup: - yocto kernel 3.8.7 (from BSP 0.7.5) - clanton image based on centos5 i386 image, using stock rpms - modified /etc/rc.d/rc.sysinit script (don't try to mount /proc/bus/usb) - modified /lib/libpthread.so.0 symlink to custom libpthread.so (I'm sure I've missed a few other things ;) share and enjoy, JJK / Jan Just Keijser -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
On Mon, Jan 01, 2001 at 12:23:05AM +, Thomas Faust wrote: Package: libc6 Version: 2.13-38 Severity: normal File: pthread Dear Maintainer, I bootstrapped a Debian base system via debootstrap --arch i386 wheezy ./newfiles http://http.debian.net/debian/; and put it on a Galileo board. On the Galileo board there new Intel Quark IA processor - which is basically a 486 with some instructions extensions from Pentium. If I boot the existing Galileo kernel with the bootstrapped fileset, many applications crash with a segfault in pthread. To reporduce, follow the instruction on https://communities.intel.com/message/220080 Here are ways to reproduce consistently - many other apps show the same behavior 1. Boot the system, create a new user (non root), connect to the board via ssh - the sshd will crash with a segfault in pthread 2. Do a 'apt-get install cowsay' - at the end, apt-get will crash with a segfault in pthread sshd[2519]: segfault at b7173107 ip b714f07b sp bf97ea94 error 0007 in libpthread-2.13.so[b714a000+15000] incorrect behavior: segfault - applications stop working expected behavior: no crash uname -a = Linux galileo 3.8.7-yocto-standard #1 Wed Jan 15 00:21:32 CET 2014 i586 GNU/Linux dpkg -s libc6 = 2.13-38 The problem happens in __nptl_setxid, at address 0x507b: 5060 __nptl_setxid: 5060: 55 push %ebp 5061: 31 c0 xor%eax,%eax 5063: 89 e5 mov%esp,%ebp 5065: b9 01 00 00 00 mov$0x1,%ecx 506a: 57 push %edi 506b: 56 push %esi 506c: 53 push %ebx 506d: 83 ec 14sub$0x14,%esp 5070: e8 fb f3 ff ff call 4470 __i686.get_pc_thunk.bx 5075: 81 c3 7f 0f 01 00 add$0x10f7f,%ebx = 507b: f0 0f b1 8b 94 21 00lock cmpxchg %ecx,0x2194(%ebx) 5082: 00 5083: 0f 85 b6 17 00 00 jne683f _L_lock_743 5089: 8b 45 08mov0x8(%ebp),%eax 508c: 8b b3 38 01 00 00 mov0x138(%ebx),%esi Despite the name __i686.get_pc_thunk.bx is fine on this CPU (it actually has been rename to __x86.get_pc_thunk.bx on recent GCC versions), as it is only get the PC through the stack with a movl instruction: 4470 __i686.get_pc_thunk.bx: 4470: 8b 1c 24mov(%esp),%ebx 4473: c3 ret So the question is if the lock cmpxchg instruction is behaving correctly on the Intel Quark. This should be supported according to the developer's manual. It might be difficult to investigate more without access to such a CPU. -- Aurelien Jarno GPG: 1024D/F1BCDB73 aurel...@aurel32.net http://www.aurel32.net -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
The default toolchain (and thus libc) in Debian has been targetting i586 for quite a while now. If this CPU doesn't provide *all* the i586 instructions, I'd be pretty surprised if anything worked. I was first trending in the same direction, but this is a different issue. I compiled code with instructions that aren't working on Quark. The issue you get then is an 'invalid instruction exception', not a segfault. If you use gcc and the '-march=i586' option, the code that is generated is fine for Quark. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
Package: libc6 Version: 2.13-38 Severity: normal File: pthread Dear Maintainer, I bootstrapped a Debian base system via debootstrap --arch i386 wheezy ./newfiles http://http.debian.net/debian/; and put it on a Galileo board. On the Galileo board there new Intel Quark IA processor - which is basically a 486 with some instructions extensions from Pentium. If I boot the existing Galileo kernel with the bootstrapped fileset, many applications crash with a segfault in pthread. To reporduce, follow the instruction on https://communities.intel.com/message/220080 Here are ways to reproduce consistently - many other apps show the same behavior 1. Boot the system, create a new user (non root), connect to the board via ssh - the sshd will crash with a segfault in pthread 2. Do a 'apt-get install cowsay' - at the end, apt-get will crash with a segfault in pthread sshd[2519]: segfault at b7173107 ip b714f07b sp bf97ea94 error 0007 in libpthread-2.13.so[b714a000+15000] incorrect behavior: segfault - applications stop working expected behavior: no crash uname -a = Linux galileo 3.8.7-yocto-standard #1 Wed Jan 15 00:21:32 CET 2014 i586 GNU/Linux dpkg -s libc6 = 2.13-38 -- System Information: Debian Release: 7.3 APT prefers stable APT policy: (500, 'stable') Architecture: i386 (i586) Kernel: Linux 3.8.7-yocto-standard Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Shell: /bin/sh linked to /bin/dash Versions of packages libc6:i386 depends on: ii libc-bin 2.13-38 ii libgcc1 1:4.7.2-5 Versions of packages libc6:i386 recommends: pn libc6-i686 none Versions of packages libc6:i386 suggests: ii debconf [debconf-2.0] 1.5.49 pn glibc-doc none pn localesnone -- debconf information: glibc/restart-services: libraries/restart-without-asking: false glibc/disable-screensaver: glibc/upgrade: true glibc/restart-failed: -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#738575: pthread: segfault in libpthread on Intel Galileo board
On Mon, Jan 01, 2001 at 12:23:05AM +, Thomas Faust wrote: which is basically a 486 with some instructions extensions from Pentium. The default toolchain (and thus libc) in Debian has been targetting i586 for quite a while now. If this CPU doesn't provide *all* the i586 instructions, I'd be pretty surprised if anything worked. ... Adam -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
