[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[STINNER Victor]

STINNER Victor  added the comment:

I looked at this old issue. First of all, "bug.py" filename is accurate: this 
program contains a bug, but it's non-obvious.

* the main thread opens a file which gets the file descriptor 3
* a new thread is spawns which indirectly closes this file descriptor on: 
sys.stdout.close()
* the main thread is not aware that the file has been closed, and so will try 
to close the same file descriptor 3 sometime during Python shutdown
* then, sometimes, the glibc fails with an internal error because the main 
thread closed a file descriptor which was just opened by the other thread to 
implement thread cancellation: that logs "libgcc_s.so.1 must be installed for 
pthread_cancel to work" and calls abort()

Note: the Python file object remains alive until the "t" local variable is 
cleared, since "t.fh" contains a strong reference to the file object.

Again: it's a bug in the program. "libgcc_s.so.1 must be installed for 
pthread_cancel to work" is just *one* example of bad thing that can happen.

What can be do on the Python side? Well, stop to ignore silently I/O errors 
when a file is closed by its destructor.

I wrote a shy PR 12786 to only log these exceptions in development mode (-X 
dev). I'm not sure if we could stop to silence these exceptions *by default*.

Python has been modified last years to first raise an exception on I/O errors 
when file.close() is called explicitly, and then a similar change has been done 
for call to socket.socket.close().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +12713
stage: resolved -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Zack Weinberg]

Zack Weinberg  added the comment:

I have observed this problem in a production application using Python 3.5 and 
3.6 (system-packaged interpreters from Ubuntu) and I would like to mention that 
this is an effective workaround:

```
import ctypes
libgcc_s = ctypes.CDLL("libgcc_s.so.1")
```

provided you do it before creating any threads, and the variable `libgcc_s` 
remains live until after all threads are terminated.

--
nosy: +zwol

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Barry Davis]

Barry Davis added the comment:

I meant my cross compiled python, not my cross compiler.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Barry Davis]

Barry Davis added the comment:

Looks like I was mistaken. My cross compiler was trying to load libgcc_s.so.1 
from the standard location and not liking the one it found. Fixed for now by 
setting LD_LIBRARY_PATH to point at the dir containing the right libgcc_s.so.1

--
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Barry Davis]

Barry Davis added the comment:

I think this is the same issue I'm getting.
I'm hitting it when compiling python 3.6.2 compiled with gcc-4.8.4.
This wasn't occasional, it was every time I tried.
As a feeble workaround I was compiling in parallel, then in serial when it 
fails, although I'm now hitting the same issue when building uwsgi-2.0.15.

I'm building on a 56 core system so that might make me more likely to hit the 
issue.

--
nosy: +Barry Davis

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Barry Davis]

Changes by Barry Davis :


--
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Nikolay Bryskin]

Changes by Nikolay Bryskin :


--
nosy: +nikicat

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Charles-François Natali]

Charles-François Natali added the comment:

> On Linux, you can try to set the LD_PRELOAD environment variable as a
> workaround.
>
> LD_PRELOAD=libgcc_s.so.1 python bug.py
>
> You may need to specify the full path.

I don't think that'll work.
Despite its name, using LD_PRELOAD won't "preload" the library. It
will only be loaded upon dlopen(). It just makes sure that symbols
will be looked for in this library first, even before the libc.

> Because in python2.x it wasn't loaded at runtime.

Yes it was. As explained above, you can get the very same crash upon
pthread_exit().

> Alright ... would it be a very big hack to preload libgcc in the thread 
> module (at import
> time) ?

IMO yes, but if someone writes a patch, I won't oppose to it :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Maries Ionel Cristian]

Maries Ionel Cristian added the comment:

Alright ... would it be a very big hack to preload libgcc in the thread module 
(at import time) ? There is platform specific code there anyway, it wouldn't be 
such a big deal would it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[STINNER Victor]

STINNER Victor added the comment:

2013/8/18 Maries Ionel Cristian :
> Well anyway, is there any way to preload libgcc ? Because in python2.x it 
> wasn't loaded at runtime.

On Linux, you can try to set the LD_PRELOAD environment variable as a
workaround.

LD_PRELOAD=libgcc_s.so.1 python bug.py

You may need to specify the full path.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Maries Ionel Cristian]

Maries Ionel Cristian added the comment:

Well anyway, is there any way to preload libgcc ? Because in python2.x it 
wasn't loaded at runtime.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[STINNER Victor]

STINNER Victor added the comment:

2013/8/17 Maries Ionel Cristian :
> I don't think it's that obscure ... uwsgi has this issue
> https://www.google.com/search?q=libgcc_s.so.1+must+be+installed+for+pthread_cancel+to+work+uwsgi+site:lists.unbit.it-
> they cause it probably different but the point is that it's not
> obscure.

The error message is maybe the same, the reason is completly different:
http://lists.unbit.it/pipermail/uwsgi/2013-July/006213.html
"IT WAS A limit-as issue"

The root cause is an arbitrary limit on the size of the virtual memory.

Here the bug is that a thread B closes a file descriptor opened in a
thread B by the glibc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Maries Ionel Cristian]

Maries Ionel Cristian added the comment:

Correct link 
https://www.google.com/search?q=libgcc_s.so.1+must+be+installed+for+pthread_cancel+to+work+uwsgi+site:lists.unbit.it

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Maries Ionel Cristian]

Maries Ionel Cristian added the comment:

> What is the version of your libc library? Try something like "dpkg -l
> libc6".
>

 2.15-0ubuntu10.4

I don't think it's that obscure ... uwsgi has this issue
https://www.google.com/search?q=libgcc_s.so.1+must+be+installed+for+pthread_cancel+to+work+uwsgi+site:lists.unbit.it-
they cause it probably different but the point is that it's not
obscure.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Charles-François Natali]

Charles-François Natali added the comment:

> Perhaps the only thing we could do would be try to "preload" libgcc by
> calling one of those APIs at startup?

Yeah, I was thinking about doing this in PyThread_init_thread() but...

> But I'm not sure it's a good idea to
> add such a platform-specific hack (for what is arguably an obscure and rare
> issue).

then thought the exact same thing.

Also, this exact problem can possibly show up anywhere dlopen() is
used (a grep through glibc code shows at least an occurrence in
nsswitch-related code).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[STINNER Victor]

STINNER Victor added the comment:

Oh ok, I'm able to reproduce the issue with the system Python 3.3:

$ while true; do echo "loop"; python3.3 bug.py || break; done
loop
...
loop
libgcc_s.so.1 must be installed for pthread_cancel to work
Abandon (core dumped)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[STINNER Victor]

STINNER Victor added the comment:

Maries Ionel Cristian> ubuntu 12.04.2

What is the version of your libc library? Try something like "dpkg -l libc6".

--

Could this issue be related to this glibc issue?
http://sourceware.org/bugzilla/show_bug.cgi?id=2644

I ran your script on Python 3.4 in a shell loop: I'm unable to reproduce the 
issue. I'm running Fedora 18 with the glibc 2.16.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Antoine Pitrou]

Antoine Pitrou added the comment:

Perhaps the only thing we could do would be try to "preload" libgcc by calling 
one of those APIs at startup? But I'm not sure it's a good idea to add such a 
platform-specific hack (for what is arguably an obscure and rare issue).

--
nosy: +pitrou
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Charles-François Natali]

Charles-François Natali added the comment:

Unfortunately, there's not much we can do about it: if dlsym() fails - which is 
the case here either because read() fails with EBADF, or because the file 
descriptor now points to another stream (i.e. not libgcc), the libc aborts 
(here upon pthread_exit(), not PyThread_delete_key()):
"""
libgcc_s.so.1 must be installed for pthread_cancel to work

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xb7b0eb70 (LWP 17152)]
0xb7fe1424 in __kernel_vsyscall ()
(gdb) bt
#0  0xb7fe1424 in __kernel_vsyscall ()
#1  0xb7e4e941 in *__GI_raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb7e51d72 in *__GI_abort () at abort.c:92
#3  0xb7e8ae15 in __libc_message (do_abort=1, fmt=0xb7f606f5 "%s") at 
../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0xb7e8af44 in *__GI___libc_fatal (message=0xb7fc75ec "libgcc_s.so.1 must be 
installed for pthread_cancel to work\n") at 
../sysdeps/unix/sysv/linux/libc_fatal.c:200
#5  0xb7fc4ffa in pthread_cancel_init () at 
../nptl/sysdeps/pthread/unwind-forcedunwind.c:65
#6  0xb7fc509d in _Unwind_ForcedUnwind (exc=0xb7b0edc0, stop=0xb7fc2bf0 
, stop_argument=0xb7b0e454) at 
../nptl/sysdeps/pthread/unwind-forcedunwind.c:126
#7  0xb7fc2b98 in *__GI___pthread_unwind (buf=) at unwind.c:130
#8  0xb7fbcce0 in __do_cancel () at pthreadP.h:265
#9  __pthread_exit (value=0x0) at pthread_exit.c:30
#10 0x08132ced in PyThread_exit_thread () at Python/thread_pthread.h:266
#11 0x08137c37 in t_bootstrap (boot_raw=0x8318aa8) at 
./Modules/_threadmodule.c:1023
#12 0xb7fbbc39 in start_thread (arg=0xb7b0eb70) at pthread_create.c:304
#13 0xb7ef978e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
"""

So if you're unlucky and end up closing the FD referring to libgcc used by 
dlopen/dlsym, you're pretty much screwed, and there's no way around this.

Note that you specific problem (upon PyThread_delete_key()) doesn't occur for 
python 2.7 because it uses and ad-hoc TLS implementation, whereas Python 3 uses 
the platform native TLS.
But as noted above, you can very well get an abort on pthread_exit(), and in 
likely many other places (pretty much everywhere libgcc can be dlopen'ed).

Unfortunately, we can't do much against this, so I'm tempted to close this as 
"wont fix".

--
nosy: +neologix
stage:  -> committed/rejected
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18748] libgcc_s.so.1 must be installed for pthread_cancel to work

[Maries Ionel Cristian]

New submission from Maries Ionel Cristian:

Running the file couple of times will make the interpreter fail with: 
libgcc_s.so.1 must be installed for pthread_cancel to work

>From what I've seen it is triggered from PyThread_delete_key (tries to load 
>libgcc_s.so at that time).

How does it happen?

The main thread will close fd 4 (because fh object is getting dereferenced to 
0) exactly at the same time libpthread will try to open and read libgcc_s.so 
with the same descriptor (4)

It's fairly obvious that the file handling in bug.py is a bug, but the 
interpreter should not crash like that !

This doesn't happen on python2.7. Also, python2.7 appears to be linked with 
libgcc_s.so.1 directly while the 3.x does not (I've tried 3.2 from ubuntu 
repos, and built 3.3 and 3.4 myself on ubuntu 12.04.2) - at least that's what 
ldd indicates.

--
components: Build, Extension Modules
files: bug.py
messages: 195253
nosy: ionel.mc
priority: normal
severity: normal
status: open
title: libgcc_s.so.1 must be installed for pthread_cancel to work
type: crash
versions: Python 3.2, Python 3.3, Python 3.4
Added file: http://bugs.python.org/file31301/bug.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com