Re: Tkinter GUI freezing, used Thread then encountered RuntimeError: threads can only be started once

[MRAB]

On 2023-01-11 00:13, Cameron Simpson wrote:

On 10Jan2023 18:32, MRAB  wrote:
I don't like how you're passing Thread...start as an argument. IMHO, it 
would be better/cleaner to pass a plain function, even if the only 
thing that function does is to start the thread.


Yes, and this is likely the thing causing the cited exception "threads
can only be started once". Your setup of the button with the action
defined as:

  Thread().start

creates a _single_ new Thread _when you define the button_, and makes
hte button callback try to start it. On the second and following
callback, you're trying to start the _same_ single Thread again.


You're right! I missed that detail. :-(


Do as MRAB suggests and have the callback create-and-start a Thread
instead of just starting an _existing_ Thread.

Also, for simple quick things there's no need to use a Thread at all. If
the final version of the programme is going to do something long running
at that point, then sure.

I can't tell what 'change_flag' is doing because of the formatting 
issue. Is it doing GUI stuff? In a thread? If yes, don't do that. The 
GUI doesn't like that. Only the main thread should do GUI stuff.


Aye. This is very important in almost all GUI toolkits.

Bit me very badly with Qt once, badly in that the segfaults (yes!
segfaults! in a Python app!) were erratic and very timing dependent,
making them hard to reproduce and understand. It wasn't until I
_realised_ it was thread/concurrency related that I could fix it.

Note that in Tk you can have a callback do GUI work, just not in a
separate thread.



--
https://mail.python.org/mailman/listinfo/python-list

Re: Tkinter GUI freezing, used Thread then encountered RuntimeError: threads can only be started once

[Cameron Simpson]

On 10Jan2023 18:32, MRAB  wrote:
I don't like how you're passing Thread...start as an argument. IMHO, it 
would be better/cleaner to pass a plain function, even if the only 
thing that function does is to start the thread.


Yes, and this is likely the thing causing the cited exception "threads 
can only be started once". Your setup of the button with the action 
defined as:


Thread().start

creates a _single_ new Thread _when you define the button_, and makes 
hte button callback try to start it. On the second and following 
callback, you're trying to start the _same_ single Thread again.


Do as MRAB suggests and have the callback create-and-start a Thread 
instead of just starting an _existing_ Thread.


Also, for simple quick things there's no need to use a Thread at all. If 
the final version of the programme is going to do something long running 
at that point, then sure.


I can't tell what 'change_flag' is doing because of the formatting 
issue. Is it doing GUI stuff? In a thread? If yes, don't do that. The 
GUI doesn't like that. Only the main thread should do GUI stuff.


Aye. This is very important in almost all GUI toolkits.

Bit me very badly with Qt once, badly in that the segfaults (yes!  
segfaults! in a Python app!) were erratic and very timing dependent, 
making them hard to reproduce and understand. It wasn't until I 
_realised_ it was thread/concurrency related that I could fix it.


Note that in Tk you can have a callback do GUI work, just not in a 
separate thread.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: Tkinter GUI freezing, used Thread then encountered RuntimeError: threads can only be started once

[MRAB]

On 2023-01-10 14:57, Abhay Singh wrote:

Here is the entire code snippet of the same.

Please help

def change_flag(top_frame, bottom_frame, button1, button2, button3, button4, 
controller): global counter, canvas, my_image, chosen, flag, directory 
canvas.delete('all') button5['state'] = DISABLED counter += 1

chosen, options_text = function_options()
right_answer_flag = get_right_answer_flag(chosen, options_text)
#pdb.set_trace()

try:
 location = directory + chosen + format_image
except:
 controller.show_frame(PlayAgainExit)
 
my_image = PhotoImage(file=location)

canvas.create_image(160, 100, anchor=CENTER, image=my_image)

button1["text"] = options_text[0]
button2["text"] = options_text[1]
button3["text"] = options_text[2]
button4["text"] = options_text[3]

button1['state'] = NORMAL
button2['state'] = NORMAL
button3['state'] = NORMAL
button4['state'] = NORMAL
##

 button5 = Button(
 next_frame,
 width=20,
 text="next",
 fg="black",
 #command=lambda: 
change_flag(top_frame,bottom_frame,button1,button2,button3,button4,controller))
 command=Thread(target=change_flag, args 
=(top_frame,bottom_frame,button1,button2,button3,button4,controller)).start)
 
 button5.pack(side=RIGHT, padx=5, pady=5)



The formatting is messed up, which doesn't help.

Some points:

You have a 'bare' except, i.e. "except:". Don't do that. It swallows 
_all_ exceptions and can hide bugs.


I don't like how you're passing Thread...start as an argument. IMHO, it 
would be better/cleaner to pass a plain function, even if the only thing 
that function does is to start the thread.


I can't tell what 'change_flag' is doing because of the formatting 
issue. Is it doing GUI stuff? In a thread? If yes, don't do that. The 
GUI doesn't like that. Only the main thread should do GUI stuff.

--
https://mail.python.org/mailman/listinfo/python-list

Tkinter GUI freezing, used Thread then encountered RuntimeError: threads can only be started once

[Abhay Singh]

Here is the entire code snippet of the same.

Please help

def change_flag(top_frame, bottom_frame, button1, button2, button3, button4, 
controller): global counter, canvas, my_image, chosen, flag, directory 
canvas.delete('all') button5['state'] = DISABLED counter += 1

chosen, options_text = function_options()
right_answer_flag = get_right_answer_flag(chosen, options_text)
#pdb.set_trace()

try:
location = directory + chosen + format_image
except:
controller.show_frame(PlayAgainExit)

my_image = PhotoImage(file=location)
canvas.create_image(160, 100, anchor=CENTER, image=my_image)

button1["text"] = options_text[0]
button2["text"] = options_text[1]
button3["text"] = options_text[2]
button4["text"] = options_text[3]

button1['state'] = NORMAL
button2['state'] = NORMAL
button3['state'] = NORMAL
button4['state'] = NORMAL
##

button5 = Button(
next_frame,
width=20,
text="next",
fg="black",
#command=lambda: 
change_flag(top_frame,bottom_frame,button1,button2,button3,button4,controller))
command=Thread(target=change_flag, args 
=(top_frame,bottom_frame,button1,button2,button3,button4,controller)).start)

button5.pack(side=RIGHT, padx=5, pady=5)

Thanks,
Abhay
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue228210] Threads using same stream blow up (Windows)

[admin]

Change by admin :


--
github: None -> 33684

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210694] PRIVATE: Threads and readline (PR#120)

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210650] [HP-UX] python with-threads core-dumps (PR#342)

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue443783] Cygwin32 builds threads, but they hang

[admin]

Change by admin :


--
github: None -> 34816

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue229995] segfaults on AIX if configured with threads

[admin]

Change by admin :


--
github: None -> 33800

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue539990] Framework.mainloop() - multiple threads

[admin]

Change by admin :


--
github: None -> 36390

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401936] Auto-detect DEC threads (which need "-threads" argument)

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue215076] threads of __del__

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue502236] Asynchronous exceptions between threads

[admin]

Change by admin :


--
github: None -> 35905

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue504224] add plan9 threads include to thread.c

[admin]

Change by admin :


--
github: None -> 35921

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210706] gcc error msg, 1.5.2, Slackware + threads (PR#44)

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210703] select with dec-threads core dumps (PR#35)

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue479981] test_socket fails when compiled with threads (HP-UX)

[admin]

Change by admin :


--
github: None -> 35496

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue478006] memory leaks in Modules & threads

[admin]

Change by admin :


--
github: None -> 35465

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue221963] IDLE hangs on Threads

[admin]

Change by admin :


--
github: None -> 33457

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue221115] Kill threads from main thread

[admin]

Change by admin :


--
github: None -> 33437

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue215076] threads of __del__

[admin]

Change by admin :


--
github: None -> 33172

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue420720] Starting many threads causes core dump

[admin]

Change by admin :


--
github: None -> 34447

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue416696] --with-threads fail for 2.1c2 on HPUX 11

[admin]

Change by admin :


--
github: None -> 34354

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue410608] popen with threads

[admin]

Change by admin :


--
github: None -> 34218

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue231540] threads and profiler don't work together

[admin]

Change by admin :


--
github: None -> 33884

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue231189] Crash when Python Re-Initialized with Daemon Threads

[admin]

Change by admin :


--
github: None -> 33849

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue221963] IDLE hangs on Threads

[admin]

Change by admin :


--
github: None -> 33457

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue221327] threads within an embedded python interpreter

[admin]

Change by admin :


--
github: None -> 33445

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue221115] Kill threads from main thread

[admin]

Change by admin :


--
github: None -> 33437

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401606] threads and __del__

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210665] Compiling python on hpux 11.00 with threads (PR#360)

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210646] threads (PR#333)

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210611] Signal processing bug (Linux threads, readline, whatever else) (PR#196)

[admin]

Change by admin :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue221327] threads within an embedded python interpreter

[admin]

Change by admin :


--
github: None -> 33445

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401936] Auto-detect DEC threads (which need "-threads" argument)

[admin]

Change by admin :


--
github: None -> 33354

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue401606] threads and __del__

[admin]

Change by admin :


--
github: None -> 33174

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210706] gcc error msg, 1.5.2, Slackware + threads (PR#44)

[admin]

Change by admin :


--
github: None -> 32776

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210694] PRIVATE: Threads and readline (PR#120)

[admin]

Change by admin :


--
github: None -> 32764

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210665] Compiling python on hpux 11.00 with threads (PR#360)

[admin]

Change by admin :


--
github: None -> 32735

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210703] select with dec-threads core dumps (PR#35)

[admin]

Change by admin :


--
github: None -> 32773

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210650] [HP-UX] python with-threads core-dumps (PR#342)

[admin]

Change by admin :


--
github: None -> 32721

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210611] Signal processing bug (Linux threads, readline, whatever else) (PR#196)

[admin]

Change by admin :


--
github: None -> 32682

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue210646] threads (PR#333)

[admin]

Change by admin :


--
github: None -> 32717

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39812] Avoid daemon threads in concurrent.futures

[Josh Rosenberg]

Change by Josh Rosenberg :


--
Removed message: https://bugs.python.org/msg416876

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39812] Avoid daemon threads in concurrent.futures

[Josh Rosenberg]

Josh Rosenberg  added the comment:

I think this is causing a regression for code that explicitly desires the 
ThreadPoolExecutor to go away abruptly when all other non-daemon threads 
complete (by choosing not to use a with statement, and if shutdown is called, 
calling it with wait=False, or even with those conditions, by creating it from 
a daemon thread of its own).

It doesn't seem like it's necessary, since the motivation was "subinterpreters 
forbid daemon threads" and the same release that contained this change 
(3.9.0alpha6) also contained #40234's change that backed out the change that 
forbade spawning daemon threads in subinterpreters (because they now support 
them by default). If the conflicts with some uses of subinterpreters that make 
it necessary to use non-daemon threads, could that be made a configurable 
option (ideally defaulting to the pre-3.9 choice to use daemon threads)?

--
nosy: +josh.r

___
Python tracker 
<https://bugs.python.org/issue39812>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47082] No protection: `import numpy` in two different threads can lead to race-condition

[Sebastian Berg]

Sebastian Berg  added the comment:

Thanks, so there should already be a lock in place (sorry, I missed that).  But 
somehow we seem to get around it?

Do you know what may cause the locking logic to fail in this case?  Recursive 
imports in NumPy itself?  Or Cython using low-level C-API?

I.e. can you think of something to investigate that may help NumPy/Cython to 
make sure that locking is successful?


/* Cythons Import code (slightly cleaned up for Python 3 only): */
static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list, int level) {
PyObject *empty_list = 0;
PyObject *module = 0;
PyObject *global_dict = 0;
PyObject *empty_dict = 0;
PyObject *list;
if (from_list)
list = from_list;
else {
empty_list = PyList_New(0);
if (!empty_list)
goto bad;
list = empty_list;
}
global_dict = PyModule_GetDict(__pyx_m);
if (!global_dict)
goto bad;
empty_dict = PyDict_New();
if (!empty_dict)
goto bad;
{
if (level == -1) {
if ((1) && (strchr(__Pyx_MODULE_NAME, '.'))) {
module = PyImport_ImportModuleLevelObject(
name, global_dict, empty_dict, list, 1);
if (!module) {
if (!PyErr_ExceptionMatches(PyExc_ImportError))
goto bad;
PyErr_Clear();
}
}
level = 0;
}
if (!module) {
module = PyImport_ImportModuleLevelObject(
name, global_dict, empty_dict, list, level);
}
}
bad:
Py_XDECREF(empty_list);
Py_XDECREF(empty_dict);
return module;
}

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47082] No protection: `import numpy` in two different threads can lead to race-condition

[Christian Heimes]

Christian Heimes  added the comment:

Python used to have a global import lock. The global import lock prevented 
recursive imports from other threads while the current thread was importing. 
The import lock was source of issues and dead locks. Antoine replaced it with a 
fine grained import lock in bpo-9260.

--
nosy: +christian.heimes

___
Python tracker 
<https://bugs.python.org/issue47082>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7946] Convoy effect with I/O bound threads and New GIL

[Sophist]

Sophist  added the comment:

> https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit

1. The steering committee hasn't given the go ahead for this yet, and we have 
no idea when such a decision will be made nor whether the decision with be yes 
or no.

2. Even after the decision is made "Removing the GIL will be a large, 
multi-year undertaking with increased risk for introducing bugs and 
regressions."

3. The promised performance gains are actually small by comparison to the 
existing project that Guido is running is hoping to achieve. It is unclear 
whether the no-gil implementation would impact those gains, and if so whether a 
small reduction in the large planned performance gains would actually more than 
wipe out the modest performance gains promised by the no-gil project.

4. Given that this "Removing the GIL will require making trade-offs and taking 
on substantial risk", it is easy to see that this project could come across an 
unacceptable risk or insurmountable technical problem at any point and thus 
fall by the wayside.

Taking all of the above points together, I think that there is still merit in 
considering the pros and cons of a GIL scheduler.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7946] Convoy effect with I/O bound threads and New GIL

[Guido van Rossum]

Guido van Rossum  added the comment:

Start here: 
https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit

AFAICT the SC hasn't made up their minds about this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7946] Convoy effect with I/O bound threads and New GIL

[Sophist]

Sophist  added the comment:

> I think that we should focus our efforts on removing the GIL, now that we 
> have a feasible solution for doing so without breaking anything

Is this really a thing? Something that is definitely happening in a reasonable 
timescale?

Or are there some big compatibility issues likely to rear up and at best create 
delays, and at worse derail it completely?

Can someone give me some links about this please?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7946] Convoy effect with I/O bound threads and New GIL

[Omer Katz]

Omer Katz  added the comment:

I think that we should focus our efforts on removing the GIL, now that we
have a feasible solution for doing so without breaking anything (hopefully)
however the removal of the GIL is still far from being complete and will
need to be rebased upon the latest Python version to be merged.

This issue would probably hurt Celery since some users use it with a thread
pool and it uses a few threads itself but it seems like fixing it is too
much effort so if we were to invest a lot of effort, I'd focus on removing
the problem entirely.

Instead, I suggest we document this with a warning in the relevant place so
that people would know to avoid or workaround the problem entirely.

On Mon, Mar 21, 2022, 20:32 Guido van Rossum  wrote:

>
> Change by Guido van Rossum :
>
>
> --
> nosy: +gvanrossum
>
> ___
> Python tracker 
> <https://bugs.python.org/issue7946>
> ___
>

--

___
Python tracker 
<https://bugs.python.org/issue7946>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7946] Convoy effect with I/O bound threads and New GIL

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7946] Convoy effect with I/O bound threads and New GIL

[Sophist]

Sophist  added the comment:

Please see also https://github.com/faster-cpython/ideas/discussions/328 for a 
proposal for a simple (much simpler than BFS) GIL scheduler only allocating the 
GIL between runable O/S threads waiting to have ownership of the GIL, and using 
the O/S scheduler for scheduling the threads.

--
nosy: +Sophist

___
Python tracker 
<https://bugs.python.org/issue7946>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47082] No protection: `import numpy` in two different threads can lead to race-condition

[Sebastian Berg]

Sebastian Berg  added the comment:

To add to this: it would seem to me that the side-effects of importing should 
be guaranteed to only be called once?

However, IO or other operations could be part of the import side-effects and 
release the GIL.  So even a simple, pure-Python, package could run into this 
same issue and probably won't even realize that they can run into it.
(Assuming I understand how this is happening correctly.)

So it would seem to me that either:
* Python should lock on the thread level or maybe the `sys.modules` dictionary?
* The `threading` module could somehow ensure safety by hooking into
  the import machinery?
* Packages that may release the GIL or have side-effects that must
  only be run once have to lock (i.e. NumPy).
  (But it seems to me that many packages will not even be aware of
  possible issues.)

--
nosy: +seberg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47082] No protection: `import numpy` in two different threads can lead to race-condition

[Jens Henrik Goebbert]

New submission from Jens Henrik Goebbert :

While using [Xpra](https://github.com/Xpra-org/xpra) we came across a bug which 
might be a Python or a NumPy issue.
Perhaps some of you can help us understanding some internals.

Calling `import numpy` at the same time in two different threads of the Python 
program can lead to a race-condition.
This happens for example with Xpra when loading the encoder nvjpeg:
```
2022-03-20 12:54:59,298  cannot load enc_nvjpeg (nvjpeg encoder)
Traceback (most recent call last):
  File "/lib/python3.9/site-packages/xpra/codecs/loader.py", line 
52, in codec_import_check
ic =  __import__(class_module, {}, {}, classnames)
  File "xpra/codecs/nvjpeg/encoder.pyx", line 8, in init 
xpra.codecs.nvjpeg.encoder
  File "/lib/python3.9/site-packages/numpy/__init__.py", line 150, 
in 
from . import core
  File "/lib/python3.9/site-packages/numpy/core/__init__.py", line 
51, in 
del os.environ[envkey]
  File "/lib/python3.9/os.py", line 695, in __delitem__
raise KeyError(key) from None
KeyError: 'OPENBLAS_MAIN_FREE'
```

Here the environment variable OPENBLAS_MAIN_FREE is set in the `numpy` code:
https://github.com/numpy/numpy/blob/maintenance/1.21.x/numpy/core/__init__.py#L18
and short after that it is deleted
https://github.com/numpy/numpy/blob/maintenance/1.21.x/numpy/core/__init__.py#L51
But this deletion fails ... perhaps because the initialization runs at the same 
time in two threads :thinking:

Shouldn't Python protect us by design?

@seberg comments 
[HERE](https://github.com/numpy/numpy/issues/21223#issuecomment-1074008386):
```
So, my current hypothesis (and I have briefly checked the Python code) is that 
Python does not do manual locking. But it effectively locks due to this going 
into C and thus holding the GIL. But somewhere during the import of NumPy, 
NumPy probably releases the GIL briefly and that could allow the next thread to 
go into the import machinery.
[..]
NumPy may be doing some worse than typical stuff here, but right now it seems 
to me that Python should be protecting us.
```

Can anyone comment on this?

--
components: Interpreter Core
messages: 415687
nosy: jhgoebbert
priority: normal
severity: normal
status: open
title: No protection: `import numpy` in two different threads can lead to 
race-condition
type: behavior
versions: Python 3.11

___
Python tracker 
<https://bugs.python.org/issue47082>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34680] [doc] asyncio event_loop fails when accessed from multiple threads

[Irit Katriel]

Change by Irit Katriel :


--
keywords: +easy
title: asyncio event_loop fails when accessed from multiple threads -> [doc] 
asyncio event_loop fails when accessed from multiple threads

___
Python tracker 
<https://bugs.python.org/issue34680>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40811] Allow to create new Event Loops on Threads

[Andrew Svetlov]

Andrew Svetlov  added the comment:

I suggest closing.
The documentation explicitly describes how asyncio works with threads.

--
resolution:  -> wont fix
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue40811>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[pmp-p]

Change by pmp-p :


--
nosy: +pmpp

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[Vidhya]

Vidhya  added the comment:

Thanks Irit for your help.

On Thu., Mar. 3, 2022, 10:17 a.m. Irit Katriel, 
wrote:

>
> Irit Katriel  added the comment:
>
> Thank you @vidhya.
>
> --
> resolution:  -> fixed
> stage: patch review -> resolved
> status: open -> closed
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[Irit Katriel]

Irit Katriel  added the comment:

Thank you @vidhya.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[Irit Katriel]

Irit Katriel  added the comment:


New changeset 09819863a3fb7092ca5cbdfcb722882ebbac806b by Miss Islington (bot) 
in branch '3.9':
bpo-6634: [doc] clarify that sys.exit() does not always exit the interpreter 
(GH-31639) (GH-31661)
https://github.com/python/cpython/commit/09819863a3fb7092ca5cbdfcb722882ebbac806b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[miss-islington]

miss-islington  added the comment:


New changeset 9d9dc59d07d51d73e5af7dd506d0da63aa336995 by Miss Islington (bot) 
in branch '3.10':
bpo-6634: [doc] clarify that sys.exit() does not always exit the interpreter 
(GH-31639)
https://github.com/python/cpython/commit/9d9dc59d07d51d73e5af7dd506d0da63aa336995


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 11.0 -> 12.0
pull_requests: +29778
pull_request: https://github.com/python/cpython/pull/31660

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[miss-islington]

Change by miss-islington :


--
pull_requests: +29779
pull_request: https://github.com/python/cpython/pull/31661

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[Irit Katriel]

Irit Katriel  added the comment:


New changeset 10117f1d8cb49ce95493555c06050faf636ccee7 by vidhya in branch 
'main':
bpo-6634: [doc] clarify that sys.exit() does not always exit the interpreter 
(GH-31639)
https://github.com/python/cpython/commit/10117f1d8cb49ce95493555c06050faf636ccee7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[Vidhya]

Vidhya  added the comment:

Thanks for your comments :). The PR for the same is:
https://github.com/python/cpython/pull/31639

--
message_count: 21.0 -> 22.0
pull_requests: +29760
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/31639

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[Irit Katriel]

Irit Katriel  added the comment:

Vidhya, I think the sentence you are suggesting to add would overlap with one 
which is already there ("Since exit() ultimately “only” raises an exception, it 
will only exit the process when called from the main thread, and the exception 
is not intercepted.")


It seems to me that what could be improved is the first paragraph, which starts 
with: "Exit from Python.".   Maybe instead it could say something like "Raise a 
SystemExit exception, which has the effect of ..." and then say what it is 
(i.e., that it exits python if you are in the main thread, unless the exception 
is caught... ).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[Vidhya]

Vidhya  added the comment:

[Entry level contributor seeking guidance]
If this is still open, I can work on this.

I plan to add the following in sys.exit() and add a reference to thread.exit() 
at https://docs.python.org/3/library/sys.html#sys.exit:

When called from a thread other than the main thread, this causes the thread to 
exit silently, and is equivalent to calling :func:`thread.exit`.

Please correct if anything wrong.

--
nosy: +vidhya

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45339] concurrent.future.ThreadPoolExecutor should parameterize class used for threads

[Andrew Svetlov]

Andrew Svetlov  added the comment:

I guess if you are asking for initialization and finalization of 
thread-specific data in a thread pool -- you need exactly these things (or a 
context manager).
A custom thread class reveals too many implementation details.
I personally prefer an explicit initializer/finalizer based approach.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34680] asyncio event_loop fails when accessed from multiple threads

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Please feel free to propose pull request for documentation tuning.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34680] asyncio event_loop fails when accessed from multiple threads

[epiphyte]

epiphyte  added the comment:

For additional context, this was encountered when porting an application from 
threading to asyncio, and at the time we were still running the ioloop in its 
own thread as opposed to the mainthread. We no longer do that :D

Updating the documentation to clarify that the loop won't work across threads, 
like what we ran into at the time, would resolve this.

--

___
Python tracker 
<https://bugs.python.org/issue34680>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34680] asyncio event_loop fails when accessed from multiple threads

[Irit Katriel]

Irit Katriel  added the comment:

Perhaps this is just a doc issue - state explicitly that a loop should be used 
only in one thread, and mention that this is checked in debug mode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34680] asyncio event_loop fails when accessed from multiple threads

[Andrew Svetlov]

Andrew Svetlov  added the comment:

BaseEventLoop has _check_loop() method that is closed in debug mode only.
UnixEventLoop doesn't call this method for unix-specific API.

Adding the check to add_signal_handler()/remove_signal_handler() doesn't hurt, 
sure. 
But it doesn't help as the script is executed in non-debug mode.

Performing a check on every call_soon() call kills the performance, that's why 
debug mode is required.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34680] asyncio event_loop fails when accessed from multiple threads

[Irit Katriel]

Irit Katriel  added the comment:

I think the script does violate the rule: 

t = threading.Thread(target=do_loop, args=(loop,))

and then do_loop closes the loop that was created in the main thread.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34680] asyncio event_loop fails when accessed from multiple threads

[Andrew Svetlov]

Andrew Svetlov  added the comment:

All asyncio loop methods except `loop.call_soon_threadsafe()` should be done 
from the same thread as been used for the loop creation.
Did you violate this rule?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34680] asyncio event_loop fails when accessed from multiple threads

[Irit Katriel]

Change by Irit Katriel :


--
title: asyncio event_loop close fails off main thread if signal handler 
registered -> asyncio event_loop fails when accessed from multiple threads

___
Python tracker 
<https://bugs.python.org/issue34680>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46493] Add an API to indicate if the process might have multiple threads

[Gregory P. Smith]

New submission from Gregory P. Smith :

It'd be handy to have a function to determine if there are multiple threads in 
the current process or not - at least on POSIXish systems.  This would be 
useful anytime a library is trying to use os.fork(), as fork() is not safe in a 
multi-threaded process.

Motivation: I want to _use_ this API to consider raising RuntimeWarnings in 
libraries that call fork() to highlight the problem to people attempting to use 
that code in multithreaded processes.

As POSIXy OSes don't usually have a simple API to get this information, one 
implementation such as this could make sense:

```
def is_process_multithreaded() -> bool:
"""Are there multiple threads in this process? OSError if no such OS API is 
available."""
threads = 0
ourself = str(os.gettid())
i_feel_seen = False
try:
# Linux, NetBSD, any others?
with os.scandir('/proc/self/task') as tasks:
for task_dir_entry in tasks:
# tid named subdirs should be the only thing that exists.
# We do a simple conformity check just in case.
if task_dir_entry.name.isdigit():
if task_dir_entry.name == ourself:
i_feel_seen = True
threads += 1
        if i_feel_seen and threads > 1:
    return True  # Multiple threads confirmed.
except EnvironmentError:
    raise OSError('Unable to count threads on this platform.')
if i_feel_seen and threads == 1:
return False
else:
raise OSError(f'Unclear. Found {threads} in /proc/self/task and did not 
find ourself.')
```

macOS has mach darwin kernel APIs that can do this. Not well documented but 
they do work - 
https://stackoverflow.com/questions/21478229/how-to-count-number-of-alive-threads-in-ios

FreeBSD has APIs that can do this (see FreeBSD usr.bin/procstat/ source).

configure.ac checks and an extension module would be needed to integrate those.

My use case is not relevant to Windows (a platform unburdened by fork) but 
Windows APIs to answer the question exist if anyone has a reason to want this 
there as well - 
https://stackoverflow.com/questions/3749668/how-to-query-the-thread-count-of-a-process-using-the-regular-windows-c-c-apis

--
components: Library (Lib)
messages: 411420
nosy: gregory.p.smith
priority: normal
severity: normal
stage: needs patch
status: open
title: Add an API to indicate if the process might have multiple threads
type: enhancement
versions: Python 3.11

___
Python tracker 
<https://bugs.python.org/issue46493>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[STINNER Victor]

STINNER Victor  added the comment:

Either reopen the issue or open a new issue. Only people subscribed to this bug 
are aware of the recent comments. Closed issues are hidden from the bugs home 
page and from the default search.

If you consider that Python must again support thread-less platforms, IMO it's 
a new feature and a new issue must be opened.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[Brett Cannon]

Brett Cannon  added the comment:

I am not opening a new discussion; this is just recording this fact here as it 
has come up in other places on the internet.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45339] concurrent.future.ThreadPoolExecutor should parameterize class used for threads

[Sergei Maertens]

Sergei Maertens  added the comment:

I was looking for some way to be able to add a thread finalizer, a piece of 
code to be called when the thread pool shuts down and all threads need cleaning 
up. Glad I came across this issue, since the example of using a Thread subclass 
with a custom run (wrapped in a context manager) would fill my needs completely.

> What are examples of your some_important_context()? Is it important to call 
> some code before destroying the thread?

I currently have a use case with a web-framework that has persistent DB 
connections - i.e. they span multiple HTTP requests. This also applies in the 
context of running a command-line script with said framework where database 
connections are opened.

We are calling external APIs (IO heavy), so using a ThreadPoolExecutor makes 
sense. However, since those "DB connection pools" are thread locals, we need to 
ensure that database connections are closed again to not leak resources.

The current workaround is to submit a job/function to the pool and have it 
close the database connections, which adds overhead since database connections 
are now opened and closed within the same thread that could have been re-used.

Using a context manager, we would be able to wrap the `super().run(...)` and 
close the database connections when the thread exits (succesfully or because of 
an Exception, even). This comes very close to having an `atexit` for individual 
threads.

Furthermore I like the idea of being able to provide the class as a context 
manager kwarg, but would also not be opposed to a class property specifying the 
Thread class to use - both would greatly enhance composability and are a 
cleaner solution than adding a finalizer option (similar to the `initializer` 
kwarg)

--
nosy: +Sergei Maertens

___
Python tracker 
<https://bugs.python.org/issue45339>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46046] I/O bound threads got to no chance to run with small CPU bound threads with new GIL

[Ken Jin]

Ken Jin  added the comment:

Hi,

Thank you for your interest in Python, and especially the GIL. David Beazley 
opened an issue for his PyCon 2010 GIL talk at 
https://bugs.python.org/issue7946. You can discuss more on that issue (I 
suggest first reading through it, there's a really long history behind the GIL).

I'm closing this issue as a duplicate, as discussion should be redirected to 
the abovementioned issue7946 instead.

--
nosy: +kj
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Concur with Victor. It is fine to add a link to other discussion in a closed 
issue, but an issue closed many years ago is not a good place for a new 
discussion at all.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[STINNER Victor]

STINNER Victor  added the comment:

> This has unfortunately turned out to be a blocker on getting WASI support as 
> there's not direct threading support in WebAssembly.

This issue is now closed and unrelated to WASI support. Can you please open a 
new separated issue for it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[Brett Cannon]

Brett Cannon  added the comment:

> https://web.dev/webassembly-threads/ suggests otherwise.

It actually doesn't as that post isn't for WASI, it's for WASM in the browser 
(I chose my acronyms carefully ).

WASI is like POSIX for WebAssembly, so it's meant for desktop usage and thus no 
browser or JS engine is involved to provide web workers.

--

___
Python tracker 
<https://bugs.python.org/issue31370>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6634] [doc] sys.exit() called from threads other than the main one: undocumented behaviour

[Irit Katriel]

Irit Katriel  added the comment:

Since 3.8 we have threading.excepthook() so the situation is different now: 

https://github.com/python/cpython/pull/13515

I think it still makes sense to change the wording of the doc for sys.exit() a 
bit, and add a reference to the relevant section of the threading doc.

--
keywords: +easy
nosy: +iritkatriel
title: sys.exit() called from threads other than the main one: undocumented 
behaviour -> [doc] sys.exit() called from threads other than the main one: 
undocumented behaviour
type:  -> behavior
versions: +Python 3.10, Python 3.11, Python 3.9 -Python 2.7, Python 3.5

___
Python tracker 
<https://bugs.python.org/issue6634>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Fwd: I/O bound threads got to no chance to run with small CPU bound threads with new GIL

[Souvik Ghosh]

*Resending this message after subscribing in python-mail-list*

-- Forwarded message -
From: Souvik Ghosh 
Date: Sat, Dec 11, 2021 at 5:10 PM
Subject: I/O bound threads got to no chance to run with small CPU bound
threads with new GIL
To: 


Hello PSF,

I'm Souvik Ghosh from India. I've been coding for Python for almost 5
years now. And, I love Python and care about it so much.

The issue is stated below,

According to David Beazley' talk in PyCon'2010 in Atlanta Georgia, he
demonstrated about a new GIL with running CPU bound and I/O bound
threads together.

He said the talk that the threads which are forced to timeout of 5ms,
will have the lower priority(which is CPU bound) and the thread which
suspends the GIL within 5ms will have higher priority (which is I/O
bound).

What happens in the following code is if I set args=(1000,) (seven
zero after 1) then only I/O bound runs and returns when CPU bound
takes much time to execute. But if I decrease that args to
args=(1000,) then I/O bound got no chance to reaquire the GIL in the
meantime even though the sys.getswitchinterval() is equal to 5ms(By
default). If I/O bound doesn't reacquire GIL with args=(1,) then
the time to execute to run
only the CPU bound takes 0.426035414 seconds. Thats means
almost ticks 0.426035414/0.005=85 (approx) times to set the
priority in between the two threads. In that case if the I/O got more
priority within that time, it should have returned the value within
that ticks. But I didn't happen.
import threading
from queue import Queue
from timeit import default_timer as timer
import urllib.request


q = Queue()  # Queue technique to pass returns among threads while running


def decrement(numbers):  # CPU bound
while numbers > 0:
numbers -= 1
if not q.empty():
"""I added this method because this thread will run most of the time
because it's mostly cpu bound"""
print(numbers)
print(q.get(block=False))
print(timer() - start)  # It tell after when exactly I/O
bound returns value after both the threads started to run


def get_data():  # I/O bound

with urllib.request.urlopen("https://www.google.com;) as dt:
q.put(dt.read(), block=False)


if __name__ == "__main__":
start = timer()
t1 = threading.Thread(target=get_data)
#t2 = threading.Thread(target=decrement, args=(1000,)) #For
this I/O responds and returns
t2 = threading.Thread(target=decrement, args=(10,)) # I/O
doesn't responds at all
t1.start()
t2.start()

t1.join()
t2.join()
print(timer() - start)

Look at the second code...

import threading
from queue import Queue
from timeit import default_timer as timer
import urllib.request
import sys


q = Queue()  # Queue technique to pass returns among threads while running


def decrement(numbers):  # CPU bound
while numbers > 0:
numbers -= 1
if not q.empty():
"""I added this method because this thread will run most of the time
because it's mostly cpu bound"""
print(numbers)
print(q.get(block=False))
print(timer() - start)  # It tell after when exactly I/O
bound returns value after both the threads started to run


def get_data():  # I/O bound

with urllib.request.urlopen("https://www.google.com;) as dt:
q.put(dt.read(), block=False)


if __name__ == "__main__":
sys.setswitchinterval(0.0001)
start = timer()
t1 = threading.Thread(target=get_data)
#t2 = threading.Thread(target=decrement, args=(100,)) #I/O
responds with this
t2 = threading.Thread(target=decrement, args=(10000,))    # I/O
doesn't responds at all even with this 0.0001
seconds of threads switching interval
t1.start()
t2.start()

t1.join()
t2.join()
print(timer() - start)

Can't we have a more better version of GIL to set I/O threads(overall)
priorities even more better and not to degrade the CPU bound and
better callbacks in response? Or, try to remove the GIL?

The issue I submitted in here:- https://bugs.python.org/issue46046

Thank you so much, great future of Python!
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue46046] I/O bound threads got to no chance to run with small CPU bound threads with new GIL

[Souvik Ghosh]

Souvik Ghosh  added the comment:

Python-ideas link in here:-
https://mail.python.org/archives/list/python-id...@python.org/message/A5MX6SQUHP65JC6V5ZFCCHMMJURM4KHB/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46046] I/O bound threads got to no chance to run with small CPU bound threads with new GIL

[Souvik Ghosh]

New submission from Souvik Ghosh :

According to David Beazley' talk in PyCon'2010 in Atlanta Georgia, he 
demonstrated about a new GIL with running CPU bound and I/O bound threads 
together.

He said the talk that the threads which are forced to timeout of 5ms, will have 
the lower priority(which is CPU bound) and the thread which suspends the GIL 
within 5ms will have higher priority (which is I/O bound). 

What happens in the following code is if I set args=(1000,) (seven zero 
after 1) then only I/O bound runs and returns when CPU bound takes much time to 
execute. But if I decrease that args to args=(1000,) then I/O bound got no 
chance to reaquire the GIL in the meantime even though the 
sys.getswitchinterval() is equal to 5ms(By default). If I/O bound doesn't 
reacquire GIL with args=(1,) then the time to execute to run 
only the CPU bound takes 0.426035414 seconds. Thats means almost ticks 
0.426035414/0.005=85 (approx) times to set the priority in between the 
two threads. In that case if the I/O got more priority within that time, it 
should have returned the value within that ticks. But I didn't happen. 

import threading
from queue import Queue
from timeit import default_timer as timer
import urllib.request


q = Queue()  # Queue technique to pass returns among threads while running


def decrement(numbers):  # CPU bound
while numbers > 0:
numbers -= 1
if not q.empty():
"""I added this method because this thread will run most of the time
because it's mostly cpu bound"""
print(numbers)
print(q.get(block=False))
print(timer() - start)  # It tell after when exactly I/O bound 
returns value after both the threads started to run


def get_data():  # I/O bound

with urllib.request.urlopen("https://www.google.com;) as dt:
q.put(dt.read(), block=False)


if __name__ == "__main__":
start = timer()
t1 = threading.Thread(target=get_data)
#t2 = threading.Thread(target=decrement, args=(1000,)) #For this I/O 
responds and returns
t2 = threading.Thread(target=decrement, args=(10,)) # I/O doesn't 
responds at all
t1.start()
t2.start()

t1.join()
t2.join()
print(timer() - start)

Look at the second code...


import threading
from queue import Queue
from timeit import default_timer as timer
import urllib.request
import sys


q = Queue()  # Queue technique to pass returns among threads while running


def decrement(numbers):  # CPU bound
while numbers > 0:
numbers -= 1
if not q.empty():
"""I added this method because this thread will run most of the time
because it's mostly cpu bound"""
print(numbers)
print(q.get(block=False))
print(timer() - start)  # It tell after when exactly I/O bound 
returns value after both the threads started to run


def get_data():  # I/O bound

with urllib.request.urlopen("https://www.google.com;) as dt:
q.put(dt.read(), block=False)


if __name__ == "__main__":
sys.setswitchinterval(0.0001)
start = timer()
t1 = threading.Thread(target=get_data)
#t2 = threading.Thread(target=decrement, args=(100,)) #I/O responds 
with this 
t2 = threading.Thread(target=decrement, args=(1,))# I/O doesn't 
responds at all even with this 0.0001 seconds of 
threads switching interval
t1.start()
t2.start()

t1.join()
t2.join()
print(timer() - start)

Can't we have a more better version of GIL to set I/O threads(overall) 
priorities even more better and not to degrade the CPU bound and better 
callbacks in response? Or, try to remove the GIL?

Thank you so much, great future of Python!

--
components: Interpreter Core
files: GIL9.py
messages: 408292
nosy: souvikghosh
priority: normal
severity: normal
status: open
title: I/O bound threads got to no chance to run with small CPU bound threads 
with new GIL
type: performance
versions: Python 3.10
Added file: https://bugs.python.org/file50487/GIL9.py

___
Python tracker 
<https://bugs.python.org/issue46046>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[Gregory P. Smith]

Gregory P. Smith  added the comment:

https://web.dev/webassembly-threads/ suggests otherwise.

--

___
Python tracker 
<https://bugs.python.org/issue31370>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue31370] Remove support for threads-less builds

[Brett Cannon]

Brett Cannon  added the comment:

This has unfortunately turned out to be a blocker on getting WASI support as 
there's not direct threading support in WebAssembly.

--
nosy: +brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7946] Convoy effect with I/O bound threads and New GIL

[Mark Dickinson]

Change by Mark Dickinson :


--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45689] Add the ability to give custom names to threads created by ThreadPoolExecutor

[Tangellapalli Sai Hanuma Rahul]

Tangellapalli Sai Hanuma Rahul  added the comment:

Yes, it's true, it's so naive method, Contextvars! may be inside the Future 
Object?

--
resolution:  -> rejected
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45689] Add the ability to give custom names to threads created by ThreadPoolExecutor

[Mark Dickinson]

Mark Dickinson  added the comment:

Sorry Rahul: I'm not the right person to help you push this forward.

I'm sympathetic to the problem: I've encountered similar issues in "Real Code", 
where we needed to associate log outputs generated by worker pool threads with 
the actual tasks that generated those logs. But I'm not convinced either that 
setting the thread name is the right mechanism to get that association (it 
doesn't extend nicely to other executor types, for example), or that the API 
you propose is the right one (I find the duplication between `submit` and 
`submit_with_name` to be a bit much).

I'm wondering whether there's some way that executors could use contextvars to 
provide a per-task context. Then a task "name" could potentially be part of 
that context, and possibly you could write a custom log handler that read the 
name from the context when emitting log messages.

If you want to find someone to help push this forward, it may be worth posting 
on the python-ideas mailing list to get discussion going.

--

___
Python tracker 
<https://bugs.python.org/issue45689>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45689] Add the ability to give custom names to threads created by ThreadPoolExecutor

[Tangellapalli Sai Hanuma Rahul]

Tangellapalli Sai Hanuma Rahul  added the comment:

In summary;
==

Current Problem:
--
 
ThreadPoolExecutor handles all threads-related activities so the user needs to 
just submit tasks and set max_workers and other Executor settings if necessary. 
If ThreadPoolExecutor allowed us to name/rename threads, that would be great.

Of Course, there's a workaround that requires the user to change the name of 
the current Thread inside the Task, but that requires the user to access the 
current thread which I believe is not the goal of Thread Pool Executor.

Request;
--

Changes Made:
~~~
In this Pull request, _work_queue puts tuple of Future Object and name (name 
can be None).  i.e, typing.Tuple[_base.Future, typing. Optional[str]] which was 
just a _base.Future before.

So, inside the _worker function,
When it gets the elements and If its name is None, it will either use the 
thread_name_prefix or custom Name of task that was used before. else it sets 
the name of the current Thread with the given name.

Problems it solves
~

Currently, ThreadExecutor uses thread_name_prefix + index number for the 
Internal Thread which is not enough for debugging when we try to submit 
different types of Tasks (types here mean general callable). 

So, using this Optional Feature which allows users to submit tasks with names 
it allows them to easily find issues using the name of Thread either in Logger 
or in the debugger.

--

___
Python tracker 
<https://bugs.python.org/issue45689>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45689] Add the ability to give custom names to threads created by ThreadPoolExecutor

[Tangellapalli Sai Hanuma Rahul]

Tangellapalli Sai Hanuma Rahul  added the comment:

There is a new function submit_with_name in _base.Executor that can accept name 
parameters before args and kwargs, submit can continue to be used as before.

submit internally calls submit_with_name with name as None. Calling 
submit_with_name in Executor causes a NotImplementedError, it was only 
implemented in ThreadPoolExecutor, but it could be implemented in any Executor

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45689] Add the ability to give custom names to threads created by ThreadPoolExecutor

[Mark Dickinson]

Mark Dickinson  added the comment:

> previously one could write .submit(function_name, *args, **kwargs)
> but now one should write 
> .submit(function_name, name_of_thread, *args, **kwargs)
> name_of_thread can be None

This approach can't work, I'm afraid: it would be a backwards-incompatible 
change to the `submit` method signature, and would break many existing uses of 
submit.

--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45689] Add the ability to give custom names to threads created by ThreadPoolExecutor

[Alex Waygood]

Change by Alex Waygood :


--
title: Custom Name for ThreadPoolExecutor -> Add the ability to give custom 
names to threads created by ThreadPoolExecutor

___
Python tracker 
<https://bugs.python.org/issue45689>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44105] tempfile.TemporaryDirectory deleted after sleep in threads

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> out of date
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45339] concurrent.future.ThreadPoolExecutor should parameterize class used for threads

[Erick]

Erick  added the comment:

It also occurs to me that even if `concurrent.futures` adopted an alternative 
Thread class, it would still be preferable to allow for composition in the 
manner that was originally proposed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com