[issue47159] multiprocessing.pool.Pool.apply block infinitely when stressed while using maxtasksperchild
New submission from Harsh Patel : We have a long-running process, which uses thread pool to get tasks and these threads then schedule the tasks and send them to multiprocessing.Pool (with maxtasksperchild limit set) using apply function. The original codebase is large but I have extracted this logic into a simple script for convenience so that you can see the issue. The issue is that after sometime the apply function call gets blocked infinitely. This only happens when we use maxtasksperchild and cause the library for multiple process recreation. How script works: Script creates an object of the Manager class which is responsible for creating a pool of threads and creating multiprocessing pool. Each thread is provided jobs to execute which is done by Scheduler.get_ready_jobs function call. I have designed this function in a way that 100 jobs are returned after every 5 calls to this function else it returns no jobs. Each thread while performing job sends processing_func to multiprocessing pool for execution which just prints "processing something" and exits. Environment: Python 3.7.10 Clang 12.0.5 MacOS v11.6 intel Steps to reproduce: 1. Download the attached Python script and execute it 2. Let it run for 10-15 seconds and interrupt the execution You will notice that some threads will exit successfully but for some, you will see the last message printed is "executing cpu_pool apply " which indicates these threads are blocked even though the actual processing_func is just calling a print function. Warning: You will also not be able to exit the process without killing it. As a note, this is my first Python issue report so please let me know if you need more information. -- components: Library (Lib) files: example.py messages: 416331 nosy: davin, harsh8398, pitrou priority: normal severity: normal status: open title: multiprocessing.pool.Pool.apply block infinitely when stressed while using maxtasksperchild type: behavior versions: Python 3.7 Added file: https://bugs.python.org/file50706/example.py ___ Python tracker <https://bugs.python.org/issue47159> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39634] Add clarification in documentation for incorrect heapq heapify naming
Change by Harsh Patel : -- title: Incorrect heapq heapify naming -> Add clarification in documentation for incorrect heapq heapify naming ___ Python tracker <https://bugs.python.org/issue39634> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39634] Incorrect heapq heapify naming
Harsh Patel added the comment: This ticket is not intended to rename heapify method in the stdlib (its too late for that). This is to merely add a clarification in the docs to state that heapify here has a different connotation than what's used in standard algorithmic literature (CLRS, Sedgewick etc). See here https://xlinux.nist.gov/dads/HTML/heapify.html References: https://algs4.cs.princeton.edu/24pq/ http://math.utoledo.edu/~codenth/Fall_16/4380/heap-hw.pdf On Fri, Feb 14, 2020 at 2:00 PM Steven D'Aprano wrote: > > Steven D'Aprano added the comment: > > The "-ify" or "-fy" suffix in English means "to make". For example, > "amplify", "magnify", "terrify", etc. > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wiktionary.org_wiki_-2Dify=DwIFaQ=slrrB7dE8n7gBJbeO0g-IQ=owI_YZl60rgj74U9tzqxcA=jUulcI-27ChSIukpI4uzg0LktqA5zRt9HI8Lz0JxRYU=o95nXSmjNMCHMbmoWeL7LildI9qwEdnusnlAOenreQ4= > > So heapify literally means "make into a heap". Not only is the name > correct, but it is a better name for an in-place operation than either of > the terms you give: > > * "build-heap" does not describe an in-place operation; > * "make-heap" is ambiguous in whether it is in-place or not. > > But even if we agreed that the name needs to change to match text books > (which text books?), since heapify is a public function, we cannot just > change the name. We would have to keep the old name around for many, many > years, possibly forever. > > -- > nosy: +steven.daprano > > ___ > Python tracker > < > https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue39634=DwIFaQ=slrrB7dE8n7gBJbeO0g-IQ=owI_YZl60rgj74U9tzqxcA=jUulcI-27ChSIukpI4uzg0LktqA5zRt9HI8Lz0JxRYU=kE5QfuakDzLUOPpeWAGCf7Z0zJeeeC1iDUSj-bMFzrE= > > > ___ > -- ___ Python tracker <https://bugs.python.org/issue39634> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39634] Incorrect heapq heapify naming
New submission from Harsh Patel : heapify method is a misnomer in that it is actually the make-heap or build-heap procedure from textbooks -- components: Library (Lib) messages: 361996 nosy: hp685 priority: normal pull_requests: 17888 severity: normal status: open title: Incorrect heapq heapify naming ___ Python tracker <https://bugs.python.org/issue39634> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11907] SysLogHandler can't send long messages
Harsh Patel added the comment: I know this has been closed as "Not a bug" but it would have been convenient if the message was broken down into multiple packets and transmitted, should it exceed the packet limit. Raising an exception [Errno 40 or Errno 90] or silently ignoring the packet (by setting logging.raiseExceptions to False) doesn't help much in terms of robustness and reliability. I eventually ended up extending these Handler classes and overriding the send to accomplish breaking messages into multiple parts -- nosy: +Harsh Patel ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue11907> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
