[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> Hi there. I get this behavior as well, although only in a venv.

Ah, thanks for the clarification.  Then it's a duplicate of issue35797.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> concurrent.futures.ProcessPoolExecutor does not work in venv on 
Windows

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[Tom Wilson]

Tom Wilson  added the comment:

In case this is a clue - the attached script "mp_hang2.py" adds a call to 
qsize() and uses only a single consumer. When I run it from the command line it 
does one of two things:


Option 1:

C:\TEMP\Py-3.7.2b-Venv\Scripts>.\python.exe 
"C:\Users\Tom.Wilson\Documents\Python-Bugs\mp_hang2.py"
Creating 1 consumers
Putting
Poisoning
Joining
Process Consumer-1:
Traceback (most recent call last):
  File 
"C:\Users\Tom.Wilson\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py",
 line 297, in _bootstrap
self.run()
  File "C:\Users\Tom.Wilson\Documents\Python-Bugs\mp_hang2.py", line 18, in run
print(f'Queue size: {self.task_queue.qsize()}')
  File 
"C:\Users\Tom.Wilson\AppData\Local\Programs\Python\Python37\lib\multiprocessing\queues.py",
 line 117, in qsize
return self._maxsize - self._sem._semlock._get_value()
PermissionError: [WinError 5] Access is denied


Option 2:

C:\TEMP\Py-3.7.2b-Venv\Scripts>.\python.exe 
"C:\Users\Tom.Wilson\Documents\Python-Bugs\mp_hang2.py"
Creating 1 consumers
Putting
Poisoning
Joining
Queue size: 2147483647
Getting task
 <<< Hangs here >>>


If I can provide anything else please let me know.

--
Added file: https://bugs.python.org/file48070/mp_hang2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[Tom Wilson]

Tom Wilson  added the comment:

Hi there. I get this behavior as well, although only in a venv.

  Main   Virtual  
v3.7.1:260ec2c36a  CompletesCompletes
v3.7.2:9a3ffc0492  Completes  Hangs

Some other details of my setup:
 - Windows 10 Pro, Version 1803 (OS Build 17134.472)
 - Python platform is AMD64 on win32
 - Running from the command line (cmd.exe)
 - The virtual environment was created from the command line like this: 
 .\python -m venv c:\temp\Py-3.7.2b-Venv

--
nosy: +tom.wilson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[STINNER Victor]

STINNER Victor  added the comment:

I ran mp_hang.py on Windows 10 with Python 3.7.2: the script completes (it 
doesn't hang). The issue might be specific to VS Code (on Windows?).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[STINNER Victor]

STINNER Victor  added the comment:

mp_hang.py: I created the example into a script, I added the __main__ section 
described by Antoine. I cannot reproduce the bug on the Python master branch on 
Linux.

@June Kim: What is the output when it hangs? Can you try your example without 
VS Code? For example, try to run it in a cmd.exe terminal?

--
Added file: https://bugs.python.org/file48023/mp_hang.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I couldn't reproduce on Ubuntu either.  I tried the "fork", "forkserver" and 
"spawn" methods (all with 3.7.2).

Terry, if you are on Windows, can you try the script?  Be sure to enclose the 
test() call in a "if __name__ == '__main__'" guard.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[Terry J. Reedy]

Change by Terry J. Reedy :


--
nosy: +davin, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[June Kim]

June Kim  added the comment:

Here is my environment

---system
CPU: Intel i5 @2.67GHz
RAM: 8G
OS: Windows 10 Home (64bit)
OS version: 1803
OS build: 17134.472

---python
version1: 3.7.1 AMD64 on win32
version2: 3.7.2 AMD64 on win32
Python path: (venv)/Scripts/python.exe
IDE: VS Code(1.30.1)
Terminal: Git Bash

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[Rémi Lapeyre]

Rémi Lapeyre  added the comment:

I just tried your script in "9a3ffc" (3.7.2final) and "260ec2c36a" (3.7.1final) 
and it worked on both without halting the main process.

I'm on MacOS Sierra, can you give more details about your environment?

--
nosy: +remi.lapeyre

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[June Kim]

New submission from June Kim :

## Test code ##
## Modified a bit from the original written by Doug Hellmann
## https://pymotw.com/3/multiprocessing/communication.html

import multiprocessing
import time


class Consumer(multiprocessing.Process):
def __init__(self, task_queue, result_queue):
multiprocessing.Process.__init__(self)
self.task_queue = task_queue
self.result_queue = result_queue

def run(self):
proc_name = self.name
while True:
print('Getting task')
next_task = self.task_queue.get()
print(f'task got: {next_task}')
if next_task is None:
print('{}: Exiting'.format(proc_name))
self.task_queue.task_done()
break
print('{}: {}'.format(proc_name, next_task))
answer = next_task()
self.task_queue.task_done()
self.result_queue.put(answer)


class Task:
def __init__(self, a, b):
self.a = a
self.b = b

def __call__(self):
time.sleep(0.1)
return '{self.a} * {self.b} = {product}'.format(
self=self, product=self.a * self.b)

def __str__(self):
return '{self.a} * {self.b}'.format(self=self)


def test():
tasks = multiprocessing.JoinableQueue()
results = multiprocessing.Queue()
num_consumers = multiprocessing.cpu_count() * 2
print('Creating {} consumers'.format(num_consumers))
consumers = [Consumer(tasks, results) for i in range(num_consumers)]
[w.start() for w in consumers]
num_jobs = 10
print('Putting')
[tasks.put(Task(i, i)) for i in range(num_jobs)]
print('Poisoning')
[tasks.put(None) for i in range(num_consumers)]
print('Joining')
tasks.join()
while num_jobs:
result = results.get()
print('Result:', result)
num_jobs -= 1

###
1. This code works perfectly in 3.7.1 but halts the main process in 3.7.2
2. It seems the JoinableQueue is empty when it is accessed by processes.
3. IMHO, resource sharing mechanism in multiprocessing.queue seems not working 
properly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35627] multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1

[June Kim]

Change by June Kim :


--
components: Library (Lib)
nosy: June Kim
priority: normal
severity: normal
status: open
title: multiprocessing.queue in 3.7.2 doesn't behave as it was in 3.7.1
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com