[ 
https://issues.apache.org/jira/browse/AIRFLOW-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277354#comment-15277354
 ] 

ASF subversion and git services commented on AIRFLOW-61:
--------------------------------------------------------

Commit 0e5fb905c2c6bb02b8a3aabe36af3189d407ed23 in incubator-airflow's branch 
refs/heads/airbnb_rb1.7.1_3 from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0e5fb90 ]

Fix corner case with joining processes/queues (#1473)

If a process places items in a queue and the process is joined before the queue 
is emptied, it can lead to a deadlock under some circumstances. Closes 
AIRFLOW-61.

See for example: 
https://docs.python.org/3/library/multiprocessing.html#all-start-methods 
("Joining processes that use queues")
http://stackoverflow.com/questions/31665328/python-3-multiprocessing-queue-deadlock-when-calling-join-before-the-queue-is-em
http://stackoverflow.com/questions/31708646/process-join-and-queue-dont-work-with-large-numbers
http://stackoverflow.com/questions/19071529/python-multiprocessing-125-list-never-finishes

> Corner case with deadlocked multiprocessing queues
> --------------------------------------------------
>
>                 Key: AIRFLOW-61
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-61
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: Airflow 1.7.0
>            Reporter: Jeremiah Lowin
>            Assignee: Jeremiah Lowin
>             Fix For: Airflow 1.7.1
>
>
> I've never experienced this before, but it came up while testing this 
> morning. Joining a process that puts items in a queue can apparently lead to 
> deadlocks. This happens in the multiprocessing-enabled scheduler. 
> See for example: 
> https://docs.python.org/3/library/multiprocessing.html#all-start-methods 
> ("Joining processes that use queues") and related issues:
> http://stackoverflow.com/questions/31665328/python-3-multiprocessing-queue-deadlock-when-calling-join-before-the-queue-is-em
> http://stackoverflow.com/questions/31708646/process-join-and-queue-dont-work-with-large-numbers
> http://stackoverflow.com/questions/19071529/python-multiprocessing-125-list-never-finishes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to