potiuk commented on issue #14896: URL: https://github.com/apache/airflow/issues/14896#issuecomment-912136260
Well, I think we are talking about two different issues with "deamonic processes" and this is the source of confusion here. 1) The original stack trace here (by @ivanrezic-maistra ) was about using PythonOperator running "multiprocessing" using LocalExecutor - and this is the one which I wrote about in the `billiard` context. The issue was about using multiprocessing inside your own customised Python operators - which I saw several people solved by simply using `import billiard as multiprocessing` in their own code. 2) The second issue added in the same thread by @damon09273 and @ahazeemi also mentioned by @kaxil have been merged and solved and released in Airflow 2.1.2 (see milestone at https://github.com/apache/airflow/pull/16700) - but this one was about CeleryKubernetes Executor and that was not a "custom" code - it was Airflow itself that failed in this case., Even if the error message was the same, those two issues have very different root cause, and while they were somewhat hijacked here - the error 1) still requires the `billiard` importing to be solved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
