potiuk commented on issue #23901: URL: https://github.com/apache/airflow/issues/23901#issuecomment-1142296307
This is correct. Airflow does not - on its own propagate SIGALRM (this is how timeout is implemented) to a subprocess. And your subprocess would have to handle the SIGALRM in this case. Airflow timeout only works if you run a code in the task process but you do more. People contxt manager is documented here: https://docs.python.org/3/library/subprocess.html#subprocess.Popen: > Popen objects are supported as context managers via the [with (https://docs.python.org/3/reference/compound_stmts.html#with) statement: on exit, standard file descriptors are closed, and the process is waited for. You are starting subprocess and waiting for it, the only way to stop the waiting is to kill the process, but airflow has no notion of the sub-process you started so it cannot do much there (short of killing all the process group forcefully which it should not do), So you should manage both timeout and killing the subprocess on your own. The easy way to do it it s to start your subprocess without context manager (i.e. with) and explicitly wait for it with timeout https://docs.python.org/3/library/subprocess.html#popen-objects and terminate the process in the way you think is appropriate after the timeout expires. Airflow simply cannot do it for you because it does not know how many and what kind of subprocesses you started. Only your code knows it and only your code can terminate those sub-processes in a gentle (or not if you choose so) way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
