hterik commented on issue #14813: URL: https://github.com/apache/airflow/issues/14813#issuecomment-972665718
Note that SubprocessHook still sends the stderr of the subprocess straight to stdout without going to logger, it also differs a lot from stdlib subprocess behavior which can be unexpected, like changing the cwd to a tempdir. Having written a similar class myself before, just have to warn that it is not as easy as one thinks to simultaneously capture and stream both stdout+stderr to a logger. You will end up more or less re-implementing POpen.communicate(), to not risk either of the streams getting full and blocking read of the other. I would recommend looking for a subprocess wrapper outside of airflow that can handle this. It's a general python logging + subprocess issue which not only affects airflow. Maybe try https://github.com/amoffat/sh, haven't used it much but it looks like it would might possible to tweak into this, right now it has the subprocess output in DEBUG logs but those logs also contain a lot of other verbose data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
