potiuk commented on issue #12969: URL: https://github.com/apache/airflow/issues/12969#issuecomment-744062884
Hey @ashb, @kaxil @mik-laj (since you were looking at similar problems before) - I think we have serious rc3 candidate here. From my investigation, it looks like the S3 Handler's close() method is never called when the task finishes. Since we are using mulitprocessing/forking, I beleive this is because the logging.shutdown() method is never executed. It is registered in logging via registering atexit, and it seems that the processes that are started via multiplrocessing technicaly do not execute the "atexit" method: https://stackoverflow.com/questions/34506638/how-to-register-atexit-function-in-pythons-multiprocessing-subprocess It looks like everything works fine until the close() method that is supposed to write the log. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
