seanghaeli opened a new issue, #76640: URL: https://github.com/apache/airflow/issues/76640
`RemoteLogIO.upload()` returns `None` (`shared/logging/src/airflow_shared/logging/remote.py`), and the handlers signal the result in different, implicit ways: - S3, GCS and WASB empty the local file when the upload works and leave it when it fails, only logging the error. - HDFS and OSS never empty the file, so a caller can't tell a stored file from a failed one. - OSS appends to the remote object, so uploading the same file twice stores its lines twice. Anything that uploads a file more than once, or needs to know whether it can delete the local copy, has to guess from the file's contents. The triggerer's watcher log files (#76527) hit this: they can only retry a failed part for S3, GCS and WASB, and OSS already stores a released watcher's file twice. Suggestion: have `upload()` return whether the file was stored (or raise on failure), and make OSS replace the object instead of appending, or skip content it already stored. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
