jvstein commented on code in PR #67144:
URL: https://github.com/apache/airflow/pull/67144#discussion_r3284531998
##########
providers/amazon/src/airflow/providers/amazon/aws/log/s3_task_handler.py:
##########
@@ -62,6 +62,8 @@ def upload(self, path: os.PathLike | str, ti: RuntimeTI):
has_uploaded = self.write(log, remote_loc)
if has_uploaded and self.delete_local_copy:
shutil.rmtree(os.path.dirname(local_loc))
+ elif has_uploaded:
+ local_loc.write_text("")
Review Comment:
> Maybe I'm looking at it wrong. I think right now if the S3 upload fails,
then the local log doesn't get cleaned up. Which means if there is a hiccup and
it works the next pass, it will upload duplicate logs.
I think in that case, that means the lines in the local log file are not
present at all in the S3 file. So when it gets appended, there is no
duplication. But maybe I'm the one thinking about it wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
