henry3260 commented on PR #60398:
URL: https://github.com/apache/airflow/pull/60398#issuecomment-3736021478
> Hey, I think I know why the test was flaky. Whats happening is in the
S3Hook, the download msg isn't appended it's reassigned, so if it fits both
criteria, meaning the size changed AND the last_modified date has changed, then
it only gets the latter message and not both which is why the test was a bit
flaky. Might be worth fixing that directly, maybe we have a more general
download message or we log both reasons.
>
> Code I'm referring to:
>
> ```
> if s3_object.size != local_stats.st_size:
> should_download = True
> download_msg = f"S3 object size ({s3_object.size}) and local file size
({local_stats.st_size}) differ."
>
> s3_last_modified = s3_object.last_modified
> if local_stats.st_mtime < s3_last_modified.timestamp():
> should_download = True
> download_msg = f"S3 object last modified
({s3_last_modified.microsecond}) and local file last modified
({local_stats.st_mtime}) differ."```
> ```
Wow, great catch! That explains everything completely.
Do you want me to include this fix in my PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
