ephraimbuddy commented on a change in pull request #16002:
URL: https://github.com/apache/airflow/pull/16002#discussion_r637637964
##########
File path: airflow/providers/microsoft/azure/log/wasb_task_handler.py
##########
@@ -180,15 +180,20 @@ def wasb_write(self, log: str, remote_log_location: str,
append: bool = True) ->
the new log is appended to any existing logs.
:type append: bool
"""
- if append and self.wasb_log_exists(remote_log_location):
- old_log = self.wasb_read(remote_log_location)
- log = '\n'.join([old_log, log]) if old_log else log
-
try:
- self.hook.load_string(
- log,
- self.wasb_container,
- remote_log_location,
- )
+ if append and self.wasb_log_exists(remote_log_location):
+ self.hook.append_string(
+ log,
+ self.wasb_container,
+ remote_log_location
+ )
+ else:
+ self.hook.load_string(
+ log,
+ self.wasb_container,
+ remote_log_location,
+ blob_type = 'AppendBlob',
+ overwrite=True
+ )
Review comment:
> or simply, use the read-overwrite method based on BlockBlob
I prefer the above with the `overwrite` being an argument to the wasb_write
method so it's not hardcoded.
Also, see
https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
