flolas commented on a change in pull request #16002:
URL: https://github.com/apache/airflow/pull/16002#discussion_r637624343
##########
File path: airflow/providers/microsoft/azure/log/wasb_task_handler.py
##########
@@ -180,15 +180,20 @@ def wasb_write(self, log: str, remote_log_location: str,
append: bool = True) ->
the new log is appended to any existing logs.
:type append: bool
"""
- if append and self.wasb_log_exists(remote_log_location):
- old_log = self.wasb_read(remote_log_location)
- log = '\n'.join([old_log, log]) if old_log else log
-
try:
- self.hook.load_string(
- log,
- self.wasb_container,
- remote_log_location,
- )
+ if append and self.wasb_log_exists(remote_log_location):
+ self.hook.append_string(
+ log,
+ self.wasb_container,
+ remote_log_location
+ )
+ else:
+ self.hook.load_string(
+ log,
+ self.wasb_container,
+ remote_log_location,
+ blob_type = 'AppendBlob',
+ overwrite=True
+ )
Review comment:
or simply, use the read-overwrite method based on BlobkBlob....
##########
File path: airflow/providers/microsoft/azure/log/wasb_task_handler.py
##########
@@ -180,15 +180,20 @@ def wasb_write(self, log: str, remote_log_location: str,
append: bool = True) ->
the new log is appended to any existing logs.
:type append: bool
"""
- if append and self.wasb_log_exists(remote_log_location):
- old_log = self.wasb_read(remote_log_location)
- log = '\n'.join([old_log, log]) if old_log else log
-
try:
- self.hook.load_string(
- log,
- self.wasb_container,
- remote_log_location,
- )
+ if append and self.wasb_log_exists(remote_log_location):
+ self.hook.append_string(
+ log,
+ self.wasb_container,
+ remote_log_location
+ )
+ else:
+ self.hook.load_string(
+ log,
+ self.wasb_container,
+ remote_log_location,
+ blob_type = 'AppendBlob',
+ overwrite=True
+ )
Review comment:
or simply, use the read-overwrite method based on BlockBlob....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
