potiuk commented on code in PR #41142:
URL: https://github.com/apache/airflow/pull/41142#discussion_r1702117633
##########
airflow/providers/microsoft/azure/operators/container_instances.py:
##########
@@ -86,6 +86,12 @@ class AzureContainerInstancesOperator(BaseOperator):
:param container_timeout: max time allowed for the execution of
the container instance.
:param tags: azure tags as dict of str:str
+ :param xcom_all: Control if logs are pushed to XCOM similarly to how
DockerOperator does.
Review Comment:
> You can see the logic controlling this around line 316 of the operator in
the PR.
Yes I see `[-1]`. But does it mean that the whole log is kept in memory
before doing that operation? Or is it somehow optimized under the hood to
retrieve only last line when it is needed ? I am afraid that if you have 1GB
log (which is not uncommon) you will download the whole log from remote service
to **just** print one line. This is not only slow and will delay completion of
the task a lot (sometimes minutes) in a way that is quite unexpected, but if
the log is entirely loaded in memory, it means that the task will grow the
memory by at least 1 GB - only for the purpose of pushing one line to XCo,mm.
Do you know if this is - or can be - optimized @perry2of5 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
