[
https://issues.apache.org/jira/browse/AIRFLOW-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822872#comment-16822872
]
jack commented on AIRFLOW-4336:
-------------------------------
[~kaxilnaik] still. haven't seen any other operator that saves entire file to
the log. The log is for messages usually.
Take for example the equivalent in Azure:
[https://github.com/apache/airflow/blob/993779f93df172f5f774f1ea2634c55e9c1212c4/airflow/contrib/hooks/wasb_hook.py#L120]
There is no such log.
> GoogleCloudStorageDownloadOperator allow not to save file to log
> ----------------------------------------------------------------
>
> Key: AIRFLOW-4336
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4336
> Project: Apache Airflow
> Issue Type: Wish
> Components: gcp, operators
> Reporter: jack
> Assignee: Kaxil Naik
> Priority: Major
> Fix For: 1.10.4
>
>
> The operator has
> {code:java}
> store_to_xcom_key {code}
> flag which allow the user to deiced if to store the file in Xcom or not but
> there is no flag to specify if the file should be saved to the task log.
> Currently the operator download the file & auto dump the file into the log.
> We use the operator in production. It dump enormousness amount of files of
> GBs which spam the log with millions of rows.
> I fail to see the point of this auto dump to the log. At the end if there is
> a place where something shouldn't be saved to the log it's here. The operator
> gives a copy of the file. There is no point in having it in the log as well.
>
> There are two approaches to fix this:
> # Add flag and allow the user to deiced.
> # Make it default not to save to the log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
