[
https://issues.apache.org/jira/browse/AIRFLOW-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823017#comment-16823017
]
Kaxil Naik commented on AIRFLOW-4336:
-------------------------------------
[~jackjack10] I actually had added a limit on this on one of my recent PRs:
https://github.com/apache/airflow/pull/5054#discussion_r272843304 but as miklaj
correctly suggested that it is used at many places, hence I had to change it.
But I agree we can change this for this operator instead of hook.
> GoogleCloudStorageDownloadOperator allow not to save file to log
> ----------------------------------------------------------------
>
> Key: AIRFLOW-4336
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4336
> Project: Apache Airflow
> Issue Type: Wish
> Components: gcp, operators
> Reporter: jack
> Assignee: Kaxil Naik
> Priority: Major
> Fix For: 1.10.4
>
>
> The operator has
> {code:java}
> store_to_xcom_key {code}
> flag which allow the user to deiced if to store the file in Xcom or not but
> there is no flag to specify if the file should be saved to the task log.
> Currently the operator download the file & auto dump the file into the log.
> We use the operator in production. It dump enormousness amount of files of
> GBs which spam the log with millions of rows.
> I fail to see the point of this auto dump to the log. At the end if there is
> a place where something shouldn't be saved to the log it's here. The operator
> gives a copy of the file. There is no point in having it in the log as well.
>
> There are two approaches to fix this:
> # Add flag and allow the user to deiced.
> # Make it default not to save to the log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
