[
https://issues.apache.org/jira/browse/AIRFLOW-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825058#comment-16825058
]
ASF subversion and git services commented on AIRFLOW-4336:
----------------------------------------------------------
Commit 6cbeb158a5aaa5a40cbdb46a7ea556331dab59aa in airflow's branch
refs/heads/v1-10-stable from Kaxil Naik
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=6cbeb15 ]
[AIRFLOW-4336] Stop showing entire GCS files bytes in log for
gcs_download_operator (#5151)
(cherry picked from commit 11dc3df4118fb767462d2b776914f44ec96fa7a4)
> GoogleCloudStorageDownloadOperator allow not to save file to log
> ----------------------------------------------------------------
>
> Key: AIRFLOW-4336
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4336
> Project: Apache Airflow
> Issue Type: Wish
> Components: gcp, operators
> Reporter: jack
> Assignee: Kaxil Naik
> Priority: Major
> Fix For: 1.10.4
>
>
> The operator has
> {code:java}
> store_to_xcom_key {code}
> flag which allow the user to deiced if to store the file in Xcom or not but
> there is no flag to specify if the file should be saved to the task log.
> Currently the operator download the file & auto dump the file into the log.
> We use the operator in production. It dump enormousness amount of files of
> GBs which spam the log with millions of rows.
> I fail to see the point of this auto dump to the log. At the end if there is
> a place where something shouldn't be saved to the log it's here. The operator
> gives a copy of the file. There is no point in having it in the log as well.
>
> There are two approaches to fix this:
> # Add flag and allow the user to deiced.
> # Make it default not to save to the log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
