[
https://issues.apache.org/jira/browse/AIRFLOW-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jack updated AIRFLOW-4336:
--------------------------
Description:
The operator has
{code:java}
store_to_xcom_key {code}
flag which allow the user to deiced if to store the file in Xcom or not but
there is no flag to specify if the file should be saved to the task log.
Currently the operator download the file & auto dump the file into the log.
We use the operator in production. It dump enormousness amount of files of GBs
which spam the log with millions of rows.
I fail to see the point of this auto dump to the log. At the end if there is a
place where something shouldn't be saved to the log it's here. The operator
gives a copy of the file. There is no point in having it in the log as well.
There are two approaches to fix this:
# Add flag and allow the user to deiced.
# Make it default not to save to the log.
was:
The operator has
{code:java}
store_to_xcom_key {code}
flag which allow the user to deiced if to store the file in Xcom or not but
there is no flag to specify if the file should be saved to the task log.
Currently the operator auto save the file to the log.
We use the operator in production. It download enormousness amount of files of
GBs which spam the log with millions of rows.
> GoogleCloudStorageDownloadOperator allow not to save file to log
> ----------------------------------------------------------------
>
> Key: AIRFLOW-4336
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4336
> Project: Apache Airflow
> Issue Type: Wish
> Components: gcp, operators
> Reporter: jack
> Priority: Major
> Fix For: 1.10.4
>
>
> The operator has
> {code:java}
> store_to_xcom_key {code}
> flag which allow the user to deiced if to store the file in Xcom or not but
> there is no flag to specify if the file should be saved to the task log.
> Currently the operator download the file & auto dump the file into the log.
> We use the operator in production. It dump enormousness amount of files of
> GBs which spam the log with millions of rows.
> I fail to see the point of this auto dump to the log. At the end if there is
> a place where something shouldn't be saved to the log it's here. The operator
> gives a copy of the file. There is no point in having it in the log as well.
>
> There are two approaches to fix this:
> # Add flag and allow the user to deiced.
> # Make it default not to save to the log.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
