jack created AIRFLOW-3499:
-----------------------------
Summary: Add flag to Opeators to write the Render to the log.
Key: AIRFLOW-3499
URL: https://issues.apache.org/jira/browse/AIRFLOW-3499
Project: Apache Airflow
Issue Type: Wish
Affects Versions: 2.0.0
Reporter: jack
*Motivation:*
I have few operators who uses Variable. The variable is updated consistently
and gets overwritten with more recent data.
Say I have this operator:
{code:java}
NEXT_ORDER_ID= Variable.get("next_order_id_to_import")
import_orders = MySqlToGoogleCloudStorageOperator(
task_id='import',
mysql_conn_id='c_production',
google_cloud_storage_conn_id='gcp_m',
approx_max_file_size_bytes = 100000000,
sql = 'Select … from … where orders_id between {{ params.next_order_id}}
and {{ ti.xcom_pull('max_order_id') }}',
params={'next_order_id_to_import': NEXT_ORDER_ID},
bucket=GCS_BUCKET_ID,
filename=file_name_orders_products,
dag=dag)
{code}
The problem is that If I can not see the parameters for this query on the
render page.
To be exact :
{code:java}
{{ ti.xcom_pull('max_order_id') }}{code}
- Saved in Database so it always show the correct one per DAG.
{code:java}
{{ params.next_order_id}} {code}
- Will always show the most recent value as this is not DAG parameter and not
saved to the database. When clicking on the render page it goes to the variable
and take the value from there, regardless if this was the value during the run
or not.
*Suggested Solution:*
Since it's unlikely that the Render tab will be change (as my use case could be
different than how other use it) the best solution is simply to allow to write
the Render as it was during the execution of the task to the task log. This
will help to traceback issues.
Basically add to all operators (Base Operator?) a flag :
{code:java}
write_render_to_log {code}
which default is False. If this flag set to true than the render content will
be flushed to the log of the task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
