Omid Vahdaty created AIRFLOW-5865:
-------------------------------------
Summary: BigQueryOperator should allow adding kev/values to xcom
Key: AIRFLOW-5865
URL: https://issues.apache.org/jira/browse/AIRFLOW-5865
Project: Apache Airflow
Issue Type: Improvement
Components: core
Affects Versions: 1.10.6
Reporter: Omid Vahdaty
I am trying to save the dag and query i run in BigQueryOperator to XCOM as
key/value so i can later on, do some analytics of cost per dag per query.
attached the code i use to send key/value (myQuery/myproject ) via
python_callable
However - what i get in the XCOM DB is
job_id | X job_CAvEphremRU2_RpEc-Hj4uwKuUPrq
i.e everything but what i am sending.
def push(**kwargs): kwargs['ti'].xcom_push(key=kwargs['random_query'],
value=kwargs['random_project'])
delete_query = """ delete from `DATA_LAKE_INGESTION_US.One_Truth4` where date
BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY) AND DATE_SUB(CURRENT_DATE(),
INTERVAL 0 DAY) """
myQuery = delete_query
myproject='OneTruth'
bq_delete_query = BigQueryOperator( task_id='bq_delete_query', bql=sql,
bigquery_conn_id='bigquery_default', use_legacy_sql=False, dag=dag,
python_callable=push, op_kwargs={'random_query': myQuery, 'random_project':
myproject}, )
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
