[
https://issues.apache.org/jira/browse/AIRFLOW-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamil Bregula updated AIRFLOW-5865:
-----------------------------------
Component/s: (was: core)
gcp
> BigQueryOperator should allow adding kev/values to xcom
> -------------------------------------------------------
>
> Key: AIRFLOW-5865
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5865
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
> Affects Versions: 1.10.6
> Reporter: Omid Vahdaty
> Priority: Minor
>
> I am trying to save the dag and query i run in BigQueryOperator to XCOM as
> key/value so i can later on, do some analytics of cost per dag per query.
> attached the code i use to send key/value (myQuery/myproject ) via
> python_callable
> However - what i get in the XCOM DB is
> job_id | X job_CAvEphremRU2_RpEc-Hj4uwKuUPrq
> i.e everything but what i am sending.
>
> def push(**kwargs): kwargs['ti'].xcom_push(key=kwargs['random_query'],
> value=kwargs['random_project'])
> delete_query = """ delete from `DATA_LAKE_INGESTION_US.One_Truth4` where
> date BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY) AND
> DATE_SUB(CURRENT_DATE(), INTERVAL 0 DAY) """
> myQuery = delete_query
> myproject='OneTruth'
> bq_delete_query = BigQueryOperator( task_id='bq_delete_query', bql=sql,
> bigquery_conn_id='bigquery_default', use_legacy_sql=False, dag=dag,
> python_callable=push, op_kwargs={'random_query': myQuery, 'random_project':
> myproject}, )
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
