Joffreybvn commented on issue #28199:
URL: https://github.com/apache/airflow/issues/28199#issuecomment-1698582168

   Hello,
   
   I am having similar issue. **I'm whilling to make a PR**. Same story as 
@PatrickfBraz: giving a try to "full taskflow" and Airflow native features, 
instead of doing everything with KubernetesPodOperator / PythonOperator. Using 
Airflow 2.7.0.
   
   I have a custom XCom backend. My dag consist of: `HttpOperator -> 
SQLExecuteQueryOperator` to pull data from an API, passing it via XCom, and 
saving it into a database. I use a SQL template which pull XCom and render a 
big query:
   ```sql
   {%- set data = ti.xcom_pull(task_ids='pull_data_from_api') -%}
   {%- for entry in data -%}
   INSERT INTO "SCHEMA_NAME"."TABLE_NAME" VALUES ('{{entry|tojson}}');
   {% endfor %}
   ```
   Resulting in: *(This is an example with mockup data)*
   ![Screenshot 2023-08-29 
164037](https://github.com/apache/airflow/assets/17571457/0a6879e2-083b-455f-9175-5d0c0cd4d6f2)
   The database I'm pushing to has limitation: I have to push data doing 
multiple queries. As a workaround, I could do an 'executemany' with `params` in 
a PythonOperator. But as soon as this chunk of code would turn into a custom 
reusable Operator, I will have a templated `parameters` which will anyway save 
all the data into the database.
   
   I also see the `AIRFLOW__CORE__MAX_NUM_RENDERED_TI_FIELDS_PER_TASK ` 
parameter. But rendered templates are great (especially for beginners), I don't 
want to disable them, I want to keep them visible in the UI.
   
   ---
   
   So, if I understand well this discussion, a PR to begin with could render 
the template two times:
   - One for the actual execution of the task, with real data - not stored in 
the db.
   - One to be stored in the db/ shown in the UI, based on orm_deserialize 
method
   ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to