Joffreybvn commented on issue #28199:
URL: https://github.com/apache/airflow/issues/28199#issuecomment-1698582168
Hello,
I am having similar issue. **I'm whilling to make a PR**. Same story as
@PatrickfBraz: giving a try to "full taskflow" and Airflow native features,
instead of doing everything with KubernetesPodOperator / PythonOperator. Using
Airflow 2.7.0.
I have a custom XCom backend. My dag consist of: `HttpOperator ->
SQLExecuteQueryOperator` to pull data from an API, passing it via XCom, and
saving it into a database. I use a SQL template which pull XCom and render a
big query:
```sql
{%- set data = ti.xcom_pull(task_ids='pull_data_from_api') -%}
{%- for entry in data -%}
INSERT INTO "SCHEMA_NAME"."TABLE_NAME" VALUES ('{{entry|tojson}}');
{% endfor %}
```
Resulting in: *(This is an example with mockup data)*

The database I'm pushing to has limitation: I have to push data doing
multiple queries. As a workaround, I could do an 'executemany' with `params` in
a PythonOperator. But as soon as this chunk of code would turn into a custom
reusable Operator, I will have a templated `parameters` which will anyway save
all the data into the database.
I also see the `AIRFLOW__CORE__MAX_NUM_RENDERED_TI_FIELDS_PER_TASK `
parameter. But rendered templates are great (especially for beginners), I don't
want to disable them, I want to keep them visible in the UI.
---
So, if I understand well this discussion, a PR to begin with could render
the template two times:
- One for the actual execution of the task, with real data - not stored in
the db.
- One to be stored in the db/ shown in the UI, based on orm_deserialize
method
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]