potiuk commented on issue #33178:
URL: https://github.com/apache/airflow/issues/33178#issuecomment-1674000928
I think my deduction and observations were quite right @uranusjr.
There is indeed a session management problem in only mapped operators.
The out-of-sync operatiors are with allmost 100% certainty caused by this
code in airflow/midels/mapped_operator.py and in particular by not passing
down the session to `render_template_fields` method:
```
# Ideally we'd like to pass in session as an argument to this
function,
# but we can't easily change this function signature since operators
# could override this. We can't use @provide_session since it closes
and
# expunges everything, which we don't want to do when we are so
"deep"
# in the weeds here. We don't close this session for the same reason.
session = settings.Session()
mapped_kwargs, seen_oids = self._expand_mapped_kwargs(context,
session)
unmapped_task = self.unmap(mapped_kwargs)
context_update_for_unmapped(context, unmapped_task)
# Since the operators that extend `BaseOperator` are not subclasses
of
# `MappedOperator`, we need to call `_do_render_template_fields` from
# the unmapped task in order to call the operator method when we
override
# it to customize the parsing of nested fields.
unmapped_task._do_render_template_fields(
parent=unmapped_task,
template_fields=self.template_fields,
```
By addind logs for opening and destroying session I caught one case when it
happeened and the only extra sessions created were there in
`render_template_fields` - which caught my attention to this ^^
https://github.com/apache/airflow/actions/runs/5826237682/job/15799895643?pr=33259#step:6:9927
I think we should find out a better solution than this hack - it is going to
cause us more troubles than it's worth IMHO. I am happy to brainstorm on that
or see whether I can come up with some proposal.
But yes - it looks like our tests are actually detecting a real problem with
our implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
