meel-ayush opened a new issue, #76553:
URL: https://github.com/apache/airflow/issues/76553
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.4.0rc3 (and main)
### What happened and how to reproduce it?
### Description
Following up on #71968 and #76547,
`Connection.get_connection_from_secrets()` currently does not accept or forward
an optional `session` parameter to secrets backends.
When internal components or extension points holding an open database
transaction read a connection via `Connection.get_connection_from_secrets()`,
they cannot pass their active session. As a result,
`MetastoreBackend.get_connection()` always falls back to opening a separate
session via `create_session(scoped=False)`.
This causes an extra connection checkout from the SQLAlchemy pool on every
lookup, adding latency and connection overhead in high-throughput loops.
Additionally, transactional callers cannot read newly added uncommitted
connections within the same transaction.
### Affected Code
In `airflow-core/src/airflow/models/connection.py:522-583`:
```python
@classmethod
def get_connection_from_secrets(cls, conn_id: str, team_name: str | None =
None) -> Connection:
...
for secrets_backend in ensure_secrets_loaded():
try:
conn = call_secrets_backend_method(
secrets_backend.get_connection, team_name=team_name,
conn_id=conn_id
)
```
By comparison, `Variable.get_variable_from_secrets()` in
`airflow-core/src/airflow/models/variable.py:494-536` already accepts `*,
session: Session | None = None` and forwards it to
`MetastoreBackend.get_variable()`.
### Steps to Reproduce
Calling `Connection.get_connection_from_secrets("my_conn", session=session)`
raises:
```text
TypeError: Connection.get_connection_from_secrets() got an unexpected
keyword argument 'session'
```
### Expected Behavior
`Connection.get_connection_from_secrets()` should accept an optional
`session: Session | None = None` keyword argument and forward it to
`MetastoreBackend.get_connection()` when supported.
### Proposed Fix
1. Add `*, session: Session | None = None` keyword-only parameter to
`Connection.get_connection_from_secrets()` in
`airflow-core/src/airflow/models/connection.py`.
2. Check `accepts_kwarg(secrets_backend.get_connection, "session")` and pass
`session` when calling `call_secrets_backend_method()`.
I have verified this behavior and am working on a PR with unit tests to add
this session propagation.
### What you think should happen instead?
_No response_
### Operating System
Kali GNU/Linux
### Deployment
None
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]