meel-ayush opened a new issue, #76553:
URL: https://github.com/apache/airflow/issues/76553

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.4.0rc3 (and main)
   
   ### What happened and how to reproduce it?
   
   ### Description
   
   Following up on #71968 and #76547, 
`Connection.get_connection_from_secrets()` currently does not accept or forward 
an optional `session` parameter to secrets backends.
   
   When internal components or extension points holding an open database 
transaction read a connection via `Connection.get_connection_from_secrets()`, 
they cannot pass their active session. As a result, 
`MetastoreBackend.get_connection()` always falls back to opening a separate 
session via `create_session(scoped=False)`.
   
   This causes an extra connection checkout from the SQLAlchemy pool on every 
lookup, adding latency and connection overhead in high-throughput loops. 
Additionally, transactional callers cannot read newly added uncommitted 
connections within the same transaction.
   
   ### Affected Code
   
   In `airflow-core/src/airflow/models/connection.py:522-583`:
   
   ```python
   @classmethod
   def get_connection_from_secrets(cls, conn_id: str, team_name: str | None = 
None) -> Connection:
       ...
       for secrets_backend in ensure_secrets_loaded():
           try:
               conn = call_secrets_backend_method(
                   secrets_backend.get_connection, team_name=team_name, 
conn_id=conn_id
               )
   ```
   
   By comparison, `Variable.get_variable_from_secrets()` in 
`airflow-core/src/airflow/models/variable.py:494-536` already accepts `*, 
session: Session | None = None` and forwards it to 
`MetastoreBackend.get_variable()`.
   
   ### Steps to Reproduce
   
   Calling `Connection.get_connection_from_secrets("my_conn", session=session)` 
raises:
   
   ```text
   TypeError: Connection.get_connection_from_secrets() got an unexpected 
keyword argument 'session'
   ```
   
   ### Expected Behavior
   
   `Connection.get_connection_from_secrets()` should accept an optional 
`session: Session | None = None` keyword argument and forward it to 
`MetastoreBackend.get_connection()` when supported.
   
   ### Proposed Fix
   
   1. Add `*, session: Session | None = None` keyword-only parameter to 
`Connection.get_connection_from_secrets()` in 
`airflow-core/src/airflow/models/connection.py`.
   2. Check `accepts_kwarg(secrets_backend.get_connection, "session")` and pass 
`session` when calling `call_secrets_backend_method()`.
   
   I have verified this behavior and am working on a PR with unit tests to add 
this session propagation.
   
   
   ### What you think should happen instead?
   
   _No response_
   
   ### Operating System
   
   Kali GNU/Linux
   
   ### Deployment
   
   None
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to