valereColleville commented on issue #53578:
URL: https://github.com/apache/airflow/issues/53578#issuecomment-3097683955
A bit more log after some digging (which seems to confirm that using
EKSPodOperator with Metastore connection is not possible):
In the first level (reation of the operator), the system work, aws
connection are retreived using taskSDK api without issue (i have backport the
fix on main to select TaskSDK if setup inside AWS_Base
```python
@classmethod
def get_connection(cls, conn_id: str) -> Connection:
"""
Get connection, given connection id.
:param conn_id: connection id
:return: connection
"""
import sys
# if SUPERVISOR_COMMS is set, we're in task sdk context
if
hasattr(sys.modules.get("airflow.sdk.execution_time.task_runner"),
"SUPERVISOR_COMMS"):
log.info("Connection search from (via task-sdk) -
airflow.sdk.definitions.connection")
from airflow.sdk.definitions.connection import Connection
conn = Connection.get(conn_id)
log.info("Connection Retrieved '%s' (via task-sdk)",
conn.conn_id)
log.info("Connection Retrieved '%s' (via task-sdk)", conn.extra)
log.info("Connection Retrieved '%s' (via task-sdk)",
conn.conn_type)
return conn
from airflow.models.connection import Connection as ConnectionModel
log.info("Connection search from ConnectionModel
get_connection_from_secrets - airflow.models.connection")
conn = ConnectionModel.get_connection_from_secrets(conn_id)
log.info("Connection Retrieved '%s' (via core Airflow)",
conn.conn_id)
return conn
````
The issue happen in the second level, once the kubeconfig is generated by
EKSHook with the commande to use eks_get_token, when using a connection_id so
that conneciton is pass to the underlaying sh exec command of the kubeconfig
(make sens).
But when the exec run, it doesn't detect the supervisor_comms (probably not
loaded), so it use the default "not-tasksdk api" to solve the connection_id
when creating the EKSHook (from the kubectl exec call that time).
https://github.com/apache/airflow/blob/3.0.3/airflow-core/src/airflow/models/connection.py#L466
So from there it test all backend_secret, but failling over metastore one so
not finding any connection_id.
Logs from the exec sh -c commande from generated kubeconfig
```
[2025-07-21T17:07:12.555+0000] {base.py:75} INFO - Connection search from
ConnectionModel get_connection_from_secrets - airflow.models.connection
[2025-07-21T17:07:12.555+0000] {connection.py:489} INFO - VAC-MOD-LOG:
enabled only if SecretCache.init() has been called first
[2025-07-21T17:07:12.555+0000] {connection.py:496} INFO - VAC-MOD-LOG:
Iterate over backends
[2025-07-21T17:07:12.556+0000] {connection.py:498} INFO - VAC-MOD-LOG:
Checking secrets backend (EnvironmentVariablesBackend) .
[2025-07-21T17:07:12.556+0000] {connection.py:498} INFO - VAC-MOD-LOG:
Checking secrets backend (MetastoreBackend) .
[2025-07-21T17:07:12.556+0000] {connection.py:510} INFO - VAC-MOD-LOG:
Unable to retrieve connection from secrets backend (MetastoreBackend). Checking
subsequent secrets backend.
[2025-07-21T17:07:12.556+0000] {base_aws.py:623} WARNING - Unable to find
AWS Connection ID 'val_aws_assume_test', switching to empty.
[2025-07-21T17:07:12.556+0000] {base_aws.py:189} INFO - No connection ID
provided. Fallback on boto3 credential strategy (region_name='eu-central-1').
See:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2025-07-21T17:07:12.563+0000] {base_aws.py:189} INFO - No connection ID
provided. Fallback on boto3 credential strategy (region_name='eu-central-1').
See:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
[2025-07-21T17:07:13.064+0000] {base.py:75} INFO - Connection search from
ConnectionModel get_connection_from_secrets - airflow.models.connection
[2025-07-21T17:07:13.064+0000] {connection.py:489} INFO - VAC-MOD-LOG:
enabled only if SecretCache.init() has been called first
[2025-07-21T17:07:13.064+0000] {connection.py:496} INFO - VAC-MOD-LOG:
Iterate over backends
[2025-07-21T17:07:13.064+0000] {connection.py:498} INFO - VAC-MOD-LOG:
Checking secrets backend (EnvironmentVariablesBackend) .
[2025-07-21T17:07:13.065+0000] {connection.py:498} INFO - VAC-MOD-LOG:
Checking secrets backend (MetastoreBackend) .
[2025-07-21T17:07:13.065+0000] {connection.py:510} INFO - VAC-MOD-LOG:
Unable to retrieve connection from secrets backend (MetastoreBackend). Checking
subsequent secrets backend.
[2025-07-21T17:07:13.065+0000] {base_aws.py:623} WARNING - Unable to find
AWS Connection ID 'aws_default', switching to empty.
[2025-07-21T17:07:13.065+0000] {base_aws.py:189} INFO - No connection ID
provided. Fallback on boto3 credential strategy (region_name='eu-central-1').
See:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
expirationTimestamp: 2025-07-21T17:21:13Z, token: k8s-aws-v1.xxxx
```
Where the exact same commande run from inside the worker directly generate
the following logs:
```
[2025-07-21T17:12:27.099+0000] {base.py:75} INFO - Connection search from
ConnectionModel get_connection_from_secrets - airflow.models.connection
[2025-07-21T17:12:27.100+0000] {connection.py:489} INFO - VAC-MOD-LOG:
enabled only if SecretCache.init() has been called first
[2025-07-21T17:12:27.100+0000] {connection.py:496} INFO - VAC-MOD-LOG:
Iterate over backends
[2025-07-21T17:12:27.101+0000] {connection.py:498} INFO - VAC-MOD-LOG:
Checking secrets backend (EnvironmentVariablesBackend) .
[2025-07-21T17:12:27.101+0000] {connection.py:498} INFO - VAC-MOD-LOG:
Checking secrets backend (MetastoreBackend) .
[2025-07-21T17:12:27.101+0000] {metastore.py:51} INFO - VAC-MOD-LOG:
METASTORE GET CONNECTION val_aws_assume_test
[2025-07-21T17:12:28.668+0000] {base.py:77} INFO - Connection Retrieved
'val_aws_assume_test' (via core Airflow)
[2025-07-21T17:12:29.547+0000] {base.py:75} INFO - Connection search from
ConnectionModel get_connection_from_secrets - airflow.models.connection
[2025-07-21T17:12:29.547+0000] {connection.py:489} INFO - VAC-MOD-LOG:
enabled only if SecretCache.init() has been called first
[2025-07-21T17:12:29.547+0000] {connection.py:496} INFO - VAC-MOD-LOG:
Iterate over backends
[2025-07-21T17:12:29.548+0000] {connection.py:498} INFO - VAC-MOD-LOG:
Checking secrets backend (EnvironmentVariablesBackend) .
[2025-07-21T17:12:29.548+0000] {connection.py:498} INFO - VAC-MOD-LOG:
Checking secrets backend (MetastoreBackend) .
[2025-07-21T17:12:29.548+0000] {metastore.py:51} INFO - VAC-MOD-LOG:
METASTORE GET CONNECTION aws_default
[2025-07-21T17:12:29.564+0000] {base.py:77} INFO - Connection Retrieved
'aws_default' (via core Airflow)
[2025-07-21T17:12:29.564+0000] {connection_wrapper.py:325} INFO - AWS
Connection (conn_id='aws_default', conn_type='aws') credentials retrieved from
login and password.
expirationTimestamp: 2025-07-21T17:26:29Z, token:
```
(from the worker we run the commande without export
PYTHON_OPERATORS_VIRTUAL_ENV_MODE=1, if we use it, we have the same result as
the sub-exec from kubeconfig)
Hope that addition of details will help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
