TheoLauw opened a new issue, #56501:
URL: https://github.com/apache/airflow/issues/56501
### Apache Airflow version
3.1.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
We are experiencing a critical issue with log upload to S3 when using
DbtTaskGroup from Cosmos in our Airflow 3.x environment (Cosmos 1.10.2,
dbt-core 1.11.x, Astronomer runtime). The airflow deployment is running on EKS
in AWS and tasks in airflow are run as KubernetesExecutor (tried with
CeleryExecutors and exact same problem was observed).
When a Cosmos DbtTaskGroup task fails and is retried, the logs for the
failed attempts are not uploaded to S3 (remote logging enabled).
Only the logs from the last retry are available in the Airflow UI and on S3.
This does not happen with standard Airflow operators (e.g. BashOperator),
only with Cosmos DbtTaskGroup. What is weird is that if we choose to put no
retry at all, we have the logs the first time. But if we choose to put one
retry on the airflow dag, we lose the first attempt but we get the second.
During the execution of the tasks, we have the logs because airflow is still
pulling them directly from the Task pod. But as soon as finished, if the pod
have an error, it crashes before being table to upload any log to S3. So logs
of the attempt are lost.
The pod running the task is stopped abruptly after failure, before logs can
be uploaded.
We see this error in the logs:
airflow.sdk.api.client.ServerResponseError: Server returned errormessage':
'Not Found', 'detail': {'detail': 'Not Found'}}...File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py",
line 258, in _from_frame
Full stack trace of the failing pod not succeeding to upload its logs to S3
:
{"timestamp":"2025-10-07T09:37:34.134606Z","level":"warning","event":"The
`airflow.models.baseoperator.BaseOperator` attribute is deprecated. Please use
`'airflow.sdk.bases.operator.BaseOperator'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:34.135126Z","level":"warning","event":"The
`airflow.hooks.base.BaseHook` attribute is deprecated. Please use
`'airflow.sdk.bases.hook.BaseHook'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.384648Z","level":"warning","event":"section/key
[openlineage/namespace] not found in
config","logger":"airflow.configuration","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:37.385521Z","level":"warning","event":"The
`airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use
`'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.387229Z","level":"warning","event":"The
`airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use
`'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.388410Z","level":"warning","event":"The
`airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use
`'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.389432Z","level":"warning","event":"The
`airflow.hooks.base.BaseHook` attribute is deprecated. Please use
`'airflow.sdk.bases.hook.BaseHook'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.403079Z","level":"warning","event":"The
`airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use
`'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.408536Z","level":"warning","event":"The
`airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use
`'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.409877Z","level":"warning","event":"The
`airflow.utils.task_group.TaskGroup` attribute is deprecated. Please use
`'airflow.sdk.TaskGroup'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:37.803284Z","level":"warning","event":"The
`airflow.hooks.base.BaseHook` attribute is deprecated. Please use
`'airflow.sdk.bases.hook.BaseHook'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:38.058447Z","level":"warning","event":"The
`airflow.utils.timezone.utcnow` attribute is deprecated. Please use
`'airflow.sdk.timezone.utcnow'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:38.383665Z","level":"info","event":"DAG
bundles loaded:
dags-folder","logger":"airflow.dag_processing.bundles.manager.DagBundlesManager","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.383986Z","level":"info","event":"Filling
up the DagBag from
/opt/airflow/dags/test/airflow/dags/simple_dag.py","logger":"airflow.models.dagbag.DagBag","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.389834Z","level":"info","event":"Parsing
DAG 'simple_dummy_dag'
(caller=/opt/airflow/dags/test/airflow/dags/simple_dag.py,
env_root=/opt/airflow/dags/test/airflow)","logger":"airflow.utils.log.logging_mixin.LoggingMixin","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.389921Z","level":"info","event":"Selected
environment:
staging","logger":"airflow.utils.log.logging_mixin.LoggingMixin","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.390126Z","level":"info","event":"Env file
candidate:
/opt/airflow/dags/test/airflow/.env.staging","logger":"airflow.utils.log.logging_mixin.LoggingMixin","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:38.390578Z","level":"info","event":"Loaded
environment variables from
/opt/airflow/dags/test/airflow/.env.staging","logger":"airflow.utils.log.logging_mixin.LoggingMixin","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.395721Z","level":"info","event":"Trying to
parse the dbt project using dbt ls cache
cosmos_cache__simple_dummy_dag__dbt_task_group...","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.420706Z","level":"info","event":"Cosmos
performance: time to calculate cache identifier
cosmos_cache__simple_dummy_dag__dbt_task_group for current version:
0.0005148248746991158","logger":"cosmos.cache","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.420837Z","level":"info","event":"Cosmos
performance [simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: The cache
size for cosmos_cache__simple_dummy_dag__dbt_task_group is
20132","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.421382Z","level":"info","event":"Cosmos
performance: Cache hit for cosmos_cache__simple_dummy_dag__dbt_task_group -
116fbfec9b8ac0541dc22732080e7505,8503c37d87e09fa9e1f60b8912e6c0c3","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.421689Z","level":"info","event":"Total
nodes: 27","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.421747Z","level":"info","event":"Total
filtered nodes:
27","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.421814Z","level":"info","event":"Cosmos
performance (simple_dummy_dag__dbt_task_group) -
[simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: It took 0.0262s to
parse the dbt project for DAG using
LoadMode.DBT_LS_CACHE","logger":"cosmos.converter","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.429799Z","level":"info","event":"Cosmos
performance (simple_dummy_dag__dbt_task_group) -
[simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: It took 0.00796s to
build the Airflow
DAG.","logger":"cosmos.converter","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.430664Z","level":"info","event":"Trying to
parse the dbt project using dbt ls cache
cosmos_cache__simple_dummy_dag__minimal_dbt_test...","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440241Z","level":"info","event":"Cosmos
performance: time to calculate cache identifier
cosmos_cache__simple_dummy_dag__minimal_dbt_test for current version:
0.00046902894973754883","logger":"cosmos.cache","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440335Z","level":"info","event":"Cosmos
performance [simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: The cache
size for cosmos_cache__simple_dummy_dag__minimal_dbt_test is
20132","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440855Z","level":"info","event":"Cosmos
performance: Cache hit for cosmos_cache__simple_dummy_dag__minimal_dbt_test -
116fbfec9b8ac0541dc22732080e7505,8503c37d87e09fa9e1f60b8912e6c0c3","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440928Z","level":"info","event":"Total
nodes: 27","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.440971Z","level":"info","event":"Total
filtered nodes:
27","logger":"cosmos.dbt.graph","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.441026Z","level":"info","event":"Cosmos
performance (simple_dummy_dag__minimal_dbt_test) -
[simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: It took 0.0104s to
parse the dbt project for DAG using
LoadMode.DBT_LS_CACHE","logger":"cosmos.converter","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.446371Z","level":"info","event":"Cosmos
performance (simple_dummy_dag__minimal_dbt_test) -
[simple-dummy-dag-dbt-task-group-issues-run-7p5aps05|23]: It took 0.00532s to
build the Airflow
DAG.","logger":"cosmos.converter","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:43.447464Z","level":"warning","event":"The
`airflow.hooks.base.BaseHook` attribute is deprecated. Please use
`'airflow.sdk.bases.hook.BaseHook'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:43.447542Z","level":"warning","event":"The
`airflow.operators.bash.BashOperator` attribute is deprecated. Please use
`'airflow.providers.standard.operators.bash.BashOperator'`.","category":"DeprecatedImportWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:43.447592Z","level":"warning","event":"Pydantic
serializer warnings:\n PydanticSerializationUnexpectedValue(Expected `str` -
serialized value may not be as expected [input_value={'version':
'116fbfec9b8a...p_id': 'dbt_task_group'},
input_type=dict])","category":"UserWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:43.447633Z","level":"warning","event":"Pydantic
serializer warnings:\n PydanticSerializationUnexpectedValue(Expected `str` -
serialized value may not be as expected [input_value={'version':
'116fbfec9b8a...id': 'minimal_dbt_test'},
input_type=dict])","category":"UserWarning","filename":"supervisor.py","lineno":1739,"logger":"py.warnings"}
{"timestamp":"2025-10-07T09:37:44.835755Z","level":"info","event":"dbtRunner
is available. Using dbtRunner for invoking
dbt.","logger":"cosmos.operators.base","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.836090Z","level":"info","event":"Cloning
project to writable temp directory /tmp/tmptfu0b0kt from
/opt/airflow/dags/test/dbt/test","logger":"cosmos.operators.base","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.838856Z","level":"info","event":"Partial
parse is enabled and the latest partial parse file is
None","logger":"cosmos.operators.base","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.839055Z","level":"info","event":"Using
user-supplied profiles.yml at
/opt/airflow/dags/test/dbt/test/profiles.yml","logger":"cosmos.config","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.840093Z","level":"info","event":"Trying to
run dbtRunner with:\n ['deps', '--project-dir', '/tmp/tmptfu0b0kt',
'--profiles-dir', '/opt/airflow/dags/test/dbt/test', '--profile', 'test',
'--target', 'staging', '--no-static-parser']\n in
/tmp/tmptfu0b0kt","logger":"cosmos.dbt.runner","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:44.895441Z","level":"info","event":"\u001b[0m09:37:44
Running with
dbt=1.11.0-b2","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.515707Z","level":"info","event":"\u001b[0m09:37:45
Updating lock file in file path:
/tmp/tmptfu0b0kt/package-lock.yml","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.523362Z","level":"info","event":"\u001b[0m09:37:45
Installing
/opt/airflow/commons/dbt/data_eng_dbt","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.524967Z","level":"info","event":"\u001b[0m09:37:45
Installed from <local @
/opt/airflow/commons/dbt/data_eng_dbt>","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.525726Z","level":"info","event":"\u001b[0m09:37:45
Installing
entechlog/dbt_snow_mask","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.765544Z","level":"info","event":"\u001b[0m09:37:45
Installed from version
0.2.7","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.765723Z","level":"info","event":"\u001b[0m09:37:45
Up to date!","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.766406Z","level":"info","event":"\u001b[0m09:37:45
Installing
get-select/dbt_snowflake_query_tags","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.930474Z","level":"info","event":"\u001b[0m09:37:45
Installed from version
2.5.1","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.930654Z","level":"info","event":"\u001b[0m09:37:45
Updated version available:
2.5.3","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:45.931518Z","level":"info","event":"\u001b[0m09:37:45
Installing
dbt-labs/dbt_utils","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.129733Z","level":"info","event":"\u001b[0m09:37:46
Installed from version
1.3.1","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.129896Z","level":"info","event":"\u001b[0m09:37:46
Up to date!","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.130716Z","level":"info","event":"\u001b[0m09:37:46","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.131005Z","level":"info","event":"\u001b[0m09:37:46
Updates available for packages:
['get-select/dbt_snowflake_query_tags']","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.131067Z","level":"info","event":"Update
your versions in packages.yml, then run dbt
deps","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.495431Z","level":"info","event":"Trying to
run dbtRunner with:\n ['run', '--models', 'issues', '--project-dir',
'/tmp/tmptfu0b0kt', '--profiles-dir', '/opt/airflow/dags/test/dbt/test',
'--profile', 'test', '--target', 'staging', '--no-static-parser']\n in
/tmp/tmptfu0b0kt","logger":"cosmos.dbt.runner","filename":"supervisor.py","lineno":1739}
{"timestamp":"2025-10-07T09:37:46.500154Z","level":"info","event":"\u001b[0m09:37:46
Running with
dbt=1.11.0-b2","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.500980Z","level":"info","event":"\u001b[0m09:37:46
[\u001b[33mWARNING\u001b[0m]: Deprecated
functionality","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.501046Z","level":"info","event":"Usage of
`--models`, `--model`, and `-m` is deprecated in favor of `--select`
or","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:46.501118Z","level":"info","event":"`-s`.","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:47.898632Z","level":"info","event":"\u001b[0m09:37:47
Registered adapter:
snowflake=1.10.2","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:48.270847Z","level":"info","event":"\u001b[0m09:37:48
Unable to do partial parsing because saved manifest not found. Starting full
parse.","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.739177Z","level":"info","event":"\u001b[0m09:37:49
Found 5 models, 22 data tests, 625
macros","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.752263Z","level":"info","event":"\u001b[0m09:37:49","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.752374Z","level":"info","event":"\u001b[0m09:37:49
Concurrency: 16 threads
(target='staging')","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.752445Z","level":"info","event":"\u001b[0m09:37:49","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.768383Z","level":"info","event":"\u001b[0m09:37:49","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.768650Z","level":"info","event":"\u001b[0m09:37:49
Finished running in 0 hours 0 minutes and 0.03 seconds
(0.03s).","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769138Z","level":"info","event":"\u001b[0m09:37:49
Encountered an
error:","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769208Z","level":"info","event":"Runtime
Error","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769273Z","level":"info","event":"
Database error while listing schemas in database
\"TEST_STAGING\"","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769338Z","level":"info","event":"
Database Error","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769402Z","level":"info","event":"
('Could not deserialize key data. The data may be in an incorrect format, it
may be encrypted with an unsupported algorithm, or it may be an unsupported key
type (e.g. EC curves with explicit parameters).',
[<OpenSSLError(code=503841036, lib=60, reason=524556,
reason_text=unsupported)>])","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769646Z","level":"info","event":"\u001b[0m09:37:49
[\u001b[33mWARNING\u001b[0m][DeprecationsSummary]: Deprecated
functionality","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769711Z","level":"info","event":"Summary
of encountered
deprecations:","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769775Z","level":"info","event":"-
ModelParamUsageDeprecation: 1
occurrence","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769857Z","level":"info","event":"To see
all deprecation instances instead of just the first occurrence of
each,","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769919Z","level":"info","event":"run
command again with the `--show-all-deprecations` flag. You may also need
to","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.769998Z","level":"info","event":"run with
`--no-partial-parse` as some deprecations are only encountered
during","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:49.770073Z","level":"info","event":"parsing.","logger":"task.stdout","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:50.159921Z","level":"error","event":"Task
failed with
exception","logger":"task","filename":"supervisor.py","lineno":1739,"error_detail":[{"exc_type":"CosmosDbtRunError","exc_value":"dbt
invocation did not complete with unhandled error: Runtime Error\n Database
error while listing schemas in database \"TEST_STAGING\"\n Database Error\n
('Could not deserialize key data. The data may be in an incorrect format, it
may be encrypted with an unsupported algorithm, or it may be an unsupported key
type (e.g. EC curves with explicit parameters).',
[<OpenSSLError(code=503841036, lib=60, reason=524556,
reason_text=unsupported)>])","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":920,"name":"run"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":1307,"name":"_execute
_task"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/base.py","lineno":310,"name":"execute"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py","lineno":853,"name":"build_and_run_cmd"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py","lineno":650,"name":"run_command"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/operators/local.py","lineno":258,"name":"handle_exception_dbt_runner"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/cosmos/dbt/runner.py","lineno":109,"name":"handle_exception_if_needed"}],"is_group":false,"exceptions":[]}]}
{"timestamp":"2025-10-07T09:37:50.203777Z","level":"error","event":"API
server error","status_code":404,"detail":{"detail":"Not Found"},"message":"Not
Found","logger":"supervisor","filename":"supervisor.py","lineno":616}
{"timestamp":"2025-10-07T09:37:50.204113Z","level":"error","event":"Top
level
error","logger":"task","filename":"supervisor.py","lineno":1739,"error_detail":[{"exc_type":"AirflowRuntimeError","exc_value":"API_SERVER_ERROR:
{'status_code': 404, 'message': 'Not Found', 'detail': {'detail': 'Not
Found'}}","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":1452,"name":"main"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py","lineno":1397,"name":"finalize"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py","lineno":207,"name":"send"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py","lineno":271,"name":"_get_response"},{"filename":"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/
comms.py","lineno":258,"name":"_from_frame"}],"is_group":false,"exceptions":[]}]}
{"timestamp":"2025-10-07T09:37:50.244332Z","level":"error","event":"/usr/python/lib/python3.12/multiprocessing/resource_tracker.py:279:
UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects
to clean up at
shutdown","logger":"task.stderr","filename":"supervisor.py","lineno":1754}
{"timestamp":"2025-10-07T09:37:50.244508Z","level":"error","event":"
warnings.warn('resource_tracker: There appear to be %d
'","logger":"task.stderr","filename":"supervisor.py","lineno":1754}
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py",
line 125, in <module>
main()
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py",
line 121, in main
execute_workload(workload)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/execute_workload.py",
line 66, in execute_workload
supervise(
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1887, in supervise
exit_code = process.wait()
^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 988, in wait
self.update_task_state_if_needed()
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1002, in update_task_state_if_needed
self.client.task_instances.finish(
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 221, in finish
self.client.patch(f"task-instances/{id}/state",
content=body.model_dump_json())
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 1218, in patch
return self.request(
^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
338, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
477, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
378, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
400, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449,
in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401,
in __get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
480, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 861, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 999, in _send_handling_redirects
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 982, in _send_handling_redirects
hook(response)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 181, in raise_on_4xx_5xx_with_note
return get_json_error(response) or response.raise_for_status()
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 171, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Server returned error
### What you think should happen instead?
Even if the task fail, for each retries, I should always get the logs
available in the UI.
### How to reproduce
1. Create a DbtTaskGroup with a failing dbt model and retries > 0
2. Set the tasks as KubernetesExecutors
3. Choose S3 for remote logging with airflow
4. Trigger the DAG
5. Observe that only the last retry's logs are available in S3/UI
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
