n-badtke-cg opened a new issue, #71772:
URL: https://github.com/apache/airflow/issues/71772
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
We are currently running Airflow 3.3.0 in Kubernetes using the Airflow Helm
Chart v1.22 for deployment. All Dags are healthy. The Dags are fetched via Git
Dag Bundle.
After deploying, the dag processor messages errors like `Processor for
DagFileInfo(rel_path=PosixPath('<filepath_in_git_repo>'),
bundle_name='<a_name>',
bundle_path=PosixPath('/tmp/airflow/dag_bundles/<a_name>/tracking_repo',
bundle_version=None) with PID NNN has been running for 50.01 seconds, exceeding
the timeout of 50.00 seconds. Killing it!
[airflow.dag_processing.manager.DagFileProcessorManager] loc=manager.py:1621`
and the overview table now shows "# Errors" > 0
We have a very basic "airflow_health_check" Dag to verify the Airflow
workers Dag runtime:
```py
from airflow import XComArg
from airflow.sdk import dag, task
from airflow.sdk.exceptions import AirflowFailException
from datetime import datetime, timedelta
HEALTH_CHECK_MESSAGE: str = "Health check successful"
HEALTHY: str = "healthy"
UNHEALTHY: str = "unhealthy"
@dag(
schedule="*/2 * * * *",
start_date=datetime(2026, 7, 1),
dagrun_timeout=timedelta(minutes=3),
is_paused_upon_creation=False,
catchup=False,
)
def airflow_health_check() -> None:
@task.bash
def health_check(text: str) -> str:
return f"echo '{text}'"
@task
def result_is_healthy(*, result: XComArg, check_against: str) -> bool:
return str(result) == check_against
@task
def print_result(is_healthy: bool) -> None:
print("is_healthy: " + str(type(is_healthy)) + " " + str(is_healthy))
print(HEALTHY if is_healthy == True else UNHEALTHY)
if not is_healthy == True:
raise AirflowFailException()
health_check_result: XComArg = health_check(HEALTH_CHECK_MESSAGE)
result_is_healthy_bool: bool = result_is_healthy(
result=health_check_result, check_against=HEALTH_CHECK_MESSAGE
)
print_result(result_is_healthy_bool)
airflow_health_check()
```
With Airflow 3.3.0, the Dag Runs are green.
Shortly after the upgrade to 3.3.1, the Dag Runs are stuck in "queued", but
the Dag itself is still active.
With the Dag processor catching up and now throwing errors, the Dag gets
"Deactivated".
### What you think should happen instead?
It should just work? Idk, do I overlook something?
### Operating System
Container image apache/airflow:3.3.1-python3.13
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
amazon, celery, cncf-kubernetes, common-compat, common-io, common-messaging,
common-sql, databricks, docker, elasticsearch, fab, ftp, git, google, grpc,
hashicorp, http, keycloak, microsoft-azure, mysql, odbc, openlineage,
opensearch, postgres, redis, sendgrid, sftp, slack, smtp, snowflake, ssh,
standard
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.34.0
apache-airflow-providers-celery==3.23.1
apache-airflow-providers-cncf-kubernetes==10.21.0
apache-airflow-providers-common-compat==1.18.0
apache-airflow-providers-common-io==1.8.0
apache-airflow-providers-common-messaging==2.0.4
apache-airflow-providers-common-sql==2.1.0
apache-airflow-providers-databricks==7.18.1
apache-airflow-providers-docker==4.5.9
apache-airflow-providers-elasticsearch==6.9.0
apache-airflow-providers-fab==3.8.0
apache-airflow-providers-ftp==3.15.2
apache-airflow-providers-git==0.4.2
apache-airflow-providers-google==22.3.0
apache-airflow-providers-grpc==3.9.5
apache-airflow-providers-hashicorp==4.8.0
apache-airflow-providers-http==6.0.5
apache-airflow-providers-keycloak==0.8.2
apache-airflow-providers-microsoft-azure==14.1.0
apache-airflow-providers-mysql==6.6.1
apache-airflow-providers-odbc==4.12.3
apache-airflow-providers-openlineage==2.20.0
apache-airflow-providers-opensearch==1.12.0
apache-airflow-providers-postgres==7.0.1
apache-airflow-providers-redis==4.5.0
apache-airflow-providers-sendgrid==4.2.4
apache-airflow-providers-sftp==6.0.1
apache-airflow-providers-slack==9.10.2
apache-airflow-providers-smtp==3.0.3
apache-airflow-providers-snowflake==6.16.0
apache-airflow-providers-ssh==6.0.1
apache-airflow-providers-standard==1.17.0
### Official Helm Chart version
1.22.0 (latest released)
### Kubernetes Version
v1.35.1
### Helm Chart configuration
Incomplete, only an excerpt. Please let me know if there might be something
other relevant to share.
```yaml
airflowVersion: 3.3.1
config:
azure_remote_logging:
remote_wasb_log_container: airflow-remote-logs
celery_broker_transport_options:
visibility_timeout: 14400
core:
auth_manager:
airflow.providers.keycloak.auth_manager.keycloak_auth_manager.KeycloakAuthManager
test_connection: Hidden
keycloak_auth_manager:
realm: application
requests_pool_size: 20
server_url: <a_valid_url>
logging:
delete_local_logs: "False"
remote_base_log_folder: wasb://logs
remote_log_conn_id: airflow-remote-logging-wasb
remote_logging: "True"
dagProcessor:
dagBundleConfigList:
- classpath: airflow.providers.git.bundles.git.GitDagBundle
kwargs:
git_conn_id: <a_name_for_conn>
refresh_interval: 30
tracking_ref: main
name: <a_name>
```
### Docker Image customizations
Dockerfile:
```dockerfile
ARG AIRFLOW_CONTAINER_VERSION=3.3.0-python3.13
FROM apache/airflow:${AIRFLOW_CONTAINER_VERSION}
USER root
COPY org_ca.pem /tmp/org_ca.pem
RUN csplit -z -f /usr/local/share/ca-certificates/org_ca_ \
--suffix-format='%03d.crt' \
/tmp/org_ca.pem \
'/-----BEGIN CERTIFICATE-----/' '{*}' \
&& update-ca-certificates \
&& rm /tmp/org_ca.pem
USER airflow
COPY upstream-requirements.txt upstream-requirements.txt
RUN --mount=type=secret,id=pip-conf,dst=/etc/pip.conf,mode=0440 \
pip install --no-cache-dir \
--constraint /home/airflow/constraints.txt \
--requirement upstream-requirements.txt
# install custom providers and packages in separate pip install command to
avoid dependency version clash with constraints
COPY custom-requirements.txt custom-requirements.txt
RUN --mount=type=secret,id=pip-conf,dst=/etc/pip.conf,mode=0440 \
pip install --no-cache-dir \
--requirement custom-requirements.txt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
```
upstream-requirements.txt:
```
# Environmernt variable ${AIRFLOW_VERSION} comes from the upstream Airflow
image and is resolved by pip
apache-airflow[cloudpickle,databricks,keycloak,microsoft.azure,otel]==${AIRFLOW_VERSION}
# Airflow components not covered via extras, versions are covered by
constraints file
virtualenv
```
custom-requirements.txt:
```
# Environment variable ${AIRFLOW_VERSION} comes from the upstream Airflow
image and is resolved by pip
# Pin Airflow version to prevent pip from changing it during dependency
resolution
apache-airflow==${AIRFLOW_VERSION}
<some propietary providers>
```
### Anything else?
To be honest, I am a bit lost where to look at else. Please let me know :)
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
