alexsanderp opened a new issue, #42831:
URL: https://github.com/apache/airflow/issues/42831
### Apache Airflow version
2.10.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Scheduler Exception:
```
/home/airflow/.local/lib/python3.11/site-packages/airflow/configuration.py:765
FutureWarning: The auth_backends setting in [api] has had
airflow.api.auth.backend.session added in the running config, which is needed
by the UI. Please update your config before Apache Airflow 3.0.
/home/airflow/.local/lib/python3.11/site-packages/airflow/metrics/validators.py:95
DeprecationWarning: The statsd_allow_list option in [metrics] has been renamed
to metrics_allow_list - the old setting has been used, but please update your
config.
/home/airflow/.local/lib/python3.11/site-packages/airflow/metrics/statsd_logger.py:184
RemovedInAirflow3Warning: The basic metric validator will be deprecated in the
future in favor of pattern-matching. You can try this now by setting config
option metrics_use_pattern_match to True.
DataHurb2LineageBackend is not available. Please install the required
dependencies.
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2024-10-08T15:26:01.247+0000] {_client.py:1026} INFO - HTTP Request: GET
https://apacheairflow.gateway.scarf.sh/scheduler?version=2.10.2&python_version=3.11&platform=Linux&arch=x86_64&database=postgresql&db_version=12.20&executor=KubernetesExecutor
"HTTP/1.1 200 OK"
/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py:143
FutureWarning: The config section [kubernetes] has been renamed to
[kubernetes_executor]. Please update your `conf.get*` call to use the new name
[2024-10-08T15:26:01.361+0000] {executor_loader.py:254} INFO - Loaded
executor: KubernetesExecutor
[2024-10-08T15:26:01.437+0000] {scheduler_job_runner.py:935} INFO - Starting
the scheduler
[2024-10-08T15:26:01.437+0000] {scheduler_job_runner.py:942} INFO -
Processing each file at most -1 times
[2024-10-08T15:26:01.438+0000] {kubernetes_executor.py:287} INFO - Start
Kubernetes executor
[2024-10-08T15:26:01.471+0000] {kubernetes_executor_utils.py:140} INFO -
Event: and now my watch begins starting at resource_version: 0
[2024-10-08T15:26:01.491+0000] {kubernetes_executor.py:208} INFO - Found 0
queued task instances
[2024-10-08T15:26:01.493+0000] {scheduler_job_runner.py:1847} INFO -
Adopting or resetting orphaned tasks for active dag runs
[2024-10-08T15:26:01.653+0000] {scheduler_job_runner.py:1001} ERROR -
Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 984, in _execute
self._run_scheduler_loop()
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1067, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py",
line 97, in wrapper
return func(*args, session=session, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1850, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
443, in __iter__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
376, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in
result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1894, in adopt_or_reset_orphaned_tasks
to_reset.extend(executor.try_adopt_task_instances(tis))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
line 571, in try_adopt_task_instances
pod_list = self._list_pods(query_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
line 162, in _list_pods
pods.extend(dynamic_client.get(resource=pod_resource,
namespace=namespace, **query_kwargs).items)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/client.py",
line 112, in get
return self.request('get', path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/client.py",
line 62, in inner
return serializer(self, json.loads(resp.data.decode('utf8')))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/resource.py",
line 294, in __init__
for item in instance['items']:
TypeError: 'NoneType' object is not iterable
[2024-10-08T15:26:01.656+0000] {kubernetes_executor.py:752} INFO - Shutting
down Kubernetes executor
[2024-10-08T15:26:01.700+0000] {scheduler_job_runner.py:1014} INFO - Exited
execute loop
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/__main__.py", line
62, in main
args.func(args)
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/cli_config.py",
line 49, in command
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/cli.py", line
115, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py",
line 55, in wrapped_function
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py",
line 59, in scheduler
run_command_with_daemon_option(
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/daemon_utils.py",
line 86, in run_command_with_daemon_option
callback()
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py",
line 62, in <lambda>
callback=lambda: _run_scheduler_job(args),
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/cli/commands/scheduler_command.py",
line 48, in _run_scheduler_job
run_job(job=job_runner.job, execute_callable=job_runner._execute)
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py",
line 97, in wrapper
return func(*args, session=session, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/job.py", line
421, in run_job
return execute_job(job, execute_callable=execute_callable)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/job.py", line
450, in execute_job
ret = execute_callable()
^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 984, in _execute
self._run_scheduler_loop()
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1067, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/session.py",
line 97, in wrapper
return func(*args, session=session, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1850, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
443, in __iter__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
376, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/tenacity/__init__.py", line
398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in
result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/jobs/scheduler_job_runner.py",
line 1894, in adopt_or_reset_orphaned_tasks
to_reset.extend(executor.try_adopt_task_instances(tis))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
line 571, in try_adopt_task_instances
pod_list = self._list_pods(query_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py",
line 162, in _list_pods
pods.extend(dynamic_client.get(resource=pod_resource,
namespace=namespace, **query_kwargs).items)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/client.py",
line 112, in get
return self.request('get', path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/client.py",
line 62, in inner
return serializer(self, json.loads(resp.data.decode('utf8')))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.11/site-packages/kubernetes/dynamic/resource.py",
line 294, in __init__
for item in instance['items']:
TypeError: 'NoneType' object is not iterable
```
### What you think should happen instead?
Scheduler running
### How to reproduce
I don't know how I reproduced it, I just started using version 2.10.2 and
this error has been occurring
### Operating System
Kubernetes GKE
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.28.0
apache-airflow-providers-celery==3.8.1
apache-airflow-providers-cncf-kubernetes==8.4.1
apache-airflow-providers-common-compat==1.2.0
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.16.0
apache-airflow-providers-docker==3.13.0
apache-airflow-providers-elasticsearch==5.5.0
apache-airflow-providers-fab==1.3.0
apache-airflow-providers-ftp==3.11.0
apache-airflow-providers-google==10.19.0
apache-airflow-providers-grpc==3.6.0
apache-airflow-providers-hashicorp==3.8.0
apache-airflow-providers-http==4.13.0
apache-airflow-providers-imap==3.7.0
apache-airflow-providers-microsoft-azure==10.4.0
apache-airflow-providers-mysql==5.7.0
apache-airflow-providers-odbc==4.7.0
apache-airflow-providers-openlineage==1.11.0
apache-airflow-providers-postgres==5.12.0
apache-airflow-providers-redis==3.8.0
apache-airflow-providers-sendgrid==3.6.0
apache-airflow-providers-sftp==4.11.0
apache-airflow-providers-slack==8.9.0
apache-airflow-providers-smtp==1.8.0
apache-airflow-providers-snowflake==5.7.0
apache-airflow-providers-sqlite==3.9.0
apache-airflow-providers-ssh==3.13.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Helm Chart + ArgoCD
### Anything else?
This issue is crashing our production environment
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
