Reddy1990 edited a comment on issue #13805:
URL: https://github.com/apache/airflow/issues/13805#issuecomment-877090063
HI All, I'm still getting the same error. While using
CeleryKubernetesExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-09 10:31:52,972] {scheduler_job.py:1251} INFO - Starting the
scheduler
[2021-07-09 10:31:52,972] {scheduler_job.py:1256} INFO - Processing each
file at most -1 times
[2021-07-09 10:31:52,978] {kubernetes_executor.py:471} INFO - Start
Kubernetes executor
[2021-07-09 10:31:52,978] {scheduler_job.py:1302} ERROR - Exception when
executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py",
line 1276, in _execute
self.executor.start()
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_kubernetes_executor.py",
line 60, in start
self.kubernetes_executor.start()
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py",
line 473, in start
raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
[2021-07-09 10:31:52,979] {dag_processing.py:453} WARNING - Ending without
manager process.
[2021-07-09 10:31:52,979] {scheduler_job.py:1305} INFO - Exited execute loop
Environment:
Using Airflow 2.0.2 Version
Kubernetes v1.20.7
Cloud provider or hardware configuration: AWS Linux
I was using KubernetesExecutor (all was working fine , faced no issues and
able to run DAG's/Jobs) and now experimenting with CeleryKubernetesExecutor the
issue started showing up and I check if any job/Dag/task is causing the issue
where the Scheduler looks for but I tried after deleting DAG's and placing any
empty dir but the issue is still there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
