Re: [D] Schedulers crashing when upgrading to 2.10.4 from 2.8.2 [airflow]

2025-04-18 Thread via GitHub 


GitHub user s-kim1003 added a comment to the discussion: Schedulers crashing 
when upgrading to 2.10.4 from 2.8.2

Interesting, it lingered for 2+ days for me. I ended up restoring the DB from a 
backup, then downgraded to 2.9.3. That worked without issues. Did you have any 
DAGs running in your environment when the issue occurred?

GitHub link: 
https://github.com/apache/airflow/discussions/47117#discussioncomment-12871508


This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Re: [D] Schedulers crashing when upgrading to 2.10.4 from 2.8.2 [airflow]

2025-04-18 Thread via GitHub 


GitHub user pingu1m added a comment to the discussion: Schedulers crashing when 
upgrading to 2.10.4 from 2.8.2

My issue was fixed by itself after 30min. I was debugging when things started 
working. I believe it has to do with db sessions. Once things are reset it will 
work.

GitHub link: 
https://github.com/apache/airflow/discussions/47117#discussioncomment-12860493


This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Re: [D] Schedulers crashing when upgrading to 2.10.4 from 2.8.2 [airflow]

2025-04-18 Thread via GitHub 


GitHub user s-kim1003 deleted a comment on the discussion: Schedulers crashing 
when upgrading to 2.10.4 from 2.8.2

Interesting, it lingered for 2+ days for me. I ended up restoring the DB from a 
backup, then downgraded to 2.9.3. That worked without issues. Did you have any 
DAGs running in your environment when the issue occurred?

GitHub link: 
https://github.com/apache/airflow/discussions/47117#discussioncomment-12871508


This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Re: [D] Schedulers crashing when upgrading to 2.10.4 from 2.8.2 [airflow]

2025-04-17 Thread via GitHub 


GitHub user nkjmsg added a comment to the discussion: Schedulers crashing when 
upgrading to 2.10.4 from 2.8.2

Interesting, it lingered for 2+ days for me. I ended up restoring the DB from a 
backup, then downgraded to 2.9.3. That worked without issues. Did you have any 
DAGs running in your environment when the issue occurred?

GitHub link: 
https://github.com/apache/airflow/discussions/47117#discussioncomment-12871514


This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Re: [D] Schedulers crashing when upgrading to 2.10.4 from 2.8.2 [airflow]

2025-04-16 Thread via GitHub 


GitHub user pingu1m added a comment to the discussion: Schedulers crashing when 
upgrading to 2.10.4 from 2.8.2

Experiencing the same error. 
```
Traceback (most recent call last):
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 999, in _execute
self._run_scheduler_loop()
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 1138, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
 
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 1244, in _do_scheduling
self._create_dagruns_for_dags(guard, session)
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/retries.py", 
line 93, in wrapped_function
for attempt in run_with_db_retries(max_retries=retries, logger=logger, 
**retry_kwargs):
   
^^^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 
443, in __iter__
do = self.iter(retry_state=retry_state)
 ^^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 
376, in iter
result = action(retry_state)
 ^^^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 
398, in 
self._add_action_func(lambda rs: rs.outcome.result())
 ^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in 
result
return self.__get_result()
   ^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in 
__get_result
raise self._exception
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/retries.py", 
line 102, in wrapped_function
return func(*args, **kwargs)
   ^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/scheduler_job_runner.py",
 line 1312, in _create_dagruns_for_dags
query, dataset_triggered_dag_info = DagModel.dags_needing_dagruns(session)
^^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/dag.py", line 
4101, in dags_needing_dagruns
if not dag_ready(dag_id, cond=ser_dag.dag.timetable.dataset_condition, 
statuses=statuses):
  ^^^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/models/serialized_dag.py",
 line 235, in dag
return SerializedDAG.from_dict(data)
   ^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serialized_objects.py",
 line 1785, in from_dict
return cls.deserialize_dag(serialized_obj["dag"])
   ^^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serialized_objects.py",
 line 1713, in deserialize_dag
v = decode_timetable(v)
^^^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/serialization/serialized_objects.py",
 line 339, in decode_timetable
return timetable_class.deserialize(var[Encoding.VAR])
   ^^
  File 
"/home/airflow/.local/lib/python3.12/site-packages/airflow/timetables/simple.py",
 line 181, in deserialize
return cls(decode_dataset_condition(data["dataset_condition"]))
^
KeyError: 'dataset_condition'
```

GitHub link: 
https://github.com/apache/airflow/discussions/47117#discussioncomment-12860221


This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]