potiuk commented on code in PR #58365:
URL: https://github.com/apache/airflow/pull/58365#discussion_r2531953601
##########
airflow-core/src/airflow/executors/local_executor.py:
##########
@@ -186,9 +193,14 @@ def _check_workers(self):
# via `sync()` a few times before the spawned process actually starts
picking up messages. Try not to
# create too much
if num_outstanding and len(self.workers) < self.parallelism:
- # This only creates one worker, which is fine as we call this
directly after putting a message on
- # activity_queue in execute_async
- self._spawn_worker()
+ if self.is_mp_using_fork:
+ # This creates the maximum number of worker processes at once
Review Comment:
BTW. The stats are **clearly** showing what's going on here and that it's
worth it - especially the USS and PSS numbers are clearly showing the overall
far less memory used in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
