jedcunningham commented on code in PR #42304:
URL: https://github.com/apache/airflow/pull/42304#discussion_r1766203101
##########
airflow/executors/base_executor.py:
##########
@@ -143,10 +144,14 @@ def __init__(self, parallelism: int = PARALLELISM):
"""
self.attempts: dict[TaskInstanceKey, RunningRetryAttemptType] =
defaultdict(RunningRetryAttemptType)
+ tracemalloc.start()
Review Comment:
We definitely don't want to do this here, tracemalloc is expensive and this
will almost never be used. Will it work if we start it on the first sigusr2
instead?
##########
airflow/executors/base_executor.py:
##########
@@ -617,6 +622,7 @@ def debug_dump(self):
len(self.event_buffer),
"\n\t".join(map(repr, self.event_buffer.items())),
)
+ self.log.debug("executor memory usgae (current: %d, peak: %d)",
*tracemalloc.get_traced_memory())
Review Comment:
It's likely way more useful to display the top x allocations, instead of
just current/peak, like
[this](https://docs.python.org/3/library/tracemalloc.html#display-the-top-10).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
