Taragolis commented on code in PR #39650:
URL: https://github.com/apache/airflow/pull/39650#discussion_r1604905076
##########
airflow/task/task_runner/standard_task_runner.py:
##########
@@ -186,3 +193,20 @@ def get_process_pid(self) -> int:
if self.process is None:
raise RuntimeError("Process is not started yet")
return self.process.pid
+
+ def _read_task_utilization(self):
+ dag_id = self._task_instance.dag_id
+ task_id = self._task_instance.task_id
+
+ while True:
+ try:
+ with self.process.oneshot():
+ mem_usage = self.process.memory_percent()
+ cpu_usage = self.process.cpu_percent(interval=1)
Review Comment:
What I found that `interval` do not work well with `oneshot()` and always
return `0`
However if we do not provide interval (`None` by default) and manually sleep
then it works as expected.
##########
airflow/task/task_runner/standard_task_runner.py:
##########
@@ -186,3 +193,20 @@ def get_process_pid(self) -> int:
if self.process is None:
raise RuntimeError("Process is not started yet")
return self.process.pid
+
+ def _read_task_utilization(self):
+ dag_id = self._task_instance.dag_id
+ task_id = self._task_instance.task_id
+
+ while True:
+ try:
+ with self.process.oneshot():
+ mem_usage = self.process.memory_percent()
+ cpu_usage = self.process.cpu_percent(interval=1)
+
+ Stats.gauge(f"task.mem_usage.{dag_id}.{task_id}",
mem_usage)
+ Stats.gauge(f"task.cpu_usage.{dag_id}.{task_id}",
cpu_usage)
+ except (psutil.NoSuchProcess, psutil.AccessDenied):
+ break
+ except AttributeError:
+ raise
Review Comment:
```suggestion
except (psutil.NoSuchProcess, psutil.AccessDenied,
AttributeError):
break
```
I think we should consider that Attribute error is fine in this case, that
mean process not exists anymore, so we could stop send metrics and stop
gracefully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
