potiuk commented on code in PR #56690:
URL: https://github.com/apache/airflow/pull/56690#discussion_r2446723039
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -835,6 +836,24 @@ def _validate_task_inlets_and_outlets(*, ti:
RuntimeTaskInstance, log: Logger) -
)
+def _get_resource_usage() -> tuple[float, float] | None:
+ """Get current resource usage for the task process."""
+ try:
+ process = psutil.Process()
+
+ # Get CPU usage
+ cpu_percent = process.cpu_percent(interval=0)
Review Comment:
Return a float representing the current system-wide CPU utilization as a
percentage. When interval is > 0.0 compares system CPU times elapsed before and
after the interval (blocking). When interval is 0.0 or None compares system CPU
times elapsed since last call or module import, returning immediately. That
means the first time this is called it will return a meaningless 0.0 value
which you are supposed to ignore. In this case it is recommended for accuracy
that this function be called with at least 0.1 seconds between calls. When
percpu is True returns a list of floats representing the utilization as a
percentage for each CPU. First element of the list refers to first CPU, second
element to second CPU and so on. The order of the list is consistent across
calls. Internally this function maintains a global map (a dict) where each key
is the ID of the calling thread
([threading.get_ident](https://docs.python.org/3/library/threading.html#threading.get_ident)).
This means it can be ca
lled from different threads, at different intervals, and still return
meaningful and independent results.
I think you need to handle the meaningless "0" in some way.
##########
task-sdk/src/airflow/sdk/execution_time/task_runner.py:
##########
@@ -835,6 +836,24 @@ def _validate_task_inlets_and_outlets(*, ti:
RuntimeTaskInstance, log: Logger) -
)
+def _get_resource_usage() -> tuple[float, float] | None:
+ """Get current resource usage for the task process."""
+ try:
+ process = psutil.Process()
+
+ # Get CPU usage
+ cpu_percent = process.cpu_percent(interval=0)
Review Comment:
> Return a float representing the current system-wide CPU utilization as a
percentage. When interval is > 0.0 compares system CPU times elapsed before and
after the interval (blocking). When interval is 0.0 or None compares system CPU
times elapsed since last call or module import, returning immediately. That
means the first time this is called it will return a meaningless 0.0 value
which you are supposed to ignore. In this case it is recommended for accuracy
that this function be called with at least 0.1 seconds between calls. When
percpu is True returns a list of floats representing the utilization as a
percentage for each CPU. First element of the list refers to first CPU, second
element to second CPU and so on. The order of the list is consistent across
calls. Internally this function maintains a global map (a dict) where each key
is the ID of the calling thread
([threading.get_ident](https://docs.python.org/3/library/threading.html#threading.get_ident)).
This means it can be
called from different threads, at different intervals, and still return
meaningful and independent results.
I think you need to handle the meaningless "0" in some way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
