Richard-Durkee opened a new pull request, #2621:
URL: https://github.com/apache/datafusion-ballista/pull/2621
# Which issue does this PR close?
Follow-up to #2581. Related to #1426.
# Rationale for this change
Pull-mode executors don't send heartbeats, only `poll_work`, and each poll
replaced the executor's stored heartbeat with an empty one. So with
`pull-staged` scheduling, `/api/executors` shows no metrics for any executor:
```
"metrics":[{"type":"peak_physical_memory","value":0},{"type":"peak_virtual_memory","value":0}]
```
# What changes are included in this PR?
- Pull-mode executors now send the same heartbeat as push mode, every
`--executor-heartbeat-interval-seconds`. Polling still keeps them alive.
- A poll only refreshes the heartbeat's timestamp, so it keeps the status,
metrics and peaks from the last heartbeat.
- If the first heartbeat arrives before the first poll, the scheduler
registers the executor the way a poll would.
- The shutdown heartbeat now includes the latest metrics instead of an empty
list, in both modes, so they stay visible after the executor stops.
- Push and pull share one function to build the heartbeat. The metric
sampling moved unchanged into `ExecutorMetricCollectionPolicy::collect`.
I used a separate heartbeat loop rather than adding metrics to `poll_work`,
since an idle executor can poll every 50ms and that's too often to sample
memory. It also needs no protocol change. Happy to switch if you'd prefer
metrics on the poll.
As a side effect, a pull executor's `Terminating` status no longer gets
cleared by its next poll, so a draining executor is treated the same as in push
mode.
Added a test that fails without the fix. I also ran a pull-mode scheduler
and executor with a TPC-H query, and `/api/executors` now shows the full set of
metrics while idle, after the query and after shutdown. This is the executor
just after the query:
```
"metrics":[
{"type":"proc_physical_memory","value":124895232},
{"type":"proc_virtual_memory","value":502005022720},
{"type":"total_memory","value":51539607552},
{"type":"available_memory","value":37897338880},
{"type":"used_memory","value":31048040448},
{"type":"memory_pool_reserved","value":0},
{"type":"memory_pool_size","value":1000000000},
{"type":"allocated_memory","value":956315},
{"type":"peak_physical_memory","value":124895232},
{"type":"peak_virtual_memory","value":502005596160}
]
```
# Are there any user-facing changes?
Pull-mode executors now report metrics in the executors REST API. No
breaking changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
