s0nskar opened a new pull request, #3710:
URL: https://github.com/apache/celeborn/pull/3710
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX]
Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a
faster review.
-->
### What changes were proposed in this pull request?
Added /servingState which supports GET and POST http methods.
- GET will just return the current serving state.
- POST can be used to force override the serving state of a worker. It
takes two params – state (override for serving state) and timeout (after which
overridden state should clear up). If timeout is not present forced state will
not clear up, unless someone overrides it to empty state.
- handling live migration scenarios and other cases where we don't want the
worker to receive new data but still want to keep it running. Or maybe where we
want to force unpause the worker.
### Why are the changes needed?
This can be used for planned maintenance of worker or cases where worker is
degraded or under high load but not having high memory pressure. This can also
be used for cases to force resume worker which can be useful in cases like
https://github.com/apache/celeborn/pull/3696
We are using this specifically during GCP live migration –
https://docs.cloud.google.com/compute/docs/instances/live-migration-process
### Does this PR resolve a correctness bug?
<!-- Check if yes. The `correctness` label will be added/removed
automatically. -->
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
<!-- Check if yes. -->
- [ ] Yes
### How was this patch tested?
Tested in our dev setup
```
>>> curl <host:port>/servingState
====================== Worker Serving State ==========================
Current state: NONE_PAUSED.
>>> curl -X POST "<host:port>/servingState" -H "Content-Type:
application/x-www-form-urlencoded" -d "state=PUSH_PAUSED" -d "timeout=1m"
====================== Set Serving State ============================
Serving state forced to: PUSH_PAUSED
Override will auto-clear after 1m.
>>> curl <host:port>/servingState
====================== Worker Serving State ==========================
Current state: PUSH_PAUSED.
Manual override active.
// Cleared after 1 min
>>> curl <host:port>/servingState
====================== Worker Serving State ==========================
Current state: NONE_PAUSED.
// Forced without timeout
>>> curl -X POST "<host:port>/servingState" -H "Content-Type:
application/x-www-form-urlencoded" -d "state=PUSH_PAUSED"
====================== Set Serving State ============================
Serving state forced to: PUSH_PAUSED
Override will persist until explicitly cleared.
>>> curl <host:port>/servingState
====================== Worker Serving State ==========================
Current state: PUSH_PAUSED.
Manual override active.
>>> curl -X POST "<host:port>/servingState" -H "Content-Type:
application/x-www-form-urlencoded" -d "state="
====================== Set Serving State ============================
Manual servingState override cleared.
>>> curl <host:port>/servingState
====================== Worker Serving State ==========================
Current state: NONE_PAUSED.
// Invalid scenarios
>>> curl -X POST "<host:port>/servingState" -H "Content-Type:
application/x-www-form-urlencoded" -d "state=PUSH_PAUSED" -d "timeout=1k"
Invalid timeout '1k'. java.lang.NumberFormatException: Time must be
specified as seconds (s), milliseconds (ms), microseconds (us), minutes (m or
min), hour (h), or day (d). E.g. 50s, 100ms, or 250us.
Invalid suffix: "k"
>>> curl -X POST "<host:port>/servingState" -H "Content-Type:
application/x-www-form-urlencoded" -d "state=INVALID"
Invalid state 'INVALID'. Legal values: PUSH_AND_REPLICATE_PAUSED,
PUSH_PAUSED, NONE_PAUSED
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
