Mike,
Thank you for taking the time to review the API!
> (Modulo question below about whether a queue is involved).
>
> There's also a question in my head as to whether all the waiting/running
> go into a "stopped" state when shard splitting is stopped on a cluster. I
> would suggest yes, in which case the above might move to:
>
> jobs: {
> total: 20,
> stopped: 15,
> completed: 5
> }
>
That's true, there is no queue currently. If there are too many jobs in the
system, new jobs won't be accepted. So total / stopped / completed would
make sense.
> > * `PUT /_shard_splits`
> >
> > Enable or disable shard splitting on the cluster. This feature that would
> ...
> >
> > An alternative for this would be to have another underscore path like
> > `_shard_splits/_state` but I feel it is better to minimize the use of
> > underscore path, they feel less REST-ful.
>
> `_shard_splits/state` is definitely more REST-like I think, as I'd say
> state is a sub-resource. I'm not sure why this would need to be `_state`.
>
> I would likely expose it as more of a state machine, with the POST
> requests having matching schema regardless of the state transition
> requested:
>
> { "state": "stopped/started", "reason": "maintenance ticket 1234" }
>
> My assumption is that this is an async call and you'd need to `GET
> /_shard_splits/state` to get whether the transition was completed (i.e.,
> all jobs in stopped state). Having the sub-resource also gives you the
> opportunity for this `GET` call.
>
>
Ah good call. I like the /_shard_split/state and { "state":
"stopped/started", "reason": "maintenance ticket 1234" } as a way to change
it.
Previously I had used a PUT to change the state but I see you suggest a
POST. Wonder about the difference. My reasoning was that a PUT modifies the
state as opposed to creating new child state so to speak.
As for _state, it is because initially there was no .../jobs endpoint so
top level always returned the list of jobs, and so we had an "underscore"
endpoint to change the state.
> > * `GET /_shard_splits/jobs`
> >
> > Get all shard split jobs
> >
> > Response body:
> >
> > {
> > "jobs": [
> > {
> > "id":
> > "001-e41e8751873b56e4beafa373823604d26a2f11ba434a040f865b48df835ccb0b",
> > "job_state": "completed",
> ...
> > }
> > ],
> > "offset": 0,
> > "total_rows": 1
> > }
>
> My main question here was whether you needed all the pieces of information
> in this response -- is for example just the job_state enough?
>
I mainly mirrored the _scheduler/jobs behavior. This would allow someone to
filter or select specific fields (using say jq '.') from all the jobs.
There are other fields that might be interesting to filter by such as node,
shard name. I think it would be simpler from that point of view than
creating a script to get a list of jobs then GET them individually.
> > * `POST /_shard_splits/jobs`
> >
> > Start a shard splitting job.
> >
> > Request body:
> >
> > {
> > "node": "dbc...@db1.sandbox001.cloudant.net",
> > "shard": "shards/00000000-FFFFFFFF/username/dbname.$timestamp"
> > }
>
> How would you see this looking if we wanted to split all replicas of a
> shard? It'd be nice to avoid having to pass in an array of nodes for this
> if possible.
>
For all shard copies, I can see just omitting the node part. Submitting 3
jobs would be easy however the complication there is if creates 3 jobs and
3 separate IDs or a single job but that job is responsible for monitoring
and tracking sub-jobs on other 2 copies.
>
> >
> > Response body:
> >
> > {
> > "id":
> > "001-e41e8751873b56e4beafa373823604d26a2f11ba434a040f865b48df835ccb0b",
> > "ok": true
> > }
> >
> > Or if there are too many shard splitting jobs (a limit inspired by
> > scheduling replicator as well) it might return an error:
> >
> > {
> > "error": "max_jobs_exceeded",
> > "reason" "There are $N jobs currently running"
> > }
>
> Is there a queue in play here, or do jobs either go straight to running or
> get rejected?
>
>
Yeah no queue in there currently. User would have to remove some jobs from
the system to be able to insert new ones.
> > If shard splitting is disabled globally, user get an error and a reason.
> > The reason here would be the reason sent in the `PUT /_shard_splits`
> body.
> >
> > {
> > "error": "stopped",
> > "reason": "Shard splitting is disabled on the cluster currently"
> > }
>
> 👍
>
> > * `GET /_shard_splits/jobs/$jobid`
>
> My thought for this one was that it'd be more useful to the user to
> enumerate the transition states and times the job has been through rather
> than have a single created/started/updated -- given that there's an
> internal state machine.
>
> {
> "id":
> "001-5f553fd2d9180c74aa39c35377fe3e1731d09ec39bbd0f02541f55148e48d888",
> "job_state": "completed",
> "node": "node1@127.0.0.1",
> "source": "shards/00000000-1fffffff/db.1548186810",
> "split_state": "completed",
> "state_info": {},
> "targets": [
> "shards/00000000-0fffffff/db.1548186810",
> "shards/10000000-1fffffff/db.1548186810"
> ],
> "transitions": [
> {"state": "created", ts: "2019-01-23T18:36:17.951228Z"},
> {"state": "running", ts: "2019-01-23T18:36:18.457231Z"},
> {"state": "stopped", ts: "2019-01-23T18:49:19.174453Z"},
> {"state": "running", ts: "2019-01-23T19:49:19.174453Z"},
> {"state": "completed", ts: "2019-01-23T19:55:19.174453Z"}
> ]
> }
>
> ts being the time the state was entered (and by definition the time the
> previous state was exited).
>
>
That sounds reasonable. I like the idea.
> > * `DELETE /_shard_splits/jobs/$jobid`
> >
> > Remove a job. After a job completes or fails, it will not be
> automatically
> > removed but will stay around to allow the user to retrieve its status.
> > After its status is inspected the user should use the DELETE method to
> > remove the job. If the job is running, it will be cancelled and removed
> > from the system.
> >
> > Response body:
> >
> > {
> > "ok": true
> > }
> >
>
> Is there a way to stop/cancel a splitting job? `POST
> /_shard_splits/jobs/$jobid/state`? I guess DELETE could handle this but I
> could imagine wanting to see that a job was stopped rather than it just
> disappearing.
>
>
There is no way to control the job state individually currently. Shard
splitting as a whole is stopped and it would reflect in the job state when
queried. I can try adding the ability to stop a job individually. Then a
POST (though I am inclined to go with PUT as discussed above) to
../$jobid/state with a {"state":"running"/"stopped", "reason":"..."} would
work.
Thanks again Mike, those were great suggestions!
-Nick
