In a previous thread
https://mail-archives.apache.org/mod_mbox/couchdb-dev/201901.mbox/%3CCAJd%3D5Hbs%2BNwrt0%3Dz%2BGN68JPU5yHUea0xGRFtyow79TmjGN-_Sg%40mail.gmail.com%3E
we discussed the possibility of adding shard splitting to CouchDB.
There was some interest in the IRC channel and the mailing list, and I
would like some help refining the new API so that the whole community has
say in it. Also, to propose to continue working on this in the ASF repo for
visibility and continued feedback from the whole community.
To recap the previous thread, this is about having the basics necessary to
split database shards. It would happen without having to stop the cluster
or put nodes in maintenance mode.
The API was partially inspired by _scheduler/jobs, another similar API
introduced along with the scheduling replicator functionality.
## API Spec
* `GET /_shard_splits`
Get a summary of shard splitting for the whole cluster. This would return
the total number of shard splitting jobs and the number of active ones,
that is the ones that are doing work at that very moment. Another piece of
information is the global state of shard splitting, if it is stopped or
running.
{
"jobs_total": 10,
"jobs_running": 2,
"states": {
"running": [
"node1@127.0.0.1",
"node2@127.0.0.1",
"node3@127.0.0.1"
]
}
}
* `PUT /_shard_splits`
Enable or disable shard splitting on the cluster. This feature that would
be helpful is to have the ability to disable or enable shard splitting
globally on a cluster. This can function as a feature flag as well used in
cases where existing tooling might for example manipulate shards and it's
desirable for the shard splitting to not interfere.
To disable request body would be:
{
"stop": "<reason>"
}
To (re)-enable:
{
"start": true
}
An alternative for this would be to have another underscore path like
`_shard_splits/_state` but I feel it is better to minimize the use of
underscore path, they feel less REST-ful.
* `GET /_shard_splits/jobs`
Get all shard split jobs
Response body:
{
"jobs": [
{
"id":
"001-e41e8751873b56e4beafa373823604d26a2f11ba434a040f865b48df835ccb0b",
"job_state": "completed",
"node": "node1@127.0.0.1",
"source": "shards/00000000-1fffffff/db.1548175503",
"split_state": "completed",
"state_info": {},
"targets": [
"shards/00000000-0fffffff/db.1548175503",
"shards/10000000-1fffffff/db.1548175503"
],
"time_created": "2019-01-23T18:36:17.951228Z",
"time_started": "2019-01-23T18:36:18.457231Z",
"time_updated": "2019-01-23T18:49:19.174453Z"
}
],
"offset": 0,
"total_rows": 1
}
The offset and total_rows here are to keep a view-like shape of the
response and to have it look more like _scheduler/jobs.
* `POST /_shard_splits/jobs`
Start a shard splitting job.
Request body:
{
"node": "dbc...@db1.sandbox001.cloudant.net",
"shard": "shards/00000000-FFFFFFFF/username/dbname.$timestamp"
}
Response body:
{
"id":
"001-e41e8751873b56e4beafa373823604d26a2f11ba434a040f865b48df835ccb0b",
"ok": true
}
Or if there are too many shard splitting jobs (a limit inspired by
scheduling replicator as well) it might return an error:
{
"error": "max_jobs_exceeded",
"reason" "There are $N jobs currently running"
}
If shard splitting is disabled globally, user get an error and a reason.
The reason here would be the reason sent in the `PUT /_shard_splits` body.
{
"error": "stopped",
"reason": "Shard splitting is disabled on the cluster currently"
}
* `GET /_shard_splits/jobs/$jobid`
Access information about a particular shard splitting job by its ID. This
is the ID returned from the POST to _shard_splits/jobs request.
{
"id":
"001-5f553fd2d9180c74aa39c35377fe3e1731d09ec39bbd0f02541f55148e48d888",
"job_state": "completed",
"node": "node1@127.0.0.1",
"source": "shards/00000000-1fffffff/db.1548186810",
"split_state": "completed",
"state_info": {},
"targets": [
"shards/00000000-0fffffff/db.1548186810",
"shards/10000000-1fffffff/db.1548186810"
],
"time_created": "2019-01-23T18:36:17.951228Z",
"time_started": "2019-01-23T18:36:18.457231Z",
"time_updated": "2019-01-23T18:49:19.174453Z"
}
* `DELETE /_shard_splits/jobs/$jobid`
Remove a job. After a job completes or fails, it will not be automatically
removed but will stay around to allow the user to retrieve its status.
After its status is inspected the user should use the DELETE method to
remove the job. If the job is running, it will be cancelled and removed
from the system.
Response body:
{
"ok": true
}
