nickva commented on a change in pull request #409: RFC for CouchDB background workers URL: https://github.com/apache/couchdb-documentation/pull/409#discussion_r293103079
########## File path: rfcs/007-background-jobs.md ########## @@ -0,0 +1,337 @@ +--- +name: Formal RFC +about: Submit a formal Request For Comments for consideration by the team. +title: 'Background jobs with FoundationDB' +labels: rfc, discussion +assignees: '' + +--- + +[NOTE]: # ( ^^ Provide a general summary of the RFC in the title above. ^^ ) + +# Introduction + +This document describes a data model, implementation, and an API for running +CouchDB background jobs with FoundationDB. + +## Abstract + +CouchDB background jobs are used for things like index building, replication +and couch-peruser processing. We present a generalized model which allows +creation, running, and monitoring of these jobs. + +## Requirements Language + +[NOTE]: # ( Do not alter the section below. Follow its instructions. ) + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", +"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this +document are to be interpreted as described in +[RFC 2119](https://www.rfc-editor.org/rfc/rfc2119.txt). + + +## General Concepts + +In the discussion below a job is considered to be an abstract unit of work. It +is identified by a `JobId` and has a `JobType`. The user creates a job, and +then it is executed by a job processor. A job processor is language-specific +execution unit that runs the job. It could be an Erlang process, a thread, or +just a function. + +The API used to create jobs is called the `Job Creation API` and the API used +by the job processors to run jobs is called the `Job Processing API`. + +### Job States + +Jobs in the system can be in a few different states. After a job is added and +is waiting to run, the job is considered to be `pending`. A job executed by +a job processor is considered to be `running`. When a job is neither `running`, +nor `pending`, it is considered to be `finished`. This is the state transition +diagram: + +``` + +------------>+ + | | + | v + -->[PENDING] [RUNNING]--->[FINISHED] + ^ | + | | + +-------------+ +``` + + + +### Typical API Usage + +The general pattern of using this API might look like: + + * Job creators: + - Call `add/3,4` to add a job + - If the job needs to be resubmitted, call `resubmit/1` + + * Job processors: + - Call `accept/1,2` and wait until it gets a job to process. + - Periodically call `update/2,3` to prevent the job from being re-enqueued + due to idleness. + - When done running a job, call `finish/2,3` + + +### Job Creation API + +``` +add(Type, JobId, JobData[, ScheduledTime]) -> {ok, Job} | {error, Error} +``` + - Add a job to be executed by a job processor + - `Job` is an opaque object used to represent a job. Users of the API MUST + NOT rely on any fields in this map as those may change at any time. + - `JobData` is map with a job type-specific data in it. It MAY contain any + data as long as it can be properly encoded as JSON. + - `ScheduledTime` is an optional parameter to schedule the job to be executed + at a later time. The format is an integer seconds since UNIX epoch. + +``` +remove(Job) -> ok | {error, Error} +``` + - Remove a job. If it is running, it will be stopped. + +``` +resubmit(Job) -> ok | {error, Error} +``` + - Indicates the job should be re-submitted. That is, after the job processor + calls `finish/2`, the job will be re-enqueued as `pending` again. If the job + is not `running` this function has no effect. + +``` +get_job_data(Job) -> {ok, JobData} | {error, Error} +``` + - Get `JobData` associated with the job. + +``` +get_job_state(Job) -> {ok, pending | running | finished} | {error, Error} +``` + - Get the job's state. + +``` +set_type_timeout(Type, TimeoutSec) -> ok +``` + + - Set the activity timeout for a job type. Review comment: Yes, good call! ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
