Hi,
On 18 April 2016 at 14:51, Bertrand Delacretaz <[email protected]>
wrote:
> Hi,
>
> I chatted with Ian last week about making our jobs engine more
> scalable (based on his SLING-5646 work) and I think a dead simple API
> for batch jobs might be useful, alongside our existing
> org.apache.sling.event Jobs API.
>
> Our jobs API provides fine-grained synchronous control on the jobs
> that it executes, like stopping jobs, querying the engine for job
> states etc.
>
> That's useful for small jobs and page approval workflows, but it's
> hard to implement in a scalable and distributed way.
>
> Heavy jobs like digital asset processing, for example, do not need
> such fine grained control.
>
> To execute such jobs in a scalable way, a "fire and almost forget"
> scenario should work fine: submit a job to process an asset, subscribe
> to an events stream about its status, check the latest status received
> after a time T and it's not done consider it failed submit it again.
> Make the jobs themselves idempotent for robustness if needed.
>
> I think this would be useful alongside our existing jobs engine, with
> an API that can be as simple as this:
>
> interface BatchEngine {
> /* Options can be relative priority, preferences for which node
> executes the job, etc. */
> JobId submit(Callable<Void> job, Map<String, Object> options);
> }
>
I am not certain how a Callable will work with a distributed
implementation, as the Callable implies that an Object implementing
Callable is provided in the submit essentially binding the Job to the JVM
where it was submitted.
>
> interface BatchEventsSource {
> /** If restrictToSpecificJobIds is not null the last known state
> of these jobs is resent, if available */
> void registerBatchEventListener(BatchEventsListener bleh, JobId
> ... restrictToSpecificJobIds);
> }
>
IIUC this is a bit on an Anti pattern with OSGi. AFAIK the Whiteboard
pattern is prefered. I recently refactored the new Jobs implementation
(SLING-5646) to eliminate listener registration in favour of whiteboard
style registration.
>
> interface BatchEventsListener {
> void onEvent(BatchEvent beh);
> }
>
> class BatchEvent {
> JobId getJobId();
> JobStatus getStatus();
> String getInfo();
> }
>
> WDYT?
>
Is there a reason that you think the implementation under SLING-5646 won't
support the batch use case ?
Best Regards
Ian
>
> We might also use an existing API if there's a good one, but I think
> we don't need more than the above.
>
> -Bertrand
>
