Hi, On Mon, Apr 18, 2016 at 4:20 PM, Timothée Maret <[email protected]> wrote: > ...One aspect that may differentiate asset processing from general job use > case is the guarantees offered from the job delivery. Some use case would > be fine with "at-least-one" or "maybe" delivery whereas some use case will > need "exactly-one" delivery....
It's not only about delivering the job submission message, in some cases it's the whole job execution that needs to be exactly once, for example. Designing (or configuring if using the SLING-5646 stuff) a batch job engine with relatively relaxed requirements about message delivery (like at least once for the job submission message) might make it much simpler to implement it in a scalable way. Of course there are some cases where exactly once *execution* of batch jobs is required, but then the whole execution chain should be considered, maybe using a distinct distributed consensus service to coordinate the whole execution chain. Ian mentioned to me that some queuing systems can play this role by having a two-phase "take message from queue" mechanism IIUC, where a job executor has to confirm success before a message is considered consume, that could also play this role. My point is that in some cases the whole job execution chain has to be considered for exactly once or at least once semantics. -Bertrand
