https://bugzilla.wikimedia.org/show_bug.cgi?id=57613
--- Comment #3 from Tyler Romeo <[email protected]> --- The specification for this needs a bit more. Some questions: * How distributed? GitHub is "distributed", but at the same time centralized. Do we just want the task execution to be distributed, or will the entire system be distributed? IMO the best solution will most likely be something that's completely distributed. * Where are tasks consumed from? The description says it needs to be configurable by traditional crontabs. Does this mean a single crontab on one computer is enough to schedule a task for the system? Or should it be entered on every crontab on every computer? Neither of these are the best options, and I don't think crontab configuration in such an environment is the most practical idea. * Are the tasks themselves distributed? Or will a task be isolated to a single machine? With most tasks I'd imagine the latter. If we assume the sort of answers I gave to the questions are correct, then this will be a fun task to undertake. The network would have to be configured as a full mesh, and we'd likely use something like Raft to determine consensus (Paxos works as well, but since Raft has leader election it's probably a better candidate, not to mention it has a Go implementation). However, consensus protocols generally do not work well with large clusters, which mean we'd need to implement some sort of task sharding so that smaller clusters can operate independently. This brings to mind a sort of worker approach, where a consensus algorithm is performed on the five or six masters, who then hand off tasks to a worker cluster they own. This would still allow for failover and network partitions while still being performant. But that also brings up the problem of how the master will handle failures in its own worker cluster. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. _______________________________________________ Wikibugs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
