Running some experiments with the sqlmaster/sqlworker system shows that running multiple independent workers results in many jobs being done multiple times. This is noted in the manual as
"If you have more than one --sqlworker jobs may be run more than once." In simple testing, this is not an occasional thing, but results in nearly all jobs being performed N times. It seems the table already has the concept of jobs already taken (exitval: -1220) which would suggest it should be locking each job. How can one create a system where multiple workers grab jobs from the sql table in a way that minimizes this? If this is a race condition, can they take jobs in a shuffled/randomized order from available jobs in order to minimize this?
