> so most folks tend to use a store which can support atomic "push" and "pop" operations so you can ensure that one and only one worker gets the job.
This usually requires a centralized resource. My scheme is 100% distributed. On Fri, Oct 5, 2012 at 4:14 PM, Dan Milon <[email protected]> wrote: > Mongo is definitely NOT always consistent. > > On Fri, Oct 5, 2012 at 10:14 PM, Evan <[email protected]> wrote: > >> You are basically proposing the schema for DelayedJobs (Ruby) [ >> https://github.com/collectiveidea/delayed_job ]. This kind of thing >> gets really weird in eventually consistant databases, but Mongo (like >> mySQL) is always consistant so you should be OK. However, a lot of folks >> have been finding some locking problems with this approach (multiple job >> execution or really aggressive table locking is needed), so most folks tend >> to use a store which can support atomic "push" and "pop" operations so you >> can ensure that one and only one worker gets the job. Redis is the most >> popular of these types of stores these day. I make use of that property in >> http://actionherojs.com/ for exactly this purpose, as does the very >> popular https://github.com/defunkt/resque and some other projects. >> >> >> On Friday, October 5, 2012 12:32:39 PM UTC-7, Mark Hahn wrote: >> >>> I may be crazy but I'm implementing a scheme where processes get the >>> tasks from a db record and then stores their process number in that record. >>> Then while they are running they periodically check to make sure that >>> their server number is still the one in the record. If another process has >>> 'stolen' the task then the process aborts and looks for another one to do. >>> >>> This is the only way I could figure out how to do task assignment when >>> faced with a db that only has "eventual consistency". The CouchDB i'm >>> using offers no atomic operations so this was the only reliable way to do >>> it. >>> >>> It works quite well. In the usual case it just grabs the task, does it, >>> and moves on. Collisions are rare, but they may be more frequent as the >>> cluster grows in size. >>> >>> >>> On Fri, Oct 5, 2012 at 9:21 AM, Dan Milon <[email protected]> wrote: >>> >>>> greelkorke ment using a job queue where jobs are put, and handed to >>>> workers. >>>> >>>> If you want to do it only with mongo, you'll need to use some "lock" >>>> document, that is set and unset by the first process which tries to >>>> initiate a task. All other processes which try to grab the lock while its >>>> held by another process should assume that the job is being worked by >>>> another process. But thats really ugly & has problems because jobs cant be >>>> acknowledged, so if a process crashes while its performing some task, >>>> you're fucked. >>>> >>>> On Fri, Oct 5, 2012 at 5:07 PM, Tom <[email protected]> wrote: >>>> >>>>> Unfortunately I'm afraid that I don't see how a scheduler can avoid >>>>> the concurrency problems. Note that the advantages (e.g. in availability) >>>>> of having a cluster should be maintained here, and so you cannot run a >>>>> scheduler in a separate process on a single server. If every server would >>>>> be running the scheduler, the same concurrency problems would arise. What >>>>> were you proposing? >>>>> >>>>> About the initialization, I guess that would work. It is not the way I >>>>> would prefer to do it, as I would like the application to be >>>>> self-controlled and usable without running special tools, but I reckon it >>>>> is an acceptable approach. >>>>> >>>>> Tom >>>>> >>>>> Op vrijdag 5 oktober 2012 19:59:33 UTC+7 schreef greelgorke het >>>>> volgende: >>>>> >>>>>> i wouldn't do it that way. when deploying your app just do a >>>>>> pre-start script, that ensures the existence of your desired data. >>>>>> If you have periodical tasks, it's best to use a lib for it, that >>>>>> triggers jobs appart of your main application. i.E >>>>>> http://stackoverflow.com/**q**uestions/3785736/is-there-a-**jo** >>>>>> b-scheduler-library-for-**node-**js<http://stackoverflow.com/questions/3785736/is-there-a-job-scheduler-library-for-node-js>, >>>>>> so you just avoid the concurrency problems. >>>>>> >>>>>> Am Freitag, 5. Oktober 2012 14:04:58 UTC+2 schrieb Tom: >>>>>>> >>>>>>> I've setup a cluster of physical servers. Each server runs exactly >>>>>>> the same code. Moreover, each server runs multiple node processes using >>>>>>> the build in cluster functionality. >>>>>>> >>>>>>> I use MongoDB (native) to share information between processes and >>>>>>> servers. However, I am having some difficulty with running a special >>>>>>> task >>>>>>> that needs to be executed only once during initialization: >>>>>>> > if a special `admin` account does not yet exist in the database, >>>>>>> it should be created >>>>>>> >>>>>>> Originally I figured that I could read from the MongoDB master >>>>>>> server on each node and check if the admin account already exists. If it >>>>>>> does not then another node has not yet created it, so this node should >>>>>>> do >>>>>>> so. However this is problematic because creating an admin password hash >>>>>>> is >>>>>>> asynchronous and takes time. Therefore there is a delay between when a >>>>>>> node >>>>>>> decides to create the account and when the account is being found by >>>>>>> other >>>>>>> nodes when querying the database. >>>>>>> >>>>>>> The code snippet that reads from the Mongo master only and creates >>>>>>> the account is available here: >>>>>>> https://gist.github.com/****3839429<https://gist.github.com/3839429> >>>>>>> >>>>>>> In the future I would also like a special task to be executed every >>>>>>> 5 minutes. This task must then only be executed by a running server, and >>>>>>> not by all servers. >>>>>>> >>>>>>> In short: when running a cluster of servers, how do you coordinate >>>>>>> between these servers which of them is going to execute a sole task >>>>>>> such as >>>>>>> the one described above? >>>>>>> >>>>>>> Tom >>>>>>> >>>>>> -- >>>>> Job Board: http://jobs.nodejs.org/ >>>>> Posting guidelines: https://github.com/joyent/** >>>>> node/wiki/Mailing-List-**Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>>> You received this message because you are subscribed to the Google >>>>> Groups "nodejs" group. >>>>> To post to this group, send email to [email protected] >>>>> >>>>> To unsubscribe from this group, send email to >>>>> nodejs+un...@**googlegroups.com >>>>> >>>>> For more options, visit this group at >>>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>>> >>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: https://github.com/joyent/**node/wiki/Mailing-List- >>>> **Posting-Guidelines<https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines> >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to [email protected] >>>> >>>> To unsubscribe from this group, send email to >>>> nodejs+un...@**googlegroups.com >>>> >>>> For more options, visit this group at >>>> http://groups.google.com/**group/nodejs?hl=en?hl=en<http://groups.google.com/group/nodejs?hl=en?hl=en> >>>> >>> >>> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
