I just finished testing with threading. It worked well when my long running process was not doing cpu intensive tasks (e.g being blocked by IO - db, filesystem), but once I started processes that are cpu intensive, than threading was not very helpful. It looked like my short tasks got blocked for 10 minutes, and once long running process started doing DB queries, then I got my short lived processes running.
I will look into reserve and see if I can have participant to have affinity towards a worker. That way I dont have to worry about multiple long running processes running in parallel and consuming a lot of RAM. Thanks, Iuri On Friday, January 4, 2013 9:56:58 AM UTC-6, Iuri G. wrote: > > Hello John, > > You are right, I have participant classes that inherit > from storage participant and implement on_workitem method. Processes have a > human participant in the end that has related information for the work done > by previous participants, that we can review later. > > I just found some docs where you talk about storage participant and how it > by default says do_not_thread => true. For some reason, I remember reading > that ruote 2.3 threads by default, and in my mind I assumed that storage > partiicpant would be running in its own thread by default. I should have > checked it. I will implement do_not_thread and set it to false. Also, We > are running MRI 1.9.3, which has green threads, unlike JRuby that has true > multithreading. I'm not sure if do_not_thread will help alot in this case, > but I will try.I will also experiment with setting current thread priority > to low inside my long running participant and see if MRI can prioritize > well for me. > > Thank you for your valuable responses. I will post results shortly. > > Iuri > > > > On Thursday, January 3, 2013 6:31:39 PM UTC-6, John Mettraux wrote: >> >> >> On Thu, Jan 03, 2013 at 05:30:37PM -0600, Iuri Gagnidze wrote: >> > >> > Let me try to explain my problem in a different way. >> > >> > Right now I have participant A that takes couple hours to run and I >> have >> > participant B that takes only couple seconds and is scheduled to run >> every >> > minute. >> > >> > When I run participant A, participant B get blocked, since the worker >> > instance is busy doing work for participant A, and participant B misses >> its >> > schedule. >> >> Hello again, >> >> is your participant using do_not_thread --> true ? By default it's false, >> so >> participant work happens in its own thread, thus that should not block. >> >> Ah, I guess you're overriding StorageParticipant... >> >> Wouldn't you be better served by a "normal" participant? One that doesn't >> extend StorageParticipant? >> StorageParticipant is meant for human task lists. >> >> An automated task is better served by something else that a storage >> participant. >> >> At least, let your #do_not_thread respond false. >> >> >> > What I'm trying to do is to have two worker instances, were one worker >> will >> > only consume participant A, and another worker only consumes >> participant B. >> > That way I am guaranteed that long tasks won't block my short tasks. >> > >> > Also, I have couple long running tasks queued up and I don't want to >> run >> > them in parallel, but if I bring up two instances of worker, then both >> of >> > them start running long running tasks that are queued up. >> >> Maybe you could override the #reserve(msg) method in your storage and >> return >> false if the msg is a "dispatch" for the wrong participant_name. >> >> The worker places a reference to itself in >> Thread.current['ruote_worker'], >> the storage could use that to determine if the reserve should fail or be >> successful. (in the reserve case, true means success, ie the worker can >> go on >> processing the msg). >> >> >> If the root issue is task a blocking task b, please use something else >> than a >> storage participant. >> >> I remember vaguely discussing with people on this list about storage >> participant and automated tasks and I came up with explanations on how to >> combine both... >> >> Try setting #do_not_thread to responding false... >> >> Do you use a StorageParticipant because you need to list the workitems >> currently being processed by this participant? >> >> -- >> John Mettraux - http://lambda.io/jmettraux >> > -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en
