Hi Wayne, Not at all, good you mention it actually. Testing the spawn depth as we speak. A scheduled task could certainly help, it could launch a queue-cron once every several hours to make sure. I added logic to prevent two cron jobs to be active at the same time. Not entirely fail-safe, but even if there would be two, not much harm could be done. The queue would just empty slightly quicker.. :)
Kind regards, Geert Van: general-boun...@developer.marklogic.com [mailto:general-boun...@developer.marklogic.com] Namens Wayne Feick Verzonden: zaterdag 15 oktober 2011 17:29 Aan: general@developer.marklogic.com Onderwerp: Re: [MarkLogic Dev General] how to purge the task server queue? Oh, nevermind, wasn't following the thread closely enough and responded before applying coffee to my brain. On 10/15/2011 08:27 AM, Wayne Feick wrote: You could use a scheduled task instead of a task that keeps respawning itself, and that'll solve the restart problem. With the respawn approach, you'll eventually run into a limit on the number of respawns (protection against runaway code). On 10/15/2011 08:22 AM, Geert Josten wrote: Nice code, thnx! I couldn't resist the temptation of writing some code wrapping the task server to create alternative queue management. Not unlikely MarkLogic Server 5.0 will contain improvements itself, but not sure it satisfies all needs. This could be a basis for adding what is yet lacking. The extensions of MarkLogic Server pretty much provide everything you need to make this work, as my code shows. The biggest problem is imitating some kind of cron job. I did so with module that spawns itself after a sleep. Does the trick, but does not guard the case when it stops. I added a manager interface that allows you to monitor it though. It is most suited for background processing anyway I think, it only uses idle threads. It I put my work at github: https://github.com/grtjn/ml-queue Kind regards, Geert Van: general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> [mailto:general-boun...@developer.marklogic.com] Namens Christopher Cieslinski Verzonden: donderdag 13 oktober 2011 20:09 Aan: General MarkLogic Developer Discussion Onderwerp: Re: [MarkLogic Dev General] how to purge the task server queue? We have run something like this to "brute force" through clearing the requests as they get initiated from the queue. Far from perfect, but it "gets the job done," so to speak. (: ##### Task Server Job Clear ##### :) xquery version "1.0-ml"; declare namespace ss = "http://marklogic.com/xdmp/status/server";<http://marklogic.com/xdmp/status/server>; declare namespace hs = "http://marklogic.com/xdmp/status/host";<http://marklogic.com/xdmp/status/host>; let $taskServerId as xs:unsignedLong := xdmp:host-status(xdmp:host())//hs:task-server-id return (: Run for around 9 minutes (if there are any queued tasks left) - hopefully all are cleared by then. If not, run the script again :) for $i as xs:integer in (1 to 5400) return for $requestId as xs:unsignedLong in xdmp:server-status(xdmp:host(), $taskServerId)//ss:request-id/text() return ( try { xdmp:request-cancel(xdmp:host(), $taskServerId, $requestId) } catch ($e) { xdmp:log("Failed to cancel requests, retrying...") }, xdmp:sleep(100) ) Chris ________________________________ From: Mike Sokolov Sent: Thursday, October 13, 2011 12:06 PM To: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] how to purge the task server queue? The "replace my script with one that does nothing, and then after things have settled down again, put it back the way it was before" hack is so useful, that it deserves to be enshrined as a function in the library with a name of its own. Is there some way that behavior could be codified meaningfully? For example: xdmp:interrupt-module ($module-name) that prevent that module from being started until some condition is met (the task server is cleared? the task server has nothing queued with that name?) -Mike On 10/11/2011 12:19 PM, Geert Josten wrote: Hi, I think what Jakob has in mind is checking some queue state as soon as a task from the queue gets initiated. At that point the task that is being initiated from the queue could decide to continue or cancel. Rather similar to replacing the module with this 'no-op' suggestion by Damon.. But I thought someone on this list mentioned pondering about a separate registration of tasks, and firing them one by one, based on priority etc. Most of it should be feasible with one itself reactivating thread, that spawns others, or perhaps using a cron.. Kind regards, Geert -----Oorspronkelijk bericht----- Van: general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> [mailto:general-boun...@developer.marklogic.com] Namens Danny Sokolsky Verzonden: dinsdag 11 oktober 2011 17:48 Aan: General MarkLogic Developer Discussion Onderwerp: Re: [MarkLogic Dev General] how to purge the task server queue? Hi Jakob, request-cancel would not work to clear the queue because the requests have not yet started and therefore do not have an ID. You need to restart the node that has the task server to clear the queue (or do something clever like Damon suggested to make it clear Real Fast). There are some RFEs around providing more control of the task queue, and I put your comments there. Also remember, if you are using CPF and you "clear the queue" by restarting, CPF keeps a persistent state and remembers where you where, continueing processing after the restart, so careful of the whack-a-mole issue. -Danny -----Original Message----- From: general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Jakob Fix Sent: Tuesday, October 11, 2011 7:53 AM To: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] how to purge the task server queue? Thanks Damon, we restarted the server. When you say "replace the invoked module with a no-op module" this will not remove tasks currently in the queue, correct? On a related note, could one write some code around xdmp:request-cancel() to purge the queue? In any case, I think it would be a useful addition to the api to be able to better control the task server. Many thanks for your answer, Jakob. On Tue, Oct 11, 2011 at 15:35, Damon Feldman <damon.feld...@marklogic.com><mailto:damon.feld...@marklogic.com> wrote: Jakob, Yes, restarting the server will clear the task queue. Also, you can replace the invoked module with a no-op module (xdmp:log("skipping queued task") or similar) if no other code is calling it. The queue will then "drain" quickly. I don't believe there is a programmatic way to remove queued tasks. Yours, Damon -----Original Message----- From: general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> [mailto:general-boun...@developer.marklogic.com] On Behalf Of Jakob Fix Sent: Tuesday, October 11, 2011 8:55 AM To: General Mark Logic Developer Discussion Subject: [MarkLogic Dev General] how to purge the task server queue? Hi, we have a long waiting list on our task server and would like to remove them programmatically. It doesn't look like there is such an option in the admin: api, or is there? Also, would restarting the server remove the waiting tasks from the queue? thanks in advance, Jakob. _______________________________________________ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> http://developer.marklogic.com/mailman/listinfo/general NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com<mailto:wayne.fe...@marklogic.com> Phone: +1 650 655 2378 www.marklogic.com<http://www.marklogic.com> This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation. -- Wayne Feick Principal Engineer MarkLogic Corporation wayne.fe...@marklogic.com<mailto:wayne.fe...@marklogic.com> Phone: +1 650 655 2378 www.marklogic.com<http://www.marklogic.com> This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this e-mail communication by others is strictly prohibited. If you are not the intended recipient, please notify us immediately by returning this message to the sender and delete all copies. Thank you for your cooperation.
_______________________________________________ General mailing list General@developer.marklogic.com http://developer.marklogic.com/mailman/listinfo/general
