Nice code, thnx!
I couldn’t resist the temptation of writing some code wrapping the
task server to create alternative queue management. Not unlikely
MarkLogic Server 5.0 will contain improvements itself, but not sure
it satisfies all needs. This could be a basis for adding what is yet
lacking. The extensions of MarkLogic Server pretty much provide
everything you need to make this work, as my code shows. The biggest
problem is imitating some kind of cron job. I did so with module that
spawns itself after a sleep. Does the trick, but does not guard the
case when it stops. I added a manager interface that allows you to
monitor it though. It is most suited for background processing anyway
I think, it only uses idle threads.
It I put my work at github: https://github.com/grtjn/ml-queue
Kind regards,
Geert
*Van:*[email protected]
[mailto:[email protected]] *Namens *Christopher
Cieslinski
*Verzonden:* donderdag 13 oktober 2011 20:09
*Aan:* General MarkLogic Developer Discussion
*Onderwerp:* Re: [MarkLogic Dev General] how to purge the task server
queue?
We have run something like this to "brute force" through clearing the
requests as they get initiated from the queue. Far from perfect, but
it "gets the job done," so to speak.
(: ##### Task Server Job Clear ##### :)
xquery version "1.0-ml";
declare namespace ss = "http://marklogic.com/xdmp/status/server";
<http://marklogic.com/xdmp/status/server>;
declare namespace hs = "http://marklogic.com/xdmp/status/host";
<http://marklogic.com/xdmp/status/host>;
let $taskServerId as xs:unsignedLong :=
xdmp:host-status(xdmp:host())//hs:task-server-id
return
(: Run for around 9 minutes (if there are any queued tasks left)
- hopefully all are cleared by then. If not, run the script again :)
for $i as xs:integer in (1 to 5400)
return
for $requestId as xs:unsignedLong in
xdmp:server-status(xdmp:host(), $taskServerId)//ss:request-id/text()
return (
try {
xdmp:request-cancel(xdmp:host(), $taskServerId,
$requestId)
} catch ($e) {
xdmp:log("Failed to cancel requests, retrying...")
},
xdmp:sleep(100)
)
Chris
------------------------------------------------------------------------
*From*: Mike Sokolov
*Sent*: Thursday, October 13, 2011 12:06 PM
*To*: General MarkLogic Developer Discussion
*Subject*: Re: [MarkLogic Dev General] how to purge the task server
queue?
The "replace my script with one that does nothing, and then after things
have settled down again, put it back the way it was before" hack is so
useful, that it deserves to be enshrined as a function in the library
with a name of its own. Is there some way that behavior could be
codified meaningfully?
For example: xdmp:interrupt-module ($module-name) that prevent that
module from being started until some condition is met (the task server
is cleared? the task server has nothing queued with that name?)
-Mike
On 10/11/2011 12:19 PM, Geert Josten wrote:
Hi,
I think what Jakob has in mind is checking some queue state as soon as a task from the queue gets initiated. At that point the task that is being initiated from the queue could decide to continue or cancel. Rather similar to replacing the module with this 'no-op' suggestion by Damon..
But I thought someone on this list mentioned pondering about a separate registration of tasks, and firing them one by one, based on priority etc. Most of it should be feasible with one itself reactivating thread, that spawns others, or perhaps using a cron..
Kind regards,
Geert
-----Oorspronkelijk bericht-----
Van:[email protected]
<mailto:[email protected]>
[mailto:[email protected]] Namens Danny Sokolsky
Verzonden: dinsdag 11 oktober 2011 17:48
Aan: General MarkLogic Developer Discussion
Onderwerp: Re: [MarkLogic Dev General] how to purge the task server queue?
Hi Jakob,
request-cancel would not work to clear the queue because the requests have not yet started and therefore do not have an ID. You need to restart the node that has the task server to clear the queue (or do something clever like Damon suggested to make it clear Real Fast).
There are some RFEs around providing more control of the task queue, and I put your comments there.
Also remember, if you are using CPF and you "clear the queue" by restarting, CPF keeps a persistent state and remembers where you where, continueing processing after the restart, so careful of the whack-a-mole issue.
-Danny
-----Original Message-----
From:[email protected]
<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Jakob Fix
Sent: Tuesday, October 11, 2011 7:53 AM
To: General MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] how to purge the task server queue?
Thanks Damon,
we restarted the server. When you say "replace the invoked module with
a no-op module" this will not remove tasks currently in the queue,
correct?
On a related note, could one write some code around
xdmp:request-cancel() to purge the queue?
In any case, I think it would be a useful addition to the api to be
able to better control the task server.
Many thanks for your answer,
Jakob.
On Tue, Oct 11, 2011 at 15:35, Damon Feldman
<[email protected]> <mailto:[email protected]> wrote:
Jakob,
Yes, restarting the server will clear the task queue. Also, you can replace the invoked module
with a no-op module (xdmp:log("skipping queued task") or similar) if no other code is
calling it. The queue will then "drain" quickly.
I don't believe there is a programmatic way to remove queued tasks.
Yours,
Damon
-----Original Message-----
From:[email protected]
<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Jakob Fix
Sent: Tuesday, October 11, 2011 8:55 AM
To: General Mark Logic Developer Discussion
Subject: [MarkLogic Dev General] how to purge the task server queue?
Hi, we have a long waiting list on our task server and would like to
remove them programmatically. It doesn't look like there is such an
option in the admin: api, or is there? Also, would restarting the
server remove the waiting tasks from the queue?
thanks in advance,
Jakob.
_______________________________________________
General mailing list
[email protected] <mailto:[email protected]>
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected] <mailto:[email protected]>
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected] <mailto:[email protected]>
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected] <mailto:[email protected]>
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected] <mailto:[email protected]>
http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________
General mailing list
[email protected] <mailto:[email protected]>
http://developer.marklogic.com/mailman/listinfo/general
NOTICE: This email message is for the sole use of the intended
recipient(s) and may contain confidential and privileged information.
Any unauthorized review, use, disclosure or distribution is
prohibited. If you are not the intended recipient, please contact the
sender by reply email and destroy all copies of the original message.
