Hi Tim, I don't think there is any way to de-prioritize the order of something on the task server queue once it is already spawned. If you wanted to do that, it would have to be before it is spawned.
What you might be able to do (and I think you hinted at this in your question) is to use a different host to spawn the tasks to. The host that a task is spawned to is the same host in which the query is evaluated (the e-node), so you can try to send higher priority tasks to a different (and less used) e-node. I am not sure what the best way to do this is, and I would guess that would depend on your application. It could be as simple as having some dispatcher code somewhere that looks at the priority (your application would have to supply this) and then redirects the query to another server. Or you could do it in a load balancer or proxy forwarder. By the time it gets to CPF, however, it is probably too late, so this would have to come before the CPF event is triggered. I don't know of another way to do this, as there is no API do remove or reorder items in the task server queue. -Danny From: [email protected] [mailto:[email protected]] On Behalf Of Tim Meagher Sent: Monday, July 26, 2010 4:13 AM To: 'General Mark Logic Developer Discussion' Subject: [MarkLogic Dev General] Prioritizing entries in the task server queue Hi Folks, I have a workflow for processing documents of various priorities using the Content Processing Framework. The problem I'm running into is that I might get 10,000 documents that need to be processed at a low priority which get submitted to the task server queue, but then maybe 10 documents come in that are of a higher priority (I'm using these counts for purposes of discussion). What I would like to be able to do is to insert the 10 high priority items in the queue so that they are processed before any outstanding low priority items in the task server queue, in other words I want to interrupt FIFO processing. I'm not concerned about the high priority processing starving low priority processing as the volume of the high priority items is relatively low, but nonetheless an elegant solution would allow me to fine-tune the process so that low priority starvation does not occur. There was some previous discussion about using tail-recursion with xdmp:spawn. That way I would hopefully be able to select the next document to process based on its relative priority. In that case I would probably want to revise the CPF process to merely fill customized priority queues, e.g. high, mid, and low priority queues and to use tail recursion to examine the queues and decide which document to process next. I get the impression that clustering could be a useful way to create task servers that are dedicated to higher and lower priority processing for the needs of an entire organization, but it seems to me that allowing for pre-emption in a given task server could be a really useful feature. Perhaps there are some existing features that are provided to deal with just this problem. There are times when I've submitted more docs to be processed by the task server and would like to be able to dequeue them - I suppose that a prioritization solution would also allow for dequeuing tasks. Thanks ahead of time for any help! Tim Meagher
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
