My error was using the list of <task> elements as the control for my loop rather than some lookup key. This was causing read locks on the <term> elements that then blocked my subsequent update attempts.
I rewrote my looping code to limit reads of the <task> elements to just those places where it was doing an update so that no read lock was held outside the updating code. My process is now almost completed for its full 500K+ task set, so memory issues resolved. Now I just need to fix a bug in my queue loading logic and it might just work for production… Cheers, E. -- Eliot Kimber http://contrext.com On 11/27/17, 2:50 PM, "[email protected] on behalf of Eliot Kimber" <[email protected] on behalf of [email protected]> wrote: I’ve tried turning on the lock tracing and I do see deadlocks both with the recursive version and the loop-with-set version, but the loop-with-set version is basically ever task doc is locked (which is what I would expect) but with the recursive version I get little bursts of five or six deadlocks at a time but the code generally runs. But this does suggest that my code is creating unexpected locks that I need to resolve. Cheers, E. -- Eliot Kimber http://contrext.com On 11/27/17, 11:45 AM, "[email protected] on behalf of Will Thompson" <[email protected] on behalf of [email protected]> wrote: Eliot, Is the controller/while-loop transaction read-only (i.e.: is xdmp:request-timestamp() nonempty)? If it is, then I think you can be sure it's not holding locks. Otherwise, I would restructure that part of the application so that any transaction responsible for dispatching jobs doesn't make any updates. Generally, you don't want a long-running update transaction to touch lots of documents. If you turn on debug logging, ML will report to the error log when it detects a deadlock (and randomly kills and retries one of the deadlocking transactions). There is also a lock trace event you can enable to get detailed output about which transactions are holding locks and which ones are waiting on them (See: https://help.marklogic.com/knowledgebase/article/View/387/0/understanding-the-lock-trace-diagnostic-trace-event). All of the reporting IIRC is based on transaction IDs, so you generally have to do your own logging elsewhere to identify which IDs are associated with which transactions. As you might expect, this can get kind of hairy. In the past I have used a task server job to check for a condition, and if it hasn't been met, sleep for a few 100ms and respawn. Similar behavior could also be accomplished with triggers or wth CPF, but both are probably overkill for your case. -Will > On Nov 27, 2017, at 10:59 AM, William Sawyer <[email protected]> wrote: > > You could recursively spawn or setup a schedule task to run every minute or faster if needed. > > -Will > > On Mon, Nov 27, 2017 at 9:56 AM, Eliot Kimber <[email protected]> wrote: > I have a client-server system where the client is spawning 100s of 1000s of jobs on the client. The client polls the servers to see when each server’s task queue is ready for more jobs. This all works fine. > > Logically this polling is a while-true() loop that will continue until either all the servers are offline or all the tasks to be submitted are consumed. > > In a procedural language this is trivial, but in XQuery 2 I’m not finding a way to do it that works. In XQuery 3 I could use the new iterate operator but that doesn’t seem to be available in MarkLogic 9. > > My first attempt was to use a recursive process, relying on tail recursion optimization to avoid blowing the stack buffer. That worked logically but I still ran into out-of-memory on the server at some point (around 200K jobs submitted) and it seems likely that it was runaway recursion doing it. > > So I tried using a simple loop with xdmp:set() to iterate over the tasks and use an exception to break out when all the tasks are done: > > try { > for $i in 1 to 1000000 (: i.e., loop forever :) > if (empty($tasks)) > then error() > else submit-task(head($tasks)) > xdmp:set($tasks, tail($tasks)) > } catch ($e) { > (: We’re done. ( > } > > Is there a better way to do this kind of looping forever? > > I’m also having a very strange behavior where in my new looping code I’m getting what I think must be a pending commit deadlock that I didn’t get in my recursive version of the code. I can trace the code to the xdmp:eval() that would commit an update to the task and that code never returns. > > Each task is a document that I update to reflect the details of the task’s status (start and end times, current processing status, etc.). Those updates are all done either in separately-run modules or via xdmp:eval(), so as far as I can tell there shouldn’t be any issues with uncommitted updates. I didn’t change anything in the logic that updates the task documents, only the loop that iterates over the tasks. > > Could it be that the use of xdmp:set() to modify the $tasks variable (a sequence of <task> elements) would be causing some kind of commit lock? > > Thanks, > > Eliot > > -- > Eliot Kimber > http://contrext.com > > > > > _______________________________________________ > General mailing list > [email protected] > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > Manage your subscription at: > https://urldefense.proofpoint.com/v2/url?u=http-3A__developer.marklogic.com_mailman_listinfo_general&d=DwICAg&c=IdrBOxAMwHPzAikPNzltHw&r=_thRNTuzvzYaEDwaA_AfnAe5hN2lWgi6qdluz6ApLYI&m=lBzqSD4gcYpoA1Wa70W-NT925su08aQxRr7d9dGMNz8&s=vZmkjApvUox0IYCvco1Y7YcYmCwsIpxoU-w1dPpXGqs&e= _______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
