Re: [MarkLogic Dev General] Tracking Spawned Tasks?
That’s a good point about comments. I’ll try to add comments for things I found lacking and subsequently discover answers to. Cheers, E. -- Eliot Kimber http://contrext.com From: on behalf of Evan Lenz Reply-To: MarkLogic Developer Discussion Date: Monday, August 14, 2017 at 12:27 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Tracking Spawned Tasks? Hi Eliot, One nice thing about the MarkLogic documentation for functions is that you can add helpful comments yourself. I've seen others write "See also" comments and have done so myself from time to time. In general I've found the comments very helpful, even if it's a user getting their newbie question answered. Evan Evan Lenz President, Lenz Consulting Group, Inc. http://lenzconsulting.com On Mon, Aug 14, 2017 at 9:57 AM, Erik Hennum wrote: Hi, Eliot and Ron: The return option is explained with the rest of the options in the eval article: http://docs.marklogic.com/xdmp:eval The second example under spawn uses the promise: http://docs.marklogic.com/xdmp:spawn As Ron notes, the server field is only useful if the polling requests go back to the same host. To allow for restarts, the polling logic should check for the persisted final status document if the server field is empty. (That's the motivation for persisting a final status document even when using server fields.) Thanks for the feedback on the documentation -- I'll pass that along. Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Ron Hitchens [r...@ronsoft.com] Sent: Monday, August 14, 2017 7:55 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Tracking Spawned Tasks? Proceed with caution when using server fields. They exist only on a single machine, they are not propagated across nodes in a cluster. If you have a cluster behind a load balancer (as most are) and you stash something in a server field to be checked later, the next request may be vectored to a different cluster node, where your stashed value will not be present. Likewise, if you put something in a field to be picked up by a spawned task, the spawned task may run on a different node. Ron Hitchens r...@overstory.co.uk, +44 7879 358212 On August 14, 2017 at 3:24:32 PM, Eliot Kimber (ekim...@contrext.com) wrote: I like using set-server-field: my requirement feels like just what server fields were intended for. Cheers, E. -- Eliot Kimber http://contrext.com On 8/14/17, 8:32 AM, "general-boun...@developer.marklogic.com on behalf of Erik Hennum" wrote: Hi, Eliot: xdmp:spawn() doesn't return an identifier because, if it is used as a future via the result option, it is obligated to return the result. The approach you sketch below -- passing in an identifier and writing tickets to a status database -- is pretty much what InfoStudio did. One refinement would be to log status in a server field via xdmp:set-server-field() and, on completion, write final status to a database (for durability in the case of a restart). Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Eliot Kimber [ekim...@contrext.com] Sent: Saturday, August 12, 2017 10:15 AM To: MarkLogic Developer Discussion Subject: [MarkLogic Dev General] Tracking Spawned Tasks? Using ML 8 I’m refining a profiling application that spawns a number of tasks and then, eventually, reports on the results once all the tasks have completed. Right now I just fire off the tasks and then refresh my app, which looks for results. It would be nice to be able to show the status of the spawned tasks but it looks like xdmp:spawn() doesn’t return anything (sort of expected to get some sort of task ID or something) and so there’s no obvious way to track spawned tasks from the spawning application. I could do something like generate private task IDs and pass those as parameters to the spawned tasks and then maintain a set of task status docs, but I was hoping there was some something easier. It seems like it would be a common requirement but I couldn’t find anything useful in the ML 8 docs or searching the web. Thanks, Eliot -- Eliot Kimber http://contrext.com ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at
Re: [MarkLogic Dev General] Tracking Spawned Tasks?
Hi Eliot, One nice thing about the MarkLogic documentation for functions is that you can add helpful comments yourself. I've seen others write "See also" comments and have done so myself from time to time. In general I've found the comments very helpful, even if it's a user getting their newbie question answered. Evan Evan Lenz President, Lenz Consulting Group, Inc. http://lenzconsulting.com On Mon, Aug 14, 2017 at 9:57 AM, Erik Hennum wrote: > Hi, Eliot and Ron: > > The return option is explained with the rest of the options in the eval > article: > > http://docs.marklogic.com/xdmp:eval > > The second example under spawn uses the promise: > > http://docs.marklogic.com/xdmp:spawn > > As Ron notes, the server field is only useful if the polling requests go > back to the same host. > > To allow for restarts, the polling logic should check for the persisted > final status document if the server field is empty. (That's the > motivation for persisting a final status document even when using server > fields.) > > Thanks for the feedback on the documentation -- I'll pass that along. > > > Erik Hennum > > -- > *From:* general-boun...@developer.marklogic.com [ > general-boun...@developer.marklogic.com] on behalf of Ron Hitchens [ > r...@ronsoft.com] > *Sent:* Monday, August 14, 2017 7:55 AM > *To:* MarkLogic Developer Discussion > *Subject:* Re: [MarkLogic Dev General] Tracking Spawned Tasks? > > >Proceed with caution when using server fields. They exist only on a > single machine, they are not propagated across nodes in a cluster. If you > have a cluster behind a load balancer (as most are) and you stash something > in a server field to be checked later, the next request may be vectored to > a different cluster node, where your stashed value will not be present. > Likewise, if you put something in a field to be picked up by a spawned > task, the spawned task may run on a different node. > > > Ron Hitchens r...@overstory.co.uk, +44 7879 358212 <+44%207879%20358212> > > > On August 14, 2017 at 3:24:32 PM, Eliot Kimber (ekim...@contrext.com) > wrote: > > I like using set-server-field: my requirement feels like just what server > fields were intended for. > > Cheers, > > E. > -- > Eliot Kimber > http://contrext.com > > > > On 8/14/17, 8:32 AM, "general-boun...@developer.marklogic.com on behalf > of Erik Hennum" erik.hen...@marklogic.com> wrote: > > Hi, Eliot: > > xdmp:spawn() doesn't return an identifier because, if it is used as a > future via the result option, it is obligated to return the result. > > The approach you sketch below -- passing in an identifier and writing > tickets to a status database -- is pretty much what InfoStudio did. > > One refinement would be to log status in a server field via > xdmp:set-server-field() and, on completion, write final status to a > database (for durability in the case of a restart). > > > Hoping that helps, > > > Erik Hennum > > ________ > From: general-boun...@developer.marklogic.com [general-bounces@developer. > marklogic.com] on behalf of Eliot Kimber [ekim...@contrext.com] > Sent: Saturday, August 12, 2017 10:15 AM > To: MarkLogic Developer Discussion > Subject: [MarkLogic Dev General] Tracking Spawned Tasks? > > Using ML 8 > > I’m refining a profiling application that spawns a number of tasks and > then, eventually, reports on the results once all the tasks have completed. > > Right now I just fire off the tasks and then refresh my app, which looks > for results. > > It would be nice to be able to show the status of the spawned tasks but it > looks like xdmp:spawn() doesn’t return anything (sort of expected to get > some sort of task ID or something) and so there’s no obvious way to track > spawned tasks from the spawning application. > > I could do something like generate private task IDs and pass those as > parameters to the spawned tasks and then maintain a set of task status > docs, but I was hoping there was some something easier. > > It seems like it would be a common requirement but I couldn’t find > anything useful in the ML 8 docs or searching the web. > > Thanks, > > Eliot > -- > Eliot Kimber > http://contrext.com > > > > > ___ > General mailing list > General@developer.marklogic.com > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general > ___ > General mailing list > General@developer.marklogic.com &g
Re: [MarkLogic Dev General] Tracking Spawned Tasks?
Hi, Eliot and Ron: The return option is explained with the rest of the options in the eval article: http://docs.marklogic.com/xdmp:eval The second example under spawn uses the promise: http://docs.marklogic.com/xdmp:spawn As Ron notes, the server field is only useful if the polling requests go back to the same host. To allow for restarts, the polling logic should check for the persisted final status document if the server field is empty. (That's the motivation for persisting a final status document even when using server fields.) Thanks for the feedback on the documentation -- I'll pass that along. Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Ron Hitchens [r...@ronsoft.com] Sent: Monday, August 14, 2017 7:55 AM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Tracking Spawned Tasks? Proceed with caution when using server fields. They exist only on a single machine, they are not propagated across nodes in a cluster. If you have a cluster behind a load balancer (as most are) and you stash something in a server field to be checked later, the next request may be vectored to a different cluster node, where your stashed value will not be present. Likewise, if you put something in a field to be picked up by a spawned task, the spawned task may run on a different node. Ron Hitchens r...@overstory.co.uk<mailto:r...@overstory.co.uk>, +44 7879 358212 On August 14, 2017 at 3:24:32 PM, Eliot Kimber (ekim...@contrext.com<mailto:ekim...@contrext.com>) wrote: I like using set-server-field: my requirement feels like just what server fields were intended for. Cheers, E. -- Eliot Kimber http://contrext.com On 8/14/17, 8:32 AM, "general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> on behalf of Erik Hennum" mailto:general-boun...@developer.marklogic.com> on behalf of erik.hen...@marklogic.com<mailto:erik.hen...@marklogic.com>> wrote: Hi, Eliot: xdmp:spawn() doesn't return an identifier because, if it is used as a future via the result option, it is obligated to return the result. The approach you sketch below -- passing in an identifier and writing tickets to a status database -- is pretty much what InfoStudio did. One refinement would be to log status in a server field via xdmp:set-server-field() and, on completion, write final status to a database (for durability in the case of a restart). Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com> [general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>] on behalf of Eliot Kimber [ekim...@contrext.com<mailto:ekim...@contrext.com>] Sent: Saturday, August 12, 2017 10:15 AM To: MarkLogic Developer Discussion Subject: [MarkLogic Dev General] Tracking Spawned Tasks? Using ML 8 I’m refining a profiling application that spawns a number of tasks and then, eventually, reports on the results once all the tasks have completed. Right now I just fire off the tasks and then refresh my app, which looks for results. It would be nice to be able to show the status of the spawned tasks but it looks like xdmp:spawn() doesn’t return anything (sort of expected to get some sort of task ID or something) and so there’s no obvious way to track spawned tasks from the spawning application. I could do something like generate private task IDs and pass those as parameters to the spawned tasks and then maintain a set of task status docs, but I was hoping there was some something easier. It seems like it would be a common requirement but I couldn’t find anything useful in the ML 8 docs or searching the web. Thanks, Eliot -- Eliot Kimber http://contrext.com ___ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com<mailto:General@developer.marklogic.com> Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Tracking Spawned Tasks?
Proceed with caution when using server fields. They exist only on a single machine, they are not propagated across nodes in a cluster. If you have a cluster behind a load balancer (as most are) and you stash something in a server field to be checked later, the next request may be vectored to a different cluster node, where your stashed value will not be present. Likewise, if you put something in a field to be picked up by a spawned task, the spawned task may run on a different node. Ron Hitchens r...@overstory.co.uk, +44 7879 358212 On August 14, 2017 at 3:24:32 PM, Eliot Kimber (ekim...@contrext.com) wrote: I like using set-server-field: my requirement feels like just what server fields were intended for. Cheers, E. -- Eliot Kimber http://contrext.com On 8/14/17, 8:32 AM, "general-boun...@developer.marklogic.com on behalf of Erik Hennum" wrote: Hi, Eliot: xdmp:spawn() doesn't return an identifier because, if it is used as a future via the result option, it is obligated to return the result. The approach you sketch below -- passing in an identifier and writing tickets to a status database -- is pretty much what InfoStudio did. One refinement would be to log status in a server field via xdmp:set-server-field() and, on completion, write final status to a database (for durability in the case of a restart). Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com [ general-boun...@developer.marklogic.com] on behalf of Eliot Kimber [ ekim...@contrext.com] Sent: Saturday, August 12, 2017 10:15 AM To: MarkLogic Developer Discussion Subject: [MarkLogic Dev General] Tracking Spawned Tasks? Using ML 8 I’m refining a profiling application that spawns a number of tasks and then, eventually, reports on the results once all the tasks have completed. Right now I just fire off the tasks and then refresh my app, which looks for results. It would be nice to be able to show the status of the spawned tasks but it looks like xdmp:spawn() doesn’t return anything (sort of expected to get some sort of task ID or something) and so there’s no obvious way to track spawned tasks from the spawning application. I could do something like generate private task IDs and pass those as parameters to the spawned tasks and then maintain a set of task status docs, but I was hoping there was some something easier. It seems like it would be a common requirement but I couldn’t find anything useful in the ML 8 docs or searching the web. Thanks, Eliot -- Eliot Kimber http://contrext.com ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Tracking Spawned Tasks?
I like using set-server-field: my requirement feels like just what server fields were intended for. Cheers, E. -- Eliot Kimber http://contrext.com On 8/14/17, 8:32 AM, "general-boun...@developer.marklogic.com on behalf of Erik Hennum" wrote: Hi, Eliot: xdmp:spawn() doesn't return an identifier because, if it is used as a future via the result option, it is obligated to return the result. The approach you sketch below -- passing in an identifier and writing tickets to a status database -- is pretty much what InfoStudio did. One refinement would be to log status in a server field via xdmp:set-server-field() and, on completion, write final status to a database (for durability in the case of a restart). Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Eliot Kimber [ekim...@contrext.com] Sent: Saturday, August 12, 2017 10:15 AM To: MarkLogic Developer Discussion Subject: [MarkLogic Dev General] Tracking Spawned Tasks? Using ML 8 I’m refining a profiling application that spawns a number of tasks and then, eventually, reports on the results once all the tasks have completed. Right now I just fire off the tasks and then refresh my app, which looks for results. It would be nice to be able to show the status of the spawned tasks but it looks like xdmp:spawn() doesn’t return anything (sort of expected to get some sort of task ID or something) and so there’s no obvious way to track spawned tasks from the spawning application. I could do something like generate private task IDs and pass those as parameters to the spawned tasks and then maintain a set of task status docs, but I was hoping there was some something easier. It seems like it would be a common requirement but I couldn’t find anything useful in the ML 8 docs or searching the web. Thanks, Eliot -- Eliot Kimber http://contrext.com ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Tracking Spawned Tasks?
Can you expand on this statement: “if it is used as a future via the result option, it is obligated to return the result.” I didn’t see anything about this in the ML 8 docs for xdmp:spawn() and it seems pretty important. One general comment I’ll make about the ML docs is that it seems to assume/require a fairly encyclopedic knowledge of many subtle details. I’ve now read pretty much all the guides at least once and spent a lot of time in the reference docs, and while it’s all easy to access and very useful, it’s still a challenge to fully understand the implications of many things. For example, the documentation for various options for ranges only shows the syntax but has no guidance about semantics or what values are actually allowed, which would be very useful. This is documentation where a few well-placed see-alsos or a bit more usage guidance would go a long way. As a professional technical writer I know how challenging this aspect of docs is but it would help a lot. Cheers, Eliot On 8/14/17, 8:32 AM, "general-boun...@developer.marklogic.com on behalf of Erik Hennum" wrote: Hi, Eliot: xdmp:spawn() doesn't return an identifier because, if it is used as a future via the result option, it is obligated to return the result. The approach you sketch below -- passing in an identifier and writing tickets to a status database -- is pretty much what InfoStudio did. One refinement would be to log status in a server field via xdmp:set-server-field() and, on completion, write final status to a database (for durability in the case of a restart). Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Eliot Kimber [ekim...@contrext.com] Sent: Saturday, August 12, 2017 10:15 AM To: MarkLogic Developer Discussion Subject: [MarkLogic Dev General] Tracking Spawned Tasks? Using ML 8 I’m refining a profiling application that spawns a number of tasks and then, eventually, reports on the results once all the tasks have completed. Right now I just fire off the tasks and then refresh my app, which looks for results. It would be nice to be able to show the status of the spawned tasks but it looks like xdmp:spawn() doesn’t return anything (sort of expected to get some sort of task ID or something) and so there’s no obvious way to track spawned tasks from the spawning application. I could do something like generate private task IDs and pass those as parameters to the spawned tasks and then maintain a set of task status docs, but I was hoping there was some something easier. It seems like it would be a common requirement but I couldn’t find anything useful in the ML 8 docs or searching the web. Thanks, Eliot -- Eliot Kimber http://contrext.com ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
Re: [MarkLogic Dev General] Tracking Spawned Tasks?
Hi, Eliot: xdmp:spawn() doesn't return an identifier because, if it is used as a future via the result option, it is obligated to return the result. The approach you sketch below -- passing in an identifier and writing tickets to a status database -- is pretty much what InfoStudio did. One refinement would be to log status in a server field via xdmp:set-server-field() and, on completion, write final status to a database (for durability in the case of a restart). Hoping that helps, Erik Hennum From: general-boun...@developer.marklogic.com [general-boun...@developer.marklogic.com] on behalf of Eliot Kimber [ekim...@contrext.com] Sent: Saturday, August 12, 2017 10:15 AM To: MarkLogic Developer Discussion Subject: [MarkLogic Dev General] Tracking Spawned Tasks? Using ML 8 I’m refining a profiling application that spawns a number of tasks and then, eventually, reports on the results once all the tasks have completed. Right now I just fire off the tasks and then refresh my app, which looks for results. It would be nice to be able to show the status of the spawned tasks but it looks like xdmp:spawn() doesn’t return anything (sort of expected to get some sort of task ID or something) and so there’s no obvious way to track spawned tasks from the spawning application. I could do something like generate private task IDs and pass those as parameters to the spawned tasks and then maintain a set of task status docs, but I was hoping there was some something easier. It seems like it would be a common requirement but I couldn’t find anything useful in the ML 8 docs or searching the web. Thanks, Eliot -- Eliot Kimber http://contrext.com ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
[MarkLogic Dev General] Tracking Spawned Tasks?
Using ML 8 I’m refining a profiling application that spawns a number of tasks and then, eventually, reports on the results once all the tasks have completed. Right now I just fire off the tasks and then refresh my app, which looks for results. It would be nice to be able to show the status of the spawned tasks but it looks like xdmp:spawn() doesn’t return anything (sort of expected to get some sort of task ID or something) and so there’s no obvious way to track spawned tasks from the spawning application. I could do something like generate private task IDs and pass those as parameters to the spawned tasks and then maintain a set of task status docs, but I was hoping there was some something easier. It seems like it would be a common requirement but I couldn’t find anything useful in the ML 8 docs or searching the web. Thanks, Eliot -- Eliot Kimber http://contrext.com ___ General mailing list General@developer.marklogic.com Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
