Re: [MarkLogic Dev General] [RESOLVE] Spawned Task Appears to Block Other Threads

[Ron Hitchens]

   I posted a few messages about this a couple of months ago with
reasonably in-depth discussion of locks, deadlocks and concurrency.

   To really gain a solid understanding of how this stuff works, get to
know and love the most excellent paper *Inside MarkLogic Server*, available
here: https://developer.marklogic.com/inside-marklogic


Ron Hitchens r...@overstory.co.uk, +44 7879 358212


On November 10, 2017 at 9:44:41 PM, Eliot Kimber (ekim...@contrext.com)
wrote:

I moved my response handler to a different web app than the web app that
starts the job submission task, but I was still having the issue where the
response handler didn’t handle responses (or didn’t get responses, hard for
me to tell which it is) until the job submission task is canceled.

I realized that my problem was the initial job submission was updating the
job record for each job, but I was doing the update as part of the main
processing, rather than using eval(), so the commit wasn’t done until the
task ended.

The response handler also wants to update the job record, but because there
is a pending commit, it is blocked.

By having the job submission do the update via eval() then the commit is
done immediately and everything works.

So the lesson is: you have to really understand the implications of updates
and commits or you will go astray. Or maybe “concurrency is always hard”.

Thanks,

Eliot
--
Eliot Kimber
http://contrext.com



On 11/10/17, 9:39 AM, "general-boun...@developer.marklogic.com on behalf of
Eliot Kimber"  wrote:

Yes, the client process is started from a Web app, so I think your analysis
is correct.

I will move the response handling to a separate Web app—probably should
have done that from the start.

Thanks,

Eliot
--
Eliot Kimber
http://contrext.com



On 11/9/17, 11:46 PM, "general-boun...@developer.marklogic.com on behalf of
Geert Josten"  wrote:

Hi Eliot,

I think you kicked off your watcher job with an HTTP request, and it keeps
the port open until it finishes. Only one thread can use the port at the
same time. Use a different port for task response traffic, or consider
running your watcher as a scheduled task.

Not super robust, and probably not used in production, but i did write an
alternative queque for MarkLogic. It might give you some ideas..

https://github.com/grtjn/ml-queue


Cheers,
Geert

On 11/10/17, 1:06 AM, "general-boun...@developer.marklogic.com on behalf
of Eliot Kimber"  wrote:

>I have a system where I have a ³client² ML server that submits jobs to a
>set of remote ML servers, checking their task queues and keeping each
>server¹s queue at a max of 100 queued items (the remote servers could go
>away without notice so the client needs to be able to restart tasks and
>not have too many things queued up that would just have to resubmitted).
>
>The remote tasks then talk back to the client to report status and return
>their final results.
>
>My job submission code use recursive functions to iterate over the set of
>tasks to be submitted, checking for free remote queue slots via the ML
>REST API and submitting jobs as the queues empty. This code is spawned
>into a separate task in the task server. It uses xdmp:sleep(1000) to
>pause between checking the job queues.
>
>This all works fine, in that my jobs are submitted correctly and the
>remote queues fill up.
>
>However, as long as the job-submission task in the task server is
>running, the HTTP app that handles the REST calls from the remote servers
>is blocked (which blocks the remote jobs, which are of course waiting for
>responses from the client).
>
>If I kill the task server task, then the remote responses are handled as
>I would expect.
>
>My question: Why would the task server task block the other app? There
>must be something I¹m doing or not doing but I have no idea what it might
>be.
>
>Thanks,
>
>Eliot
>--
>Eliot Kimber
>http://contrext.com
>
>
>
>
>___
>General mailing list
>General@developer.marklogic.com
>Manage your subscription at:
>http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
___
General mailing list
General@developer.marklogic.com
Manage your 

Re: [MarkLogic Dev General] [RESOLVE] Spawned Task Appears to Block Other Threads

[Eliot Kimber]

I moved my response handler to a different web app than the web app that starts 
the job submission task, but I was still having the issue where the response 
handler didn’t handle responses (or didn’t get responses, hard for me to tell 
which it is) until the job submission task is canceled.

I realized that my problem was the initial job submission was updating the job 
record for each job, but I was doing the update as part of the main processing, 
rather than using eval(), so the commit wasn’t done until the task ended.

The response handler also wants to update the job record, but because there is 
a pending commit, it is blocked. 

By having the job submission do the update via eval() then the commit is done 
immediately and everything works.

So the lesson is: you have to really understand the implications of updates and 
commits or you will go astray. Or maybe “concurrency is always hard”.

Thanks,

Eliot
--
Eliot Kimber
http://contrext.com
 


On 11/10/17, 9:39 AM, "general-boun...@developer.marklogic.com on behalf of 
Eliot Kimber"  wrote:

Yes, the client process is started from a Web app, so I think your analysis 
is correct.

I will move the response handling to a separate Web app—probably should 
have done that from the start.

Thanks,

Eliot
--
Eliot Kimber
http://contrext.com
 


On 11/9/17, 11:46 PM, "general-boun...@developer.marklogic.com on behalf of 
Geert Josten"  wrote:

Hi Eliot,

I think you kicked off your watcher job with an HTTP request, and it 
keeps
the port open until it finishes. Only one thread can use the port at the
same time. Use a different port for task response traffic, or consider
running your watcher as a scheduled task.

Not super robust, and probably not used in production, but i did write 
an
alternative queque for MarkLogic. It might give you some ideas..

https://github.com/grtjn/ml-queue


Cheers,
Geert

On 11/10/17, 1:06 AM, "general-boun...@developer.marklogic.com on behalf
of Eliot Kimber"  wrote:

>I have a system where I have a ³client² ML server that submits jobs to 
a
>set of remote ML servers, checking their task queues and keeping each
>server¹s queue at a max of 100 queued items (the remote servers could 
go
>away without notice so the client needs to be able to restart tasks and
>not have too many things queued up that would just have to 
resubmitted).
>
>The remote tasks then talk back to the client to report status and 
return
>their final results.
>
>My job submission code use recursive functions to iterate over the set 
of
>tasks to be submitted, checking for free remote queue slots via the ML
>REST API and submitting jobs as the queues empty. This code is spawned
>into a separate task in the task server. It uses xdmp:sleep(1000) to
>pause between checking the job queues.
>
>This all works fine, in that my jobs are submitted correctly and the
>remote queues fill up.
>
>However, as long as the job-submission task in the task server is
>running, the HTTP app that handles the REST calls from the remote 
servers
>is blocked (which blocks the remote jobs, which are of course waiting 
for
>responses from the client).
>
>If I kill the task server task, then the remote responses are handled 
as
>I would expect.
>
>My question: Why would the task server task block the other app? There
>must be something I¹m doing or not doing but I have no idea what it 
might
>be.
>
>Thanks,
>
>Eliot
>--
>Eliot Kimber
>http://contrext.com
> 
>
>
>
>___
>General mailing list
>General@developer.marklogic.com
>Manage your subscription at:
>http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list

Re: [MarkLogic Dev General] Retrieving collections based on document uri

[Pooja Rajput]

Thanks Justin.

From: general-boun...@developer.marklogic.com 
[mailto:general-boun...@developer.marklogic.com] On Behalf Of Justin Makeig
Sent: Thursday, November 9, 2017 6:02 PM
To: MarkLogic Developer Discussion 
Subject: Re: [MarkLogic Dev General] Retrieving collections based on document 
uri

You can build a collection query dynamically from the result sequence of 
collection URIs that xdmp:document-get-collections() returns. For example,

cts:and-query((
  …
  cts:collection-query(xdmp:document-get-collections($uri)),
  …
)…)

On Nov 9, 2017, at 3:17 PM, Pooja Rajput 
> wrote:

Thanks for the response Ron.
I need to use this inside cts-query so that I can filter the type of 
collections as well.
Hence I am looking for similar cts query.

Thanks,
Pooja

From: 
general-boun...@developer.marklogic.com
 [mailto:general-boun...@developer.marklogic.com] On Behalf Of Ron Hitchens
Sent: Thursday, November 9, 2017 4:14 PM
To: MarkLogic Developer Discussion 
>
Subject: Re: [MarkLogic Dev General] Retrieving collections based on document 
uri


https://docs.marklogic.com/xdmp:document-get-collections


Ron Hitchens r...@overstory.co.uk, +44 7879 358212



On November 9, 2017 at 11:12:18 PM, Pooja Rajput 
(pooja.raj...@healthbi.com) wrote:
Hi Team,
I need to retrieve the collections associated with particular document.
All I have is document uri. How can I retrieve this info using cts query like 
xdmp:Document-get-collections provides results.
Any example is appreciated.

Thanks
Pooja


___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
___
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

Re: [MarkLogic Dev General] Spawned Task Appears to Block Other Threads

[Eliot Kimber]

Yes, the client process is started from a Web app, so I think your analysis is 
correct.

I will move the response handling to a separate Web app—probably should have 
done that from the start.

Thanks,

Eliot
--
Eliot Kimber
http://contrext.com
 


On 11/9/17, 11:46 PM, "general-boun...@developer.marklogic.com on behalf of 
Geert Josten"  wrote:

Hi Eliot,

I think you kicked off your watcher job with an HTTP request, and it keeps
the port open until it finishes. Only one thread can use the port at the
same time. Use a different port for task response traffic, or consider
running your watcher as a scheduled task.

Not super robust, and probably not used in production, but i did write an
alternative queque for MarkLogic. It might give you some ideas..

https://github.com/grtjn/ml-queue


Cheers,
Geert

On 11/10/17, 1:06 AM, "general-boun...@developer.marklogic.com on behalf
of Eliot Kimber"  wrote:

>I have a system where I have a ³client² ML server that submits jobs to a
>set of remote ML servers, checking their task queues and keeping each
>server¹s queue at a max of 100 queued items (the remote servers could go
>away without notice so the client needs to be able to restart tasks and
>not have too many things queued up that would just have to resubmitted).
>
>The remote tasks then talk back to the client to report status and return
>their final results.
>
>My job submission code use recursive functions to iterate over the set of
>tasks to be submitted, checking for free remote queue slots via the ML
>REST API and submitting jobs as the queues empty. This code is spawned
>into a separate task in the task server. It uses xdmp:sleep(1000) to
>pause between checking the job queues.
>
>This all works fine, in that my jobs are submitted correctly and the
>remote queues fill up.
>
>However, as long as the job-submission task in the task server is
>running, the HTTP app that handles the REST calls from the remote servers
>is blocked (which blocks the remote jobs, which are of course waiting for
>responses from the client).
>
>If I kill the task server task, then the remote responses are handled as
>I would expect.
>
>My question: Why would the task server task block the other app? There
>must be something I¹m doing or not doing but I have no idea what it might
>be.
>
>Thanks,
>
>Eliot
>--
>Eliot Kimber
>http://contrext.com
> 
>
>
>
>___
>General mailing list
>General@developer.marklogic.com
>Manage your subscription at:
>http://developer.marklogic.com/mailman/listinfo/general

___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general



___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general

[MarkLogic Dev General] start/stop with CENTOS-7 and systemd

[APEL Holger]

Hello,

I've installed the latest MarkLogic 9 on CENTOS 7 which uses systemd to 
bootstrap the system. But since the installer only provides 
/etc/init.d/MarkLogic scripts I have created the following system unit file to 
integrate it with systemd

[Unit]
Description=MarkLogic Server
After=syslog.target

[Service]
Type=forking
ExecStart=/etc/init.d/MarkLogic start
ExecStop=/etc/init.d/MarkLogic stop

[Install]
WantedBy=multi-user.target

That all seems to work and I can start/stop MarkLogic with systemctl now, but 
the automatic restart triggered by the admin console doesn't work and 
systemd-journald reports:
Broadcast message from systemd-journ...@vmlinmlc06.iso.ch (Fri 2017-11-10 
10:05:11 CET):
MarkLogic[7536]: Initialization: XDMP-DISABLED: Service disabled on 
vmlinmlc06.iso.ch: Stopping by SIGTERM from pid 7535

Has anyone experience with MarkLogic and systemd?

Holger
___
General mailing list
General@developer.marklogic.com
Manage your subscription at: 
http://developer.marklogic.com/mailman/listinfo/general