Hi Danny,
This confirms our observations exactly. Thanks.
We have chosen the following approach for the moment, might be of interest for
other users..
All xquery requests that concern locking, start with the following statement:
xdmp:document-insert("synchronize-lock-requests", <x/>)
(Actually, we have a delegating Xquery, but as soon as it identified the call
as lock related, this statement is executed. The statement itself is tucked
away in a function called synchronize-lock-requests() ofcourse..)
After that statement, it is safe to check for existing locks in any way you
prefer, there will be no challenging transactions, they will all be handled
sequentially.
Note that it is not possible to enforce this on all Xqueries with one
instruction, it is up to the programmer to check all lock related xqueries. In
our case, we have only one externally accessible Xquery to handle, and in that
one all lock related code is concentrated on one specific spot, which made the
work for us very easy. It was quite literally one additional line of code.
We are still using xdmp:lock-acquire, xdmp:lock-release and
xdmp:document-locks, just out of convenience. We had written that code already,
so basically we only needed to add the document-insert. To make clear those
calls do not provide exclusive locks in our case, we pass the 'shared' option
and have the exclusiveness being checked in the application logic (and some
wrapping functions which we also already had written). The same can probably be
achieved using document properties, or even extra lock documents, one for each
real document, but this was quicker for us.. :)
Kind regards,
Geert
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of
> Danny Sokolsky
> Sent: woensdag 11 februari 2009 18:35
> To: General Mark Logic Developer Discussion
> Subject: RE: [MarkLogic Dev General]
> Atomicandexclusivelockondocuments(accross a cluster)
>
> Hi Geert,
>
> Locks that happen as a result of an update operation
> (xdmp:document-insert, xdmp:node-replace,
> xdmp:document-set-properties,
> and so on--in general, these are the functions that are in the Update
> Built-Ins section of the apidoc) have a few important differences from
> the ones you get via xdmp:lock-acquire. First, they are
> implicit; that
> is, they happen automatically. Second, they exist only for
> the duration
> of the transaction; when the transaction completes, the lock is
> automatically released. Third, they lock against *any* other
> transaction, including another transaction from the same user.
>
> xdmp:lock-acquire locks must be added explicitly (via an xquery
> operation, for example), and they only lock against other
> named *users*
> performing transactions. Also, they must be removed explicitly (or
> timed out, depending on how they were created). If you are
> running as a
> single user, xdmp:lock-acquire does not really do what you are looking
> for.
>
> Hope that helps,
> -Danny
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Geert
> Josten
> Sent: Wednesday, February 11, 2009 12:06 AM
> To: General Mark Logic Developer Discussion
> Subject: RE: [MarkLogic Dev General]
> Atomicandexclusivelockondocuments(accross a cluster)
>
> Danny,
>
> I think you explained almost all. I ran the examples from the
> dev guide
> and created some of my own, understanding a lot more about
> undetectable
> deadlocks etc.
>
> But one question still remains: what is the difference with
> document-locks instead of get-document-properties and lock-acquire
> instead of document-insert (or set-document-properties for instance)..
>
> Kind regards,
> Geert
>
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]] On Behalf Of
> > Danny Sokolsky
> > Sent: dinsdag 10 februari 2009 23:34
> > To: General Mark Logic Developer Discussion
> > Subject: RE: [MarkLogic Dev General] Atomic
> > andexclusivelockondocuments(accross a cluster)
> >
> > To explain why storing the "lock" on the property will work
> > transactionally, let's briefly review the transaction
> > semantics of MarkLogic Server (most of this is described in
> > the Developer's Guide "understanding Transactions in
> > MarkLogic Server" chapter--
> > http://developer.marklogic.com/pubs/4.0/books/dev_guide.pdf).
> >
> > MarkLogic Server has two types of requests (we'll call them
> > statements):
> > query statements and update statements. Query statements run
> > at a timestamp and simply look at the latest versions of
> > documents up to the timestamp in which the query runs. Query
> > statements do not lock any documents.
> >
> > An update statement occurs if there is any kind of update
> > operation in the statement. This is determined during static
> > analysis. Update statements run in what is called
> > readers/writers mode. During the statement evaluation, each
> > time a document is referenced the system automatically
> > attempts to get a read lock on the document. It does this
> > for each document the statement accesses. When it comes time
> > to update the document, the system tries to promote the read
> > lock on any documents that will be updated to a write lock.
> > If it cannot get the write lock, the statement will either
> > retry until it can get the lock or will throw an exception
> > and the update will fail.
> >
> > So if you have the property stored with the document URI and
> > you use the technique of checking the property for a known
> > value, that value will be checked while the system has a lock
> > on the document, it will not be allowed to be updated unless
> > it can get a write lock on the document.
> > Any other thread trying to update this document while it is
> > locked will have to wait until the lock is released, and as
> > long as you update the document and the property (or however
> > you are storing the "user" name) in the same transaction,
> > they will remain in sync.
> >
> >
> > -Danny
> >
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]] On Behalf Of
> > Geert Josten
> > Sent: Tuesday, February 10, 2009 1:37 PM
> > To: General Mark Logic Developer Discussion
> > Subject: RE: [MarkLogic Dev General] Atomic
> > andexclusivelockondocuments(accross a cluster)
> >
> >
> > > Yes, and to be clear, what I was talking about was to *not* use
> > > xdmp:lock-acquire at all, but to use the property (or
> > another document
> > > as Frank is suggesting) instead.
> >
> > I know you were not talking about lock-acquire, but do you
> > mean that it makes big difference and I really should not use
> > it to make it work like we want? (Or only use it in
> > combination with a document-insert as described by Frank)
> >
> > And next to this, if I supply the uri of the document that is
> > to be locked as a request param, how does MarkLogic Server
> > know which document to write-lock for the transaction? Or is
> > the write-lock started at the moment the document-insert is called?
> >
> > It sounds to me that by using a document-insert on a helper
> > doc, you are forcing the MarkLogic Server to synchronize the
> > lock requests, right? So if all lock requesting calls would
> > start with a document-insert to 'lock.xml' (not really doing
> > anything actually), it will result in all lock requests being
> > synchronized?
> >
> > I appreciate the input, I am really learning from these details..
> >
> > Kind regards,
> > Geert
> >
> >
> > > -----Original Message-----
> > > From: [email protected]
> > > [mailto:[email protected]] On
> Behalf Of Frank
> > > Sanders
> > > Sent: Tuesday, February 10, 2009 1:23 PM
> > > To: General Mark Logic Developer Discussion
> > > Subject: RE: [MarkLogic Dev General] Atomic and
> > > exclusivelockondocuments(accross a cluster)
> > >
> > > Geert,
> > >
> > > Using a process similar to what Danny described, and
> you've already
> > > implemented, those (micro?)seconds shouldn't matter.
> > > As he mentioned, MarkLogic Server guarantees that for any
> > two queries
> > > trying to update a given document the second query will see the
> > > results of the first query.
> > > We can use this to our advantage, but we need to make sure
> > that we're
> > > seen as an update query by the server:
> > >
> > > let $force-update-mode :=
> > xdmp:document-insert("/mutex-doc.xml", <foo
> > > />) let $user := "user-the-first"
> > > return
> > > (: do lock checking and updating here :)
> > >
> > > For this to be performant you'll have to create multiple
> mutex docs
> > > that in some way correspond to the documents you wish to
> > update. But
> > > all queries that want to update a given document are going
> > to have to
> > > block on the call to
> > > xdmp:document-insert() until the current query, with the
> > lock on the
> > > mutex, has finished executing.
> > >
> > > -fs
> > >
> > > -----Original Message-----
> > > From: [email protected]
> > > [mailto:[email protected]] On
> Behalf Of Geert
> > > Josten
> > > Sent: Tuesday, February 10, 2009 3:18 PM
> > > To: General Mark Logic Developer Discussion
> > > Subject: RE: [MarkLogic Dev General] Atomic and exclusive
> > > lockondocuments(accross a cluster)
> > >
> > > Hi Danny,
> > >
> > > Thanks for your input.
> > >
> > > Actually, we have similar logic, though we use the 'owner'
> > > property of locks themselves. It is more or less like this:
> > >
> > > Acquiring the lock:
> > >
> > > if (not(document-locks($uri))) then
> > > lock-acquire($uri, 'exclusive', 0, $owner, ()) else
> > > error('already locked')
> > >
> > > Performing update:
> > >
> > > if (document-locks($uri)//lock:owner = $owner) then
> > > do-update($uri)
> > > Else
> > > error('locked by someone else')
> > >
> > > The issue is in acquiring the lock. There is time between
> > the moment
> > > of checking for locks and acquiring the actual lock.
> > > It is very small (perhaps just a few CPU cycles), but time
> > nontheless,
> > > which leaves room for other threads or requests to make a
> > same call to
> > > lock-acquire at 'the same' time.
> > > Lock-acquire is atomic across a cluster, but since all
> lock-acquire
> > > calls are executed by the same user, it never fails, not
> > providing us
> > > the exclusive lock we hoped to get.
> > >
> > > Kind regards,
> > > Geert
> > >
> > > > -----Original Message-----
> > > > From: [email protected]
> > > > [mailto:[email protected]] On
> > Behalf Of Danny
> > > > Sokolsky
> > > > Sent: dinsdag 10 februari 2009 20:16
> > > > To: General Mark Logic Developer Discussion
> > > > Subject: RE: [MarkLogic Dev General] Atomic and exclusive lock
> > > > ondocuments(accross a cluster)
> > > >
> > > > You can store your locks in a separate document (or in a
> > > property on
> > > > the same document), and have your application logic only
> > > allow updates
> > >
> > > > if the "user" (that is, from your app's perspective,
> > since you are
> > > > only using one MarkLogic
> > > > user) is the one with the lock. The main difference
> > > between this and
> > > > the lock-acquire approach is that your application must
> > manage the
> > > > locks.
> > > >
> > > > If you implement this correctly, it should solve the
> > problem of the
> > > > second update request seeing what happened from the first update
> > > > request. Internally in MarkLogic Server, updates must
> > get a write
> > > > lock on a document before it can be updated. If someone
> > > else has the
> > > > write lock, then another update request must wait for the
> > > lock to be
> > > > released before it can proceed with its transaction.
> > > >
> > > > Imagine this scenario:
> > > >
> > > > /doc/mydoc.xml has the following property:
> > > >
> > > > <owner>user-a</owner>
> > > >
> > > > Your application code has a variable $user that is set to
> > > the current
> > > > user. Before any updates can occur, your code has some
> logic like
> > > > this:
> > > >
> > > > let $user := "user-a"
> > > > return
> > > > if ($user ne fn:data(
> > > > xdmp:document-get-properties("/doc/mydoc.xml",
> fn:QName("",
> > > > "owner"))) )
> > > > then fn:error("NOUPDATE", "You cannot update because you
> > > are the wrong
> > > > user")
> > > > else "put update code here"
> > > >
> > > > I might be missing something, but it seems like this
> > approach would
> > > > work. Of course implementing a content management system is
> > > > complicated, and you might find that managing the
> security in the
> > > > application layer will quickly get very complex. For the
> > > simple case,
> > >
> > > > however, it seems doable from my naive perspective.
> > > >
> > > > -Danny
> > > >
> > > > -----Original Message-----
> > > > From: [email protected]
> > > > [mailto:[email protected]] On
> > Behalf Of Geert
> > > > Josten
> > > > Sent: Monday, February 09, 2009 11:52 PM
> > > > To: General Mark Logic Developer Discussion
> > > > Subject: RE: [MarkLogic Dev General] Atomic and exclusive lock
> > > > ondocuments(accross a cluster)
> > > >
> > > > Hi Danny,
> > > >
> > > > Tjerk is my colleague. We already have something like that,
> > > using the
> > > > owner field of locks themselves, as we needed locks over
> > > > multi-transactions anyhow. But we need absolute 100 percent
> > > guarantee
> > > > that if two requests are challenging for an exclusive lock
> > > on the same
> > >
> > > > document, only one can be acquired. Unfortunately, since we
> > > are using
> > > > an application user, the xdmp:lock-acquire function never
> > > fails. And
> > > > however small, there is time between checking for a lock
> > > and acquiring
> > >
> > > > the lock itself, so leaving a tiny gap for two requests to
> > > get a lock
> > > > on the same document.
> > > >
> > > > Moreover, I noticed that the first lock is being
> > overwritten by the
> > > > second, effectively extending the lock-period (which could be
> > > > usefull).
> > > > If the second request comes from some other user, and the first
> > > > response is is a success, the evidence of the first request
> > > will get
> > > > lost. We really would like to prevent this to happen.
> > > >
> > > > One workaround we thought of ourselves is to create
> > temporary users
> > > > (one for each lock) and use xdmp:eval to acquire the
> lock as that
> > > > user. But that sounds like unnecessary overhead and is
> > likely to be
> > > > error prone (error handling, releasing locks and removing
> > > those temp
> > > > users is tricky).
> > > >
> > > > Any suggestion at all would be welcome..
> > > >
> > > > Kind regards,
> > > > Geert
> > > >
> > > > > -----Original Message-----
> > > > > From: [email protected]
> > > > > [mailto:[email protected]] On
> > > Behalf Of Danny
> > > > > Sokolsky
> > > > > Sent: maandag 9 februari 2009 20:07
> > > > > To: General Mark Logic Developer Discussion
> > > > > Subject: RE: [MarkLogic Dev General] Atomic and
> > exclusive lock on
> > > > > documents(accross a cluster)
> > > > >
> > > > > Hi Tjerk,
> > > > >
> > > > > One approach you can take is to store the current owner of the
> > > > > document somewhere in the document, or in the document
> > properties.
> > > > > Then, in your application, you can have logic that only
> > > allows that
> > > > > user to update the document. For example, if you had a
> > > > property that
> > > > > looked something like
> > > > > this:
> > > > >
> > > > > <owner>user-a</owner>
> > > > >
> > > > > Then you might have some logic in your update module that
> > > > checked that
> > > > > the user was user-a before allowing him to update the
> > > document. Of
> > > > > course it would be up to you to write that logic correctly.
> > > > >
> > > > > During the update in MarkLogic Server, the document is
> > > > locked across
> > > > > the cluster--that is a key part of the transaction
> semantics of
> > > > > MarkLogic Server.
> > > > >
> > > > > -Danny
> > > > >
> > > > > -----Original Message-----
> > > > > From: [email protected]
> > > > > [mailto:[email protected]] On
> > > Behalf Of Tjerk
> > > > > Stroband
> > > > > Sent: Monday, February 09, 2009 6:31 AM
> > > > > To: [email protected]
> > > > > Subject: [MarkLogic Dev General] Atomic and exclusive lock on
> > > > > documents(accross a cluster)
> > > > >
> > > > > Hi all,
> > > > >
> > > > > During development of a particular application a number of
> > > > > architectural design decisions have been made. One of these
> > > > decisions
> > > > > was to not used the internal ML user-database since that
> > > > would require
> > > > > the client application to do user-impersonation etc
> and require
> > > > > complex synchronization procedures with the LDAP (which
> > > is used to
> > > > > store all the users). Instead the application will
> use a single
> > > > > application user-account to access ML.
> > > > >
> > > > > However, the application requires support for exclusively and
> > > > > atomically (across the cluster) locking documents
> > > > (regardless of the
> > > > > user since only one user accesses ML) as part of a pessimistic
> > > > > locking-scheme for use by an external workflow.
> > > > > The logical way to achieve this would be to use the
> > > > xdmp:lock-acquire
> > > > > and xdmp:lock-release functions. However, these functions
> > > > only lock a
> > > > > document for a particular user.
> > > > > Two consecutive calls to the xdmp:lock-acquire function
> > > > using the same
> > > > > user will not cause the second call to fail, regardless of the
> > > > > description.
> > > > > Instead they both succeed and the description will have
> > > that of the
> > > > > last call.
> > > > >
> > > > > This poses a significant challenge as it seems it is
> > > > impossible to get
> > > > > an exclusive and atomic lock on a document when not using
> > > > named users
> > > > > in ML.
> > > > >
> > > > > Is our analysis correct or are we missing something?
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Tjerk Stroband
> > > > > Daidalos
> > > > > The Netherlands
> > > > >
> > > > >
> > > > >
> > > > > T. I. Stroband
> > > > > Senior consultant
> > > > > GSM: +31 (0)6 22 52 44 45
> > > > >
> > > > >
> > > > > Daidalos BV
> > > > > Source of Innovation
> > > > > Hoekeindsehof 1-4
> > > > > 2665 JZ Bleiswijk
> > > > > Tel.: +31 (0) 10 850 1200
> > > > > Fax: +31 (0) 10 850 1199
> > > > > http://www.daidalos.nl/
> > > > > KvK 27164984
> > > > > De informatie - verzonden in of met dit emailbericht - is
> > > afkomstig
> > > > > van Daidalos BV en is uitsluitend bestemd voor de
> geadresseerde.
> > > > > Indien u dit bericht onbedoeld hebt ontvangen, verzoeken
> > > > wij u het te
> > > > > verwijderen. Aan dit bericht kunnen geen rechten worden
> > ontleend.
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > General mailing list
> > > > > [email protected]
> > > > > http://xqzone.com/mailman/listinfo/general
> > > > > _______________________________________________
> > > > > General mailing list
> > > > > [email protected]
> > > > > http://xqzone.com/mailman/listinfo/general
> > > > > _______________________________________________
> > > > General mailing list
> > > > [email protected]
> > > > http://xqzone.com/mailman/listinfo/general
> > > > _______________________________________________
> > > > General mailing list
> > > > [email protected]
> > > > http://xqzone.com/mailman/listinfo/general
> > > > _______________________________________________
> > > General mailing list
> > > [email protected]
> > > http://xqzone.com/mailman/listinfo/general
> > > _______________________________________________
> > > General mailing list
> > > [email protected]
> > > http://xqzone.com/mailman/listinfo/general
> > > _______________________________________________
> > General mailing list
> > [email protected]
> > http://xqzone.com/mailman/listinfo/general
> > _______________________________________________
> > General mailing list
> > [email protected]
> > http://xqzone.com/mailman/listinfo/general
> >
> _______________________________________________
> General mailing list
> [email protected]
> http://xqzone.com/mailman/listinfo/general
> _______________________________________________
> General mailing list
> [email protected]
> http://xqzone.com/mailman/listinfo/general
>
_______________________________________________
General mailing list
[email protected]
http://xqzone.com/mailman/listinfo/general
