I like the approach of a module that does the validation rather than a
pre-commit trigger. The overhead of the trigger seems unnecessary. Couldn't you
model your logic so that only those documents that pass your conditions are
loaded into the database, rather than rolling back a transaction if the
conditions are not met?
For example, you could have a module that goes something like the following:
xquery version "1.0-ml";
declare $candidate-document as node() external;
let $test1 := $candidate-document/meta/pub-date castable as xs:dateTime
let $test2 := $candidate-document/meta/category = ("A","B","C","D")
return
if($test1 and $test2)
then
(xdmp:document-insert($candidate-document/meta/uri/text(),$candidate-document),"document
inserted")
else "document failed"
In this case the overhead for each document is parsing the XML and evaluating
the conditions. Only those documents that satisfy the conditions are inserted
into the database.
Kelly
Message: 1
Date: Fri, 12 Aug 2011 14:31:05 -0700
From: Danny Sokolsky <danny.sokol...@marklogic.com>
Subject: Re: [MarkLogic Dev General] How to validate a document before
inserting in the DB
To: General MarkLogic Developer Discussion
<general@developer.marklogic.com>
Message-ID:
<c9924d15b04672479b089f7d55ffc1320290948...@exchg-be.marklogic.com>
Content-Type: text/plain; charset="us-ascii"
Hi Manoj,
Yes, much of the work to create them is done because the indexes for any
updates in MarkLogic are created as part of each transaction. But in the case
that an exception is thrown, no new fragments will be created for that
transaction.
I recommend testing it with your content to see how long it takes for the case
where you do not do the updates. It might be that it is all fast enough that
it does not matter.
-Danny
From: general-boun...@developer.marklogic.com
[mailto:general-boun...@developer.marklogic.com] On Behalf Of Manoj
Sent: Friday, August 12, 2011 2:24 PM
To: General MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] How to validate a document before
inserting in the DB
Thanks Danny. What I want to understand is that whether indexes or fragments
would be created even before the trigger fires. If I understood correctly,
indexes and fragments would have been created before the trigger fires. Is my
understanding correct?
Thanks & Regards
Manoj
On Fri, Aug 12, 2011 at 4:17 PM, Danny Sokolsky
<danny.sokol...@marklogic.com<mailto:danny.sokol...@marklogic.com>> wrote:
You can think of a pre-commit trigger like a multi-statement transaction. The
flow goes something like this:
* The update occurs and it is prepared for committing.
* The pre-commit trigger is fired.
* The action module that the pre-commit trigger invokes sees the
changed (but not yet committed to the rest of the world's) view of the update.
* After the pre-commit trigger module finishes, the transaction is
committed.
* If an exception is thrown anywhere during this process, then the
transaction rolls back.
So there is a bunch of work that is done to prepare the transaction before
running the pre-commit trigger. The impact in terms of system usage is similar
to any transaction that does similar work. How much work it takes depends a
lot on the details of what the transaction is doing.
-Danny
From:
general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>
[mailto:general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>]
On Behalf Of Manoj
Sent: Friday, August 12, 2011 8:30 AM
To: General MarkLogic Developer Discussion
Subject: Re: [MarkLogic Dev General] How to validate a document before
inserting in the DB
Hi Danny,
Thanks for the pointer.Yes that's what I had in mind. I have few question
regarding to the pre-commit trigger. In our case out of 100 documents that are
coming to the DB for loading say around 10% would pass the validation and go on
to get saved in the DB rest all will fail. Based on this I have the following
questions
1) Before the pre-commit triggers are invoked, where will the documents be
held? will the document be in the memory or on the disk?
2) Will ML create index for the documents before the pre-commit triggers is
invoked and completed? because if the document is going to fail the validation,
creating indexes will be unnecessary as the documents will not be saved to the
DB.
3) What is the impact of using pre-commit triggers in terms of memory usage,
performance etc in general and with respect to what we are trying.
We also thought about having the logic in the module that loads the document,
but since that module is generic one including this will not go very well with
it.
Thanks & Regards
Manoj
On Wed, Aug 10, 2011 at 5:39 PM, Danny Sokolsky
<danny.sokol...@marklogic.com<mailto:danny.sokol...@marklogic.com>> wrote:
Hi Manoj,
It is possible to create a pre-commit trigger to do this. That trigger could,
for example, check your pre-conditions and, if they are not satisfied, throw an
error, which would cause the transaction to fail.
Is that what you had in mind?
You could also put that logic in your module that loads the document and avoid
using a pre-commit trigger.
-Danny
From:
general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>
[mailto:general-boun...@developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>]
On Behalf Of Manoj
Sent: Wednesday, August 10, 2011 1:59 PM
To: General MarkLogic Developer Discussion
Subject: [MarkLogic Dev General] How to validate a document before inserting in
the DB
Hi All,
We have a requirement where in a common service will keep loading documents
into 2 ML DB through xcc. We need to allow only some of the documents to be
loaded in the 2nd DB based on certain conditions. Is it possible to check for
these condition when the service is trying to load the document into the DB. I
mean something like before-insert trigger or some other approach using which we
can stop the document from loading if it fails certain condition. The condition
are not static.
Thanks & Regards
Manoj
_______________________________________________
General mailing list
General@developer.marklogic.com
http://developer.marklogic.com/mailman/listinfo/general
