Manoj, To your question about being held in memory - yes, MarkLogic uses Multi-version Concurrency Control which writes data as "append only" modifications to the in-memory stand, which is then written to disk once full, so all writes are to memory and only get written to disk if they exist when the stand hits a size threshold. Stands are the smaller data structures inside Forests that are merged together and hold all the data+indexes.
Note journaling writes to disk immediately to ensure commits survive a system failure. But if you have a dynamic condition in your trigger with code to check the document and throw an exception, can't you invoke that same code in the generic ingest module? I'm interested to know how it works out if you profile both approaches. Yours, Damon From: [email protected] [mailto:[email protected]] On Behalf Of Manoj Sent: Friday, August 12, 2011 5:24 PM To: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] How to validate a document before inserting in the DB Thanks Danny. What I want to understand is that whether indexes or fragments would be created even before the trigger fires. If I understood correctly, indexes and fragments would have been created before the trigger fires. Is my understanding correct? Thanks & Regards Manoj On Fri, Aug 12, 2011 at 4:17 PM, Danny Sokolsky <[email protected]<mailto:[email protected]>> wrote: You can think of a pre-commit trigger like a multi-statement transaction. The flow goes something like this: * The update occurs and it is prepared for committing. * The pre-commit trigger is fired. * The action module that the pre-commit trigger invokes sees the changed (but not yet committed to the rest of the world's) view of the update. * After the pre-commit trigger module finishes, the transaction is committed. * If an exception is thrown anywhere during this process, then the transaction rolls back. So there is a bunch of work that is done to prepare the transaction before running the pre-commit trigger. The impact in terms of system usage is similar to any transaction that does similar work. How much work it takes depends a lot on the details of what the transaction is doing. -Danny From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Manoj Sent: Friday, August 12, 2011 8:30 AM To: General MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] How to validate a document before inserting in the DB Hi Danny, Thanks for the pointer.Yes that's what I had in mind. I have few question regarding to the pre-commit trigger. In our case out of 100 documents that are coming to the DB for loading say around 10% would pass the validation and go on to get saved in the DB rest all will fail. Based on this I have the following questions 1) Before the pre-commit triggers are invoked, where will the documents be held? will the document be in the memory or on the disk? 2) Will ML create index for the documents before the pre-commit triggers is invoked and completed? because if the document is going to fail the validation, creating indexes will be unnecessary as the documents will not be saved to the DB. 3) What is the impact of using pre-commit triggers in terms of memory usage, performance etc in general and with respect to what we are trying. We also thought about having the logic in the module that loads the document, but since that module is generic one including this will not go very well with it. Thanks & Regards Manoj On Wed, Aug 10, 2011 at 5:39 PM, Danny Sokolsky <[email protected]<mailto:[email protected]>> wrote: Hi Manoj, It is possible to create a pre-commit trigger to do this. That trigger could, for example, check your pre-conditions and, if they are not satisfied, throw an error, which would cause the transaction to fail. Is that what you had in mind? You could also put that logic in your module that loads the document and avoid using a pre-commit trigger. -Danny From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Manoj Sent: Wednesday, August 10, 2011 1:59 PM To: General MarkLogic Developer Discussion Subject: [MarkLogic Dev General] How to validate a document before inserting in the DB Hi All, We have a requirement where in a common service will keep loading documents into 2 ML DB through xcc. We need to allow only some of the documents to be loaded in the 2nd DB based on certain conditions. Is it possible to check for these condition when the service is trying to load the document into the DB. I mean something like before-insert trigger or some other approach using which we can stop the document from loading if it fails certain condition. The condition are not static. Thanks & Regards Manoj _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
