Hi Justin
I'm simply copying some of the values from first-collection into second
colleciton on finding the values in 2nd collection based on search criteria.
Here is my entire code for one record. If there are multiple records in
first-collection, then this conflict is occuring.
---- Code
declareUpdate();
var json = "/MarkLogic/json/json.xqy";
var cpf = require("/MarkLogic/cpf/cpf.xqy");
var dls = require("/MarkLogic/dls.xqy");
var jsearch = require("/MarkLogic/jsearch.sjs");
var transition;
//var doc = cts.doc(uri).toObject();
//var id = doc.personId;
var id = "1234";
var rawColls =
require("/MarkLogic/jsearch.sjs").collections(["first-collection"]);
var matColl =
require("/MarkLogic/jsearch.sjs").collections(['second-collection']);
var rawJson;
var count=0;
var matViewUri =null;
var rawUri = null;
// Reading from RAW Sources.
var tempObj =
rawColls.documents()
.where(rawColls.byExample({personId: id}))
.result();
var length = 0;
var rawResponse =
rawColls.documents()
.where(rawColls.byExample({personId: id}))
.result('iterator');
if ( tempObj.estimate != 0 ) {
rawUri = tempObj.results.uri;
var materialView = matColl.documents()
.where(matColl.byExample({personId: id})).result();
if (materialView.results!=null) { // If records found in the second
collection View.
matViewUri = materialView.results[0].uri;
for (var result of rawResponse.results) {
var newObj = {};
var matOriginalDoc = cts.doc(matViewUri);
var dateNow =new Date().toUTCString();
// if (doc.firstname !=null)
newObj.firstName = result.document.firstname;
// if (doc.lastName !=null)
newObj.lastName = result.document.lastname;
// if (doc.dateOfBirth !=null)
newObj.dateOfBirth = result.document.dob;
// if (doc.city !=null)
newObj.city = result.document.city;
newObj.modifiedDate = new Date().toUTCString();
count++;
xdmp:eval('xdmp.nodeReplace(matOriginalDoc,newObj)',newObj,{
"isolation" : "different-transaction",
"preventDeadlocks" : true
});
}
}
else
{
// create new document in 2nd collection for that PersonId;
xdmp.log(">>>> Create a new document");
}
}
else {
xdmp.log(">>>> ERROR: NO RECORDS FOUND IN first collection");
}
On Mon, Oct 3, 2016 at 3:28 PM, Justin Makeig <justin.mak...@marklogic.com>
wrote:
> I still don't understand what your code is trying to do. Can you please
> describe in words what you're trying to accomplish? Where does `matViewUri`
> come from? How does it relate to `{personId: id}`?
>
> Justin
>
>
> > On Oct 3, 2016, at 7:08 AM, Shiv Shankar <shiv.shivshan...@gmail.com>
> wrote:
> >
> > Hi Justin,
> > Here is the actual code that returns XDMP-CONFLICTINGUPDATES Error:.
> >
> > ---
> > declareUpdate();
> > var mainDataResponse =
> > rawColls.documents()
> > .where(rawColls.byExample({personId: id}))
> > .result('iterator');
> >
> > for (var result of mainDataResponse.results) {
> >
> > var normalizedViewObj = cts.doc(matViewUri).toObject();
> >
> > var matOriginalDoc = cts.doc(matViewUri);
> >
> > var dateNow =new Date().toUTCString();
> >
> > // if (doc.firstname !=null)
> > normalizedViewObj.firstName = result.document.firstname;
> >
> > // if (doc.lastName !=null)
> > normalizedViewObj.lastName = result.document.lastname;
> > // if (doc.dateOfBirth !=null)
> > normalizedViewObj.dateOfBirth = result.document.dob;
> >
> > xdmp.nodeReplace(cts.doc(matViewUri),normalizedViewObj),
> > normalizedViewObj = null;
> >
> > }
> >
> > ---
> >
> > On Thu, Sep 29, 2016 at 3:00 PM, <general-request@developer.
> marklogic.com> wrote:
> > Send General mailing list submissions to
> > general@developer.marklogic.com
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> > http://developer.marklogic.com/mailman/listinfo/general
> > or, via email, send a message with subject or body 'help' to
> > general-requ...@developer.marklogic.com
> >
> > You can reach the person managing the list at
> > general-ow...@developer.marklogic.com
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of General digest..."
> >
> >
> > Today's Topics:
> >
> > 1. Re: General Digest, Vol 147, Issue 51 (Justin Makeig)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Thu, 29 Sep 2016 16:32:44 +0000
> > From: Justin Makeig <justin.mak...@marklogic.com>
> > Subject: Re: [MarkLogic Dev General] General Digest, Vol 147, Issue 51
> > To: Shiv Shankar <shiv.shivshan...@gmail.com>
> > Cc: "general@developer.marklogic.com"
> > <general@developer.marklogic.com>
> > Message-ID: <22fded82-fc67-4aad-b1f9-2eecb5b6a...@marklogic.com>
> > Content-Type: text/plain; charset="us-ascii"
> >
> > Can you show your actual code?
> >
> > Justin
> >
> >
> > --
> > Justin Makeig
> > Director, Product Management
> > MarkLogic
> > jmak...@marklogic.com
> >
> >
> > > On Sep 29, 2016, at 7:09 AM, Shiv Shankar <shiv.shivshan...@gmail.com>
> wrote:
> > >
> > > HI Justin,
> > > Thanks for your quick reply. However, I used both
> declareUpdate({explicitCommit: true}); and declareUpdate(); Still same
> issue.
> > >
> > > I am just reading from one collection and updating into another
> document in a different collection. I managed with insertDocument, but this
> over writes the docs which I don't want. I just want to update the
> attributes ( it is like UPDATE SET field1:=value, field2=value in loop in
> oracle world)
> > >
> > > Note: I used lock/unlock options, but getting no lock on that target
> record. I used isolation level as part of eval. Couldn't get through it.
> > > XDMP-CONFLICTINGUPDATES: xdmp.eval("//{explicitCommit:
> true}\ndeclareUpdate(); \nvar json = \"/Mark...", {}, <options
> xmlns="xdmp:eval"><database>67298589469423423423</database><root>/</root><isolation...</options>)
> --Conflicting updates xdmp.nodeReplace(cts.doc("
> > >
> > >
> > > Any thoughts?
> > >
> > > On Thu, Sep 29, 2016 at 7:55 AM, <general-request@developer.
> marklogic.com> wrote:
> > > Send General mailing list submissions to
> > > general@developer.marklogic.com
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > > http://developer.marklogic.com/mailman/listinfo/general
> > > or, via email, send a message with subject or body 'help' to
> > > general-requ...@developer.marklogic.com
> > >
> > > You can reach the person managing the list at
> > > general-ow...@developer.marklogic.com
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of General digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > > 1. Re: XDMP-CONFLICTINGUPDATES: while using
> xdmp.nodeReplace
> > > (Justin Makeig)
> > > 2. Re: mlcp Transaction Errors - SVC-EXTIME and XDMP-NOTXN #CGO#
> > > (Jain, Abhishek)
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Wed, 28 Sep 2016 21:02:26 +0000
> > > From: Justin Makeig <justin.mak...@marklogic.com>
> > > Subject: Re: [MarkLogic Dev General] XDMP-CONFLICTINGUPDATES: while
> > > using xdmp.nodeReplace
> > > To: MarkLogic Developer Discussion <general@developer.marklogic.com>
> > > Message-ID: <4136b8ca-113a-4373-b6d3-e54d34f15...@marklogic.com>
> > > Content-Type: text/plain; charset="us-ascii"
> > >
> > > The general pattern for doing updates on persisted JSON documents in
> JavaScript is to
> > > 1) get the document
> > > 2) turn it into an object
> > > 3) update the object
> > > 4) replace the document node with the updated object
> > >
> > > Here's an example that updates the `balance` property of every
> document in the `accounts` collection:
> > >
> > > declareUpdate();
> > > for(var doc of fn.collection('accounts')) {
> > > var account = doc.toObject();
> > > account.balance = account.balance * 1.05;
> > > xdmp.nodeReplace(doc, account);
> > > }
> > >
> > > Note that you also need a declareUpdate(). The transaction is
> committed (or rolledback) automatically, so you don't need the
> xdmp.commit().
> > >
> > > Justin
> > >
> > > --
> > > Justin Makeig
> > > Director, Product Management
> > > MarkLogic
> > > jmak...@marklogic.com
> > >
> > > > On Sep 28, 2016, at 1:46 PM, Shiv Shankar <
> shiv.shivshan...@gmail.com> wrote:
> > > >
> > > > Hi Rob,
> > > > Thanks for quick reply,
> > > >
> > > > Here is the sample scenario.
> > > >
> > > > I would be building random data and put in to newObjData in a for
> loop.
> > > >
> > > > 01.json = {"firstName": "abc", "lastName": "xyz"}
> > > >
> > > > newObjectData.firstName= "Donald";
> > > >
> > > > for Loop
> > > > {
> > > > xdmp.nodeReplace(cts.doc("/test/01.json"),newObjectData);
> > > > xdmp.commit();
> > > > }
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Sep 28, 2016 at 4:37 PM, Rob Szkutak <
> rob.szku...@marklogic.com> wrote:
> > > > Hi,
> > > >
> > > > Do you have a reproducible case of this I could look at?
> > > >
> > > > Best,
> > > > Rob
> > > >
> > > > Rob Szkutak
> > > > Senior Consultant
> > > > MarkLogic Corporation
> > > > rob.szku...@marklogic.com
> > > > www.marklogic.com
> > > >
> > > > From: Shiv Shankar [shiv.shivshan...@gmail.com]
> > > > Sent: Wednesday, September 28, 2016 3:35 PM
> > > > To: general@developer.marklogic.com; Rob Szkutak
> > > > Subject: XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace
> > > >
> > > > Hi Rob,
> > > >
> > > > I am getting XDMP-CONFLICTINGUPDATES: while using xdmp.nodeReplace
> to update JSON attributes in FOR loop, instead of over-writing entire
> document. Any advice.
> > > >
> > > > BTW, I used xdmp.eval, but it is not identifying the java script
> variables.
> > > >
> > > > Thanks
> > > > ShivShankar.
> > > >
> > > > _______________________________________________
> > > > General mailing list
> > > > General@developer.marklogic.com
> > > > Manage your subscription at:
> > > > http://developer.marklogic.com/mailman/listinfo/general
> > >
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > Message: 2
> > > Date: Thu, 29 Sep 2016 11:54:54 +0000
> > > From: "Jain, Abhishek" <abhishek.b.j...@capgemini.com>
> > > Subject: Re: [MarkLogic Dev General] mlcp Transaction Errors -
> > > SVC-EXTIME and XDMP-NOTXN #CGO#
> > > To: MarkLogic Developer Discussion <general@developer.marklogic.com>
> > > Message-ID:
> > > <5A37843F322E824AA0D8359ED88CD01F276FF29D@IN-HYD-DAGND2.
> corp.capgemini.com>
> > >
> > > Content-Type: text/plain; charset="utf-8"
> > >
> > > Great , Expected !! As it was pretty optimized MLCP Query.
> > > BTW, addressing the second part of your question about logging
> errors/failed files ?
> > > I?m also working on similar stuff, this can be done in three ways ?
> > >
> > >
> > > 1. Modifying MLCP library to write all the failed IDs and
> corresponding files in a separate log file
> > >
> > > However that requires additional work(java program or x-query )
> program to create new input files of failed record.(Tedious)
> > >
> > > 2. Second way, we need to write a pre-commit trigger, and attach
> to database. In pre-commit action module we can
> > >
> > > Write custom logs.
> > >
> > > 3. Similar module can also be written for MLCP Transform.
> > >
> > >
> > >
> > > All three approaches have limitations , which one to choose depends on
> what exactly we want to log.
> > >
> > > In my case it pre-commit solution does well.
> > >
> > > I wonder MLCP doesn?t provide RE-TRY option.
> > >
> > > Thanks and Regards,
> > > [Email_CBE.gif]Abhishek Jain
> > > Associate Consultant
> > >
> > > From: general-boun...@developer.marklogic.com [mailto:general-bounces@
> developer.marklogic.com] On Behalf Of Stuart Myles
> > > Sent: Friday, September 23, 2016 8:45 PM
> > > To: MarkLogic Developer Discussion
> > > Subject: Re: [MarkLogic Dev General] mlcp Transaction Errors -
> SVC-EXTIME and XDMP-NOTXN #CGO#
> > >
> > > Thanks! This helped me prevent the errors from occurring and - as a
> bonus - significantly sped up my ingestion.
> > >
> > > I couldn't use exactly the mlcp command line you suggested, since - in
> the version of mlcp I'm using - -input_file_type xml isn't allowed, I had
> to use -input_file_type documents instead. Also, my input files don't need
> to be split. However, bumping up the threads used (to 30 in my case) made
> the transaction / timeout complaints go away. And now I'm ingesting 100,000
> documents in 12 minutes, rather than one hour. Much better!
> > >
> > > Regards,
> > >
> > > Stuart
> > >
> > >
> > >
> > > On Fri, Sep 23, 2016 at 3:34 AM, Jain, Abhishek <
> abhishek.b.j...@capgemini.com<mailto:abhishek.b.j...@capgemini.com>>
> wrote:
> > > Hi Stuart,
> > >
> > > MLCP comes with various options, and can be used in various
> combinations depending on the file size, memory available and
> > > Other number of nodes, forest etc.
> > >
> > > If you want to try a quick solution you can try this mlcp command :
> > > mlcp import -host yourhost -port 8000 -username userName -password
> PASSWORD -input_file_type xml -input_file_path TempData -thread_count
> -thread_count_per_split 3 -batch_size 200 -transaction_size 20
> -max_split_size 33554432 -split_input true
> > > change username, input file type etc accordingly.
> > > It?s always good to use splits and threads when working with huge
> dataset.
> > > Some performance matrix you can consider while using above mlcp :
> > >
> > > 1. In app server settings you can check if connection time out
> is set to 0.
> > >
> > > 2. Default spilt size is 32MB, if you can change -max_split_size
> 33554432 ( it take in bytes, if your file is bigger )
> > >
> > > 3. Make sure split and thread ratio remains 1:2 or 1:3 for
> example
> > > If your document size is 10 MB, and your split size is 1000,000 (1 MB)
> then 10/1 = 10 splits
> > > Then you should create 20 or 30 thread for best CPU utilization.
> > >
> > > 4. The above mlcp does well with 150 Million rows, should work
> for you as well.
> > >
> > > 5. I assume you have a nice good RAM > 4GB alteast.
> > >
> > > Thanks and Regards,
> > > [Email_CBE.gif]Abhishek Jain
> > > Associate Consultant
> > > Capgemini India | Hyderabad
> > >
> > > From: general-boun...@developer.marklogic.com<mailto:general-
> boun...@developer.marklogic.com> [mailto:general-bounces@
> developer.marklogic.com<mailto:general-boun...@developer.marklogic.com>]
> On Behalf Of Stuart Myles
> > > Sent: Thursday, September 22, 2016 11:52 PM
> > > To: MarkLogic Developer Discussion
> > > Subject: [MarkLogic Dev General] mlcp Transaction Errors - SVC-EXTIME
> and XDMP-NOTXN
> > >
> > > When I'm loading directories of slightly fewer than 100,000 XML files
> into a large MarkLogic instance, I often get timeout and transaction
> errors. If I re-run the same directory of files which got those errors, I
> typically don't get any errors.
> > >
> > > So, I have a few questions:
> > >
> > > * Can I get prevent the errors from happening in the first place -
> e.g. by tuning MarkLogic parameters or altering my use of mlcp?
> > > * If I do get errors, what is the best way to get a report on the
> files which failed, so I can retry just those ones? Is the best option for
> me to write some code to pick out the errors from the log file? And, if so,
> am I guaranteed to get all of the files reported?
> > >
> > > Some Details
> > >
> > > The command line template is
> > >
> > > mlcp.sh import -username {1} -password {2} -host localhost -port {4}
> -input_file_path {5} -output_uri_replace \"{6},'{7}'\"
> > >
> > > Sometimes, the imports run just fine. However, often I get a large
> number of SVC-EXTIME errors followed by a XDMP-NOTXN error. For example:
> > >
> > > 16/09/22 17:54:03 ERROR mapreduce.ContentWriter: SVC-EXTIME: Time
> limit exceeded
> > > 16/09/22 17:54:03 WARN mapreduce.ContentWriter: Failed document
> 029ccd8ac3323658277ca28fead7a73d.0.xml in file:/mnt/ingestion/
> MarkLogicIngestion/smyles/todo/2014_0005.done/
> 029ccd8ac3323658277ca28fead7a73d.0.xml
> > > 16/09/22 17:54:03 ERROR mapreduce.ContentWriter: SVC-EXTIME: Time
> limit exceeded
> > > 16/09/22 17:54:03 WARN mapreduce.ContentWriter: Failed document
> 02eb4562784255e249c4ec3ed472f9aa.1.xml in file:/mnt/ingestion/
> MarkLogicIngestion/smyles/todo/2014_0005.done/
> 02eb4562784255e249c4ec3ed472f9aa.1.xml
> > > 16/09/22 17:54:04 INFO contentpump.LocalJobRunner: completed 33%
> > > 16/09/22 17:54:21 ERROR mapreduce.ContentWriter: XDMP-NOTXN: No
> transaction with identifier 9076269665213828952
> > >
> > > So far, I'm just rerunning the entire directory again. Most of the
> time, it ingests fine on the second attempt. However, I have thousands of
> these directories to process. So, I would prefer to avoid getting the
> errors in the first place. Failing that, I would like to capture the errors
> and just retry the files which failed.
> > >
> > > Any help much appreciated.
> > >
> > > Regards,
> > >
> > > Stuart
> > >
> > >
> > >
> > > This message contains information that may be privileged or
> confidential and is the property of the Capgemini Group. It is intended
> only for the person to whom it is addressed. If you are not the intended
> recipient, you are not authorized to read, print, retain, copy,
> disseminate, distribute, or use this message or any part thereof. If you
> receive this message in error, please notify the sender immediately and
> delete all copies of this message.
> > >
> > > _______________________________________________
> > > General mailing list
> > > General@developer.marklogic.com<mailto:General@developer.marklogic.com
> >
> > > Manage your subscription at:
> > > http://developer.marklogic.com/mailman/listinfo/general
> > >
> > > -------------- next part --------------
> > > An HTML attachment was scrubbed...
> > > URL: http://developer.marklogic.com/pipermail/general/
> attachments/20160929/e5f311a1/attachment.html
> > > -------------- next part --------------
> > > A non-text attachment was scrubbed...
> > > Name: image001.gif
> > > Type: image/gif
> > > Size: 1616 bytes
> > > Desc: image001.gif
> > > Url : http://developer.marklogic.com/pipermail/general/
> attachments/20160929/e5f311a1/attachment.gif
> > >
> > > ------------------------------
> > >
> > > _______________________________________________
> > > General mailing list
> > > General@developer.marklogic.com
> > > Manage your subscription at:
> > > http://developer.marklogic.com/mailman/listinfo/general
> > >
> > >
> > > End of General Digest, Vol 147, Issue 51
> > > ****************************************
> > >
> >
> >
> >
> > ------------------------------
> >
> > _______________________________________________
> > General mailing list
> > General@developer.marklogic.com
> > Manage your subscription at:
> > http://developer.marklogic.com/mailman/listinfo/general
> >
> >
> > End of General Digest, Vol 147, Issue 53
> > ****************************************
> >
>
>
_______________________________________________
General mailing list
General@developer.marklogic.com
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
