Fragmentation is not a per-database constant so you do not waste any more or less space by putting a small document in one database with big documents or another with only small documents.
-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of helen chen Sent: Monday, November 08, 2010 3:14 PM To: Jason Booth Cc: General Mark Logic Developer Discussion; helen chen Subject: Re: [MarkLogic Dev General] question about logging Hi Jason, I tested it, it seems very fast for the logging part, and very easy to search. I think I can use it. I don't have much understanding for fragment, maybe I'm asking some silly questions here, but I want to ask before I use it : My impression for marklogic is: I put articles' xml into database, this article's xml contains almost everything for this article. Depending on each article, the xml file can be very big, some are relatively small, but much bigger than this tiny log document. And because it is hard to say that these elements will fit in one fragment or not, we didn't do fragmentation for article. Now I have these tiny documents in the same database, that is pretty much to say: my xml can be very big and also can be tiny, each of these tiny document will use one fragment (I think this way, let me know if I'm wrong). Then do these tiny document waste the fragment? Should I put these log document in a separate database ? Or maybe I totally mixed the concept for fragmentation? Thanks, Helen On Nov 5, 2010, at 9:50 PM, Jason Booth wrote: > > Hi Helen, > > As far as an xdmp:invoke I would highly recommend it (instead of xdmp:eval) as it will prove beneficial for the performance gains with module caching. > > My suggestion for creating a document for each step will benefit you in a few ways. First, you won't need to keep updating the same document with more steps - unnecessary overhead. Second, if you kept updating a single document with many steps you could possibly get into a situation where you have an extremely large document (say, 1GB or more) and you certainly don't want that. Lastly, when you have many small documents you will have a more efficient search - many step documents vs a single document with many steps works best for a search on your data. Placing your smaller step documents into a collection allows you to organize them in a way that you can search against them (e.g. perform a search on the "step-123" collection giving me back all steps, in date-time order, descending). Think of it like this, if you were using an RDBMS your collection would be a table and the documents would be rows in a table. > > I recommend you experiment in with the above approach in a sandbox database to see what I mean. Then, once you feel comfortable with it move your code into your application. > > Best Regards, > > Jason > > > ________________________________________ > From: [email protected] [[email protected]] On Behalf Of Helen Chen [[email protected]] > Sent: Friday, November 05, 2010 8:34 PM > To: [email protected] > Cc: Helen Chen > Subject: Re: [MarkLogic Dev General] question about logging > > Hi Jason, > > The invoke() needs to use main module, I prefer to not use main module so I can > call functions. > > And if each step is a document, I feel that will create too many documents, and > these documents will be tiny. I prefer one batch as a document, that's kind of > say I need to do a lot of update to the document. Not sure if it sounds good > using eval. > > Thanks, Helen > > >>>> Jason Booth <[email protected]> 11/05/10 4:39 PM >>> > > I would use xdmp:invoke() vs xdmp:eval() for better performance, esp. w/heavy > use - you take advantage of caching. > > To follow-up with what Walter said, you could add each document to a collection > when you create them. Each "step" could be a document, and a series of steps > could belong to the same collection, say based off an article id. And, on top of > that, they could belong to a general collection as a bigger umbrella to search > on all "steps". Below is some psuedo-code: > > let $article-id := "123" > let $step := > <step> > <step-datetime>{fn:current-dateTime()}</step-datetime> > <article-id>{$article-id}</article-id> > <action>Published article</action> > </step> > > return xdmp:document-insert( > fn:concat("/steps/", xdmp:hash64(xdmp:quote($step))), $step, (), > (fn:concat("step-", $article-id), "steps")) > > > ________________________________________ > From: [email protected] > [[email protected]] On Behalf Of helen chen > [[email protected]] > Sent: Friday, November 05, 2010 4:36 PM > To: General Mark Logic Developer Discussion > Subject: Re: [MarkLogic Dev General] question about logging > > Hi Wunder, > > This sounds reasonable to me and might work for me. > > we have two database, one for original format, and one for working format. > During publishing process, I need to move data around, and I might also need to > insert some data back to both working version and original version, then I want > to verify the data after modification, that means I need to use xdmp:eval() for > transaction. And depending on which database I'm in or to avoid lock, if I want > to write the log message to a document, I think I also need to use xdmp:eval(). > > I tested xdmp:eval() with some document change, it seems fast. I want to ask: > should xdmp:eval() be used whenever I need to guarantee the data change and > whenever I want get the latest data information? If I use a lot is it going to > have impact? > > Thanks, Helen > > > > On Nov 5, 2010, at 4:12 PM, Walter Underwood wrote: > >> You could log that information to a document in MarkLogic. One history > document for each document that is processed. --wunder >> >> On Nov 5, 2010, at 12:46 PM, helen chen wrote: >> >>> Hi David, >>> >>> What happened here is: when the article get published, I have to do something > and move data around. I want to log the important steps and information to a > separate file, and this file will be kept as a reference in case something > wrong. This publishing can involve a few hundred articles at one time, that > means the message I'm going to create can be a lot at some point. If the > message is not much, I think web service can do it, but when I have a lot, and > those messages are from each steps, not at one time, it means I have to call web > service many many times, I'm not sure the impact of this to the program. It > sounds to me that it is going to slow down a lot. But I never tried this way. >>> >>> In what situation did you use this way? >>> >>> Thanks, >>> Helen >>> >>> On Nov 5, 2010, at 3:23 PM, Lee, David wrote: >>> >>>> Another alternative is you could log to a web service. >>>> That requires some substantial infrastructure but it may be worth it >>>> depending on what your doing. >>>> Technically its quite easy to do, but it adds one more big piece to the >>>> puzzle, and depending on how frequent your logs are may slow things down >>>> a lot. >>>> >>>> >>>> ---------------------------------------- >>>> David A. Lee >>>> Senior Principal Software Engineer >>>> Epocrates, Inc. >>>> [email protected] >>>> 812-482-5224 >>>> >>>> >>>> >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: [email protected] >>>> [mailto:[email protected]] On Behalf Of helen chen >>>> Sent: Friday, November 05, 2010 3:18 PM >>>> To: General Mark Logic Developer Discussion >>>> Subject: Re: [MarkLogic Dev General] question about logging >>>> >>>> Hi Wunder, >>>> >>>> Thanks for this information. I have to study it and see if I can make it >>>> work. >>>> >>>> Helen >>>> >>>> >>>> On Nov 5, 2010, at 2:58 PM, Walter Underwood wrote: >>>> >>>>> Using the admin UI, navigate to the group your hosts are in. On that >>>> page, near the bottom, there is an choice for "system log level". That >>>> chooses what level of log events will be sent to the system log. On >>>> Unix, that is syslog. There is a separate choice for "file log level", >>>> which controls what is logged in ErrorLog.txt. >>>>> >>>>> syslog_ng can use patterns to match log messages and route them to a >>>> particular log. >>>>> >>>>> wunder >>>>> >>>>> On Nov 5, 2010, at 11:44 AM, helen chen wrote: >>>>> >>>>>> Hi Walter, >>>>>> >>>>>> When you say "configure MarkLogic so the system log level includes >>>> your extra log messages", where can I do the configuration? Does that >>>> mean the log that write to ErrorLog.txt will also be written to syslog? >>>> is there any special format that I need to do for the message that I >>>> want to log? If it is in some document, can you point me which document >>>> I can find it? >>>>>> >>>>>> Thanks, Helen >>>>>> >>>>>> >>>>>> On Nov 5, 2010, at 1:37 PM, Walter Underwood wrote: >>>>>> >>>>>>> If you are on a system that uses syslog_ng, you can do this with >>>> that tool. >>>>>>> >>>>>>> Log messages normally, but configure MarkLogic so the system log >>>> level includes your extra log messages. Configure syslog_ng to route >>>> those log messages to the file you want. >>>>>>> >>>>>>> wunder >>>>>>> == >>>>>>> Walter Underwood >>>>>>> [email protected] >>>>>>> >>>>>>> On Nov 5, 2010, at 8:33 AM, helen chen wrote: >>>>>>> >>>>>>>> Maybe I didn't say it clearly. >>>>>>>> >>>>>>>> fn:concat() is for the message part. I also want to write this >>>> message to a separate file on the file system, the file name is >>>> specified dynamically. And if this file already exists on file system, >>>> it should be the append , not overwrite. It is similar to the unix >>>> script that I write my log to some file I want. >>>>>>>> >>>>>>>> In the meantime I don't want to stop the xdmp:log(), if I use >>>> xdmp:log, it should still write to ErrorLog.txt file. >>>>>>>> >>>>>>>> Thanks, Helen >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Nov 5, 2010, at 11:19 AM, Tim Meagher wrote: >>>>>>>> >>>>>>>>> I just embed fn:concat() within the call the xdmp:log() and >>>> concatenation >>>>>>>>> the various message parts, e.g. >>>>>>>>> >>>>>>>>> xdmp:log(concat("Path: ", {$path})) >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: [email protected] >>>>>>>>> [mailto:[email protected]] On Behalf Of >>>> helen chen >>>>>>>>> Sent: Friday, November 05, 2010 11:16 AM >>>>>>>>> To: General Mark Logic Developer Discussion >>>>>>>>> Subject: [MarkLogic Dev General] question about logging >>>>>>>>> >>>>>>>>> Hello there, >>>>>>>>> >>>>>>>>> In Marklogic, I use xdmp:log() to log message to ErrorLog.txt >>>> file. I want >>>>>>>>> to do some logging similar to script, like I specify the path and >>>> file name, >>>>>>>>> then I write just the message I want to this file and then keep >>>> appending >>>>>>>>> message to this file. I expect that this should not stop the >>>> normal logging >>>>>>>>> of xdmp:log(). >>>>>>>>> >>>>>>>>> Does anyone have suggestion on how to do it? >>>>>>>>> >>>>>>>>> Thanks, Helen >>>>>>>>> _______________________________________________ >>>>>>>>> General mailing list >>>>>>>>> [email protected] >>>>>>>>> http://developer.marklogic.com/mailman/listinfo/general >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> General mailing list >>>>>>>>> [email protected] >>>>>>>>> http://developer.marklogic.com/mailman/listinfo/general >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> General mailing list >>>>>>>> [email protected] >>>>>>>> http://developer.marklogic.com/mailman/listinfo/general >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> General mailing list >>>>>>> [email protected] >>>>>>> http://developer.marklogic.com/mailman/listinfo/general >>>>>> >>>>>> _______________________________________________ >>>>>> General mailing list >>>>>> [email protected] >>>>>> http://developer.marklogic.com/mailman/listinfo/general >>>>> >>>>> -- >>>>> Walter Underwood >>>>> [email protected] >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> General mailing list >>>>> [email protected] >>>>> http://developer.marklogic.com/mailman/listinfo/general >>>> >>>> _______________________________________________ >>>> General mailing list >>>> [email protected] >>>> http://developer.marklogic.com/mailman/listinfo/general >>>> _______________________________________________ >>>> General mailing list >>>> [email protected] >>>> http://developer.marklogic.com/mailman/listinfo/general >>> >>> _______________________________________________ >>> General mailing list >>> [email protected] >>> http://developer.marklogic.com/mailman/listinfo/general >> >> -- >> Walter Underwood >> [email protected] >> >> >> >> _______________________________________________ >> General mailing list >> [email protected] >> http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
