Hi, I agree that it does not necessarily make sense to do this within MarkLogic. On the other hand, you might have a good reason. Particularly when there is really lots of information to analyse, and you have to store your information somewhere..
Have you considered taking an asynchronized approach? You can have one query gather all the uri's that need processing and store that somewhere, perhaps in batches. Then use Triggers or CPF to process those batches. MarkLogic is capable of handling those triggers in multiple threads, though if all uri's point to the same website, you perhaps don't want to overload it that way. Perhaps a small sleep would be appreciated by the website hoster.. You could also use scripts and programming languages to do stuff, but then it might be better to do that part outside MarkLogic all together, and only insert the log reports for analysis purposes.. Kind regards, Geert > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of > Lee, David > Sent: donderdag 25 juni 2009 13:50 > To: General Mark Logic Developer Discussion > Subject: RE: [MarkLogic Dev General] force a "buffer write"? > > I've done link-checking programs before and I suggest this > may be best done *outside* of ML. > What I would do if I were to do this > > 1) Use ML to generate an XML document with the info you need > (xml file with list nodes) > 2) Use a scripting language, or programming language that > supports multithreads or multiprocessors > 3) In batches of N threads/processes test the links and write > the results to non-conflicting output files > 4) wait for each batch to continue and aggregate the results > (maybe send this bit back to ML ?) > 5) Goto 3 until done > 6) Send the aggregated results back to ML > > > For the scripting or programming language for #2, there are > many options . > My personal bias, of course, is xmlsh which runs background > tasks as threads, but sh, perl, java , C++ or any language > that lets you do background processing of URL fetches will work. > One that will use threads instead of processes is desirable > but I've done this kind of thing with sh before and its > acceptable. You may want a URL fetch command that has a > settable timeout, something like wget, many urls' that are > inaccessible may take 30 seconds or more to time-out using > default timeouts. > OTOH if you use a language with effecient trheading you can > do large batches (say 100 or more in parallel) and the > individual timeouts wont matter as much. > > -David Lee > > > > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of > Jakob Fix > Sent: Thursday, June 25, 2009 7:39 AM > To: General Mark Logic Developer Discussion > Subject: Re: [MarkLogic Dev General] force a "buffer write"? > > Thanks once more, Geert! > > >From an architectural point of view, does it make sense to > have a loop > over many thousand URLs run in an xquery which may take 5 > seconds each? How does Mark Logic handle long-running > queries? The goal is to assemble the information about the > accessibility of these URLs in an XML document that will be > stored in Mark Logic and used for analytical output. Should > the list be fragmented into smaller sub-lists and be > processed separately? > > Would it be possible to have several threads run simultaneously? > Somehow I doubt it as there would probably be issues with the > final aggregation of the different thread sub-documents into > a big one. > > > What about timeouts, especially if the function is called > from inside a web page, how does Mark Logic handle this issue > (I saw that one can tweak the timeout), or would this be a > browser timeout problem? > > I hope these questions are not too off-topic for the list. > > Jakob. > > > On Thu, Jun 25, 2009 at 08:11, Geert Josten > <[email protected]> wrote: > > > > Hi Jakob, > > > > You are looking for unbuffered response streams, but > sending of the response is handled fully by the HTTP server. > I don't believe you can influence that. > > > > Giving it some more thought, I am afraid that allowing > unbuffered responses would break the idea of transactions. > You don't want to send back response, unless you can > guarantee no exceptions will be thrown. And I don't think > that can be guaranteed. > > > > Perhaps a MarkLogic expert would like to comment? > > > > Hasn't this been discussed before? It vaguely rings a bell.. > > > > Kind regards, > > Geert > > > > > > > > > > > Drs. G.P.H. Josten > > Consultant > > > > > > http://www.daidalos.nl/ > > Daidalos BV > > Source of Innovation > > Hoekeindsehof 1-4 > > 2665 JZ Bleiswijk > > Tel.: +31 (0) 10 850 1200 > > Fax: +31 (0) 10 850 1199 > > http://www.daidalos.nl/ > > KvK 27164984 > > De informatie - verzonden in of met dit emailbericht - is > afkomstig van Daidalos BV en is uitsluitend bestemd voor de > geadresseerde. Indien u dit bericht onbedoeld hebt ontvangen, > verzoeken wij u het te verwijderen. Aan dit bericht kunnen > geen rechten worden ontleend. > > > > > > > From: [email protected] > > > [mailto:[email protected]] On > Behalf Of Jakob > > > Fix > > > Sent: donderdag 25 juni 2009 1:39 > > > To: General Mark Logic Developer Discussion > > > Subject: [MarkLogic Dev General] force a "buffer write"? > > > > > > So, I've written a function that looks at one URL at a time and > > > returns true or false depending on its accessibility. > > > Now, my problem is that the result is returned only when all (and > > > that means potentially many) URLs have been checked. > > > Isn't there a way to "force a write"? I'm not sure I'm expressing > > > myself correctly, but hopefully you'll understand what I mean. > > > Thanks. > > > > > > (: consider this: if the timeout is 10 seconds, and I have three > > > "bad" URLs, I may have to wait up to 30 seconds before seeing the > > > results, isn't there a way to see a result every ten seconds > > > instead? :) > > > > > > for $url in $urls > > > return <xh:li>{$url}: > > > {utils:http-resource-available($url, $timeout)}</xh:li> > > > > > > > > > (: function that checks accessibility of a URL, written for DOIs > > > which are "placeholder" URLs which forward to the real URL :) > > > > > > declare function utils:http-resource-available > > > ($doi as xs:string, $oldtimeout as xs:integer?) as > xs:boolean { > > > try { > > > > > > let $timeout := if ($oldtimeout) then $oldtimeout else 10 > > > > > > let $head := > > > xdmp:http-get(fn:concat($utils:doi-resolver, $doi), > > > <options > > > xmlns="xdmp:http"><timeout>{$timeout}</timeout></options>) > > > let $code := $head//xdh:code cast as xs:integer > > > let $location := $head//xdh:location > > > return ((fn:contains($location, > > > $utils:location-part-to-match)) and > > > ($code < 400)) (: we want 3XX or 2XX? :) > > > } catch ($ex) { > > > if ($ex/error:code eq 'SVC-SOCRECV') > > > then > > > fn:false() > > > else > > > xdmp:rethrow() > > > } > > > }; > > > > > > > > > > > > > _______________________________________________ > > General mailing list > > [email protected] > > http://xqzone.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://xqzone.com/mailman/listinfo/general > _______________________________________________ > General mailing list > [email protected] > http://xqzone.com/mailman/listinfo/general > _______________________________________________ General mailing list [email protected] http://xqzone.com/mailman/listinfo/general
