Hi Danny -- Yeah, I thought it was strange too that the 3rd-party tool wouldn't be able to handle multiple requests, but that turned out to be the problem. Fortunately, it was only because of a config setting -- it has a max threads setting which defaults to 1. I increased that setting, and everything worked. Thanks again for your help.
John On 6/8/2011 12:10 PM, Danny Sokolsky wrote: > Hi John, > > You can set the task server queue to 1, which would effectively single-thread > your CPF processes on that host (I think...). > > But it seems strange that the http process you are calling out to would not > be able to handle multiple requests. > > Are you sure you are sending a single document at a time to the web service? > Maybe something in your code is sending multiple documents? > > At any rate, if setting the task server queue to 1 fixes this problem, that > should help you figure out what the problem is (hopefully). > > You can also try testing the web service from outside MarkLogic by sending > multiple requests to it. If you see the same issue there, then it might be > that the service is not handling multiple requests the way you expect it to. > > -Danny > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of John Munson > Sent: Wednesday, June 08, 2011 9:04 AM > To: General MarkLogic Developer Discussion > Subject: Re: [MarkLogic Dev General] CPF and multithreading > > Danny, thanks very much for the info. My pipeline calls out to a > 3rd-party text processing tool. When I submit each document to ML as a > separate load, the 3rd-party tool marks up the text correctly, but when > I submit multiple documents in one load, it only marks up the first > one. I'm thinking maybe the tool isn't able to handle multithreaded > requests, though I haven't been able to confirm that yet. In light of > that, is there a config way to get CPF / the task server to stop > multithreading and process each document sequentially? If not, maybe > I'll have to write something to submit each document separately. > > Thanks again, > John > > > On 6/7/2011 7:46 PM, Danny Sokolsky wrote: >> Hi John, >> >> CPF spawns tasks onto the task server, and the task server processes with >> multiple threads. So that is normal to have interleaved log messages. >> >> So I guess the question is what is not working correctly when you have >> multiple documents? >> >> 2 tips for debugging CPF apps are: >> >> 1) use the CPF trace event (which throws many log messages into the log) and >> 2) look at the properties of documents after processing to see if any errors >> occurred (any errors are logged into the properties). >> >> -Danny >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of John Munson >> Sent: Tuesday, June 07, 2011 4:15 PM >> To: [email protected] >> Subject: [MarkLogic Dev General] CPF and multithreading >> >> Hi -- >> >> I'm very new to MarkLogic and to XQuery. Didn't see anything about this >> in the archives... >> >> I've set up a Content Processing Framework pipeline using one of the >> samples supplied in the installation. It works fine when I load a >> single document, but not when I load more than one. I used xdmp:log() >> to trace what's going on via the error log, and I was surprised to find >> that my debug output was interleaved between the two documents I was >> loading. This suggests that CPF handed the documents off to my >> processing in separate threads. Does anyone know if that is how CPF is >> designed to work? If so, then maybe the bad output I'm getting is due >> to some other problem, but I'm wondering if it might be due to a >> multithreading problem, given that it shows up only with multiple documents. >> >> Thanks, >> John >> _______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
