My bottleneck is in the time to fetch the page. I sync about 30-40 pages for my company's conduit which takes about 60 - 90 seconds. I was thinking that fetching 2 - 3 pages at a time would cut the time in half, but this was more about running the spider.process 2 or 3 at a time.
Bill
Your mouse has moved. Windows has to reboot for changes to take effect.
To: "Plucker Development List"
"Robert <[EMAIL PROTECTED]>
O'Connor" cc: (bcc: Bill Nalen/Towers Perrin)
<rob@medicalmnem Subject: RE: Plucker conduit -- Syncing
multiple 'units'
onics.com> simultaneously
10/24/2001 03:39
PM
Please respond
to Plucker
Development List
> Adding multi-process fetching of Web pages would indeed speed things
> up, but it would also increase the likelihood of bugs significantly.
I was wondering, where exactly is the usual bottleneck in the Parser in
terms of parsing speed?
I was interested in this, since was wondering if there would be a net
increase in speed if a manager was allowed to sync, say 2 or 3
(channels|databases|streams) simultaneously (ie run three instances of the
parser outputting 3 separate.pdb files).
Best wishes,
Robert
