Regarding the performance problem: The schema of my application is:
1. I execute perl script which performs a search in a public database. It gets total results in *several pages*. Pressing "Next Page" button (with perl script) i get a list of all the links related to my query (70.000 more or less) I write down all these links in a unique text file. 2. From the Java i read each of the 70.000 links and i create a new file containing the current i'm reading. Then i call a perl script which uses this link as input parameter. It browses it and get website content saving it in a local html file. I'm having performance problems ,,,, i've tried to don't create a single file containing url for each of the 70.000 links and pass it automatically to perl script as input parameter but it fails... I've heard about LWP module? do you recomend me to use it?? Have you ever done something similar to this? can you give me some advice? Thanks T. On 1/15/07, Rob Dixon <[EMAIL PROTECTED]> wrote:
Tatiana Lloret Iglesias wrote: > Hi all, > from my java applicaation i invoke a perl script which downloads a huge > quantity of files from an external database using WWW-mechanize library and > my problem is that I have big CPU performance problems ... can you give me > any advice to avoid this? Hi Tatiana Do you really mean "CPU performance problems"? If you're downloading a lot from the Internet then your problem is more likely to be limited by the speed of the network, in which case you need to look at what you're downloading and see if you can get the information you require without moving as much extraneous data. Can you anticipate the URLs you need to access instead of following links from other pages, for instance? If your process is indeed CPU-bound then it is most liley to be an error in your coding, which we would need to see before we could help. HTH, Rob
