Re: how to get 20000 html pages content quickly from one server?

2006-03-16 Thread Bruno Desthuilliers
JuHui a écrit : in fact, I want to do a script to get news on others site. Then ask the webmasters of theses sites if they do have a ress feed... -- http://mail.python.org/mailman/listinfo/python-list

how to get 20000 html pages content quickly from one server?

2006-03-15 Thread JuHui
Hi I want to get 20 html pages content from one server, you know urllib.urlopen need construct network connection, it will be very slowly, how to speed up this function? I try to using multi-thread, it speed up, but I want to quickly more, any idea about it? Thanks! --

Re: how to get 20000 html pages content quickly from one server?

2006-03-15 Thread Zachery Bir
On Mar 15, 2006, at 11:22 AM, JuHui wrote: Hi I want to get 20 html pages content from one server, you know urllib.urlopen need construct network connection, it will be very slowly, how to speed up this function? I try to using multi-thread, it speed up, but I want to quickly more,

Re: how to get 20000 html pages content quickly from one server?

2006-03-15 Thread Larry Bates
JuHui wrote: Hi I want to get 20 html pages content from one server, you know urllib.urlopen need construct network connection, it will be very slowly, how to speed up this function? I try to using multi-thread, it speed up, but I want to quickly more, any idea about it? Thanks!

Re: how to get 20000 html pages content quickly from one server?

2006-03-15 Thread JuHui
... I will do it later. but i want to optimize the script first. after useing muti-thread, the time speed up from 8s to 2.3s per page. any other suggestions? -- http://mail.python.org/mailman/listinfo/python-list

Re: how to get 20000 html pages content quickly from one server?

2006-03-15 Thread OMouse
JuHui wrote: Hi I want to get 20 html pages content from one server, you know urllib.urlopen need construct network connection, it will be very slowly, how to speed up this function? I try to using multi-thread, it speed up, but I want to quickly more, any idea about it? Thanks!

Re: how to get 20000 html pages content quickly from one server?

2006-03-15 Thread gene tani
JuHui wrote: Hi I want to get 20 html pages content from one server, you know urllib.urlopen need construct network connection, it will be very slowly, how to speed up this function? I try to using multi-thread, it speed up, but I want to quickly more, any idea about it? Thanks!

Re: how to get 20000 html pages content quickly from one server?

2006-03-15 Thread JuHui
in fact, I want to do a script to get news on others site. I must use script get the content and analyze the html code, where is the title, where is the body so, I can't ask permission, use wget and Physically remove the harddrive and reinstall it locally :) --

Re: how to get 20000 html pages content quickly from one server?

2006-03-15 Thread Zachery Bir
On Mar 15, 2006, at 11:58 AM, JuHui wrote: in fact, I want to do a script to get news on others site. I must use script get the content and analyze the html code, where is the title, where is the body so, I can't ask permission, use wget and Physically remove the harddrive and reinstall