I need to parse sites. So... as we know it`s better to
connect different servers in one time than the same in the
queue.
I have 3 packages:
package 1) gets each site main url from a file and send id
to package 2
package 2) get main url then send it to package 3 to
parse, then get new urls (in postback) from this site and
send it again to parse until all the urls will be parsed
package 3) get url and parse it, postback to package 2
So, start...
parsing site 1
parsing site 2
parsing site 3
thats ok 3 diferent servers, but then...
parsing site`s 1 url 2
parsing site`s 1 url 3
parsing site`s 1 url 4
parsing site`s 1 url 5
parsing site`s 1 url 6
# then
parsing site`s 2 url 2
parsing site`s 2 url 3
parsing site`s 2 url 4
# and
parsing site`s 3 url 2
parsing site`s 3 url 3
parsing site`s 3 url 4
so, it connects the same server, it`s very slow, it should
be like:
parsing site`s 1 url 2
parsing site`s 2 url 2
parsing site`s 3 url 2
parsing site`s 1 url 3
parsing site`s 2 url 3
parsing site`s 3 url 3
How to do it ?
Thank you very much!
---
Professional hosting for everyone - http://www.host.ru
